Grow UFS in Solaris

One discussion in toolbox.com give me the idea to write this article. The subject was full root  (/) filesystem and how to extend it. Most of the participants do not believe this is possible, but i will demonstrate how to do it. My demonstration is based on Oracle Solaris 10 Generic_142910-17 i386.

1. What is my OS

# showrev
Hostname: sun02
Hostid: 10b69b13
Release: 5.10
Kernel architecture: i86pc
Application architecture: i386
Hardware provider:
Domain:
Kernel version: SunOS 5.10 Generic_142910-17

2. Check the map of harddisk
#  prtvtoc /dev/dsk/c1t0d0s2
* /dev/dsk/c1t0d0s2 partition map
<snip>
*                          First     Sector    Last
* Partition  Tag  Flags    Sector     Count    Sector  Mount Directory
       0      2    00      16065   2104515   2120579   /
       1      4    00    2570400   6297480   8867879   /usr
       2      5    00          0  33495525  33495524
       3      3    01   11309760   1060290  12370049
       8      1    01          0     16065     16064

As you can see I have some unallocated cylinders after each partition (/. /usr and swap) and this is done on time of installation for the reason of demonstration.

3. Check the exact sizes of filesystems
# df -k / /usr
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c1t0d0s0    1019856  377425  581240    40%    /
/dev/dsk/c1t0d0s1    3100362 2251708  786647    75%    /usr
# swap -l
swapfile             dev  swaplo blocks   free
/dev/dsk/c1t0d0s3   30,3       8 1060280 1060280

4. Create two control files, filled with random bytes for control of the integrity of filesystems
# dd if=/dev/urandom of=/checkfileroot bs=1024 count=10240
10240+0 records in
10240+0 records out
# dd if=/dev/urandom of=/usr/checkfileusr bs=1024 count=10240
10240+0 records in
10240+0 records out

5. And get checksums ot the files
# digest -a sha1 /checkfileroot /usr/checkfileusr
(/checkfileroot) = c5ee33c68b147c58e6190a99a647a9baf35581a8
(/usr/checkfileusr) = 77bb739b734ab01a43578479ec4a3abe92e6c4bd

6. Extend slice 0, 1 and 3 with some amount of cylinders
# format
Searching for disks...done

AVAILABLE DISK SELECTIONS:
       0. c1t0d0 <DEFAULT cyl 2085 alt 2 hd 255 sec 63>
          /pci@0,0/pci1000,30@10/sd@0,0
Specify disk (enter its number): 0
selecting c1t0d0
[disk formatted]
Warning: Current Disk has mounted partitions.
/dev/dsk/c1t0d0s0 is currently mounted on /. Please see umount(1M).
/dev/dsk/c1t0d0s1 is currently mounted on /usr. Please see umount(1M).
/dev/dsk/c1t0d0s3 is currently used by swap. Please see swap(1M).

FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        fdisk      - run the fdisk program
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        save       - save new disk/partition definitions
        inquiry    - show vendor, product and revision
        volname    - set 8-character volume name
        !<cmd>     - execute <cmd>, then return
        quit
format> p

PARTITION MENU:
        0      - change `0' partition
        1      - change `1' partition
        2      - change `2' partition
        3      - change `3' partition
        4      - change `4' partition
        5      - change `5' partition
        6      - change `6' partition
        7      - change `7' partition
        select - select a predefined table
        modify - modify a predefined partition table
        name   - name the current table
        print  - display the current table
        label  - write partition map and label to the disk
        !<cmd> - execute <cmd>, then return
        quit
partition> 0
Part      Tag    Flag     Cylinders        Size            Blocks
  0       root    wm       1 -  131        1.00GB    (131/0/0)   2104515
Enter partition id tag[root]:
Enter partition permission flags[wm]:
Enter new starting cyl[1]:
Enter partition size[2104515b, 131c, 131e, 1027.60mb, 1.00gb]: 140c
partition> 1
Part      Tag    Flag     Cylinders        Size            Blocks
  1        usr    wm     160 -  551        3.00GB    (392/0/0)   6297480
Enter partition id tag[usr]:
Enter partition permission flags[wm]:
Enter new starting cyl[160]:
Enter partition size[6297480b, 392c, 551e, 3074.94mb, 3.00gb]: 400c
partition> 3
Part      Tag    Flag     Cylinders        Size            Blocks
  3       swap    wu     704 -  769      517.72MB    (66/0/0)    1060290
Enter partition id tag[swap]:
Enter partition permission flags[wu]:
Enter new starting cyl[704]:
Enter partition size[1060290b, 66c, 769e, 517.72mb, 0.51gb]: 80c
partition> la
Ready to label disk, continue? y
partition> q
FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        fdisk      - run the fdisk program
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        save       - save new disk/partition definitions
        inquiry    - show vendor, product and revision
        volname    - set 8-character volume name
        !<cmd>     - execute <cmd>, then return
        quit
format> q

7. And the moment of true, extend root (/) filesystem
# growfs -M / /dev/rdsk/c1t0d0s0
Warning: 5748 sector(s) in last cylinder unallocated
/dev/rdsk/c1t0d0s0:     2249100 sectors in 367 cylinders of 48 tracks, 128 sectors
        1098.2MB in 23 cyl groups (16 c/g, 48.00MB/g, 11648 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
 32, 98464, 196896, 295328, 393760, 492192, 590624, 689056, 787488, 885920,
 1279648, 1378080, 1476512, 1574944, 1673376, 1771808, 1870240, 1968672,
 2067104, 2165536
8. Then grow /usr
# growfs -M /usr /dev/rdsk/c1t0d0s1
Warning: 624 sector(s) in last cylinder unallocated
/dev/rdsk/c1t0d0s1:     6426000 sectors in 1046 cylinders of 48 tracks, 128 sectors
        3137.7MB in 66 cyl groups (16 c/g, 48.00MB/g, 5824 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
 32, 98464, 196896, 295328, 393760, 492192, 590624, 689056, 787488, 885920,
 5512224, 5610656, 5709088, 5807520, 5905952, 6004384, 6102816, 6201248,
 6291488, 6389920
9. And swap
# swap -d /dev/dsk/c1t0d0s3
/dev/dsk/c1t0d0s3 was dump device --
invoking dumpadm(1M) -d swap to select new dump device
dumpadm: no swap devices are available
# swap -a /dev/dsk/c1t0d0s3
operating system crash dump was previously disabled --
invoking dumpadm(1M) -d swap to select new dump device
# dumpadm -d swap
      Dump content: kernel pages
       Dump device: /dev/dsk/c1t0d0s3 (swap)
Savecore directory: /var/crash/sun02
  Savecore enabled: yes
   Save compressed: on

For the swap I just delete and add it again. And update dump device, this is important. Of course it is not always possible just to delete virtual memory on production, but its possible to play with creation of new swap device, delete old, add old and delete new. This can take long time on production system, but its relatively safe operation

10. So, let check again the sizes of filesystems
# df -k / /usr
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c1t0d0s0    1090677  387745  641741    38%    /
/dev/dsk/c1t0d0s1    3163878 2262020  839851    73%    /usr
# swap -l
swapfile             dev  swaplo blocks   free
/dev/dsk/c1t0d0s3   30,3       8 1285192 1285192

As you can see they are bigger that few minutes a go

11. But what is the situation with control files
# digest -a sha1 /checkfileroot /usr/checkfileusr
(/checkfileroot) = c5ee33c68b147c58e6190a99a647a9baf35581a8
(/usr/checkfileusr) = 77bb739b734ab01a43578479ec4a3abe92e6c4bd

As you can see they are the same

12. Et voila, we successfully extend our filesystems on the fly. By the way in official Oracle Solaris documentations you can see this
--------------
LIMITATIONS
    Only UFS file systems (either mounted or unmounted) can be expanded using the growfs command. Once a file system is expanded, it cannot be decreased in size. The following conditions prevent you from expanding file systems: When acct is activated and the accounting file is on the target device. When C2 security is activated and the logging file is on the target file system. When there is a local swap file in the target file system. When the file system is root (/), /usr, or swap.

Solaris x86 root filesystem mirroring

Preamble
This document is directed to give idea how to create mirroring of root filesystem in Solaris x86 with the help of Solaris Volume Manager. Here will be used already installed OS and all the work will be done without need of reinstall

Prerequisites
1. First we should be sure have two identical harddisks in the server

# format
 Searching for disks...done
 AVAILABLE DISK SELECTIONS:
       0. c0d0 <DEFAULT cyl 1563 alt 2 hd 255 sec 63>
          /pci@0,0/pci-ide@7,1/ide@0/cmdk@0,0
       1. c0d1 <DEFAULT cyl 1563 alt 2 hd 255 sec 63>
          /pci@0,0/pci-ide@7,1/ide@0/cmdk@1,0
 Specify disk (enter its number): ^C


2. Its need to create small slice for metadb information (usually slice 7) like:

   partition> p
 Current partition table (unnamed):
 Total disk cylinders available: 1563 + 2 (reserved cylinders)

 Part      Tag    Flag     Cylinders        Size            Blocks
   0       root    wm     518 - 1562        8.01GB    (1045/0/0) 16787925
   1       swap    wu       3 -  133        1.00GB    (131/0/0)   2104515
   2     backup    wm       0 - 1562       11.97GB    (1563/0/0) 25109595
   3 unassigned    wm       0               0         (0/0/0)           0
   4 unassigned    wm       0               0         (0/0/0)           0
   5 unassigned    wm       0               0         (0/0/0)           0
   6 unassigned    wm       0               0         (0/0/0)           0
   7 unassigned    wm     134 -  135       15.69MB    (2/0/0)       32130
   8       boot    wu       0 -    0        7.84MB    (1/0/0)       16065
   9 alternates    wu       1 -    2       15.69MB    (2/0/0)       32130

3. Next step is to create the same partitions on the second disk. To avoid human error it is much better to use some kind of automations:

 prtvtoc /dev/rdsk/c0d0s2 > /tmp/c0d0s2.toc
 fmthard -s /tmp/c0d0s2.toc /dev/rdsk/c0d1s2

4. Then we should identify the partitions need to be mirrored:

 # egrep "ufs|swap" /etc/vfstab|grep "/dev/dsk"
 /dev/dsk/c0d0s1 -       -       swap    -       no      -
 /dev/dsk/c0d0s0 /dev/rdsk/c0d0s0        /       ufs     1       no      -


LVM
1. Let’s create few copies of metadb on the partitions we create for this puspose:

 # metadb -a -f -c 2 c0d0s7 c0d1s7

2. It’s time to put our disks where OS reside under management of SVM

 # metainit -f d10 1 1 c0d0s0
 d10: Concat/Stripe is setup
 # metainit -f d11 1 1 c0d0s1
 d11: Concat/Stripe is setup
 # metainit d0 -m d10
 d0: Mirror is setup
 # metainit d1 -m d11
 d1: Mirror is setup
 # metaroot d0

3. Check newly created devices:

 # ls -l /dev/md/rdsk
 total 8
 lrwxrwxrwx   1 root     root          36 Aug 30 18:29 d0 -> ../../../devices/pseudo/md@0:0,0,raw
 lrwxrwxrwx   1 root     root          36 Aug 30 18:29 d1 -> ../../../devices/pseudo/md@0:0,1,raw
 lrwxrwxrwx   1 root     root          37 Aug 30 18:28 d10 -> ../../../devices/pseudo/md@0:0,10,raw
 lrwxrwxrwx   1 root     root          37 Aug 30 18:28 d11 -> ../../../devices/pseudo/md@0:0,11,raw
 # ls -l /dev/md/dsk
 total 8
 lrwxrwxrwx   1 root     root          36 Aug 30 18:29 d0 -> ../../../devices/pseudo/md@0:0,0,blk
 lrwxrwxrwx   1 root     root          36 Aug 30 18:29 d1 -> ../../../devices/pseudo/md@0:0,1,blk
 lrwxrwxrwx   1 root     root          37 Aug 30 18:28 d10 -> ../../../devices/pseudo/md@0:0,10,blk
 lrwxrwxrwx   1 root     root          37 Aug 30 18:28 d11 -> ../../../devices/pseudo/md@0:0,11,blk

4. Make appropriate changes in /etc/vfstab to get boot from mirror, not standard disks

 # egrep "ufs|swap" /etc/vfstab|grep "/dev/md/dsk"
 /dev/md/dsk/d1  -       -       swap    -       no      -
 /dev/md/dsk/d0  /dev/md/rdsk/d0 /       ufs     1       no      -
 
5. The next step is to flush cache buffers and reboot

 # sync;sync;sync
 # init 6

6. It is time to put second disk under management of SVN

 # metainit -f d20 1 1 c0d1s0
 d20: Concat/Stripe is setup
 # metainit -f d21 1 1 c0d1s1
 d21: Concat/Stripe is setup

7. And to add them to created previously mirrors. Be aware process of synchronisation will continue in  background and you can check the process

 # metattach d0 d20
 d0: submirror d20 is attached
 # metattach d1 d21
 d1: submirror d21 is attached

8. Check the process of building mirrors and wait till they finnish

 # metastat
 d1: Mirror
     Submirror 0: d11
       State: Okay
     Submirror 1: d21
       State: Resyncing
     Resync in progress: 96 % done
     Pass: 1
     Read option: roundrobin (default)
     Write option: parallel (default)
     Size: 2104515 blocks (1.0 GB)

 d11: Submirror of d1
     State: Okay
     Size: 2104515 blocks (1.0 GB)
     Stripe 0:
         Device   Start Block  Dbase        State Reloc Hot Spare
         c0d0s1          0     No            Okay   Yes


 d21: Submirror of d1
     State: Resyncing
     Size: 2104515 blocks (1.0 GB)
     Stripe 0:
         Device   Start Block  Dbase        State Reloc Hot Spare
         c0d1s1          0     No            Okay   Yes


 d0: Mirror
     Submirror 0: d10
       State: Okay
     Submirror 1: d20
       State: Resyncing
     Resync in progress: 13 % done
     Pass: 1
     Read option: roundrobin (default)
     Write option: parallel (default)
     Size: 16787925 blocks (8.0 GB)

 d10: Submirror of d0
     State: Okay
     Size: 16787925 blocks (8.0 GB)
     Stripe 0:
         Device   Start Block  Dbase        State Reloc Hot Spare
         c0d0s0          0     No            Okay   Yes


 d20: Submirror of d0
     State: Resyncing
     Size: 16787925 blocks (8.0 GB)
     Stripe 0:
         Device   Start Block  Dbase        State Reloc Hot Spare
         c0d1s0          0     No            Okay   Yes


 Device Relocation Information:
 Device   Reloc  Device ID
 c0d1   Yes      id1,cmdk@AVMware_Virtual_IDE_Hard_Drive=01000000000000000001
 c0d0   Yes      id1,cmdk@AVMware_Virtual_IDE_Hard_Drive=00000000000000000001


Boot

1. Next step is to check if partition on second disk is active

 # fdisk /dev/rdsk/c0d1p0
             Total disk size is 1566 cylinders
             Cylinder size is 16065 (512 byte) blocks

                                               Cylinders
      Partition   Status    Type          Start   End   Length    %
      =         =   =      =
          1       Active    Solaris2          1  1565    1565    100



2. And to add boot record to the second disk to make it bootable

 # /sbin/installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c0d1s0
 stage1 written to partition 0 sector 0 (abs 16065)
 stage2 written to partition 0, 233 sectors starting at 50 (abs 16115)

3. Add new item in the boot menu (/boot/grub/menu.lst) to have alternative way to boot

 title Alternate boot
 root (hd1,0,a)
 kernel /platform/i86pc/multiboot
 module /platform/i86pc/boot_archive

4. Check if the new item is added to the boot menu

 # bootadm list-menu
 The location for the active GRUB menu is: /boot/grub/menu.lst
 default 0
 timeout 10
 0 Solaris 10 5/08 s10x_u5wos_10 X86
 1 Solaris failsafe
 2 Alternate boot

5. That’s all, you have already mirrored root partition (plus swap)

Conclusion
In the document are not mentioned all the options and possibilities of SVM, but only short set, need to done the work. For further information, please consult official Oracle documentation.

Suggestions and ideas about filesystem sizing on Solaris

Remark
This article have mainly historical value as it is created at 2008. Now the situation is quite different, diskspace is much cheaper and you don't need to be precise and careful about sizes. Also RAM is cheaper and in most of the cases you can create swap as 1GB or less.

Preambul

This document is intended to give some ideas and suggestions how to size the filesystems (and slices) in Solaris OS. Use it with cautious and do not forget to consult with official SUN documentation and your company documents, related to the subject.

Sugestions

    For / (root) filesystem normally you do not need so much disk space. Most of the time you can allocate 256 MB or ever less, but 512 MB is one very good value. Do not attempt to allocate big amount of space and put all directories as /, usr, var, opt, export/home in one filesystem. This may look very sample, but can put you in trouble after some time.
    For /usr filesystem you need average 2048 MB. On time of installation, installer will show you how much you need. Don't put much more space ever if you think to install big amount of software in /usr/local. If this happen, just create separate filesystem.
    For virtual memory i suggest to use this schema:
        < 1 GB – RAMx2.5
        <2 GB – RAMx2
        <4 GB – RAMx1,5
        >4 GB – RAMx1
        Here you should get above just as starting point, because many applications have own requirements. For example Oracle+SAP need swap to be four times RAM, almost independently of size of real memory, but Oracle itself (if it's alone) “prefer to live” in RAM.
    For /var filesystem you can allocate as minimum 512 MB, but if you expect big log files – from system, applications, etc, you should allocate more. The same is true if you expect long life and many updates on your system. Do not forget log rotate utilities, they can help you save some disk space. If your server will be used for jobs, generating big amount of data in /var as mail server or print server you can allocate more disk space, but because most of the print and mail software use /var/spool you can create dedicated filesystem for this purpose.
    Normaly /tmp filesystem is “part” of swap, but for some products as Oracle you can set dedicated filesystem. Normally 512 megabytes are enough.
    For /opt filesystem you should decide the size by self, because in this filesystem normally in Solaris are located installations of external software. 256 MB should be enough for many purposes, but some software (Oracle by example) can fill up to 1.5 GB (or more) only for executable files.
    And last but not least /export/home. Do not set this filesystem very big. Later if it's need you can move home directories and files to new filesystem or create new one like /export/home2 for new users


Ideas

•     Do not allocated entire disk even if you use some volume manager because later when you need diskspace for new filesystem or extend existing one can be hard to get new harddisk
•     Get indeep knowledge about the applications will run on this machine for better filesystems sizing
•     Do not oversize your filesystems and do not set them as little as possible. 20-30% over from the requirements is OK as general rule