Solaris commands for process management - ps

ps - print process information and status. In Solaris exist SYSV and BSD variants, respectively /usr/bin/ps and /usr/ucb/ps

bash-3.2# /usr/bin/ps
   PID TTY         TIME CMD
   866 pts/3       0:00 bash
   786 pts/3       0:00 sh
  1007 pts/3       0:00 ps
 

 bash-3.2# /usr/ucb/ps
   PID TT       S  TIME COMMAND
   362 console  S  0:00 /usr/lib/saf/ttymon -g -d /dev/console -l console -T sun-color -m l
   707 pts/2    S  0:00 /usr/dt/bin/sdt_shell -c      unset DT;     DISPLAY=:0;       /usr/
   709 pts/2    S  0:00 -sh -c      unset DT;     DISPLAY=:0;       /usr/dt/bin/dtsession_r
   721 pts/2    S  0:00 /bin/ksh /usr/dt/config/Xsession2.jds
   723 pts/2    S  0:00 /usr/bin/gnome-session
   786 pts/3    S  0:00 sh
   866 pts/3    S  0:00 bash
  1008 pts/3    O  0:00 /usr/ucb/ps

Solaris commands for process management -prstat

 prstat – fullscreen view of processes sorted by different criteria, similar to Linux top command
  PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
   502 root       48M   36M sleep   59    0   0:00:09 0.2% Xorg/1
   779 root       68M   12M sleep   58    0   0:00:08 0.1% mixer_applet2/1
   605 noaccess  162M  116M sleep   59    0   0:00:13 0.1% java/18
   783 root       75M   16M sleep   49    0   0:00:03 0.0% gnome-terminal/2
   729 root       12M 9752K sleep   59    0   0:00:04 0.0% gconfd-2/1
  1004 root     3808K 2960K cpu0    59    0   0:00:00 0.0% prstat/1
   734 root     6644K 4008K sleep   59    0   0:00:00 0.0% xscreensaver/1
   760 root       73M   18M sleep   59    0   0:00:03 0.0% gnome-panel/1
   756 root       67M   14M sleep   59    0   0:00:00 0.0% metacity/1
   777 root       71M   15M sleep   59    0   0:00:01 0.0% gnome-netstatus/1
   281 root     3780K 1544K sleep   59    0   0:00:01 0.0% VBoxService/7
     5 root        0K    0K sleep   99  -20   0:00:02 0.0% zpool-rpool/138
   751 root     7076K 3184K sleep   59    0   0:00:00 0.0% gnome-smproxy/1
   866 root     2992K 1772K sleep   37    0   0:00:00 0.0% bash/1
   965 root     4376K 2528K sleep   59    0   0:00:00 0.0% nscd/19
   867 smmsp    9184K 2112K sleep   59    0   0:00:00 0.0% sendmail/1
   559 root     3064K 2064K sleep   59    0   0:00:00 0.0% vold/5
   707 root     4616K 1760K sleep   59    0   0:00:00 0.0% sdt_shell/1
   623 root     1608K 1008K sleep   59    0   0:00:00 0.0% Xsession/1
<snip>
Total: 69 processes, 347 lwps, load averages: 0.02, 0.01, 0.01



The program in default view is less informative that top, but can give you info for zones, projects and even for processes bind to specific processor. Use the -u option to limit display to specific users. Use the -p option to limit display to specific processes

Solaris commands for process management - preap

preap – try to kick-off zombie process



# ps -efl|grep 551 
0 Z root 551 550 0 0 - - 0 - - ? 0:00 <defunct> 
# preap 551 
551: exited with status 0 
# ps -efl|grep 551



In above example, this tool successfully removes zombie process. Otherwise the only way to kill them is to reboot

Solaris commands for process management - pmap

pmap – get memory map (segments) of process

bash-3.2# pmap 867
867:    /usr/lib/sendmail -Ac -q15m
08037000      68K rw---    [ stack ]
08050000     780K r-x--  /usr/lib/sendmail
08123000      72K rw---  /usr/lib/sendmail
08135000     748K rw---    [ heap ]
FE5B0000      64K rwx--    [ anon ]
FE5D0000      64K rwx--    [ anon ]
FE5F0000      64K rwx--    [ anon ]
FE610000     388K r-x--  /usr/lib/mps/libssl3.so
FE680000      12K rwx--  /usr/lib/mps/libssl3.so
FE690000      12K r-x--  /usr/lib/mps/libplds4.so
FE6A2000       4K rwx--  /usr/lib/mps/libplds4.so
FE6B0000     116K r-x--  /usr/lib/mps/libnssutil3.so
FE6DC000      16K rwx--  /usr/lib/mps/libnssutil3.so
FE700000    1816K r-x--  /usr/lib/mps/libnss3.so
FE8D5000      32K rwx--  /usr/lib/mps/libnss3.so
FE8E0000      12K r-x--  /usr/lib/mps/libplc4.so
FE8F2000       8K rwx--  /usr/lib/mps/libplc4.so
FE900000      64K r-x--  /usr/lib/libsasl.so.1
FE910000       4K rw---  /usr/lib/libsasl.so.1
FE920000      12K r-x--  /lib/libmp.so.2
FE933000       4K rw---  /lib/libmp.so.2
FE940000       4K r-x--  /lib/libdl.so.1
FE951000       4K rw---  /lib/libdl.so.1
FE960000     288K r-x--  /usr/lib/mps/libnspr4.so
FE9B7000       8K rwx--  /usr/lib/mps/libnspr4.so
FE9B9000       8K rwx--  /usr/lib/mps/libnspr4.so
FE9C0000      40K r-x--  /usr/sfw/lib/libcrypto_extra.so.0.9.7
FE9D0000       4K rwx--    [ anon ]
FE9DA000       4K rw---  /usr/sfw/lib/libcrypto_extra.so.0.9.7
FE9E0000      40K r-x--  /usr/sfw/lib/libssl_extra.so.0.9.7
FE9F0000       4K rwx--    [ anon ]
FE9FA000       4K rw---  /usr/sfw/lib/libssl_extra.so.0.9.7
FEA00000      24K r-x--  /lib/libgen.so.1
FEA16000       4K rw---  /lib/libgen.so.1
FEA20000      24K r-x--  /lib/libuutil.so.1
FEA30000       4K rwx--    [ anon ]
FEA36000       4K rw---  /lib/libuutil.so.1
FEA40000      56K r-x--  /lib/libmd.so.1
FEA50000      12K r-x--  /lib/libpthread.so.1
FEA5E000       4K rw---  /lib/libmd.so.1
FEA60000      32K r-x--  /lib/libaio.so.1
FEA70000      16K r-x--  /lib/libthread.so.1
FEA78000       4K rw---  /lib/libaio.so.1
FEA79000       4K rw---  /lib/libaio.so.1
FEA80000       8K r-x--  /lib/libdoor.so.1
FEA90000       4K r--s-  dev:307,3 ino:4043961197
FEA92000       4K rw---  /lib/libdoor.so.1
FEAA0000      92K r-x--  /lib/libscf.so.1
FEAC0000       4K rwx--    [ anon ]
FEAC7000       4K rw---  /lib/libscf.so.1
FEAD0000    1088K r-x--  /lib/libc.so.1
FEBE0000      32K rwx--  /lib/libc.so.1
FEBE8000       8K rwx--  /lib/libc.so.1
FEBF0000     956K r-x--  /usr/sfw/lib/libcrypto.so.0.9.7
FECE0000      24K rwx--    [ anon ]
FECEF000      80K rw---  /usr/sfw/lib/libcrypto.so.0.9.7
FED03000       8K rw---  /usr/sfw/lib/libcrypto.so.0.9.7
FED10000     208K r-x--  /usr/sfw/lib/libssl.so.0.9.7
FED50000       4K rwx--    [ anon ]
FED54000      12K rw---  /usr/sfw/lib/libssl.so.0.9.7
FED60000      24K r-x--  /lib/librt.so.1
FED70000       4K rwx--    [ anon ]
FED76000       4K rw---  /lib/librt.so.1
FED80000      88K r-x--  /lib/libumem.so.1
FEDA0000       4K rwx--    [ anon ]
FEDA6000      20K rw---  /lib/libumem.so.1
FEDAB000      24K rw---  /lib/libumem.so.1
FEDC0000      24K r-x--  /usr/sfw/lib/libwrap.so.1.0
FEDD0000       4K rwx--    [ anon ]
FEDD6000       4K rw---  /usr/sfw/lib/libwrap.so.1.0
FEDD7000       4K rw---  /usr/sfw/lib/libwrap.so.1.0
FEDE0000     192K r-x--  /usr/lib/libsldap.so.1
FEE10000      12K rw---  /usr/lib/libsldap.so.1
FEE20000     204K r-x--  /usr/lib/libldap.so.5
FEE60000       4K rwx--    [ anon ]
FEE63000      12K rw---  /usr/lib/libldap.so.5
FEE70000     516K r-x--  /lib/libnsl.so.1
FEF01000      20K rw---  /lib/libnsl.so.1
FEF06000      32K rw---  /lib/libnsl.so.1
FEF10000      44K r-x--  /lib/libsocket.so.1
FEF20000       4K rwx--    [ anon ]
FEF2B000       4K rw---  /lib/libsocket.so.1
FEF30000       4K rwx--    [ anon ]
FEF40000     216K r-x--  /lib/libresolv.so.2
FEF80000       4K rwx--    [ anon ]
FEF86000       8K rw---  /lib/libresolv.so.2
FEF90000       4K rwx--    [ anon ]
FEFA0000       4K rw---    [ anon ]
FEFB0000       4K rw---    [ anon ]
FEFBA000       4K rwxs-    [ anon ]
FEFBE000     176K r-x--  /lib/ld.so.1
FEFF0000       4K rwx--    [ anon ]
FEFFA000       8K rwx--  /lib/ld.so.1
FEFFC000       8K rwx--  /lib/ld.so.1
 total      9184K



This command helps in the process of fine-tuning large applications, especially for resolving memory leak issues

Solaris commands for process management - pldd

pldd – list dynamic libraries, associated with process, similar to ldd for executable

 bash-3.2# ps -efl|grep send
 0 S    smmsp   867     1   0  40 20        ?   2296        ? 18:02:34 ?           0:00 /usr/lib/sendmail -Ac -q15m
bash-3.2# pldd 867
867:    /usr/lib/sendmail -Ac -q15m
/lib/libumem.so.1
/lib/libresolv.so.2
/lib/libsocket.so.1
/lib/libnsl.so.1
/usr/lib/libldap.so.5
/usr/lib/libsldap.so.1
/usr/sfw/lib/libwrap.so.1.0
/lib/librt.so.1
/usr/sfw/lib/libssl.so.0.9.7
/usr/sfw/lib/libcrypto.so.0.9.7
/lib/libc.so.1
/lib/libscf.so.1
/lib/libdoor.so.1
/lib/libaio.so.1
/lib/libmd.so.1
/lib/libuutil.so.1
/lib/libgen.so.1
/usr/sfw/lib/libssl_extra.so.0.9.7
/usr/sfw/lib/libcrypto_extra.so.0.9.7
/usr/lib/mps/libnspr4.so
/lib/libpthread.so.1
/lib/libdl.so.1
/lib/libmp.so.2
/usr/lib/libsasl.so.1
/usr/lib/mps/libplc4.so
/usr/lib/mps/libnss3.so
/usr/lib/mps/libnssutil3.so
/usr/lib/mps/libplds4.so
/usr/lib/mps/libssl3.so
/lib/libthread.so.1

But let execure ldd command for the same file:
bash-3.2# ldd /usr/lib/sendmail
        libresolv.so.2 =>        /lib/libresolv.so.2
        libsocket.so.1 =>        /lib/libsocket.so.1
        libnsl.so.1 =>   /lib/libnsl.so.1
        libldap.so.5 =>  /usr/lib/libldap.so.5
        libsldap.so.1 =>         /usr/lib/libsldap.so.1
        libwrap.so.1 =>  /usr/sfw/lib/libwrap.so.1
        libumem.so.1 =>  /lib/libumem.so.1
        librt.so.1 =>    /lib/librt.so.1
        libssl.so.0.9.7 =>       /usr/sfw/lib/libssl.so.0.9.7
        libcrypto.so.0.9.7 =>    /usr/sfw/lib/libcrypto.so.0.9.7
        libc.so.1 =>     /lib/libc.so.1
        libmp.so.2 =>    /lib/libmp.so.2
        libmd.so.1 =>    /lib/libmd.so.1
        libscf.so.1 =>   /lib/libscf.so.1
        libsasl.so.1 =>  /usr/lib/libsasl.so.1
        libnspr4.so =>   /usr/lib/mps/libnspr4.so
        libplc4.so =>    /usr/lib/mps/libplc4.so
        libnss3.so =>    /usr/lib/mps/libnss3.so
        libssl3.so =>    /usr/lib/mps/libssl3.so
        libdoor.so.1 =>  /lib/libdoor.so.1
        libaio.so.1 =>   /lib/libaio.so.1
        libuutil.so.1 =>         /lib/libuutil.so.1
        libgen.so.1 =>   /lib/libgen.so.1
        libpthread.so.1 =>       /lib/libpthread.so.1
        libdl.so.1 =>    /lib/libdl.so.1
        libnssutil3.so =>        /usr/lib/mps/libnssutil3.so
        libplds4.so =>   /usr/lib/mps/libplds4.so
        libthread.so.1 =>        /lib/libthread.so.1
        libssl_extra.so.0.9.7 =>         /usr/sfw/lib/libssl_extra.so.0.9.7
        libcrypto_extra.so.0.9.7 =>      /usr/sfw/lib/libcrypto_extra.so.0.9.7
        libm.so.2 =>     /lib/libm.so.2


Results from both commands are almost the same :-) Why? Ask SUN/Oracle

Solaris commands for process management - pkill

pkill – send signal to process. For example pkill -9 init :-P


# pgrep named 
628 
# pkill named 
# pgrep named 
636


As you see, the named daemon was killed and respawned (with new PID)

Solaris commands for process management - pgrep

pgrep – get the PID's of processes by name i.e. Something like ps -efl|grep -v grep|grep process_name



# pgrep named 
586

Something like ps -efl|grep -v grep|grep process_name|awk '{print $1}'

# ps -e|grep -v grep|grep named|awk '{print $1}' 
586

Solaris commands for process management - pargs

pargs – get list of arguments and environment variables with which process was started

bash-3.2# pargs 889
889:    /usr/lib/sendmail -bl -q15m
argv[0]: sendmail: accepting connections
argv[1]: <NULL>
argv[2]: nnections
bash-3.2# pargs 867
867:    /usr/lib/sendmail -Ac -q15m
argv[0]: sendmail: Queue runner@00:15:00 for /var/spool/clientmqueue
argv[1]: <NULL>
argv[2]: @00:15:00 for /var/spool/clientmqueue


in above example you see the arguments for starting sendmail as a daemon

Solaris commands for process management - pfiles

pfiles – list of file descriptors, associated with process


bash-3.2# pfiles 889
889:    /usr/lib/sendmail -bl -q15m
  Current rlimit: 1024 file descriptors
   0: S_IFCHR mode:0666 dev:303,0 ino:6815752 uid:0 gid:3 rdev:13,2
      O_RDONLY|O_LARGEFILE
      /devices/pseudo/mm@0:null
   1: S_IFCHR mode:0666 dev:303,0 ino:6815752 uid:0 gid:3 rdev:13,2
      O_WRONLY|O_LARGEFILE
      /devices/pseudo/mm@0:null
   2: S_IFCHR mode:0666 dev:303,0 ino:6815752 uid:0 gid:3 rdev:13,2
      O_WRONLY|O_LARGEFILE
      /devices/pseudo/mm@0:null
   3: S_IFDOOR mode:0444 dev:312,0 ino:55 uid:0 gid:0 size:0
      O_RDONLY|O_LARGEFILE FD_CLOEXEC  door to nscd[144]
      /var/run/name_service_door
   4: S_IFCHR mode:0000 dev:303,0 ino:57176 uid:0 gid:0 rdev:21,33
      O_WRONLY FD_CLOEXEC
      /devices/pseudo/log@0:conslog
   5: S_IFSOCK mode:0666 dev:310,0 ino:61053 uid:0 gid:0 size:0
      O_RDWR FD_CLOEXEC
        SOCK_STREAM
        SO_REUSEADDR,SO_KEEPALIVE,SO_SNDBUF(49152),SO_RCVBUF(49152)
        sockname: AF_INET 127.0.0.1  port: 25
   6: S_IFSOCK mode:0666 dev:310,0 ino:61054 uid:0 gid:0 size:0
      O_RDWR FD_CLOEXEC
        SOCK_STREAM
        SO_REUSEADDR,SO_KEEPALIVE,SO_SNDBUF(49152),SO_RCVBUF(49152)
        sockname: AF_INET 127.0.0.1  port: 587
   7: S_IFREG mode:0600 dev:307,3 ino:4059345121 uid:0 gid:25 size:32
      O_WRONLY|O_CREAT|O_EXCL|O_LARGEFILE
      /var/run/sendmail.pid



Here you see a list of files as NULL, console log, some PID files. On the top you can see the maximum number of file handlers for this process, with info that can be helpful for troubleshooting of applications

Solaris commands for process management - apptrace

apptrace – this commend is used for tracing library calls

bash-3.2# apptrace man
-> man      -> libc.so.1:int atexit(int (*)() = 0xfefce738)
<- man      -> libc.so.1:atexit()
-> man      -> libc.so.1:int atexit(int (*)() = 0x8054eac)
<- man      -> libc.so.1:atexit()
-> man      -> libc.so.1:void __fpstart(void)
<- man      -> libc.so.1:__fpstart() = 0xfecd8754
-> man      -> libc.so.1:access(0x80559a4, 0x1, 0x8047608) ** NR
-> man      -> libc.so.1:char * setlocale(int = 0x6, const char * = 0x80559a0 "")
<- man      -> libc.so.1:setlocale() = 0xfec7c045
-> man      -> libc.so.1:char * setlocale(int = 0x5, const char * = 0x0 <NULL>)
<- man      -> libc.so.1:setlocale() = 0xfec7c045
-> man      -> libc.so.1:strcpy(0x80669e8, 0xfec7c045, 0x8047608) ** NR
-> man      -> libc.so.1:int strcmp(const char * = 0x805599c "C", const char * = 0x80669e8 "C")
<- man      -> libc.so.1:strcmp()
-> man      -> libc.so.1:char * textdomain(const char * = 0x8055988 "SUNW_OST_OSCMD")
<- man      -> libc.so.1:textdomain() = 0x80683e0
-> man      -> libc.so.1:strcpy(0x80665e8, 0x8055980, 0x8047608) ** NR
-> man      -> libc.so.1:char * getenv(const char * = 0x8054fe0 "MANPATH")
<- man      -> libc.so.1:getenv() = 0x80476d5
-> man      -> libc.so.1:strrchr(0x8047678, 0x2f, 0x8047608) ** NR
-> man      -> libc.so.1:int strcmp(const char * = 0x8047678 "man", const char * = 0x8054fd8 "apropos")
<- man      -> libc.so.1:strcmp() = 0xc
-> man      -> libc.so.1:int strcmp(const char * = 0x8047678 "man", const char * = 0x8055978 "whatis")
<- man      -> libc.so.1:strcmp() = 0xfffffff6
-> man      -> libc.so.1:int strcmp(const char * = 0x8047678 "man", const char * = 0x8055964 "catman")
<- man      -> libc.so.1:strcmp() = 0xa
-> man      -> libc.so.1:int getopt(int = 0x1, char *const * = 0x8047540, const char * = 0x8066420 "FfkrP:M:T:ts:lad")
<- man      -> libc.so.1:getopt() = 0xffffffff
-> man      -> libc.so.1:char * gettext(const char * = 0x8055858 "usage:        man [-] [-adFlrt] [-M path] [-T macro-package ] [ -s section ] name ...
        man [-M path] -k keyword ...
        man [-M path] -f file ...")
<- man      -> libc.so.1:gettext() = 0x8055858
-> man      -> libc.so.1:int fprintf(FILE * = 0x8067610, const char * = 0x8054f9c "%s
", void * = 0x8055858, ...)
usage:  man [-] [-adFlrt] [-M path] [-T macro-package ] [ -s section ] name ...
        man [-M path] -k keyword ...
        man [-M path] -f file ...
<- man      -> libc.so.1:fprintf() = 0x88
-> man      -> libc.so.1:exit(0x2, 0x8047608, 0x80474fc) ** NR


in the above example you can clearly see library calls like setlocale(), fprintf(), strcmp(), getenv(), etc. 

Create backup boot diskette in Solaris 10 x86

Introduction

This manual is directed to explain the process of creation of boot diskette for Solaris 10 (x86pc architecture). I know diskettes are rare those days, but it can be helpful  in case of lose boot record, overwrite by other OS, etc.

Steps

  1. Put empty diskette in drive
  2. Create FAT filesystem on diskette: mkfs -F pcfs /dev/rdiskette
  3. Stop volume manager: /etc/init.d/volmgr stop
  4. mount diskette to /mnt for example: mount -F pcfs /dev/diskette /mnt
  5. Create directory for grub: mkdir -p /mnt/boot/grub
  6. Copy boot grub files: cp -r /boot/grub/* /mnt/boot/grub
  7. Unmount diskette: umount /mnt
  8. Go to grub directory: cd /boot/grub
  9. Install grub on the diskette (use raw device): /sbin/installgrub stage1 stage2 /dev/rdiskette
  10. Eject the diskette
  11. Start volume manager (if you use it)

Patch management in AIX

Preamble

This manual is directed to give some ideas and some framework for process of management of patches for AIX. In all the cases, not cover in this manual you should contact your local UNIX guru or search for help from consultant. Information in this article is based of AIX 5L v5.2 and 5.3. For Your version please refer appropriate manuals.

Preinstallation checks

1. Check consistency and TL/ML
Let's 1st check if packages, installed on the machine are OK

# lppchk -v

If you get report of something wrong try to get more info about the case and resolve the problem before continue with installation!

# lppchk -v -m3

2. Check the current installed ML/TL
# instfix -i|grep ML

or

# instfix -i |grep TL


Keep the above information in file (not on the server, please)

3. Check for applicability

Check if APAR you want to install is applicable to your system. Check if fileset, mentioned in APAR is installed on your system

# lslpp -L <filesetname>

If not you can skip the rest of the document and feel happy, because you have to do nothing on this machine :-) But if fileset is installed check if APAR is installed

# insfix -ivk <APAR number>


If it's installed be happy and jump to the next server, otherwise check if software is in use. For example you have installed drivers for ATM card, but you have no such card in the server or have installed software for IPsec, but not configured and not in use. In this case you can (again) be happy and continue with other tasks. Otherwise continue with steps in this document.

4. Check rootvg

See if rootvg is mirrored and all lv's are mirrored correctly (excluding dump and boot volumes). If your rootvg is not mirrored you can skip later in document part for alt_disk_install, but in generally you should thing twice if situation like this is OK for the business

# lsvg -p rootvg 

# lsvg rootvg 
# lsvg -l rootvg

5. Check for cluster


Check if the server is node of cluster, because some of the operations can disrupt the proper work of cluster software.

# smit hacmp

Check if the cluster processes are active

# lssrc -g cluster

Check if cluster software is installed

# lslpp -l | grep -i cluster


6. Preinstallation tasks

Create a new LV to keep ML/TL, Service pack and APAR's you will download from IBM web site. Be aware this filesystem can need up to 3.6 gigabytes. If you do not have enough free LP you can use any (almost) already existing partition on the server or arrange some disk space on other machine and export it via NFS. After finish the installation do not forget to remove the LV or directory if you do not need the files for other installation. In case of usage of NFS you should be aware process of installation is slower and any network problem can disrupt your installation and put you in situation of recovery system from backup or alternate disk installation. Upload (download) in to this directory TL (last one), Service pack (only last one) and APARs for prepare the installation. Extract them in current directory because they normally come as tar.gz file use this trick to save some diskspace

# gzip -d -c TL...tar.gz|tar xvf -

Sound's good to extract and install Service pack separately (in different directory)

7. Backup methods


For safe yourself from problems after installation, boot problems, etc you have few opportunities. I personally know 3:

  • make alternate disk installation. This is actually process of splitting mirrored rootvg, create "copy" of current installation to unused disk and add the second one in boot list to be sure you have working copy of your operating system in case of problem
  • if you have tape device on your server use mksysb to backup you rootvg on the tape and use this backup in case of problem
  • if you have in the network TSM (Tivoli Storage Manager) and NIM (Network Installation manager) servers installed you can backup system on the TSM and in case of problem reinstall operating system from NIM server

If you can't use any of the above methods you should be prepared for installation from CD/DVD of the entire operating system and recover it to the level to the situation before installation of APAR. Honestly speaking you can use mksysb, saving backup file on NFS mounted directory, but this will be no so helpful for easy and fast restore of the system

8. Backup with alt_disk_install

So lets start with the 1st method - alternate disk installation Remove the secondary dump device. On normal systems with mirrored rootvg exist 2 different dump devices. So to free one of the disks we should remove the dump device, resided on that disk. First we should find this device. To see the name of the secondary dump device

# lsvg -l rootvg | grep dump

Then let's change the settings for secondary dump device to /dev/sysdumpnull . Do not use /dev/null

# smit dump

And now we can remove the dump Logical Volume

# smit lvm


Next step is to unmirror the rootvg. To do this we need to check which physical disks are included in rootvg. Here and below we assume disks, included in our rootvg are hdisk0 and hdisk1. In case of more that 2 disks in your rootvg process of doing this operation become more complex and it's out of scope of this document. The similar is the situation if you have many unmirrored LV in rootvg and they can't be allocated only onone PV.

# lsvg -p rootvg


Choose what disk to be leaved in rootvg based on free space, boot list and logical volumes that reside on only one of the disks

# lspv hdisk0 ; lspv hdisk1 ; lspv -l hdisk0 ; lspv -l hdisk1

See the current bootlist

# bootlist -m normal -o

Remove the second disk from the mirror

# unmirrorvg rootvg hdisk1

Check if there are any LVs left on the second disk

# lspv -l hdisk1


And if exist migrate the remaining LVs to the first disk.

# migratepv hdisk1 hdisk0

Remove the second disk from the root VG. For do this it's need to clear the boot record of the second PV

# chpv -c hdisk1

Check if the bootlist is set first to the disk with the current rootvg, and on second place the disk with the altinst_rootvg (and optionally cd0, if exist in current bootlist)

# bootlist -m normal hdisk0 hdisk1

Add new boot image to the first PV to have “fresh” boot record and be not afraid of not boot from rootvg

# bosboot -ad /dev/hdisk0

Removes the second PV from rootvg

# reducevg rootvg hdisk1

Now we have one free PV on which we can create alternative disk installation. And it is time to check the existing FS and exclude the unnecessary ones from the backup, including the FS created for the download of the TL/ML. You can check if there is already an exclude list made for tape backup

# lsvg -l rootvg 
# cat /etc/exclude.rootvg 
# cp /etc/exclude.rootvg /etc/exclude.altdisk 
# vi /etc/exclude.altdisk

The file should look like this

/opt/myfiles/tmp/*

Do not put in exclude list /tmp, /var/tmp and similar directories, because this can cause problems if you will need to use alternate disk installation for recovery purpose. If you have names of LV more that 11 characters you should rename them to be no more that 11, because of limitation of software for alternate disk installation. To do this you should first unmount the filesystem, rename the LV and mount it again. You also should avoid naming VG, LV and devices starting with letters “alt” to avoid modification, deletion or damage the items mentioned above. The command to create alternate disk installation is sample, but you should be careful of syntax, because the results can be catastrophic

# nohup alt_disk_install -BC -e /etc/exclude.altdisk hdisk1 &

We put all in background to avoid disruption of operation in case of network problems. If you do not have/use exclude files you can omit -e key and path to exclude file. For more information about the command, please see “Commands Reference, Volume 1, a – c” of AIX OS. To monitoring the process of creation use

# tail -f nohup.out

After successful end of above operation you can test alternate disk installation (using wakeup key)

# alt_disk_install -W hdisk1

List the content of PV

# lspv -l hdisk1

And if everything is OK put the installation in “sleep”

# alt_disk_install -S


9. Backup to tape

The second way to backup our rootvg is to use mksysb utility. To do this we should be sure have tape device in the machine

# lsdev -Cc tape

And we have tape in to device

# tctl -f /dev/rmt0 rewind

or

# tctl -f /dev/rmt0 status

Alternatively you can use mt instead of tctl. Then we can create exclude list for filesystems we do not want to exist on the tape image

# vi /etc/exclude.rootvg

After this we can run the backup from command line or via smit

# smit mksysb 
# nohup mksysb -m -e -i /dev/rmt0 &

If your tape device is not /dev/rmt0 change it in to the above command. If you want to create image on NFS mounted directory do the above operations, changing the target device to file instead of tape device

# smit mksysb 

# nohup /usr/bin/mksysb '-m' '-e' '-i' '-X' /backup/image_date &


10. Backup to TSM

If You want to use TSM for backup use command

# smit sysback

For the rest of process consult your local storage/backup administrator and/or documentation

11. Installation of ML/TL, Service pack and APAR

Check for some processes you should stop by hand in time of installation to avoid problems and crashes like Oracle Database server, Tivoli Gateway or TEC server. Before start installation you should commit all the software, installed on the server to make easy roll-back in case of problem in time of installation

# smit commit

or

# installp -c all


Before install TM/ML and Service pack it is wise to install/update installp software itself to have better overview later when you try to preview the process of installation of TL/ML

# smit install - bos.rte.install

or

# installp -a -g bos.rte.install


Next step is to update all the software from TL/ML and Service pack

# smit update_all
Do not forget to set only apply the package, but not commit and keep the copy of replaced files. After installation of packages check again your system:

# lppchk -v 
# oslevel -r 
# instfix -i|grep ML

or

# instfix -i|grep TL 
# instfix -i|grep SP 
# errpt |more 
# bootlist –m normal -o

And if everything looks fine reboot your server

# shutdown -Fr


12. Final steps


Recreate the mirror of rootvg. After few days of stable work and some tests from application users it's time to recreate mirror of rootvg (if you are using this way of backup). Let's destroy alternate disk installation

# alt_disk_install -X

Check which harddisk was previously included in rootvg, and add it

# extendvg -f rootvg hdisk1

Recreate secondary dump device. Extract the name and size of the primary dump device

# lsvg –l rootvg

Calculate the size of the necessary paging space with the command (in some cases is good idea to consult with your application administrators and to increase the size of secondary dump device):

# sysdumpdev -e

Create LV similar to the first one with type of LV sysdump

# smit lvm

Do not forget to change the secondary dump device from /dev/sysdumpnull to freshly created one

# smit dump

Then you can mirror rootvg. To avoid “locking” of your terminal set process in background

# smit lvm 

# nohup mirrorvg '-S' rootvg hdisk1 &

Create boot image on hdisk1

# bosboot -ad /dev/hdisk1 

 Add hdisk1 to bootlist
# bootlist -m normal hdisk0 hdisk1

Please check the bootlist you get when you check the machine before installation and add other devices to the bootlist if it's necessary. Synchronize LVs on both disks

# nohup syncvg -v rootvg &


13. Conslusion

This manual is not intended to replace your company policies/procedures, but just to give you tested way to do the work :-)

Suggestions and ideas about filesystem sizing on Solaris

Remark
This article have mainly historical value as it is created at 2008. Now the situation is quite different, diskspace is much cheaper and you don't need to be precise and careful about sizes. Also RAM is cheaper and in most of the cases you can create swap as 1GB or less.

Preambul

This document is intended to give some ideas and suggestions how to size the filesystems (and slices) in Solaris OS. Use it with cautious and do not forget to consult with official SUN documentation and your company documents, related to the subject.

Sugestions

    For / (root) filesystem normally you do not need so much disk space. Most of the time you can allocate 256 MB or ever less, but 512 MB is one very good value. Do not attempt to allocate big amount of space and put all directories as /, usr, var, opt, export/home in one filesystem. This may look very sample, but can put you in trouble after some time.
    For /usr filesystem you need average 2048 MB. On time of installation, installer will show you how much you need. Don't put much more space ever if you think to install big amount of software in /usr/local. If this happen, just create separate filesystem.
    For virtual memory i suggest to use this schema:
        < 1 GB – RAMx2.5
        <2 GB – RAMx2
        <4 GB – RAMx1,5
        >4 GB – RAMx1
        Here you should get above just as starting point, because many applications have own requirements. For example Oracle+SAP need swap to be four times RAM, almost independently of size of real memory, but Oracle itself (if it's alone) “prefer to live” in RAM.
    For /var filesystem you can allocate as minimum 512 MB, but if you expect big log files – from system, applications, etc, you should allocate more. The same is true if you expect long life and many updates on your system. Do not forget log rotate utilities, they can help you save some disk space. If your server will be used for jobs, generating big amount of data in /var as mail server or print server you can allocate more disk space, but because most of the print and mail software use /var/spool you can create dedicated filesystem for this purpose.
    Normaly /tmp filesystem is “part” of swap, but for some products as Oracle you can set dedicated filesystem. Normally 512 megabytes are enough.
    For /opt filesystem you should decide the size by self, because in this filesystem normally in Solaris are located installations of external software. 256 MB should be enough for many purposes, but some software (Oracle by example) can fill up to 1.5 GB (or more) only for executable files.
    And last but not least /export/home. Do not set this filesystem very big. Later if it's need you can move home directories and files to new filesystem or create new one like /export/home2 for new users


Ideas

  •     Do not allocated entire disk even if you use some volume manager because later when you need diskspace for new filesystem or extend existing one can be hard to get new harddisk
  •     Get indeep knowledge about the applications will run on this machine for better filesystems sizing
  •     Do not oversize your filesystems and do not set them as little as possible. 20-30% over from the requirements is OK as general rule

Fast guide to MySQL - part 2

Next part of subset of MySQL commands (for UNIX/Linux admins)

1. Display list of databases

SHOW DATABASES;

2. Display list of tables in current database

SHOW TABLES;


3. Display list of tables in noncurrent database

SHOW TABLES FROM database_name;

Fast guide to MySQL

This is just very little subset of command to help beginners with MySQL. Do not expect something extraordinary :)
All the command must be executed from mysql command prompt which you can reach with command:
mysql -u username -p password


1. Create database

CREATE DATABASE mybase;

2. Switch to use database

USE mybase;


3. Create user

CREATE USER myuser IDENTIFIED BY 'password';


4. Assign all right of user to database

GRANT ALL PRIVILEGES ON mybase.* TO 'myuser';

5. Flush privileges (to make them work at the moment)

FLUSH PRIVILEGES;

Be aware command are all caps, with small caps are parameters

whodo on Linux way

Many times I become angry because of the absence of one UNIX command in Linux - whodo. So I try to write sample script to imitate it. See below:

for i in `who|tail -3|awk '{print $1}'`
do
who|grep -w $i
pstree -pu $i
done


I know the output is different, but for my personal purpose this is enough :)

Dealing with variables in shell environment - canonical way

One of the very important elements of interactive work with UNIX/Linux is a shell, especially variables in shell environment. To work with variables sometime can be easy, sometime hard. So here I will try to show some of the ideas how to deal with them.

1. how to see the variable: use one of the commands:

set|grep VAR
or

echo $VAR

2. how to set variable:

VAR=value;export VAR

or if you want to add some value to the existing in the variable

VAR=$VAR:value;export VAR

Of course export is need only in case you want to populate value to subinstrances of current shell. I use this way because it work on bourne shell, korn shell and bash. If you are sure script will work only in bash and/or korn shell use short form:

export VAR=value

3. how to destroy the variable:

unset VAR

I can use

VAR="";export VAR

but this only will "zero" the content and not destroy the variable

4. how to change (inside) the content of variable. You can make it entering the entire string (or number), but this little script will help you do it on the fly. Do not forget script is just an example and you should "tune" it to do the real work

VAR=`echo $VAR|sed s/string//g`;export VAR

Shell in nutshell or little tips and tricks

Today I will try to show  some tricks in UNIX/Linux shells maybe you do not know:

- use . (dot) to execute command or script in the current instance of the shell, this help much when you try to set some variables or change to directory with long and complicated path

- use - (dash) to go to previous working directory. this do not work in bourne shell

- use ${n..nn} in bash to generate sequence of numbers. Maybe was something similar in korn shell, but I can't remember.

$ printf "%03d " {1..100}


Also you can use seq program (mainly available in Linux)

$ printf "%03d " `seq 1 100`


- use bc to make arithmetic calculations on the fly:

# echo 2^15|bc

32768


- printf is very helpfull when you try to create well formatted output. Just follow the same format as in C

To be continued (maybe)...

Setting IP on Solaris - canonical way

To set IP address in Solaris you should first check if there is such network adapter. This can be done on many ways:

dmesg or from /var/adm/messages:

Sep 18 11:50:10 sun3 mac: [ID 469746 kern.info] NOTICE: e1000g3 registered


dladm:

e1000g3 link: up speed: 1000 Mbps duplex: full

prtconf:

dev_path=/pci@0,0/pci15ad,750@14:e1000g3 dev_link=/dev/e1000g3


Set in /etc/hosts record for this IP/hostname

10.0.1.32 yourhostname


Set netmask for this IP in /etc/netmasks
10.0.1.0 255.255.255.0

Next step is to set in /etc the file hostname.e1000g3 with content:

yourhostname netmask + broadcast + up

The previous three steps are done to have interface up on boot of the system

Plumb the interface

ifconfig e1000g3 plumb

And get it up and running

ifconfig e1000g3 `cat /etc/hostname.e1000g3`

To set your default gateway create file /etc/defaultrouter and put inside the IP address of your gateway

My best friend - cron

I love cron, but there are few details missed from most of the users:

1. It's wise to set only one command to be executed, do not play with one long line, just a little script. This will help to avoid errors and problems.

2. Force set environment variables you need for script. Script run by cron have very limited set of variables. Better add as first line of script:

. /my/home/.profile
or startup file for your shell (.bash_profile for example if you use bash as default shell)

3. As first line in your script set shell need for execution, do not rely on default one. This is named shebang and here you can find very good explanation what is this and how to use it.

4. Do not play with tricky sets of minutes, hours, etc. Use full citation, this will work on all UNIXes and Linux and give much better visibility:

0,10,20,30,40,50 * * * * /path/myscript


5. Be careful when you use the days of week.  Beside other time variables which "use" AND as logical function between them Day of week use OR to the day of month.

Tar - the game continue

Sometime, some sysadmins forget one little detail about UNIX tar and make archives with leading slash. This can be helpful when you try to restore fast some files or directories, but in most of the cases is very bad habit. In many cases it's just need to extract somewhere (but not on the original place) file and compare it with the current file. So you have archive with leading slash and want to extract something. Let do it on a little complicated, but safe way:

mkdir -p /tmp/lib
for i in `ldd /usr/bin/tar|awk  '{print $3}'`;
do
cp $i /tmp/lib
done
cp /usr/bin/tar /tmp
mkdir -p /tmp/usr/lib
cp /usr/lib/ld.so.1 /tmp/usr/lib
chroot /tmp ./tar xvf archive.tar


In the man page, I get the idea, (SUN Solaris chroot) was mentioned full list of so libraries, but I am lazy and hate to write. And want to avoid mistakes. BTW in the documentation is not mentioned ld.so library and without this library tar do not want to run :-)

P.S. The script was tested only on Solaris 10, but (depend of format of ldd) will work on other UNIXes and Linux
P.P.S. You can use pax or GNU tar but as they are not a standard in UNIX world I try to make my solution universal

Playing with tar

Every sysadmin know that tar program do not accept wildcards for extract files so if you want to extract only one or few files you should play with construction like this:
 tar xvf archive.tar `tar tvf archive.tar|grep string` 
But on complex requests will be need to do it several times and if archive reside on tape (do not forget from where come the name of utility tar) this can take long time. Instead is much better to deal with files:
tar tvf archive.tar>/tmp/somefilename 
make some filtering on the listing
tar xvf archive.tar `cat /tmp/somefilename` 
P.S. I use cat to make command work independently (almost) of version of tar, UNIX flavour or Linux

Play with compressed tar files

Time to time every system administrator or user should extract file or entire archive from compressed tar, but quite often in filesystems there is not enough free space. In this case it's possible to extract file(s) on the fly without first uncompress and then untar:

gzip -dc archive.tar.gz|tar xf -

The above line can be modified when you should deal with broken archives. If you try to decompress such archive gzip just stop when find error in compression. But you want to save what's possible. So let's make it on this way:

cat archive.tar.gz|gzip -dc|tar xf -

I know it is possible to use GNU tar to extract files from compressed archive, but in UNIX world you can't be sure this software is installed :-)

Few remarks:
  1. It's wise to use 'f -' key because in some Unixes without explicit 'f' tar will use tape device or whatever is defiined in TAPE variable
  2. In above examples beside less diskpace usage you benefit from parallel execution (from pipe)

Welcome

Welcome to my technical blog. Here I will try to provide little tips, tricks and ideas, related to UNIX/Linux. And here is the first: Use this code to get size of directories. This will help you easy find where diskspace disappear:
du -sk /directory/*
Here is one example
du -sk /var/*
0       /var/account
0       /var/adm
149752  /var/cache
0       /var/crash
8       /var/db
0       /var/empty
0       /var/games
0       /var/gopher
0       /var/kerberos
208928  /var/lib
0       /var/local
0       /var/lock
5868    /var/log
0       /var/mail
0       /var/nis
0       /var/opt
0       /var/preserve
0       /var/run
4       /var/spool
0       /var/target
179572  /var/tmp

But we should browse a lot so let sort the output by size
du -sk /var/*|sort -g
0       /var/account
0       /var/adm
0       /var/crash
0       /var/empty
0       /var/games
0       /var/gopher
0       /var/kerberos
0       /var/local
0       /var/lock
0       /var/mail
0       /var/nis
0       /var/opt       
0       /var/preserve  
0       /var/run       
0       /var/target    
0       /var/yp        
4       /var/spool     
8       /var/db        
5868    /var/log       
149752  /var/cache     
179572  /var/tmp       
208928  /var/lib  
 
And we do not need so much lines, so let get only few last
du -sk /var/*|sort -g|tail -5                       
8       /var/db
5872    /var/log
149752  /var/cache
179572  /var/tmp
208928  /var/lib

E voila, we have candidate directories to search for space eater

Compressed tar archive

There are some cases when you want to create compressed tar archive but you do not have enough disk space to keep original files and tar arc...