Solaris commands for process management - truss

truss – for tracing user/library/system calls and signals

bash-3.2# truss ls
execve("/usr/bin/ls", 0x0804759C, 0x080475A4)  argc = 1
sysinfo(SI_MACHINE, "i86pc", 257)               = 6
mmap(0x00000000, 32, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFEFF0000
mmap(0x00000000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFEFB0000
mmap(0x00000000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFEFA0000
memcntl(0xFEFBE000, 13408, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
memcntl(0x08050000, 3784, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
resolvepath("/usr/lib/ld.so.1", "/lib/ld.so.1", 1023) = 12
resolvepath("/usr/bin/ls", "/usr/bin/ls", 1023) = 11
sysconfig(_CONFIG_PAGESIZE)                     = 4096
stat64("/usr/bin/ls", 0x08047180)               = 0
open("/var/ld/ld.config", O_RDONLY)             Err#2 ENOENT
stat64("/lib/libsec.so.1", 0x08046A30)          = 0
resolvepath("/lib/libsec.so.1", "/lib/libsec.so.1", 1023) = 16
open("/lib/libsec.so.1", O_RDONLY)              = 3

Shows you system calls made by a process along with parameters of the function - very good for program execution traces. Use -u a.out to see user-level function calls. Note "a.out" is typed literally as "a.out" - not replaced with the name of the program

Solaris commands for process management - pwdx

pwdx – list working directory for process, like pwd command

bash-3.2# pwdx 867
867:    /var/spool/clientmqueue

Can be helpful in case of strange errors when unmount some filesystem

Solaris commands for process management - pwait

pwait – wait till process finish

# ps -efl|grep name 0 S root 875 1 0 45 20 d4188950 1298 d553ab16 22:17:58 ? 0:00 /usr/sbin/named # pwait 875 (in this moment from another terminal I execute svcadm disable dns/server)

Maybe this command will be great in some specific scripts

Solaris commands for process management - ptree

ptree - print the tree of processes
No need for comments :-)


bash-3.2# ptree
11    /lib/svc/bin/svc.startd
  343   /usr/lib/saf/sac -t 300
    356   /usr/lib/saf/ttymon
  362   /usr/lib/saf/ttymon -g -d /dev/console -l console -T sun-color -m ldterm,ttcomp
13    /lib/svc/bin/svc.configd
74    devfsadmd
128   /usr/lib/sysevent/syseventd
129   /usr/lib/crypto/kcfd
150   /usr/lib/power/powerd
155   /usr/lib/picl/picld
214   /lib/svc/method/iscsi-initiator
230   /usr/sbin/cron
281   /usr/bin/VBoxService
304   /usr/sbin/rpcbind
308   /usr/lib/autofs/automountd
  309   /usr/lib/autofs/automountd
340   /usr/lib/inet/inetd start
355   /usr/lib/utmpd
363   /usr/lib/fm/fmd/fmd
375   /usr/lib/ssh/sshd
405   /usr/sbin/syslogd
421   /usr/sadm/lib/smc/bin/smcboot
  424   /usr/sadm/lib/smc/bin/smcboot
  425   /usr/sadm/lib/smc/bin/smcboot
493   /usr/dt/bin/dtlogin -daemon -udpPort 0
  494   /usr/openwin/bin/fbconsole -n -d :0
  502   /usr/X11/bin/Xorg :0 -depth 24 -nolisten tcp -nobanner -auth /var/dt/A:0-iUaO9a
  564   /usr/dt/bin/dtlogin -daemon -udpPort 0
    623   /bin/ksh /usr/dt/bin/Xsession
      707   /usr/dt/bin/sdt_shell -c       unset DT;      DISPLAY=:0;        /usr/dt/bin/dt
        709   -sh -c       unset DT;      DISPLAY=:0;        /usr/dt/bin/dtsession_res -mer
          721   /bin/ksh /usr/dt/config/Xsession2.jds
            723   /usr/bin/gnome-session
<snip>

Solaris commands for process management - pstop

pstop – temporary hold process

# ps -efl|grep ssh 
0 S root 554 356 0 40 20 d419b870 1800 d4c87e72 21:03:52 ? 0:00 /usr/lib/ssh/sshd 
0 S root 356 1 0 40 20 d4195320 834 d49f3432 21:02:46 ? 0:00 /usr/lib/ssh/sshd 
0 S root 560 554 0 40 20 d41931a0 1747 d4c87ab2 21:05:03 ? 0:02 /usr/lib/ssh/sshd 
0 S root 812 809 0 40 20 d418a270 1746 d50402f2 22:09:00 ? 0:00 /usr/lib/ssh/sshd 
0 S root 809 356 0 40 20 d4189a10 1801 d4c874f2 22:08:47 ? 0:00 /usr/lib/ssh/sshd 
# pstop 356 
# ssh localhost (here I stop with ctrl-C ssh client after 5 minutes of waiting) 
# prun 356 
# ssh localhost 
The authenticity of host 'localhost (127.0.0.1)' can't be established. RSA key fingerprint is da:14:db:bc:2f:e0:0f:ef:f6:3f:12:0e:f4:77:0c:00. Are you sure you want to continue connecting (yes/no)? yes 
Warning: Permanently added 'localhost' (RSA) to the list of known hosts. 
Password: 
Last login: Wed Jun 13 22:09:00 2007 from gateway Sun Microsystems Inc. SunOS 5.10 Generic January 2005 

Solaris commands for process management - psig

psig – list signals that can be handled by process

bash-3.2# psig 867
867:    /usr/lib/sendmail -Ac -q15m
HUP     caught  0x809b340       RESTART,SIGINFO
INT     ignored
QUIT    ignored
ILL     default
TRAP    default
ABRT    default
EMT     default
FPE     default
KILL    default
BUS     default
SEGV    default
SYS     default
PIPE    ignored
ALRM    caught  sm_tick         RESTART,SIGINFO
TERM    caught  0x809b2af       RESTART,SIGINFO
USR1    caught  0x8081a32       RESTART,SIGINFO
USR2    default
CLD     caught  reapchild       RESTART,SIGINFO
<snip>

Solaris commands for process management - pstack

pstack – get backtrace stack of process for debugging purposes

bash-3.2# pstack 706
706:    /usr/dt/bin/dsdm
 fedcc3e5 pollsys  (8046b60, 1, 0, 0)
 fed7559a pselect  (4, 8046c00, fee323d8, fee323d8, 0, 0) + 18e
 fed75890 select   (4, 8046c00, 0, 0, 0) + 82
 fee5ab6b _XWaitForReadable (80651d0) + cf
 fee5a991 _XRead   (80651d0, 8046cf0, 20) + a9
 fee89811 _XReadEvents (80651d0) + 1e9
 fee65b12 XNextEvent (80651d0, 80475a0) + 46
 08052976 main     (1, 80476a0, 80476a8) + 1cc
 080517fa ???????? (1, 80477b0, 0, 80477c1, 80477d4, 8047808)



This command can sometimes be very helpful for debugging daemons. You can see the function call stack by thread, in real-time

Solaris commands for process management - ps

ps - print process information and status. In Solaris exist SYSV and BSD variants, respectively /usr/bin/ps and /usr/ucb/ps

bash-3.2# /usr/bin/ps
   PID TTY         TIME CMD
   866 pts/3       0:00 bash
   786 pts/3       0:00 sh
  1007 pts/3       0:00 ps
 
 bash-3.2# /usr/ucb/ps
   PID TT       S  TIME COMMAND
   362 console  S  0:00 /usr/lib/saf/ttymon -g -d /dev/console -l console -T sun-color -m l
   707 pts/2    S  0:00 /usr/dt/bin/sdt_shell -c      unset DT;     DISPLAY=:0;       /usr/
   709 pts/2    S  0:00 -sh -c      unset DT;     DISPLAY=:0;       /usr/dt/bin/dtsession_r
   721 pts/2    S  0:00 /bin/ksh /usr/dt/config/Xsession2.jds
   723 pts/2    S  0:00 /usr/bin/gnome-session
   786 pts/3    S  0:00 sh
   866 pts/3    S  0:00 bash
  1008 pts/3    O  0:00 /usr/ucb/ps

Solaris commands for process management - prun

prun – continue holded with pstop process, see example of pstop

Solaris commands for process management -prstat

 prstat – fullscreen view of processes sorted by different criteria, similar to Linux top command
  PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
   502 root       48M   36M sleep   59    0   0:00:09 0.2% Xorg/1
   779 root       68M   12M sleep   58    0   0:00:08 0.1% mixer_applet2/1
   605 noaccess  162M  116M sleep   59    0   0:00:13 0.1% java/18
   783 root       75M   16M sleep   49    0   0:00:03 0.0% gnome-terminal/2
   729 root       12M 9752K sleep   59    0   0:00:04 0.0% gconfd-2/1
  1004 root     3808K 2960K cpu0    59    0   0:00:00 0.0% prstat/1
   734 root     6644K 4008K sleep   59    0   0:00:00 0.0% xscreensaver/1
   760 root       73M   18M sleep   59    0   0:00:03 0.0% gnome-panel/1
   756 root       67M   14M sleep   59    0   0:00:00 0.0% metacity/1
   777 root       71M   15M sleep   59    0   0:00:01 0.0% gnome-netstatus/1
   281 root     3780K 1544K sleep   59    0   0:00:01 0.0% VBoxService/7
     5 root        0K    0K sleep   99  -20   0:00:02 0.0% zpool-rpool/138
   751 root     7076K 3184K sleep   59    0   0:00:00 0.0% gnome-smproxy/1
   866 root     2992K 1772K sleep   37    0   0:00:00 0.0% bash/1
   965 root     4376K 2528K sleep   59    0   0:00:00 0.0% nscd/19
   867 smmsp    9184K 2112K sleep   59    0   0:00:00 0.0% sendmail/1
   559 root     3064K 2064K sleep   59    0   0:00:00 0.0% vold/5
   707 root     4616K 1760K sleep   59    0   0:00:00 0.0% sdt_shell/1
   623 root     1608K 1008K sleep   59    0   0:00:00 0.0% Xsession/1
<snip>
Total: 69 processes, 347 lwps, load averages: 0.02, 0.01, 0.01


The program in default view is less informative that top, but can give you info for zones, projects and even for processes bind to specific processor. Use the -u option to limit display to specific users. Use the -p option to limit display to specific processes

Solaris commands for process management - preap

preap – try to kick-off zombie process



# ps -efl|grep 551 
0 Z root 551 550 0 0 - - 0 - - ? 0:00 <defunct> 
# preap 551 
551: exited with status 0 
# ps -efl|grep 551



In above example, this tool successfully removes zombie process. Otherwise the only way to kill them is to reboot

Solaris commands for process management - pmap

pmap – get memory map (segments) of process

bash-3.2# pmap 867
867:    /usr/lib/sendmail -Ac -q15m
08037000      68K rw---    [ stack ]
08050000     780K r-x--  /usr/lib/sendmail
08123000      72K rw---  /usr/lib/sendmail
08135000     748K rw---    [ heap ]
FE5B0000      64K rwx--    [ anon ]
FE5D0000      64K rwx--    [ anon ]
FE5F0000      64K rwx--    [ anon ]
FE610000     388K r-x--  /usr/lib/mps/libssl3.so
FE680000      12K rwx--  /usr/lib/mps/libssl3.so
FE690000      12K r-x--  /usr/lib/mps/libplds4.so
FE6A2000       4K rwx--  /usr/lib/mps/libplds4.so
FE6B0000     116K r-x--  /usr/lib/mps/libnssutil3.so
FE6DC000      16K rwx--  /usr/lib/mps/libnssutil3.so
FE700000    1816K r-x--  /usr/lib/mps/libnss3.so
FE8D5000      32K rwx--  /usr/lib/mps/libnss3.so
FE8E0000      12K r-x--  /usr/lib/mps/libplc4.so
FE8F2000       8K rwx--  /usr/lib/mps/libplc4.so
FE900000      64K r-x--  /usr/lib/libsasl.so.1
FE910000       4K rw---  /usr/lib/libsasl.so.1
FE920000      12K r-x--  /lib/libmp.so.2
FE933000       4K rw---  /lib/libmp.so.2
FE940000       4K r-x--  /lib/libdl.so.1
FE951000       4K rw---  /lib/libdl.so.1
FE960000     288K r-x--  /usr/lib/mps/libnspr4.so
FE9B7000       8K rwx--  /usr/lib/mps/libnspr4.so
FE9B9000       8K rwx--  /usr/lib/mps/libnspr4.so
FE9C0000      40K r-x--  /usr/sfw/lib/libcrypto_extra.so.0.9.7
FE9D0000       4K rwx--    [ anon ]
FE9DA000       4K rw---  /usr/sfw/lib/libcrypto_extra.so.0.9.7
FE9E0000      40K r-x--  /usr/sfw/lib/libssl_extra.so.0.9.7
FE9F0000       4K rwx--    [ anon ]
FE9FA000       4K rw---  /usr/sfw/lib/libssl_extra.so.0.9.7
FEA00000      24K r-x--  /lib/libgen.so.1
FEA16000       4K rw---  /lib/libgen.so.1
FEA20000      24K r-x--  /lib/libuutil.so.1
FEA30000       4K rwx--    [ anon ]
FEA36000       4K rw---  /lib/libuutil.so.1
FEA40000      56K r-x--  /lib/libmd.so.1
FEA50000      12K r-x--  /lib/libpthread.so.1
FEA5E000       4K rw---  /lib/libmd.so.1
FEA60000      32K r-x--  /lib/libaio.so.1
FEA70000      16K r-x--  /lib/libthread.so.1
FEA78000       4K rw---  /lib/libaio.so.1
FEA79000       4K rw---  /lib/libaio.so.1
FEA80000       8K r-x--  /lib/libdoor.so.1
FEA90000       4K r--s-  dev:307,3 ino:4043961197
FEA92000       4K rw---  /lib/libdoor.so.1
FEAA0000      92K r-x--  /lib/libscf.so.1
FEAC0000       4K rwx--    [ anon ]
FEAC7000       4K rw---  /lib/libscf.so.1
FEAD0000    1088K r-x--  /lib/libc.so.1
FEBE0000      32K rwx--  /lib/libc.so.1
FEBE8000       8K rwx--  /lib/libc.so.1
FEBF0000     956K r-x--  /usr/sfw/lib/libcrypto.so.0.9.7
FECE0000      24K rwx--    [ anon ]
FECEF000      80K rw---  /usr/sfw/lib/libcrypto.so.0.9.7
FED03000       8K rw---  /usr/sfw/lib/libcrypto.so.0.9.7
FED10000     208K r-x--  /usr/sfw/lib/libssl.so.0.9.7
FED50000       4K rwx--    [ anon ]
FED54000      12K rw---  /usr/sfw/lib/libssl.so.0.9.7
FED60000      24K r-x--  /lib/librt.so.1
FED70000       4K rwx--    [ anon ]
FED76000       4K rw---  /lib/librt.so.1
FED80000      88K r-x--  /lib/libumem.so.1
FEDA0000       4K rwx--    [ anon ]
FEDA6000      20K rw---  /lib/libumem.so.1
FEDAB000      24K rw---  /lib/libumem.so.1
FEDC0000      24K r-x--  /usr/sfw/lib/libwrap.so.1.0
FEDD0000       4K rwx--    [ anon ]
FEDD6000       4K rw---  /usr/sfw/lib/libwrap.so.1.0
FEDD7000       4K rw---  /usr/sfw/lib/libwrap.so.1.0
FEDE0000     192K r-x--  /usr/lib/libsldap.so.1
FEE10000      12K rw---  /usr/lib/libsldap.so.1
FEE20000     204K r-x--  /usr/lib/libldap.so.5
FEE60000       4K rwx--    [ anon ]
FEE63000      12K rw---  /usr/lib/libldap.so.5
FEE70000     516K r-x--  /lib/libnsl.so.1
FEF01000      20K rw---  /lib/libnsl.so.1
FEF06000      32K rw---  /lib/libnsl.so.1
FEF10000      44K r-x--  /lib/libsocket.so.1
FEF20000       4K rwx--    [ anon ]
FEF2B000       4K rw---  /lib/libsocket.so.1
FEF30000       4K rwx--    [ anon ]
FEF40000     216K r-x--  /lib/libresolv.so.2
FEF80000       4K rwx--    [ anon ]
FEF86000       8K rw---  /lib/libresolv.so.2
FEF90000       4K rwx--    [ anon ]
FEFA0000       4K rw---    [ anon ]
FEFB0000       4K rw---    [ anon ]
FEFBA000       4K rwxs-    [ anon ]
FEFBE000     176K r-x--  /lib/ld.so.1
FEFF0000       4K rwx--    [ anon ]
FEFFA000       8K rwx--  /lib/ld.so.1
FEFFC000       8K rwx--  /lib/ld.so.1
 total      9184K


This command helps in the process of fine-tuning large applications, especially for resolving memory leak issues

Solaris commands for process management - pldd

pldd – list dynamic libraries, associated with process, similar to ldd for executable

 bash-3.2# ps -efl|grep send
 0 S    smmsp   867     1   0  40 20        ?   2296        ? 18:02:34 ?           0:00 /usr/lib/sendmail -Ac -q15m
bash-3.2# pldd 867
867:    /usr/lib/sendmail -Ac -q15m
/lib/libumem.so.1
/lib/libresolv.so.2
/lib/libsocket.so.1
/lib/libnsl.so.1
/usr/lib/libldap.so.5
/usr/lib/libsldap.so.1
/usr/sfw/lib/libwrap.so.1.0
/lib/librt.so.1
/usr/sfw/lib/libssl.so.0.9.7
/usr/sfw/lib/libcrypto.so.0.9.7
/lib/libc.so.1
/lib/libscf.so.1
/lib/libdoor.so.1
/lib/libaio.so.1
/lib/libmd.so.1
/lib/libuutil.so.1
/lib/libgen.so.1
/usr/sfw/lib/libssl_extra.so.0.9.7
/usr/sfw/lib/libcrypto_extra.so.0.9.7
/usr/lib/mps/libnspr4.so
/lib/libpthread.so.1
/lib/libdl.so.1
/lib/libmp.so.2
/usr/lib/libsasl.so.1
/usr/lib/mps/libplc4.so
/usr/lib/mps/libnss3.so
/usr/lib/mps/libnssutil3.so
/usr/lib/mps/libplds4.so
/usr/lib/mps/libssl3.so
/lib/libthread.so.1
But let execure ldd command for the same file:
bash-3.2# ldd /usr/lib/sendmail
        libresolv.so.2 =>        /lib/libresolv.so.2
        libsocket.so.1 =>        /lib/libsocket.so.1
        libnsl.so.1 =>   /lib/libnsl.so.1
        libldap.so.5 =>  /usr/lib/libldap.so.5
        libsldap.so.1 =>         /usr/lib/libsldap.so.1
        libwrap.so.1 =>  /usr/sfw/lib/libwrap.so.1
        libumem.so.1 =>  /lib/libumem.so.1
        librt.so.1 =>    /lib/librt.so.1
        libssl.so.0.9.7 =>       /usr/sfw/lib/libssl.so.0.9.7
        libcrypto.so.0.9.7 =>    /usr/sfw/lib/libcrypto.so.0.9.7
        libc.so.1 =>     /lib/libc.so.1
        libmp.so.2 =>    /lib/libmp.so.2
        libmd.so.1 =>    /lib/libmd.so.1
        libscf.so.1 =>   /lib/libscf.so.1
        libsasl.so.1 =>  /usr/lib/libsasl.so.1
        libnspr4.so =>   /usr/lib/mps/libnspr4.so
        libplc4.so =>    /usr/lib/mps/libplc4.so
        libnss3.so =>    /usr/lib/mps/libnss3.so
        libssl3.so =>    /usr/lib/mps/libssl3.so
        libdoor.so.1 =>  /lib/libdoor.so.1
        libaio.so.1 =>   /lib/libaio.so.1
        libuutil.so.1 =>         /lib/libuutil.so.1
        libgen.so.1 =>   /lib/libgen.so.1
        libpthread.so.1 =>       /lib/libpthread.so.1
        libdl.so.1 =>    /lib/libdl.so.1
        libnssutil3.so =>        /usr/lib/mps/libnssutil3.so
        libplds4.so =>   /usr/lib/mps/libplds4.so
        libthread.so.1 =>        /lib/libthread.so.1
        libssl_extra.so.0.9.7 =>         /usr/sfw/lib/libssl_extra.so.0.9.7
        libcrypto_extra.so.0.9.7 =>      /usr/sfw/lib/libcrypto_extra.so.0.9.7
        libm.so.2 =>     /lib/libm.so.2

Results from both commands are almost the same :-) Why? Ask SUN/Oracle

Solaris commands for process management - pkill

pkill – send signal to process. For example pkill -9 init :-P


# pgrep named 
628 
# pkill named 
# pgrep named 
636


As you see, the named daemon was killed and respawned (with new PID)

Solaris commands for process management - pgrep

pgrep – get the PID's of processes by name i.e. Something like ps -efl|grep -v grep|grep process_name



# pgrep named 
586

Something like ps -efl|grep -v grep|grep process_name|awk '{print $1}'

# ps -e|grep -v grep|grep named|awk '{print $1}' 
586

Solaris commands for process management - pargs

pargs – get list of arguments and environment variables with which process was started

bash-3.2# pargs 889
889:    /usr/lib/sendmail -bl -q15m
argv[0]: sendmail: accepting connections
argv[1]: <NULL>
argv[2]: nnections
bash-3.2# pargs 867
867:    /usr/lib/sendmail -Ac -q15m
argv[0]: sendmail: Queue runner@00:15:00 for /var/spool/clientmqueue
argv[1]: <NULL>
argv[2]: @00:15:00 for /var/spool/clientmqueue

in above example you see the arguments for starting sendmail as a daemon

Solaris commands for process management - pfiles

pfiles – list of file descriptors, associated with process


bash-3.2# pfiles 889
889:    /usr/lib/sendmail -bl -q15m
  Current rlimit: 1024 file descriptors
   0: S_IFCHR mode:0666 dev:303,0 ino:6815752 uid:0 gid:3 rdev:13,2
      O_RDONLY|O_LARGEFILE
      /devices/pseudo/mm@0:null
   1: S_IFCHR mode:0666 dev:303,0 ino:6815752 uid:0 gid:3 rdev:13,2
      O_WRONLY|O_LARGEFILE
      /devices/pseudo/mm@0:null
   2: S_IFCHR mode:0666 dev:303,0 ino:6815752 uid:0 gid:3 rdev:13,2
      O_WRONLY|O_LARGEFILE
      /devices/pseudo/mm@0:null
   3: S_IFDOOR mode:0444 dev:312,0 ino:55 uid:0 gid:0 size:0
      O_RDONLY|O_LARGEFILE FD_CLOEXEC  door to nscd[144]
      /var/run/name_service_door
   4: S_IFCHR mode:0000 dev:303,0 ino:57176 uid:0 gid:0 rdev:21,33
      O_WRONLY FD_CLOEXEC
      /devices/pseudo/log@0:conslog
   5: S_IFSOCK mode:0666 dev:310,0 ino:61053 uid:0 gid:0 size:0
      O_RDWR FD_CLOEXEC
        SOCK_STREAM
        SO_REUSEADDR,SO_KEEPALIVE,SO_SNDBUF(49152),SO_RCVBUF(49152)
        sockname: AF_INET 127.0.0.1  port: 25
   6: S_IFSOCK mode:0666 dev:310,0 ino:61054 uid:0 gid:0 size:0
      O_RDWR FD_CLOEXEC
        SOCK_STREAM
        SO_REUSEADDR,SO_KEEPALIVE,SO_SNDBUF(49152),SO_RCVBUF(49152)
        sockname: AF_INET 127.0.0.1  port: 587
   7: S_IFREG mode:0600 dev:307,3 ino:4059345121 uid:0 gid:25 size:32
      O_WRONLY|O_CREAT|O_EXCL|O_LARGEFILE
      /var/run/sendmail.pid



Here you see a list of files as NULL, console log, some PID files. On the top you can see the maximum number of file handlers for this process, with info that can be helpful for troubleshooting of applications

Solaris commands for process management - apptrace

apptrace – this commend is used for tracing library calls

bash-3.2# apptrace man
-> man      -> libc.so.1:int atexit(int (*)() = 0xfefce738)
<- man      -> libc.so.1:atexit()
-> man      -> libc.so.1:int atexit(int (*)() = 0x8054eac)
<- man      -> libc.so.1:atexit()
-> man      -> libc.so.1:void __fpstart(void)
<- man      -> libc.so.1:__fpstart() = 0xfecd8754
-> man      -> libc.so.1:access(0x80559a4, 0x1, 0x8047608) ** NR
-> man      -> libc.so.1:char * setlocale(int = 0x6, const char * = 0x80559a0 "")
<- man      -> libc.so.1:setlocale() = 0xfec7c045
-> man      -> libc.so.1:char * setlocale(int = 0x5, const char * = 0x0 <NULL>)
<- man      -> libc.so.1:setlocale() = 0xfec7c045
-> man      -> libc.so.1:strcpy(0x80669e8, 0xfec7c045, 0x8047608) ** NR
-> man      -> libc.so.1:int strcmp(const char * = 0x805599c "C", const char * = 0x80669e8 "C")
<- man      -> libc.so.1:strcmp()
-> man      -> libc.so.1:char * textdomain(const char * = 0x8055988 "SUNW_OST_OSCMD")
<- man      -> libc.so.1:textdomain() = 0x80683e0
-> man      -> libc.so.1:strcpy(0x80665e8, 0x8055980, 0x8047608) ** NR
-> man      -> libc.so.1:char * getenv(const char * = 0x8054fe0 "MANPATH")
<- man      -> libc.so.1:getenv() = 0x80476d5
-> man      -> libc.so.1:strrchr(0x8047678, 0x2f, 0x8047608) ** NR
-> man      -> libc.so.1:int strcmp(const char * = 0x8047678 "man", const char * = 0x8054fd8 "apropos")
<- man      -> libc.so.1:strcmp() = 0xc
-> man      -> libc.so.1:int strcmp(const char * = 0x8047678 "man", const char * = 0x8055978 "whatis")
<- man      -> libc.so.1:strcmp() = 0xfffffff6
-> man      -> libc.so.1:int strcmp(const char * = 0x8047678 "man", const char * = 0x8055964 "catman")
<- man      -> libc.so.1:strcmp() = 0xa
-> man      -> libc.so.1:int getopt(int = 0x1, char *const * = 0x8047540, const char * = 0x8066420 "FfkrP:M:T:ts:lad")
<- man      -> libc.so.1:getopt() = 0xffffffff
-> man      -> libc.so.1:char * gettext(const char * = 0x8055858 "usage:        man [-] [-adFlrt] [-M path] [-T macro-package ] [ -s section ] name ...
        man [-M path] -k keyword ...
        man [-M path] -f file ...")
<- man      -> libc.so.1:gettext() = 0x8055858
-> man      -> libc.so.1:int fprintf(FILE * = 0x8067610, const char * = 0x8054f9c "%s
", void * = 0x8055858, ...)
usage:  man [-] [-adFlrt] [-M path] [-T macro-package ] [ -s section ] name ...
        man [-M path] -k keyword ...
        man [-M path] -f file ...
<- man      -> libc.so.1:fprintf() = 0x88
-> man      -> libc.so.1:exit(0x2, 0x8047608, 0x80474fc) ** NR

in the above example you can clearly see library calls like setlocale(), fprintf(), strcmp(), getenv(), etc. 

whodo on Linux way

Many times I become angry because of the absence of one UNIX command in Linux - whodo. So I try to write sample script to imitate it. See below:

for i in `who|tail -3|awk '{print $1}'`
do
who|grep -w $i
pstree -pu $i
done

I know the output is different, but for my personal purpose this is enough :)