Audit (Solaris) - something you have, something you do not know - part 1

Audit subsystem in Solaris is very powerful and can give you a lot of information. In this first article I will give you example how to play with audit for monitoring user actions
1. Check the user i want to monitor
# id test1
uid=200(test1) gid=1(other)

2. And the actions I want to record
# cat /etc/security/audit_class
#
.....
# File Format:
#
#       mask:name:description
#
0x00000000:no:invalid class
...
0xffffffff:all:all classes (meta-class)

Of course in above file you will see much more classes and you can set to monitor and audit only part of them, but for this example I will use monitoring all
3. Set user and classes
# cat /etc/security/audit_user
#
...
#       username:always:never
#
root:lo:no
test1:all:no

So here I will monitor everything and no exceptions
4. Run bsmconv because of first usage of audit
# /etc/security/bsmconv
This script is used to enable the Basic Security Module (BSM).
Shall we continue with the conversion now? [y/n] y
bsmconv: INFO: checking startup file.
bsmconv: INFO: turning on audit module.
bsmconv: INFO: initializing device allocation.

The Basic Security Module is ready.
If there were any errors, please fix them now.
Configure BSM by editing files located in /etc/security.
Reboot this system now to come up with BSM enabled.

5. And reboot
# init 6
updating /platform/i86pc/boot_archive
6. Login as test1 do some actions and exit
login as: test1
Password:
Last login: Sun Sep 26 10:13:00 2010 from 10.1.0.1
Sun Microsystems Inc.   SunOS 5.10      Generic January 2005
$ pwd
/export/home/test1
$ ls
$ df -k .
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c1t0d0s7     248647    1042  222741     1%    /export/home
$ /usr/sbin/mkfile 10m aa
$ ls -l
total 20496
-rw-------   1 test1    other    10485760 Sep 26 10:13 aa
$ exit

7. Check the audit file(s)
# cd /var/audit
# ls -l
total 128
-rw-------   1 root     root       64370 Sep 26 10:16 20100926081119.20100926081640.sol01
8. Browse the content of this audit file
# praudit 20100926081119.20100926081640.sol01
...
header,134,2,execve(2),,sol01,2010-09-26 10:13:35.371 +02:00
path,/usr/sbin/mkfile
attribute,100555,root,bin,3,853,0
subject,test1,test1,other,test1,other,704,3761557250,9027 71168 10.1.0.1
return,success,0
header,102,2,open(2) - read,fe,sol01,2010-09-26 10:13:35.371 +02:00
path,/var/ld/ld.config
subject,test1,test1,other,test1,other,704,3761557250,9027 71168 10.1.0.1
return,failure: No such file or directory,-1
header,114,2,munmap(2),,sol01,2010-09-26 10:13:35.371 +02:00
argument,1,0xfef8e000,addr
argument,2,0x10000,len
subject,test1,test1,other,test1,other,704,3761557250,9027 71168 10.1.0.1
return,success,0
header,168,2,memcntl(2),,sol01,2010-09-26 10:13:35.371 +02:00
argument,1,0xfee80000,base
argument,2,0x1e4b0,len
argument,3,0x4,cmd
argument,4,0x3,arg
argument,5,0x0,attr
argument,6,0x0,mask
subject,test1,test1,other,test1,other,704,3761557250,9027 71168 10.1.0.1
return,success,0
header,114,2,munmap(2),,sol01,2010-09-26 10:13:35.372 +02:00
argument,1,0xfefb0000,addr
argument,2,0x8000,len
subject,test1,test1,other,test1,other,704,3761557250,9027 71168 10.1.0.1
return,success,0
header,139,2,open(2) - read,write,creat,trunc,,sol01,2010-09-26 10:13:35.386 +02:00
path,/export/home/test1/aa
attribute,100600,test1,other,7,5,0
subject,test1,test1,other,test1,other,704,3761557250,9027 71168 10.1.0.1
return,success,3
header,150,2,close(2),,sol01,2010-09-26 10:13:35.429 +02:00
argument,1,0x3,fd
path,/export/home/test1/aa
attribute,100600,test1,other,7,5,0
subject,test1,test1,other,test1,other,704,3761557250,9027 71168 10.1.0.1
return,success,0
...

As you can see clear here is executed command mkfile and created file, named aa in to the home directory of the user. Of course in above example I use only few of the features in the audit subsystem of Solaris, but in next article will try to present more in deep the options

Audit - something you have, something you do not know, part 2

5. Add some monitored files and directories
I will add monitoring of /var/run directory for read and write actions, /etc/rc.d/init.d for write and execution and /etc/shadow file for attribute change and write
[root@centos ~]# auditctl -w /etc/shadow -p wa -k shadow-file
[root@centos ~]# auditctl -w /var/run/ -p rw -k var-run-pids
[root@centos ~]# auditctl -w /etc/rc.d/init.d -p xw -k init-scripts

and check just in case the rules i define
[root@centos ~]# auditctl -l
LIST_RULES: exit,always dir=/var/run (0x8) perm=rw key=var-run-pids
LIST_RULES: exit,always watch=/etc/shadow perm=wa key=shadow-file
LIST_RULES: exit,always dir=/etc/rc.d/init.d (0x10) perm=wx key=init-scripts

6. Do some actions which will activate some audit trails
6.1. Create user (do change) in /etc/shadow file
[root@centos ~]# useradd test
[root@centos ~]# id test
uid=501(test) gid=501(test) groups=501(test)
[root@centos ~]# passwd test
Changing password for user test.
New UNIX password:
Retype new UNIX password:
passwd: all authentication tokens updated successfully
.
6.2. Start httpd daemon to get trails for execution in /etc/rc.d/init.d and write in /var/run
[root@centos ~]#  /etc/rc.d./init.d/httpd restart
Stopping httpd:                                            [FAILED]
Starting httpd:                                            [  OK  ]

7. Checks of audit logs
7.1. General check
[root@centos ~]# tail -20 /var/log/audit/audit.log
type=CONFIG_CHANGE msg=audit(1279423171.091:97): auid=0 ses=1 op="updated rules" path="/etc/shadow" key="shadow-file" list=4 res=1
type=SYSCALL msg=audit(1279423171.091:98): arch=40000003 syscall=38 success=yes exit=0 a0=b21317 a1=b211d6 a2=b235e4 a3=0 items=5 ppid=1921 pid=2034 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="passwd" exe="/usr/bin/passwd" key="shadow-file"
type=CWD msg=audit(1279423171.091:98):  cwd="/root"
type=PATH msg=audit(1279423171.091:98): item=0 name="/etc/" inode=229377 dev=fd:00 mode=040755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1279423171.091:98): item=1 name="/etc/" inode=229377 dev=fd:00 mode=040755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1279423171.091:98): item=2 name="/etc/nshadow" inode=229819 dev=fd:00 mode=0100400 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1279423171.091:98): item=3 name="/etc/shadow" inode=230260 dev=fd:00 mode=0100400 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1279423171.091:98): item=4 name="/etc/shadow" inode=229819 dev=fd:00 mode=0100400 ouid=0 ogid=0 rdev=00:00
type=USER_CHAUTHTOK msg=audit(1279423171.103:99): user pid=2034 uid=0 auid=0 msg='PAM: chauthtok acct="test" : exe="/usr/bin/passwd" (hostname=?, addr=?, terminal=pts/0 res=success)'
type=USER_CHAUTHTOK msg=audit(1279423171.106:100): user pid=2034 uid=0 auid=0 msg='op=change password id=501 exe="/usr/bin/passwd" (hostname=?, addr=?, terminal=pts/0 res=success)'
type=SYSCALL msg=audit(1279423197.516:101): arch=40000003 syscall=10 success=no exit=-2 a0=bfde9f8a a1=0 a2=805287c a3=bfde8594 items=1 ppid=2040 pid=2043 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="rm" exe="/bin/rm" key="var-run-pids"
type=CWD msg=audit(1279423197.516:101):  cwd="/"
type=PATH msg=audit(1279423197.516:101): item=0 name="/var/run/httpd.pid" inode=426024 dev=fd:00 mode=040755 ouid=0 ogid=0 rdev=00:00
type=SYSCALL msg=audit(1279423197.536:102): arch=40000003 syscall=10 success=no exit=-2 a0=bfd3bf8a a1=0 a2=805287c a3=bfd3bd24 items=1 ppid=2040 pid=2044 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="rm" exe="/bin/rm" key="var-run-pids"
type=CWD msg=audit(1279423197.536:102):  cwd="/"
type=PATH msg=audit(1279423197.536:102): item=0 name="/var/run/httpd.pid" inode=426024 dev=fd:00 mode=040755 ouid=0 ogid=0 rdev=00:00
type=SYSCALL msg=audit(1279423199.900:103): arch=40000003 syscall=5 success=yes exit=15 a0=82c4aa0 a1=8241 a2=1a4 a3=8241 items=2 ppid=1 pid=2048 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=1 comm="httpd" exe="/usr/sbin/httpd" key="var-run-pids"
type=CWD msg=audit(1279423199.900:103):  cwd="/"
type=PATH msg=audit(1279423199.900:103): item=0 name="/etc/httpd/run/" inode=426024 dev=fd:00 mode=040755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1279423199.900:103): item=1 name="/etc/httpd/run/httpd.pid" inode=426093 dev=fd:00 mode=0100644 ouid=0 ogid=0 rdev=00:00

7.2. Check for trails of actions, related to shadow file
[root@centos ~]# ausearch -f /etc/shadow
----
time->Sun Jul 18 05:19:21 2010
type=PATH msg=audit(1279423161.665:95): item=4 name="/etc/shadow" inode=230260 dev=fd:00 mode=0100400 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1279423161.665:95): item=3 name="/etc/shadow" inode=230225 dev=fd:00 mode=0100400 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1279423161.665:95): item=2 name="/etc/shadow+" inode=230260 dev=fd:00 mode=0100400 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1279423161.665:95): item=1 name="/etc/" inode=229377 dev=fd:00 mode=040755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1279423161.665:95): item=0 name="/etc/" inode=229377 dev=fd:00 mode=040755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1279423161.665:95):  cwd="/root"
type=SYSCALL msg=audit(1279423161.665:95): arch=40000003 syscall=38 success=yes exit=0 a0=bfc58c08 a1=80593a0 a2=bfc57b40 a3=0 items=5 ppid=1921 pid=2028 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="useradd" exe="/usr/sbin/useradd" key="shadow-file"
----
time->Sun Jul 18 05:19:31 2010
type=PATH msg=audit(1279423171.091:98): item=4 name="/etc/shadow" inode=229819 dev=fd:00 mode=0100400 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1279423171.091:98): item=3 name="/etc/shadow" inode=230260 dev=fd:00 mode=0100400 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1279423171.091:98): item=2 name="/etc/nshadow" inode=229819 dev=fd:00 mode=0100400 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1279423171.091:98): item=1 name="/etc/" inode=229377 dev=fd:00 mode=040755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1279423171.091:98): item=0 name="/etc/" inode=229377 dev=fd:00 mode=040755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1279423171.091:98):  cwd="/root"
type=SYSCALL msg=audit(1279423171.091:98): arch=40000003 syscall=38 success=yes exit=0 a0=b21317 a1=b211d6 a2=b235e4 a3=0 items=5 ppid=1921 pid=2034 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="passwd" exe="/usr/bin/passwd" key="shadow-file"

As you can see very clear here i have information about the userid, command, directory from where is executed command and of course timestamp of the action

7.3. From the log is visible run of httpd, create file, named /var/run/httpd.pid and i can use this file to reveal the action behind creation
[root@centos ~]# ausearch -f /var/run/httpd.pid
----
time->Sun Jul 18 05:19:57 2010
type=PATH msg=audit(1279423197.536:102): item=0 name="/var/run/httpd.pid" inode=426024 dev=fd:00 mode=040755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1279423197.536:102):  cwd="/"
type=SYSCALL msg=audit(1279423197.536:102): arch=40000003 syscall=10 success=no exit=-2 a0=bfd3bf8a a1=0 a2=805287c a3=bfd3bd24 items=1 ppid=2040 pid=2044 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="rm" exe="/bin/rm" key="var-run-pids"
----
time->Sun Jul 18 05:19:57 2010
type=PATH msg=audit(1279423197.516:101): item=0 name="/var/run/httpd.pid" inode=426024 dev=fd:00 mode=040755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1279423197.516:101):  cwd="/"
type=SYSCALL msg=audit(1279423197.516:101): arch=40000003 syscall=10 success=no exit=-2 a0=bfde9f8a a1=0 a2=805287c a3=bfde8594 items=1 ppid=2040 pid=2043 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="rm" exe="/bin/rm" key="var-run-pids"

7.4. Because of the way httpd run i see actual place of pid file is /etc/httpd/run/httpd.pid so lets check for this file too
[root@centos ~]# ausearch -f /etc/httpd/run/httpd.pid
----
time->Sun Jul 18 05:19:59 2010
type=PATH msg=audit(1279423199.900:103): item=1 name="/etc/httpd/run/httpd.pid" inode=426093 dev=fd:00 mode=0100644 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1279423199.900:103): item=0 name="/etc/httpd/run/" inode=426024 dev=fd:00 mode=040755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1279423199.900:103):  cwd="/"
type=SYSCALL msg=audit(1279423199.900:103): arch=40000003 syscall=5 success=yes exit=15 a0=82c4aa0 a1=8241 a2=1a4 a3=8241 items=2 ppid=1 pid=2048 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=1 comm="httpd" exe="/usr/sbin/httpd" key="var-run-pids"

7.5. Aha, so here is command to run /usr/sbin/httpd. I will check for run of some init script
[root@centos ~]# ausearch -k init-scripts
----
time->Sun Jul 18 05:34:19 2010
type=CONFIG_CHANGE msg=audit(1279424059.616:109): auid=0 op=add rule key="init-scripts" list=4 res=1
----
time->Sun Jul 18 05:35:33 2010
type=PATH msg=audit(1279424133.097:110): item=2 name=(null) inode=32807 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1279424133.097:110): item=1 name=(null) inode=196644 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1279424133.097:110): item=0 name="/etc/init.d/httpd" inode=230082 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1279424133.097:110):  cwd="/"
type=EXECVE msg=audit(1279424133.097:110): argc=2 a0="/bin/bash" a1="/etc/init.d/httpd"
type=EXECVE msg=audit(1279424133.097:110): argc=3 a0="/bin/bash" a1="/etc/init.d/httpd" a2="restart"
type=SYSCALL msg=audit(1279424133.097:110): arch=40000003 syscall=11 success=yes exit=0 a0=bfafde5a a1=bfafd558 a2=94a4858 a3=5 items=3 ppid=2190 pid=2195 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="httpd" exe="/bin/bash" key="init-scripts"
8. Est, voila, we reveal very detailed output from the actions, related to the areas/directories/files we think are sensitive and critical for our system

9. And at the end lets get summary report of audit sybsystem
[root@centos ~]# aureport

Summary Report
======================
Range of time in logs: 05/22/2010 11:01:19.717 - 07/18/2010 05:35:35.467
Selected time for report: 05/22/2010 11:01:19 - 07/18/2010 05:35:35.467
Number of changes in configuration: 56
Number of changes to accounts, groups, or roles: 21
Number of logins: 30
Number of failed logins: 21
Number of authentications: 12
Number of failed authentications: 1
Number of users: 1
Number of terminals: 11
Number of host names: 7
Number of executables: 12
Number of files: 45
Number of AVC's: 1
Number of MAC events: 1
Number of failed syscalls: 76
Number of anomaly events: 0
Number of responses to anomaly events: 0
Number of crypto events: 0
Number of keys: 4
Number of process IDs: 123
Number of events: 540

Of course in this short article it is not possible to show all the options we can use for audit one Linux system. For more detailed information please refer your documentation

Audit - something you have, something you do not know, part 1

Audit subsystem is installed usually on each Linux, but very often not used. In this article will try to show you what you can do with it. Be aware here will be used CentOS/RHEL, so the commands and files in your Linux may differ and you should consult documentation
1. Instalation. Lets check if the package is installed (usually it is)
[root@centos ~]# rpm -q audit
audit-1.7.17-3.el5

In this case the package is installed, but if you do not have it you can install with command
you install audit

Here are prerequisites for the package (nothing unusual)
[root@centos ~]# rpm -qR audit
/bin/bash
/bin/sh
/bin/sh
/bin/sh
audit-libs = 1.7.17-3.el5
chkconfig
config(audit) = 1.7.17-3.el5
coreutils
libaudit.so.0
libauparse.so.0
libc.so.6
libc.so.6(GLIBC_2.0)
libc.so.6(GLIBC_2.1)
libc.so.6(GLIBC_2.1.3)
libc.so.6(GLIBC_2.2)
libc.so.6(GLIBC_2.3)
libc.so.6(GLIBC_2.3.2)
libc.so.6(GLIBC_2.3.4)
libc.so.6(GLIBC_2.4)
libm.so.6
libm.so.6(GLIBC_2.0)
libnsl.so.1
libpthread.so.0
libpthread.so.0(GLIBC_2.0)
libpthread.so.0(GLIBC_2.1)
libpthread.so.0(GLIBC_2.3.2)
librt.so.1
libwrap.so.0
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rtld(GNU_HASH)

2. Next step is to check if audit daemon is started
[root@centos ~]# service auditd status
auditd (pid  1432) is running...

if it is not you can start it with
[root@centos ~]# service auditd start
Starting auditd:
[ OK ]
3. Configuration. One important point is to decide do you want to monitor system call or not. I will set to yes setting in file /etc/sysconfig/auditd will add line (or change if its exist)
AUDITD_DISABLE_CONTEXTS="no"

and restart the daemon
Next we should set where will be located audit log file. This is done in /etc/audit/auditd.conf with parameter log_file in format
log_file = /path/file

for example in my config this is
[root@centos ~]# grep ^log_file  /etc/audit/auditd.conf
log_file = /var/log/audit/audit.log

In some high security environments you will want to create dedicated filesystem for audit log files and you should care if this filesystem become full. This is done by parameter in /etc/audit/auditd.conf file
disk_full_action

This parameter can have values: ignore, syslog, exec, suspend, single, and halt
ignore - do nothing
syslog - write warning to syslog
exec /path/to/script - execute script
suspend - stop audit daemon for writing to disk
single - put server in single user mode
halt - shutdown the server
Of course there are many other parameters,but i just touch the most important (in my understanding)

4. Check the current situation
4.1. List current rules
[root@centos ~]# auditctl -l
No rules
4.2. Check current status
[root@centos ~]# auditctl -s
AUDIT_STATUS: enabled=1 flag=1 pid=2062 rate_limit=0 backlog_limit=320 lost=0 backlog=0