Oracle ASM - yet another LVM (management)

6. Next step is to start ASM instance. Do not forget to set SID to be +ASM when you try to login with sqlplus in to the instance
$ export ORACLE_SID=+ASM
$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.1.0 Production on Mon Apr 5 09:28:21 2010
Copyright (c) 1982, 2009, Oracle.  All rights reserved.
Connected to an idle instance.
SQL> startup
ASM instance started
Total System Global Area  284565504 bytes
Fixed Size                  1336036 bytes
Variable Size             258063644 bytes
ASM Cache                  25165824 bytes
ASM diskgroups mounted
SQL>

7. Lets check the status of our diskgroups
SQL> SELECT name, type, total_mb, free_mb FROM V$ASM_DISKGROUP;
DATA                           NORMAL      32756      32575

8. Check the list of available disks
SQL> select name,  path  from V$ASM_DISK;
NAME
------------------------------
PATH
--------------------------------------------------------------------
ORCL:DFDISK4

ORCL:DFDISK5
DFDISK0
ORCL:DFDISK0
DFDISK1
ORCL:DFDISK1
DFDISK2
ORCL:DFDISK2
DFDISK3
ORCL:DFDISK3
FBDISK0
ORCL:FBDISK0
FBDISK1
ORCL:FBDISK1

8 rows selected.

9. And create new diskgroup for flashback
SQL>  CREATE DISKGROUP FBA NORMAL REDUNDANCY  FAILGROUP fba_fb1 disk 'ORCL:FBDISK0'  FAILGROUP fba_fb2  disk 'ORCL:FBDISK1';         
Diskgroup created.

10. and see the new list of diskgroups
SQL>  select name,total_mb from V$ASM_DISKGROUP;
NAME                             TOTAL_MB
------------------------------ ----------
DATA                                32756
FBA                                 16378

11. Add two new disks to diskgroup DATA
SQL>  ALTER DISKGROUP data ADD DISK 'ORCL:DFDISK4', 'ORCL:DFDISK5';
Diskgroup altered.
SQL> select name,total_mb from V$ASM_DISKGROUP where name='DATA';
NAME                             TOTAL_MB
------------------------------ ----------
DATA                                49134

12. Remove one of the disks from diskgroup
SQL>  ALTER DISKGROUP data DROP  DISK DFDISK5;
Diskgroup altered.
SQL>  select name,total_mb from V$ASM_DISKGROUP where name='DATA';
NAME                             TOTAL_MB
------------------------------ ----------
DATA                                40945
and size is already 40GB

For further information about migration of database to ASM, filepaths, etc, please consult official Oracle documentation about ASM: Oracle® Database Storage Administrator's Guide

Oracle ASM - yet another LVM (system configuration)

4. Its time for some system administrator tasks. Oracle ASM need special mark of disks will work with

4.1. Because of some reason (i don't know why) ASM can work only with partitions, but not with entire disks. So it is need to create one big partition on each disk
[root@rh-asm-ora ~]# fdisk /dev/sda
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.


The number of cylinders for this disk is set to 1044.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1044, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-1044, default 1044):
Using default value 1044

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
And the similar for the rest of the disks /dev/sdb, /dev/sdc, /dev/sdd, /dev/sde and /dev/sdf

4.2. Next step is to configure the ASMlib. This is done via init script
[root@rh-asm-ora ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting  without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface []: oracle
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
If you do not get OK on the last line check /var/log/messages. Usual reason is you do not install the correct version of ASMlib. Check on this site Oracle ASMLib. If you cant find modules for your version of kernel you should compile them from source you can get from here: http://oss.oracle.com/projects/oracleasm/

4.3. And check if the kernel module is loaded
[root@rh-asm-ora ~]# lsmod |grep ora
oracleasm              46356  1
4.4. The module is loaded so lets label the disks to be recognized by Oracle ASM. I will dedicate 4 disks for tablespace and 2 disks for flashback
[root@rh-asm-ora ~]# /etc/init.d/oracleasm createdisk dfdisk0 /dev/sda1
Marking disk "dfdisk0" as an ASM disk:                     [  OK  ]
[root@rh-asm-ora ~]# /etc/init.d/oracleasm createdisk dfdisk1 /dev/sdb1
Marking disk "dfdisk1" as an ASM disk:                     [  OK  ]
[root@rh-asm-ora ~]# /etc/init.d/oracleasm createdisk dfdisk2 /dev/sdc1
Marking disk "dfdisk2" as an ASM disk:                     [  OK  ]
[root@rh-asm-ora ~]# /etc/init.d/oracleasm createdisk dfdisk3 /dev/sdd1
Marking disk "dfdisk3" as an ASM disk:                     [  OK  ]
[root@rh-asm-ora ~]# /etc/init.d/oracleasm createdisk fbdisk0 /dev/sde1
Marking disk "fbdisk0" as an ASM disk:                     [  OK  ]
[root@rh-asm-ora ~]# /etc/init.d/oracleasm createdisk fbdisk1 /dev/sdf1
Marking disk "fbdisk1" as an ASM disk:                    [  OK  ]
 
4.5. Check the ASM volumes
[root@rh-asm-ora ~]# /etc/init.d/oracleasm listdisks
DFDISK0
DFDISK1
DFDISK2
DFDISK3
FBDISK0
FBDISK1

5. The next step is install Oracle ASM software. This is mostly straight-forward process, so just read the installation guide and do it. Please do not forget in version 11gR2 Oracle ASM is part of Grid installation package and not standard Oracle Database installation package

Oracle ASM - yet another LVM (Instalation)

1. OS installation
For OS I will use CentOS with config:
- 1.5 GB RAM
- 20 GB harddisk for OS and software
- 6*8GB harddisks for ASM
I install only server cluster and exclude some packages like PCMCIA support, smartcard, power management, etc. They are not need for server.

1.1. After installation the first task is to update your system. For this I will use command
yum update yum* rpm* kernel*

At the moment I do not need more updates. After update the kernel its need to reboot server.

1.2. Install prerequisites packages. Here is the list of the packages:
xdpyinfo
libXmp
binutils
compat-libstdc++-33
elfutils-libelf
elfutils-libelf-devel
elfutils-libelf-devel-static
gcc
gcc-c++
glibc
glibc-common
glibc-devel
glibc-headers
kernel-headers
ksh
libaio
libaio-devel
libgcc libgomp
libstdc++
libstdc++-devel
make
sysstat
unixODBC
unixODBC-devel

Here and below the information is get directly from Oracle Technical Network, Documentation section.
First and second packages are not need according to the documentation, but my personal experience show me they are need.

2. Configure OS
2.1. Set kernel parameters
this is the list of parameters you should set in /etc/sysctl.conf to make software run. If some ot the parameters exist in original file just set recommended values
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmall = 2097152
kernel.shmmax = 536870912
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048586

Do not forget to execute sysctl -p or reboot the server to get parameters set.

2.2. Add groups
oinstall, oper, dba, asmadmin, asmdba, asmoper

2.3. Create oracle user to have primary group oinstall and member of dba and oper and shell KornShell

2.4. Set in /etc/security/limits.conf number of files and number of processes for oracle user as follow
oracle              soft    nproc   2047
oracle              hard    nproc   16384
oracle              soft    nofile  1024
oracle              hard    nofile  65536

2.5. Add in /etc/profile the settings for ulimits for oracle user
if [ $USER = "oracle" ]; then
        if [ $SHELL = "/bin/ksh" ]; then
              ulimit -u 16384
              ulimit -n 65536
        else
              ulimit -u 16384 -n 65536
        fi
umask 022
fi

On my CentOS option, provided by Oracle documentation (ulimit -p 16384) because of version of ksh, so if you are with RHEL 5.x and if the version of ksh is like ksh-20080202-14.
_4.2 use ulimit -u 16384

3. Install Oracle ASM libraries and kernel modules. The packages should be downloaded from https://www.oracle.com/technetwork/database/enterprise-edition/downloads/oracle18c-linux-180000-5022980.html

Search for "Oracle Database XXX Grid Infrastructure"

Do not forget you should download and install libraries for your distribution and version of kernel. You can use this command to check the version of kernel
[root@rh-asm-ora rpm]# uname -r
2.6.18-164.15.1.

Solaris x86 root filesystem mirroring

Preamble

This document is directed to give idea how to create mirroring of root filesystem in Solaris x86 with the help of Solaris Volume Manager. Here will be used already installed OS and all the work will be done without need of reinstall

Prerequisites

1. First we should be sure have two identical harddisks in the server

 # format
 Searching for disks...done
 AVAILABLE DISK SELECTIONS:
       0. c0d0 <DEFAULT cyl 1563 alt 2 hd 255 sec 63>
          /pci@0,0/pci-ide@7,1/ide@0/cmdk@0,0
       1. c0d1 <DEFAULT cyl 1563 alt 2 hd 255 sec 63>
          /pci@0,0/pci-ide@7,1/ide@0/cmdk@1,0
 Specify disk (enter its number): ^C

2. Its need to create small slice for metadb information (usually slice 7) like:

   partition> p
 Current partition table (unnamed):
 Total disk cylinders available: 1563 + 2 (reserved cylinders)

 Part      Tag    Flag     Cylinders        Size            Blocks
   0       root    wm     518 - 1562        8.01GB    (1045/0/0) 16787925
   1       swap    wu       3 -  133        1.00GB    (131/0/0)   2104515
   2     backup    wm       0 - 1562       11.97GB    (1563/0/0) 25109595
   3 unassigned    wm       0               0         (0/0/0)           0
   4 unassigned    wm       0               0         (0/0/0)           0
   5 unassigned    wm       0               0         (0/0/0)           0
   6 unassigned    wm       0               0         (0/0/0)           0
   7 unassigned    wm     134 -  135       15.69MB    (2/0/0)       32130
   8       boot    wu       0 -    0        7.84MB    (1/0/0)       16065
   9 alternates    wu       1 -    2       15.69MB    (2/0/0)       32130

3. Next step is to create the same partitions on the second disk. To avoid human error it is much better to use some kind of automations:

 prtvtoc /dev/rdsk/c0d0s2 > /tmp/c0d0s2.toc
 fmthard -s /tmp/c0d0s2.toc /dev/rdsk/c0d1s2

4. Then we should identify the partitions need to be mirrored:

 # egrep "ufs|swap" /etc/vfstab|grep "/dev/dsk"
 /dev/dsk/c0d0s1 -       -       swap    -       no      -
 /dev/dsk/c0d0s0 /dev/rdsk/c0d0s0        /       ufs     1       no      -

LVM

1. Let’s create few copies of metadb on the partitions we create for this puspose:

 # metadb -a -f -c 2 c0d0s7 c0d1s7

2. It’s time to put our disks where OS reside under management of SVM

 # metainit -f d10 1 1 c0d0s0
 d10: Concat/Stripe is setup
 # metainit -f d11 1 1 c0d0s1
 d11: Concat/Stripe is setup
 # metainit d0 -m d10
 d0: Mirror is setup
 # metainit d1 -m d11
 d1: Mirror is setup
 # metaroot d0 

3. Check newly created devices:

 # ls -l /dev/md/rdsk
 total 8
 lrwxrwxrwx   1 root     root          36 Aug 30 18:29 d0 -> ../../../devices/pseudo/md@0:0,0,raw
 lrwxrwxrwx   1 root     root          36 Aug 30 18:29 d1 -> ../../../devices/pseudo/md@0:0,1,raw
 lrwxrwxrwx   1 root     root          37 Aug 30 18:28 d10 -> ../../../devices/pseudo/md@0:0,10,raw
 lrwxrwxrwx   1 root     root          37 Aug 30 18:28 d11 -> ../../../devices/pseudo/md@0:0,11,raw
 # ls -l /dev/md/dsk
 total 8
 lrwxrwxrwx   1 root     root          36 Aug 30 18:29 d0 -> ../../../devices/pseudo/md@0:0,0,blk
 lrwxrwxrwx   1 root     root          36 Aug 30 18:29 d1 -> ../../../devices/pseudo/md@0:0,1,blk
 lrwxrwxrwx   1 root     root          37 Aug 30 18:28 d10 -> ../../../devices/pseudo/md@0:0,10,blk
 lrwxrwxrwx   1 root     root          37 Aug 30 18:28 d11 -> ../../../devices/pseudo/md@0:0,11,blk

4. Make appropriate changes in /etc/vfstab to get boot from mirror, not standard disks

 # egrep "ufs|swap" /etc/vfstab|grep "/dev/md/dsk"
 /dev/md/dsk/d1  -       -       swap    -       no      -
 /dev/md/dsk/d0  /dev/md/rdsk/d0 /       ufs     1       no      -

5. The next step is to flush cache buffers and reboot

 # sync;sync;sync
 # reboot

6. It is time to put second disk under management of SVN

 # metainit -f d20 1 1 c0d1s0
 d20: Concat/Stripe is setup
 # metainit -f d21 1 1 c0d1s1
 d21: Concat/Stripe is setup

7. And to add them to created previously mirrors. Be aware process of synchronisation will continue in background and you can check the process

 # metattach d0 d20
 d0: submirror d20 is attached
 # metattach d1 d21
 d1: submirror d21 is attached

8. Check the process of building mirrors and wait till they finnish

 # metastat
 d1: Mirror
     Submirror 0: d11
       State: Okay
     Submirror 1: d21
       State: Resyncing
     Resync in progress: 96 % done
     Pass: 1
     Read option: roundrobin (default)
     Write option: parallel (default)
     Size: 2104515 blocks (1.0 GB) 

 d11: Submirror of d1
     State: Okay
     Size: 2104515 blocks (1.0 GB)
     Stripe 0:
         Device   Start Block  Dbase        State Reloc Hot Spare
         c0d0s1          0     No            Okay   Yes 


 d21: Submirror of d1
     State: Resyncing
     Size: 2104515 blocks (1.0 GB)
     Stripe 0:
         Device   Start Block  Dbase        State Reloc Hot Spare
         c0d1s1          0     No            Okay   Yes
 

 d0: Mirror
     Submirror 0: d10
       State: Okay
     Submirror 1: d20
       State: Resyncing
     Resync in progress: 13 % done
     Pass: 1
     Read option: roundrobin (default)
     Write option: parallel (default)
     Size: 16787925 blocks (8.0 GB) 

 d10: Submirror of d0
     State: Okay
     Size: 16787925 blocks (8.0 GB)
     Stripe 0:
         Device   Start Block  Dbase        State Reloc Hot Spare
         c0d0s0          0     No            Okay   Yes


 d20: Submirror of d0
     State: Resyncing
     Size: 16787925 blocks (8.0 GB)
     Stripe 0:
         Device   Start Block  Dbase        State Reloc Hot Spare
         c0d1s0          0     No            Okay   Yes


 Device Relocation Information:
 Device   Reloc  Device ID
 c0d1   Yes      id1,cmdk@AVMware_Virtual_IDE_Hard_Drive=01000000000000000001
 c0d0   Yes      id1,cmdk@AVMware_Virtual_IDE_Hard_Drive=00000000000000000001

Boot

1. Next step is to check if partition on second disk is active

 # fdisk /dev/rdsk/c0d1p0
             Total disk size is 1566 cylinders
             Cylinder size is 16065 (512 byte) blocks

                                               Cylinders
      Partition   Status    Type          Start   End   Length    %
      =========   ======    ============  =====   ===   ======   ===
          1       Active    Solaris2          1  1565    1565    100


 SELECT ONE OF THE FOLLOWING:
    1. Create a partition
    2. Specify the active partition
    3. Delete a partition
    4. Change between Solaris and Solaris2 Partition IDs
    5. Exit (update disk configuration and exit)
    6. Cancel (exit without updating disk configuration)
 Enter Selection: 5

2. And to add boot record to the second disk to make it bootable

 # /sbin/installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c0d1s0
 stage1 written to partition 0 sector 0 (abs 16065)
 stage2 written to partition 0, 233 sectors starting at 50 (abs 16115)

3. Add new item in the boot menu (/boot/grub/menu.lst) to have alternative way to boot

 title Alternate boot
 root (hd1,0,a)
 kernel /platform/i86pc/multiboot
 module /platform/i86pc/boot_archive

4. Check if the new item is added to the boot menu

 # bootadm list-menu
 The location for the active GRUB menu is: /boot/grub/menu.lst
 default 0
 timeout 10
 0 Solaris 10 5/08 s10x_u5wos_10 X86
 1 Solaris failsafe
 2 Alternate boot

5. That’s all, you have already mirrored root partition (plus swap)

Conclusion

In the document are not mentioned all the options and possibilities of SVM, but only short set, need to done the work. For further information, please consult official Oracle documentation.