Oracle 11gR2 RAC Installation on Oracle Linux 6.5

In this article we will be installing 2 node 11gR2 RAC on Oracle Linux 6.5. We will be looking at below steps

• Configure Virtual machine for Node 1

• Configure Virtual machine for Node 2

• OS prerequisites on all nodes

• Setup public & private network

• Configure shared storage

• Oracle Grid Installation

• Oracle RAC software installation

• Create RAC Database

Configure VM for Node 1

Open virtual box and create a new VM for node 1. We will be installing OEL 6.5 on both RAC nodes.

Create New VM

Open virtual box and click on New. Create first virtual machine with oraracn1 as the name

Min RAM allocation should be 4 GB per RAC node

Create a virtual disk now

Select VDI and click next

Select Dynamically allocated option and click next

Allocate Min 80 GB HDD to your virtual machine. You can allocate even more if you have good hard disk space. Click on Create

Now your virtual machine should be created. This is how your virtual box will look like

Right click on virtual machine and select settings. Under System settings, make sure to set below bootorder

Under storage > Optical Drive > choose OEL 6.5 iso file from your system.

Under Network, make sure attached to is set to Bridged Adapter and Enable the Network Adapter

In RAC, we need to have two network adapters. So, enable the second adapter same like the first one.

Click on OK to complete the RAC node 1 setup.

Install OEL 6.5 on Node 1

Now that our virtual machine is ready, go ahead, start the virtual machine and install OEL 6.5

Skip media test

Click next to continue

Go with default language > click next

Go with default keyboard type and click on next

Select Basic Storage Devices > Click Next

Select Yes, Discard and data

Give a hostname to your machine oraracn1.oraclgenesis.com

Click on configure network and assign a static IP

Assign public IP to your machine. In my case, I have assigned 192.168.0.101 as the machine IP

Select second network card and click on edit

Assign private IP to your node 1 on a different subnet mask. Leave Gateway blank

Click next and select your timezone

Give a root password

Select create custom layout and click next

Select Free disk space and click on create

Choose standard partition and click Create

Create boot partition with 512 MB size

Once again choose free disk space and click on create

Select file system type as Swap and give 15 GB size (Approx 15000 MB)

Again select free space and click on create

Give mount point as root (/) and chose option Fill to maximum allowable size

Click next and choose Format option

Select Write changes to disk

No changes on below screen, just click next

Choose customize now and just click next

From the list, select packages as below:

Base System

• Base

• Client management tools

• Compatibility libraries

• Hardware monitoring utilities

• Large Systems Performance

• Network file system client

• Performance Tools

• Perl Support

Servers

• Server Platform

• System administration tools

• Desktops

• Desktop

• Desktop Platform

• Fonts

• General Purpose Desktop

• Graphical Administration Tools

• Input Methods

• X Window System

Development

• Additional Development

• Development Tools

Applications

• Internet Browser

Click next and OEL installation will begin

Once done, click on Reboot

After reboot, there are few post installation steps. Click on Forward

Accept the license and click on Forward

Choose No and click on Forward

Select No thanks and proceed

Click forward

Do not create any user at this stage, click forward

Choose Yes

Check the Date and Time settings and click on forward

Nothing to choose under Kdump, click forward

OEL 6.5 installation is completed.

Configure VM for Node 2

You need to create Node 2 like how you have created Node 1 in virtual box. This will be a separate new virtual machine. I have used below configuration for node 2:

Install OEL 6.5 on Node 2

Now that our virtual machine is ready, go ahead and start the virtual machine.

Start virtual machine and install OEL 6.5

Skip media test

Click next to continue

Go with default language > click next

Go with default keyboard type and click on next

Select Basic Storage Devices > Click Next

Select Yes, Discard and data

Give a hostname to your machine oraracn2.oraclgenesis.com

Click on configure network and assign a static IP

Assign public IP to your machine. In my case, I have assigned 192.168.0.102 as the machine IP.

Select second network card and click on edit

Assign private IP to your node 2 on a different subnet mask. Leave Gateway blank

Click next and select your timezone

Give a root password

Select create custom layout and click next

Select Free disk space and click on create

Choose standard partition and click Create

Create boot partition with 512 MB size

Once again choose free disk space and click on create

Select file system type as Swap and give 15 GB size (Approx 15000 MB)

Again select free space and click on create

Give mount point as root (/) and chose option Fill to maximum allowable size

Click next and chose Format option

Select Write changes to disk

No changes on below screen, just click next

Choose customize now and just click next

From the list, select packages as below

Base System

Servers:Desktops

Desktops

Applications

Development

Click next and OEL installation will begin

Once done, click on Reboot

After reboot, there are few post installation steps. Click on Forward

Accept the license and click on Forward

Choose No and click on Forward

Select No thanks and proceed

Click forward

Do not create any user at this stage, click forward

Choose Yes

Check the Date and Time settings and click on forward

Nothing to choose under Kdump, click forward

OEL 6.5 installation on Node 2 is completed.

OS Prerequisites – All Nodes

Connect to both RAC nodes via putty and execute all the below commands on both nodes.

Check if your machine is able to speak to internet. You can ping yahoo.com. Then install below package to perform all oracle installation pre-requisites

yum -y install oracle-rdbms*

Install ASM packages

yum -y install oracleasm*
yum -y install kmod-oracleasm

Change password for oracle user

passwd oracle

Disable firewall

/etc/rc.d/init.d/iptables status
/etc/rc.d/init.d/iptables stop
service iptables stop
chkconfig iptables off
echo 0 > /selinux/enforce

Create directories to install grid and oracle software

mkdir -p /u01/app/11.2.0/grid mkdir -p /u01/app/oracle/product/11.2.0/db_1 chown -R oracle:oinstall /u01 chmod -R 775 /u01/

mkdir -p /u02
chown -R oracle:oinstall /u02

Download Oracle 11.2.0.4 gird and database binaries. Copy it to /u02 location (Only on Node 1)

Setup Public & Private Network

We have below public and private network configuration for our RAC setup

Also, below are the Virtual IP and Scan IP details:

Depending upon your environment, change IP addresses and put below under /etc/hosts file on both oraracn1 and oraracn2 nodes

On both nodes:
==============
127.0.0.1 localhost.localdomain localhost

# Public Network	- (eth0)
192.168.0.101	oraracn1.oraclegenesis.com	oraracn1
192.168.0.102	oraracn2.oraclegenesis.com	oraracn2

# Private Interconnect - (eth1)
10.10.10.101	oraracn1-priv.oraclegenesis.com	oraracn1-priv
10.10.10.102	oraracn2-priv.oraclegenesis.com	oraracn2-priv

# Public Virtual	IP (VIP) addresses - (eth0:1)	
192.168.0.105	oraracn1-vip.oraclegenesis.com	oraracn1-vip
192.168.0.106	oraracn2-vip.oraclegenesis.com	oraracn2-vip

# Single Client Access Name (SCAN)	
192.168.0.110	orarac-scan.oraclegenesis.com	orarac-scan

Test the network configuration

On oraracn1:	
============	
# ping oraracn1          --> use Ctrl+C to exit from ping
# ping oraracn2	
# ping oraracn1-priv	
# ping oraracn2-priv	

On oraracn2:	
============	
# ping oraracn1          --> use Ctrl+C to exit from ping
# ping oraracn2	
# ping oraracn1-priv	
# ping oraracn2-priv

Note: You cannot ping VIP and SCAN IP. Those IPs can only be used by an application and are not ping-able!

Configure Shared Storage

At this stage, shutdown both the virtual machines. We will allocate shared disks to RAC nodes. Open oraracn1 virtual machine settings and select storage. Click on Adds Hard Disk button under SATA controller

Choose Create new disk

Choose VDI and click Next

Select Fixed size and click Next

Allocate 100 GB of shared storage to your RAC setup

This will take some time to add shared disk to Node 1

Once disk is added, you can see the disk under Node 1 settings > storage

Now our goal is to make newly added disk on node 1 shared with node 2. For this, go to File > Virtual Media Manager

Select your shared disk > right click > choose Modify

Change the property from Normal to Shareable and click on OK

Now we are ready to share the disk with Node 2. Open oraracn2 settings > Storage > Adds hard disk icon

Do not create a new disk, Choose existing disk

Locate your shared disk at OS level. It must mostly be stored under your Node 1 location

The disk must now be shared with node 2 as well. You will see the RAC Shared Storage below

Create Shared Partitions

Now that we have 100 GB of shared storage, we need to create three partitions for the main three ASM disk groups

Create three partitions under the newly added 100 GB shared disk via node 1

On Node 1:
==========
fdisk /dev/sdb
n
p
1
<enter>
+10G

============ Frist partition is created, create second one ============

n
p
1
<enter>
+45G

============ Second partition is created, create third one =============

n
p
1
<enter>
+45G

======= All partitions are created, now save the partition table ==========

w	 this command will save the partition table

Configure ASM & ASM Disks

As we have created 3 partitions, we must allocate each partition to single ASM disks, which will then be allocated to ASM Disk groups

Configure ASM on both nodes

On Node 1 as root user:
=======================
/usr/sbin/oracleasm configure -i

Default user to own the driver interface []: oracle
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y

On Node 2 as root user:
=======================
/usr/sbin/oracleasm configure -i

Default user to own the driver interface []: oracle
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y

Load oracle-asm kernel module on both nodes. This command Creates the /etc/sysconfig/oracleasm configuration file, creates the /dev/oracleasm mount point and mounts the ASMLib driver file system

On Node 1 as root user:
=======================
/usr/sbin/oracleasm init


On Node 2 as root user:
=======================
/usr/sbin/oracleasm init

Create ASM disks – Any one Node

On Node 1 as root user:
=======================
oracleasm createdisk CRS1 /dev/sdb5
oracleasm createdisk DATA1 /dev/sdb6
oracleasm createdisk FRA1 /dev/sdb7

Run below commands on both nodes to scan and list ASM disks

oracleasm scandisks
oracleasm listdisks

Oracle Grid Installation

Oracle grid software is the one which binds multiple independent servers into a cluster. You can configure ASM only when Oracle grid is installed because ASM is bundled with Oracle Grid Software.

Configure NTP – All Nodes

On Node 1 and Node 2:
=====================
#	yum install ntp
#	service ntpd start
#	chkconfig ntpd on

# vi /etc/sysconfig/ntpd	-> Open file delete all and put below line

OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid -g"

#	service ntpd stop
#	ntpdate 1.tw.pool.ntp.org
#	service ntpd start

Now its good time to clean-up the YUM repository to release some space from the disk

On both nodes as root user:
===========================
# yum clean all

Set Bash Profile

Setup oracle user bash profile on node 1

#	su - oracle
#	vi .bash_profile

#	Oracle Settings
TMP=/tmp; export TMP
TMPDIR=$TMP; export TMPDIR
ORACLE_HOSTNAME=oraracn1; export ORACLE_HOSTNAME
ORACLE_UNQNAME=RAC; export ORACLE_UNQNAME
ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
GRID_HOME=/u01/app/11.2.0/grid; export GRID_HOME
DB_HOME=$ORACLE_BASE/product/11.2.0/db_1; export DB_HOME
ORACLE_HOME=$DB_HOME; export ORACLE_HOME
ORACLE_SID=RAC1; export ORACLE_SID
ORACLE_TERM=xterm; export ORACLE_TERM
BASE_PATH=/usr/sbin:$PATH; export BASE_PATH
PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPAT H

alias grid_env='. /home/oracle/grid_env'
alias db_env='. /home/oracle/db_env'

Create a file to set grid variables on node 1. This file will set grid environmental variables making it easy to switch between DB home and grid home.

# vi /home/oracle/grid_env

ORACLE_SID=+ASM1; export ORACLE_SID
ORACLE_HOME=$GRID_HOME; export ORACLE_HOME
PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPAT H

Create below file to set database environmental variables on Node 1

# vi /home/oracle/db_env

ORACLE_SID=RAC1; export ORACLE_SID
ORACLE_HOME=$DB_HOME; export ORACLE_HOME
PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPAT H

Setup oracle user bash profile on node 2

# su - oracle
# vi .bash_profile

# Oracle Settings
TMP=/tmp; export TMP
TMPDIR=$TMP; export TMPDIR

ORACLE_HOSTNAME=oraracn2; export ORACLE_HOSTNAME
ORACLE_UNQNAME=RAC; export ORACLE_UNQNAME
ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
GRID_HOME=/u01/app/11.2.0/grid; export GRID_HOME
DB_HOME=$ORACLE_BASE/product/11.2.0/db_1; export DB_HOME
ORACLE_HOME=$DB_HOME; export ORACLE_HOME
ORACLE_SID=RAC2; export ORACLE_SID
ORACLE_TERM=xterm; export ORACLE_TERM
BASE_PATH=/usr/sbin:$PATH; export BASE_PATH
PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPAT H

alias grid_env='. /home/oracle/grid_env'
alias db_env='. /home/oracle/db_env'

Create a file to set grid variables on node 2. This file will set grid environmental variables making it easy to switch between DB home and grid home.

# vi /home/oracle/grid_env

ORACLE_SID=+ASM2; export ORACLE_SID
ORACLE_HOME=$GRID_HOME; export ORACLE_HOME
PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPAT H

Create below file to set database environmental variables on Node 2

# vi /home/oracle/db_env

ORACLE_SID=RAC2; export ORACLE_SID
ORACLE_HOME=$DB_HOME; export ORACLE_HOME
PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPAT H

At this stage, exit as oracle user and the open another terminal with new session OR switch to root user and then back to oracle user. This will make the new environmental variables in effect.

Now it’s time to test our environment variables and check if they are working fine or not. As oracle user, execute below commands on both nodes to check if environmental variables are set properly

Node	1:
-------
su –	oracle
# . grid_env	     --To set grid env variables
# env |grep ORA
ORACLE_UNQNAME=RAC
ORACLE_SID=+ASM1
ORACLE_BASE=/u01/app/oracle
ORACLE_HOSTNAME=oraracn1
ORACLE_TERM=xterm
ORACLE_HOME=/u01/app/11.2.0/grid

# . db_env	     --To set DB env variables
# env |grep ORA
ORACLE_UNQNAME=RAC
ORACLE_SID=RAC1
ORACLE_BASE=/u01/app/oracle
ORACLE_HOSTNAME=oraracn1
ORACLE_TERM=xterm
ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1

Node	2:		
-------		
su –	oracle		
# . grid_env	     --To set grid env variables	
# env |grep ORA
ORACLE_UNQNAME=RAC
ORACLE_SID=+ASM2
ORACLE_BASE=/u01/app/oracle
ORACLE_HOSTNAME=oraracn2
ORACLE_TERM=xterm
ORACLE_HOME=/u01/app/11.2.0/grid

# . db_env	     --To set DB env variables
# env |grep ORA
ORACLE_UNQNAME=RAC
ORACLE_SID=RAC2
ORACLE_BASE=/u01/app/oracle
ORACLE_HOSTNAME=oraracn2
ORACLE_TERM=xterm
ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1

Configure SSH Between Nodes

Configure SSH connectivity between the nodes as oracle user. There are two methods to setup SSH connectivity between the nodes is: Manual method and automatic method. The manual method is very long and you might encounter errors while doing it. The easy way is the automatic method via script provided by oracle.

As you have already copied the grid and database software files under /u02, unzip the grid software file. This will create a new folder name grid under /u02. Oracle provides automatic ssh setup script

On node 1 as oracle user:
=========================
cd /u02
unzip <grid_software_file_name>

cd grid/sshsetup

./sshUserSetup.sh -user oracle -hosts "oraracn1 oraracn2" -noPromptPassphrase -confir m -advanced

Run cluvfy utility

Oracle provide a pre-installation utility which is called as Cluster Verification Utility (CLUVFY). This utility performs cluster verification before, during and after install of grid software. Let us run CLUVFY utility to check cluster before installation of grid

# /u01/softwares/grid
# ./runcluvfy.sh stage -pre crsinst -n oraracn1,oraracn2 -fixup -verbose

Below is the sample output

[oracle@oraracn1 grid]$ ./runcluvfy.sh stage -pre crsinst -n oraracn1,oraracn2 -fixup -verbose 
 
Performing pre-checks for cluster services setup 
 
Checking node reachability... 
 
Check: Node reachability from node "oraracn1"   
Destination Node                      Reachable?   
------------------------------------  ------------------------   
oraracn2                              yes   
oraracn1                              yes 
Result: Node reachability check passed from node "oraracn1"
 
Checking user equivalence... 
 
Check: User equivalence for user "oracle"   
Node Name                             Comment   
------------------------------------  ------------------------   
oraracn1                              passed   
oraracn2                              passed 
Result: User equivalence check passed for user "oracle" 
 
Checking node connectivity... 
 
Checking hosts config file...   
Node Name     Status                    Comment   
------------  ------------------------  ------------------------   
oraracn1      passed   
oraracn2      passed 
 
Verification of the hosts config file successful

Interface information for node "oraracn1"  
Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU  
------ --------------- --------------- --------------- --------------- ---------------- ------  
eth0   192.168.0.151   192.168.0.0     0.0.0.0         192.168.0.1     00:0C:29:B6:0 F:87 1500  
eth1   10.10.10.151    10.10.10.0      0.0.0.0         192.168.0.1     00:0C:29:B6:0 F:91 1500 
 
 
Interface information for node "oraracn2"  
Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU  
------ --------------- --------------- --------------- --------------- ---------------- ------  
eth0   192.168.0.152   192.168.0.0     0.0.0.0         192.168.0.1     00:0C:29:81:9 1:FA 1500  
eth1   10.10.10.152    10.10.10.0      0.0.0.0         192.168.0.1     00:0C:29:81:9 1:04 1500 
 
 
Check: Node connectivity of subnet "192.168.0.0"
Source                          Destination                     Connected?
------------------------------  ------------------------------  ---------
oraracn1:eth0                   oraracn2:eth0                   yes
Result: Node connectivity passed for subnet "192.168.0.0" with node(s) oraracn1,orara cn2
 
 
Check: TCP connectivity of subnet "192.168.0.0"   
Source                          Destination                     Connected?   
--------------------            ------------------------        --------
oraracn1:192.168.0.151          oraracn2:192.168.0.152          passed 
Result: TCP connectivity check passed for subnet "192.168.0.0" 
 
 
Check: Node connectivity of subnet "10.10.10.0"   
Source                          Destination                     Connected?   
------------------------------  -------------------             -----------  
oraracn1:eth1                   oraracn2:eth1                   yes 
Result: Node connectivity passed for subnet "10.10.10.0" with node(s) oraracn1,orarac n2 
 
 
Check: TCP connectivity of subnet "10.10.10.0"   
Source                          Destination                     Connected?   
------------------------------  -------------                   ----------         
oraracn1:10.10.10.151           oraracn2:10.10.10.152           passed 
Result: TCP connectivity check passed for subnet "10.10.10.0" 
 
 
Interfaces found on subnet "192.168.0.0" that are likely candidates for VIP are: oraracn1 eth0:192.168.0.151 
oraracn2 eth0:192.168.0.152

Interfaces found on subnet "10.10.10.0" that are likely candidates for a private inte rconnect are: 
oraracn1 eth1:10.10.10.151 
oraracn2 eth1:10.10.10.152 
 
Result: Node connectivity check passed 
 
 
Check: Total memory   
Node Name     Available                 Required                  Comment   
------------  ------------------------  ------------------------  ----------
oraracn1      2.44GB (2559180.0KB)      1.5GB (1572864.0KB)       passed
oraracn2      2.44GB (2559180.0KB)      1.5GB (1572864.0KB)       passed
Result: Total memory check passed 
 
Check: Available memory   
Node Name     Available                 Required                  Comment   
------------  ------------------------  ------------------------  ----------
oraracn1      2.11GB (2211952.0KB)      50MB (51200.0KB)          passed
oraracn2      2.19GB (2294412.0KB)      50MB (51200.0KB)          passed
Result: Available memory check passed 
 
Check: Swap space   
Node Name     Available                 Required                  Comment   
------------  ------------------------  ------------------------  ----------
oraracn1      4.88GB (5116692.0KB)      2.44GB (2559180.0KB)      passed
oraracn2      4.88GB (5116692.0KB)      2.44GB (2559180.0KB)      passed
Result: Swap space check passed 
 
Check: Free disk space for "oraracn1:/tmp"   
Path              Node Name     Mount point   Available     Required      Comment   
----------------  ------------  ------------  ------------  ------------  --------
/tmp              oraracn1      /             4.94GB        1GB           passed
Result: Free disk space check passed for "oraracn1:/tmp" 
 
Check: Free disk space for "oraracn2:/tmp"   
Path              Node Name     Mount point   Available     Required      Comment   
----------------  ------------  ------------  ------------  ------------  ---------
/tmp              oraracn2      /             4.71GB        1GB           passed
Result: Free disk space check passed for "oraracn2:/tmp" 
 
Check: User existence for "oracle"   
Node Name     Status                    Comment   
------------  ------------------------  ------------------------   
oraracn1      exists                    passed   
oraracn2      exists                    passed 
Result: User existence check passed for "oracle" 
 
Check: Group existence for "oinstall"   
Node Name     Status                    Comment   
------------  ------------------------  ------------------------   
oraracn1      exists                    passed   
oraracn2      exists                    passed 
Result: Group existence check passed for "oinstall"

Check: Group existence for "dba"
Node Name     Status                    Comment
------------  ------------------------  ------------------------
oraracn1      exists                    passed
oraracn2      exists                    passed
Result: Group existence check passed for "dba" 
 
Check: Membership of user "oracle" in group "oinstall" [as Primary]
Node Name    User Exists Group Exists  User in Group  Primary      Comment
-----------  ---------   ---------     ------------   ------------ -------
oraracn1     yes         yes           yes            yes          passed
oraracn2     yes         yes           yes            yes          passed
Result: Membership check for user "oracle" in group "oinstall" [as Primary] passed 
 
Check: Membership of user "oracle" in group "dba"
Node Name         User Exists   Group Exists  User in Group  Comment
----------------  ------------  ------------  ------------  ----------------
oraracn1          yes           yes           yes           passed
oraracn2          yes           yes           yes           passed
Result: Membership check for user "oracle" in group "dba" passed 
 
Check: Run level   
Node Name     run level    Required                Comment
------------  -----------  -----------             -----------
oraracn1      5            3,5                     passed
oraracn2      5            3,5                     passed
Result: Run level check passed 
 
Check: Hard limits for "maximum open file descriptors"
Node Name         Type          Available     Required      Comment
----------------  ------------  ------------  ------------  ----------------
oraracn1          hard          131072        65536         passed
oraracn2          hard          131072        65536         passed
Result: Hard limits check passed for "maximum open file descriptors" 
 
Check: Soft limits for "maximum open file descriptors"
Node Name         Type          Available     Required      Comment
----------------  ------------  ------------  ------------  ----------------
oraracn1          soft          131072        1024          passed
oraracn2          soft          131072        1024          passed
Result: Soft limits check passed for "maximum open file descriptors" 
 
Check: Hard limits for "maximum user processes"
Node Name         Type          Available     Required      Comment
----------------  ------------  ------------  ------------  ----------------
oraracn1          hard          131072        16384         passed
oraracn2          hard          131072        16384         passed
Result: Hard limits check passed for "maximum user processes" 
 
Check: Soft limits for "maximum user processes"
Node Name         Type          Available     Required      Comment
----------------  ------------  ------------  ------------  ----------------
oraracn1          soft          131072        2047          passed
oraracn2          soft          131072        2047          passed
Result: Soft limits check passed for "maximum user processes"

Check: System architecture
Node Name     Available                 Required                  Comment
-----------  ------------------------  ------------------------  ----------
oraracn1     x86_64                    x86_64                    passed
oraracn2     x86_64                    x86_64                    passed
Result: System architecture check passed 
 
Check: Kernel version
Node Name     Available                 Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      2.6.18-164.el5            2.6.18                    passed
oraracn2      2.6.18-164.el5            2.6.18                    passed
Result: Kernel version check passed 
 
Check: Kernel parameter for "semmsl"
Node Name     Configured                Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      250                       250                       passed
oraracn2      250                       250                       passed
Result: Kernel parameter check passed for "semmsl" 
 
Check: Kernel parameter for "semmns"
Node Name     Configured                Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      32000                     32000                     passed
oraracn2      32000                     32000                     passed
Result: Kernel parameter check passed for "semmns" 
 
Check: Kernel parameter for "semopm"
Node Name     Configured                Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      100                       100                       passed
oraracn2      100                       100                       passed
Result: Kernel parameter check passed for "semopm" 
 
Check: Kernel parameter for "semmni"
Node Name     Configured                Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      142                       128                       passed
oraracn2      142                       128                       passed
Result: Kernel parameter check passed for "semmni" 
 
Check: Kernel parameter for "shmmax"
Node Name     Configured                Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      4398046511104             536870912                 passed
oraracn2      4398046511104             536870912                 passed
Result: Kernel parameter check passed for "shmmax" 
 
Check: Kernel parameter for "shmmni"
Node Name     Configured                Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      4096                      4096                      passed
oraracn2      4096                      4096                      passed
Result: Kernel parameter check passed for "shmmni"

Check: Kernel parameter for "shmall"
Node Name     Configured                Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      4294967296                2097152                   passed
oraracn2      4294967296                2097152                   passed
Result: Kernel parameter check passed for "shmall" 
 
Check: Kernel parameter for "file-max"
Node Name     Configured                Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      6815744                   6815744                   passed
oraracn2      6815744                   6815744                   passed
Result: Kernel parameter check passed for "file-max" 
 
Check: Kernel parameter for "ip_local_port_range"
Node Name     Configured                Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      between 9000 & 65500      between 9000 & 65500      passed
oraracn2      between 9000 & 65500      between 9000 & 65500      passed
Result: Kernel parameter check passed for "ip_local_port_range" 
 
Check: Kernel parameter for "rmem_default"
Node Name     Configured                Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      262144                    262144                    passed
oraracn2      262144                    262144                    passed
Result: Kernel parameter check passed for "rmem_default" 
 
Check: Kernel parameter for "rmem_max"
Node Name     Configured                Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      4194304                   4194304                   passed
oraracn2      4194304                   4194304                   passed
Result: Kernel parameter check passed for "rmem_max" 
 
Check: Kernel parameter for "wmem_default"
Node Name     Configured                Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      262144                    262144                    passed
oraracn2      262144                    262144                    passed
Result: Kernel parameter check passed for "wmem_default" 
 
Check: Kernel parameter for "wmem_max"
Node Name     Configured                Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      1048576                   1048576                   passed
oraracn2      1048576                   1048576                   passed
Result: Kernel parameter check passed for "wmem_max" 
 
Check: Kernel parameter for "aio-max-nr"
Node Name     Configured                Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      3145728                   1048576                   passed
oraracn2      3145728                   1048576                   passed
Result: Kernel parameter check passed for "aio-max-nr"


Check: Package existence for "make-3.81"
Node Name     Available                 Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      make-3.81-3.el5           make-3.81                 passed
oraracn2      make-3.81-3.el5           make-3.81                 passed
Result: Package existence check passed for "make-3.81" 
 
Check: Package existence for "binutils-2.17.50.0.6"
Node Name     Available                 Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      binutils-2.17.50.0.6-12.el5  binutils-2.17.50.0.6      passed
oraracn2      binutils-2.17.50.0.6-12.el5  binutils-2.17.50.0.6      passed
Result: Package existence check passed for "binutils-2.17.50.0.6" 
 
Check: Package existence for "gcc-4.1.2"
Node Name     Available                 Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      gcc-4.1.2-46.el5          gcc-4.1.2                 passed
oraracn2      gcc-4.1.2-46.el5          gcc-4.1.2                 passed
Result: Package existence check passed for "gcc-4.1.2" 
 
Check: Package existence for "libaio-0.3.106 (i386)"
Node Name     Available                 Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      libaio-0.3.106-5 (i386)   libaio-0.3.106 (i386)     passed
oraracn2      libaio-0.3.106-5 (i386)   libaio-0.3.106 (i386)     passed
Result: Package existence check passed for "libaio-0.3.106 (i386)" 
 
Check: Package existence for "libaio-0.3.106 (x86_64)"
Node Name     Available                 Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      libaio-0.3.106-5 (x86_64)  libaio-0.3.106 (x86_64)   passed
oraracn2      libaio-0.3.106-5 (x86_64)  libaio-0.3.106 (x86_64)   passed
Result: Package existence check passed for "libaio-0.3.106 (x86_64)" 
 
Check: Package existence for "glibc-2.5-24 (i686)"
Node Name     Available                 Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      glibc-2.5-42 (i686)       glibc-2.5-24 (i686)       passed
oraracn2      glibc-2.5-42 (i686)       glibc-2.5-24 (i686)       passed
Result: Package existence check passed for "glibc-2.5-24 (i686)" 
 
Check: Package existence for "glibc-2.5-24 (x86_64)"
Node Name     Available                 Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      glibc-2.5-42 (x86_64)     glibc-2.5-24 (x86_64)     passed
oraracn2      glibc-2.5-42 (x86_64)     glibc-2.5-24 (x86_64)     passed
Result: Package existence check passed for "glibc-2.5-24 (x86_64)" 
 
Check: Package existence for "compat-libstdc++-33-3.2.3 (i386)"
Node Name     Available                            Required                   Comment  
------------  ------------------------             -----------------------    -------
oraracn1      compat-libstdc++-33-3.2.3-61 (i386)  compat-libstdc++-33-3.2.3 (i386)
passed
oraracn2      compat-libstdc++-33-3.2.3-61 (i386)  compat-libstdc++-33-3.2.3 (i386)
passed
Result: Package existence check passed for "compat-libstdc++-33-3.2.3 (i386)" 


Check: Package existence for "compat-libstdc++-33-3.2.3 (x86_64)"
Node Name     Available                 Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      compat-libstdc++-33-3.2.3-61 (x86_64)  compat-libstdc++-33-3.2.3 (x86 _64)  passed
oraracn2      compat-libstdc++-33-3.2.3-61 (x86_64)  compat-libstdc++-33-3.2.3 (x86 _64)  passed
Result: Package existence check passed for "compat-libstdc++-33-3.2.3 (x86_64)" 
 
Check: Package existence for "elfutils-libelf-0.125 (x86_64)"
Node Name     Available                 Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      elfutils-libelf-0.137-3.el5 (x86_64)  elfutils-libelf-0.125 (x86_64)  passed
oraracn2      elfutils-libelf-0.137-3.el5 (x86_64)  elfutils-libelf-0.125 (x86_64)  passed
Result: Package existence check passed for "elfutils-libelf-0.125 (x86_64)" 
 
Check: Package existence for "elfutils-libelf-devel-0.125"
Node Name     Available                 Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      elfutils-libelf-devel-0.137-3.el5  elfutils-libelf-devel-0.125  passed
oraracn2      elfutils-libelf-devel-0.137-3.el5  elfutils-libelf-devel-0.125  passed Result: Package existence check passed for "elfutils-libelf-devel-0.125" 
 
Check: Package existence for "glibc-common-2.5"
Node Name     Available                 Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      glibc-common-2.5-42       glibc-common-2.5          passed
oraracn2      glibc-common-2.5-42       glibc-common-2.5          passed
Result: Package existence check passed for "glibc-common-2.5" 
 
Check: Package existence for "glibc-devel-2.5 (i386)"
Node Name     Available                 Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      glibc-devel-2.5-42 (i386)  glibc-devel-2.5 (i386)    passed
oraracn2      glibc-devel-2.5-42 (i386)  glibc-devel-2.5 (i386)    passed
Result: Package existence check passed for "glibc-devel-2.5 (i386)" 
 
Check: Package existence for "glibc-devel-2.5 (x86_64)"
Node Name     Available                 Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      glibc-devel-2.5-42 (x86_64)  glibc-devel-2.5 (x86_64)  passed
oraracn2      glibc-devel-2.5-42 (x86_64)  glibc-devel-2.5 (x86_64)  passed
Result: Package existence check passed for "glibc-devel-2.5 (x86_64)" 
 
Check: Package existence for "glibc-headers-2.5"
Node Name     Available                 Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      glibc-headers-2.5-42      glibc-headers-2.5         passed
oraracn2      glibc-headers-2.5-42      glibc-headers-2.5         passed
Result: Package existence check passed for "glibc-headers-2.5"


Check: Package existence for "gcc-c++-4.1.2"
Node Name     Available                 Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      gcc-c++-4.1.2-46.el5      gcc-c++-4.1.2             passed
oraracn2      gcc-c++-4.1.2-46.el5      gcc-c++-4.1.2             passed
Result: Package existence check passed for "gcc-c++-4.1.2" 
 
Check: Package existence for "libaio-devel-0.3.106 (i386)"
Node Name     Available                 Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      libaio-devel-0.3.106-5 (i386)  libaio-devel-0.3.106 (i386)  passed
oraracn2      libaio-devel-0.3.106-5 (i386)  libaio-devel-0.3.106 (i386)  passed
Result: Package existence check passed for "libaio-devel-0.3.106 (i386)" 
 
Check: Package existence for "libaio-devel-0.3.106 (x86_64)"
Node Name     Available                 Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      libaio-devel-0.3.106-5 (x86_64)  libaio-devel-0.3.106 (x86_64)  passed
oraracn2      libaio-devel-0.3.106-5 (x86_64)  libaio-devel-0.3.106 (x86_64)  passed
Result: Package existence check passed for "libaio-devel-0.3.106 (x86_64)" 
 
Check: Package existence for "libgcc-4.1.2 (i386)"
Node Name     Available                 Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      libgcc-4.1.2-46.el5 (i386)  libgcc-4.1.2 (i386)       passed
oraracn2      libgcc-4.1.2-46.el5 (i386)  libgcc-4.1.2 (i386)       passed
Result: Package existence check passed for "libgcc-4.1.2 (i386)" 
 
Check: Package existence for "libgcc-4.1.2 (x86_64)"
Node Name     Available                 Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      libgcc-4.1.2-46.el5 (x86_64)  libgcc-4.1.2 (x86_64)     passed
oraracn2      libgcc-4.1.2-46.el5 (x86_64)  libgcc-4.1.2 (x86_64)     passed
Result: Package existence check passed for "libgcc-4.1.2 (x86_64)" 
 
Check: Package existence for "libstdc++-4.1.2 (i386)"
Node Name     Available                 Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      libstdc++-4.1.2-46.el5 (i386)  libstdc++-4.1.2 (i386)    passed
oraracn2      libstdc++-4.1.2-46.el5 (i386)  libstdc++-4.1.2 (i386)    passed
Result: Package existence check passed for "libstdc++-4.1.2 (i386)" 
 
Check: Package existence for "libstdc++-4.1.2 (x86_64)"
Node Name     Available                 Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      libstdc++-4.1.2-46.el5 (x86_64)  libstdc++-4.1.2 (x86_64)  passed
oraracn2      libstdc++-4.1.2-46.el5 (x86_64)  libstdc++-4.1.2 (x86_64)  passed
Result: Package existence check passed for "libstdc++-4.1.2 (x86_64)" 
 
Check: Package existence for "libstdc++-devel-4.1.2 (x86_64)"
Node Name     Available                 Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      libstdc++-devel-4.1.2-46.el5 (x86_64)  libstdc++-devel-4.1.2 (x86_64)
passed
oraracn2      libstdc++-devel-4.1.2-46.el5 (x86_64)  libstdc++-devel-4.1.2 (x86_64)
passed
Result: Package existence check passed for "libstdc++-devel-4.1.2 (x86_64)"


Check: Package existence for "sysstat-7.0.2"
Node Name     Available                 Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      sysstat-7.0.2-13.0.1.el5  sysstat-7.0.2             passed
oraracn2      sysstat-7.0.2-13.0.1.el5  sysstat-7.0.2             passed
Result: Package existence check passed for "sysstat-7.0.2" 
 
Check: Package existence for "unixODBC-2.2.11 (i386)"
Node Name     Available                 Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      unixODBC-2.2.11-10.el5 (i386)  unixODBC-2.2.11 (i386)    passed
oraracn2      unixODBC-2.2.11-10.el5 (i386)  unixODBC-2.2.11 (i386)    passed
Result: Package existence check passed for "unixODBC-2.2.11 (i386)" 
 
Check: Package existence for "unixODBC-2.2.11 (x86_64)"
Node Name     Available                 Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      unixODBC-2.2.11-10.el5 (x86_64)  unixODBC-2.2.11 (x86_64)  passed
oraracn2      unixODBC-2.2.11-10.el5 (x86_64)  unixODBC-2.2.11 (x86_64)  passed
Result: Package existence check passed for "unixODBC-2.2.11 (x86_64)" 
 
Check: Package existence for "unixODBC-devel-2.2.11 (i386)"
Node Name     Available                 Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      unixODBC-devel-2.2.11-10.el5 (i386)  unixODBC-devel-2.2.11 (i386)  passed
oraracn2      unixODBC-devel-2.2.11-10.el5 (i386)  unixODBC-devel-2.2.11 (i386)  passed
Result: Package existence check passed for "unixODBC-devel-2.2.11 (i386)" 
 
Check: Package existence for "unixODBC-devel-2.2.11 (x86_64)"
Node Name     Available                 Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      unixODBC-devel-2.2.11-10.el5 (x86_64)  unixODBC-devel-2.2.11 (x86_64)  passed
oraracn2      unixODBC-devel-2.2.11-10.el5 (x86_64)  unixODBC-devel-2.2.11 (x86_64)  passed
Result: Package existence check passed for "unixODBC-devel-2.2.11 (x86_64)" 
 
Check: Package existence for "ksh-20060214"
Node Name     Available                 Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      ksh-20080202-14.el5       ksh-20060214              passed
oraracn2      ksh-20080202-14.el5       ksh-20060214              passed
Result: Package existence check passed for "ksh-20060214" 
 
Checking for multiple users with UID value 0 Result: Check for multiple users with UID value 0 passed 
 
Check: Current group ID Result: Current group ID check passed Checking Core file name pattern consistency... Core file name pattern consistency check passed. 
 
Checking to make sure user "oracle" is not in "root" group
Node Name     Status                    Comment
------------  ------------------------  ------------------------
oraracn1      does not exist            passed
oraracn2      does not exist            passed

Result: User "oracle" is not part of "root" group. Check passed 
 
Check default user file creation mask
Node Name     Available                 Required                  Comment
------------  ------------------------  ------------------------  ----------
oraracn1      0022                      0022                      passed
oraracn2      0022                      0022                      passed
Result: Default user file creation mask check passed 
 
Starting Clock synchronization checks using Network Time Protocol(NTP)... 
 
NTP Configuration file check started... Network Time Protocol(NTP) configuration file not found on any of the nodes. Oracle C luster Time Synchronization Service(CTSS) can be used instead of NTP for time synchro nization on the cluster nodes 
 
Result: Clock synchronization check using Network Time Protocol(NTP) passed 
 
 
Pre-check for cluster services setup was successful. 

Note: You must see Pre-check for cluster services setup was successful at the end of the cluvfy output.

Install Grid Software

The following steps will guide you to install Oracle Grid software with Oracle ASM binaries. As oracle user, start the grid runInstaller

On Node 1: 
==========
su – oracle 

# cd /u02/grid/ 
# ./runInstaller

Note: sometimes there might be display variable error. issues below command as root user to enable GUI

# xhost +

Under Select Installation Option, choose Install and Configure Grid Infrastructure for a Cluster and Click Next

Select Advanced Installation and click Next

Select Advanced Installation and click Next

Input below details and click Next

Below screen will list out the nodes which you want to install and configure as part of Grid

Click on Add and enter details of Node 2. Click on OK

In case you have more than two nodes, Add each node by clicking on Add. Click Next to proceed

Specify network interface by selecting public and private interfaces. Click Next

Select storage option as Automatic Storage Management (ASM). Click Next

You need to assign a disk where the grid installer will keep OCR and Voting disks. Click on Change Discovery Path, type /dev/oracleasm/disks/* and click OK. Under Add Disks, you must see the three disks which we have created as part of Shard Storage.

Before you proceed, sometimes you might not see the lists of disks. In such cases, follow below:

1. Check if /dev/sdb1, /dev/sdc1, /dev/sdd1 had ownership as oracle user. If not then issue below commands as root user on both nodes

# chown oracle:oinstall /dev/sdb1
# chown oracle:oinstall /dev/sdc1
# chown oracle:oinstall /dev/sdd1

2. Also give full permissions on above disks

# chmod 777 /dev/sdb1
# chmod 777 /dev/sdc1
# chmod 777 /dev/sdd1

Again go to Change Discovery Path, type /dev/oracleasm/disks/* and click OK. Now all the disks will be listed. Type Disk Group Name as CRS, Redundancy as External and under Disk Path /dev/oracleasm/disks/CRSVOL1

Now provide password for sys user and click Next

It will prompt you for password standard, just click on Yes

Select Do Not Use IPMI and click Next

Go with default operating system groups and click Next. If prompted with notification, just click on Yes

Provide Oracle Base and Software Installation Location. Click on Next

Go with default inventory location. Click Next

The run Installer will perform pre-requsites test again. If any step is failed, you must fix it before proceeding ahead

The final screen displays installation summary. Important things to note here:

• Oracle Base Location

• Oracle Home – This should actually be grid home

Click on Finish to start the installation

Now the setup will start installing Grid Software on both nodes

Now the installer will prompt you for two root.sh scripts to be run on each node. Execute them as root user on both nodes one by one.

Your grid installation will finish and Cluster Verification Utility Filed error will be displayed. Just ignore the error.

Oracle RAC Installation

During the installation of Oracle grid infrastructure, we configured one ASM disk group named +CRS which was used to store the Oracle clusterware files (OCR and voting disk). Now we will configure ASM disk groups for DATA and FRA

Create DATA & FRA Diskgroups

Launch asmca form the command prompt as oracle user

On node 1:
==========

su – oracle
asmca

ASM Configuration Assistant window will launch. From the Disk Groups tab, click on Create.

The Create Disk Group window will open and it should show two of the ASMLib volumes we created earlier

Click on Create. Give a name to your DATA disk group. This disk group will hold database files. Then click on OK

Install Oracle RAC Software – Node 1 only

Unzip the database software files under /u02. There will be two files downloaded. Unzip them as oracle user one by one

#	su - oracle
#	cd /u02
#	unzip linux.x64_11gR2_database_1of2.zip
#	unzip linux.x64_11gR2_database_2of2.zip

Unzipping files will create a new folder as database under same location

# ls -lrt
drwxr-xr-x 8	oracle oinstall	4096	Aug 21	2009	database
-rwxr-xr-x 1	oracle oinstall 1052897657	Nov	2	09:41	linux.x64_11gR2_grid.zip
-rwxr-xr-x 1	oracle oinstall 1239269270	Dec	1	01:00	linux.x64_11gR2_database_1of2.zip
-rwxr-xr-x 1	oracle oinstall	1111416131	Dec	1	01:01	linux.x64_11gR2_database_2of2.zip

Go to database folder and start the runInstaller

#	cd database
#	./runInstaller

Create RAC Database

From node 1, start dbca as oracle user

On node 1:
==========
su - oracle
dbca

Done! Your 2 node RAC database is ready to use!