Building an Oracle10g RAC using VMWare Workstation 5.0

1. Setting up VMWare for Clustering
   1. Introduction
   2. Architecture
   3. Install VmWare 5.0
   4. Create virtual machines
      1. Node 1 Config File
      2. Node 2 Config File
2. Install Redhat Linux 3.0
   1. Introduction
   2. Install Linux
      1. Disk Configuration
      2. Network Configuration
      3. Select Packages to Install
   3. Disable all services that are not needed
   4. Configure Kernel Parameters
   5. /etc/hosts file configuration
   6. rsh configuration
   7. Create Oracle Groups/ User
   8. Create Mount Point for Oracle Software
   9. bash profile configuration
   10. Raw Devices Configuration
3. Install Clusterware (Oracle CRS): http://docs.oracle.com/cd/B28359_01/install.111/b28263/crsunix.htm
4. Install Oracle 10g database software
5. Create a new database using DBCA
6. Downloads and additional scripts: …

————————————————————

Detail:

1. Setting up VMWare for Clustering

1. Introduction
2. Architecture
3. Install VmWare 5.0
4. Create virtual machines
1. Node 1 Config File
2. Node 2 Config File

1.a Introduction

Usually Oracle 10g RAC database is installed on multiple machines connected to shared storage (such as an EMC) via fiber channel. We will try to do the same thing using VMWare on one machine. The trick is we could emulate all the components of the cluster using Vmware. The computer which runs the VMWare emulation software is called as the HOST computer and the virtual machines are called GUEST computers. So basically if you are planning to build a RAC on a single workstation, you will end up with 3 operating systems (2 virtual and 1 host) running on the same machine. Needless to say you will need atleast 2 GB of RAM and a good processor to try this setup.

Here are the minimum requirements.

Component	Minimm	Recommended
Processor	2 GHZ Pentium 4	Dual Core Pentium 3.2 GHZ
Memory	2 GB	4GB
Hard Disk	20 GB Free	60 GB free

 

1.b Architecture

A typical Real application cluster installation would look like the image below. You have multiple nodes each running an instance of the database communicating over a private network (interconnect) and using virtual IP addresses to communicate with the clients. Usually these virtual IP addresses are on the public network. We will try to emulate all of the following components using VMWare

• Private Network
• Public Network
• Nodes
• Shared Storage
• Client that is outside of the cluster

 

When using VmWare you could create multiple networks internally by making the nodes connect to virtual switches. Connect both the nodes to the same switch and give them consecutive IP addresses and they will be able to communicate immediately.

For the public network I have chosen to use the NAT feature, although i chose the IP addresses instead of relying on DHCP.

Node Name	Public Interface	Private Interface
Node1	192.168.107.101	10.0.0.101
Node2	192.168.107.102	10.0.0.102

Your DHCP 3rd octet might be different. I noticed it was changing on different machines with VMWare installed.

 

1.c Install VMWare 5.0

Installing VMWare is straight forward. Download and Install version 5.0 workstation from http://www.vmware.com .

1.d Create Virtual Machines

The best way to organize your virtual machines is as shown in the image below. You will have one Root folder called RAC – Each Node will have its own folder. Remember each node has its own OS / Oracle installation disk. This is named sda.vmdk. You will also need an OCR (Oracle cluster registry) and voting disk , I created both of them around 500 MB. There are advantages in creating them as seperate disks instead of partitions of a bigger disk

• More Stable
• Easy to backup the entire disk.

Also create ASM disks as you prefer. Here i have 3 ASM disks each with 10 GB capacity.

Component	File Name	Size
Entire Structure	RAC.zip	24 KB
Empty 10GB Disk	10G.zip	3 KB
Empty 20GB Disk	20G.zip	6 KB
Empty 500M Disk	500M.zip	1 KB

The file sizes are small as VMWare does not allocate disk space(It is an option if you want to preallocate space) when creating the virtual machine. Below is the VMX file for each vmware node (Node 1 and Node 2).

1.d.i Node 1 Configuration File

Node 1 ( Location : C:\RAC\Node1\rhel3.vmx )

config.version = “8″ virtualHW.version = “4″ scsi0.present = “TRUE” scsi0.virtualDev = “lsilogic” memsize = “1024″ scsi0:0.present = “TRUE” scsi0:0.fileName = “sda.vmdk” ide1:0.present = “TRUE” ide1:0.fileName = “auto detect” ide1:0.deviceType = “cdrom-raw” floppy0.present = “FALSE” ethernet0.present = “TRUE” ethernet0.connectionType = “nat” usb.present = “FALSE” sound.present = “FALSE” sound.virtualDev = “es1371″ displayName = “Node1″ guestOS = “rhel3″ nvram = “rhel3.nvram”ethernet1.present = “TRUE” ethernet1.connectionType = “custom” ethernet1.vnet = “VMnet2″ scsi1.present = “TRUE” scsi1.virtualDev = “lsilogic” scsi1.sharedBus = “VIRTUAL” scsi1:1.present = “TRUE” scsi1:1.mode = “independent-persistent” scsi1:1.fileName = “..\Shared\ocr.vmdk” scsi1:1.deviceType = “disk” scsi1:2.present = “TRUE” scsi1:2.mode = “independent-persistent” scsi1:2.fileName = “..\Shared\votingdisk.vmdk” scsi1:2.deviceType = “disk” scsi1:3.present = “TRUE” scsi1:3.mode = “independent-persistent” scsi1:3.fileName = “..\Shared\spfileasm.vmdk” scsi1:3.deviceType = “disk” scsi1:4.present = “TRUE” scsi1:4.mode = “independent-persistent” scsi1:4.fileName = “..\Shared\ASM1.vmdk” scsi1:4.deviceType = “disk” scsi1:5.present = “TRUE” scsi1:5.mode = “independent-persistent” scsi1:5.fileName = “..\Shared\ASM2.vmdk” scsi1:5.deviceType = “disk” scsi1:6.present = “TRUE” scsi1:6.mode = “independent-persistent” scsi1:6.fileName = “..\Shared\ASM3.vmdk” scsi1:6.deviceType = “disk” disk.locking = “FALSE” diskLib.dataCacheMaxSize = “0″ diskLib.dataCacheMaxReadAheadSize = “0″ diskLib.dataCacheMinReadAheadSize = “0″ diskLib.dataCachePageSize = “4096″ diskLib.maxUnsyncedWrites = “0″ scsi0:0.redo = “” scsi1:1.redo = “” ethernet0.addressType = “generated” ethernet1.addressType = “generated” uuid.location = “56 4d 72 58 f2 c4 75 75-bd 79 8c 2b b4 75 ad ca” uuid.bios = “56 4d 72 58 f2 c4 75 75-bd 79 8c 2b b4 75 ad ca” tools.remindInstall = “TRUE” ethernet0.generatedAddress = “00:0c:29:75:ad:ca” ethernet0.generatedAddressOffset = “0″ ethernet1.generatedAddress = “00:0c:29:75:ad:d4″ ethernet1.generatedAddressOffset = “10″ ide1:0.startConnected = “TRUE” tools.syncTime = “FALSE” scsi1:2.redo = “” scsi1:3.redo = “” scsi1:4.redo = “” scsi1:5.redo = “” scsi1:6.redo = “” scsi1:7.redo = “”

1.d.ii Node 2 Configuration File

Node 2 ( Location : C:\RAC\Node2\rhel3.vmx )

config.version = “8″ virtualHW.version = “4″ scsi0.present = “TRUE” scsi0.virtualDev = “lsilogic” memsize = “1024″ scsi0:0.present = “TRUE” scsi0:0.fileName = “sda.vmdk” ide1:0.present = “TRUE” ide1:0.fileName = “auto detect” ide1:0.deviceType = “cdrom-raw” floppy0.present = “FALSE” ethernet0.present = “TRUE” ethernet0.connectionType = “nat” usb.present = “FALSE” sound.present = “FALSE” sound.virtualDev = “es1371″ displayName = “Node2″ guestOS = “rhel3″ nvram = “rhel3.nvram” ethernet1.present = “TRUE” ethernet1.connectionType = “custom” ethernet1.vnet = “VMnet2″ scsi1.present = “TRUE” scsi1.virtualDev = “lsilogic” scsi1.sharedBus = “VIRTUAL” scsi1:1.present = “TRUE” scsi1:1.mode = “independent-persistent” scsi1:1.fileName = “..\Shared\ocr.vmdk” scsi1:1.deviceType = “disk” scsi1:2.present = “TRUE” scsi1:2.mode = “independent-persistent” scsi1:2.fileName = “..\Shared\votingdisk.vmdk” scsi1:2.deviceType = “disk” scsi1:3.present = “TRUE” scsi1:3.mode = “independent-persistent” scsi1:3.fileName = “..\Shared\spfileasm.vmdk” scsi1:3.deviceType = “disk” scsi1:4.present = “TRUE” scsi1:4.mode = “independent-persistent” scsi1:4.fileName = “..\Shared\ASM1.vmdk” scsi1:4.deviceType = “disk” scsi1:5.present = “TRUE” scsi1:5.mode = “independent-persistent” scsi1:5.fileName = “..\Shared\ASM2.vmdk” scsi1:5.deviceType = “disk” scsi1:6.present = “TRUE” scsi1:6.mode = “independent-persistent” scsi1:6.fileName = “..\Shared\ASM3.vmdk” scsi1:6.deviceType = “disk” disk.locking = “FALSE” diskLib.dataCacheMaxSize = “0″ diskLib.dataCacheMaxReadAheadSize = “0″ diskLib.dataCacheMinReadAheadSize = “0″ diskLib.dataCachePageSize = “4096″ diskLib.maxUnsyncedWrites = “0″ uuid.bios = “56 4d 88 e1 d8 25 00 e0-e3 ae f1 4f b1 65 76 26″ scsi0:0.redo = “” scsi1:1.redo = “” ethernet0.addressType = “generated” ethernet1.addressType = “generated” uuid.location = “56 4d 88 e1 d8 25 00 e0-e3 ae f1 4f b1 65 76 26″ tools.remindInstall = “TRUE” ethernet0.generatedAddress = “00:0c:29:65:76:26″ ethernet0.generatedAddressOffset = “0″ ethernet1.generatedAddress = “00:0c:29:65:76:30″ ethernet1.generatedAddressOffset = “10″ tools.syncTime = “FALSE” scsi1:2.redo = “” scsi1:3.redo = “” scsi1:4.redo = “” scsi1:5.redo = “” scsi1:6.redo = “”

Open the virtual machines now in VmWare. You should see the screens like below.

Freshly opened VM’s in VMWare

All the hardware components in each of your virtual machines. Now both nodes are ready for RHEL 3.0 installations.

2. Installing RHEL 3.0 for Oracle on VMWare 5.0

---

1. Introduction
2. Install Linux
   1. Disk Configuration
   2. Network Configuration
   3. Select Packages to Install
3. Disable all services that are not needed
4. Configure Kernel Parameters
5. /etc/hosts file configuration
6. rsh configuration
7. Create Oracle Groups/ User
8. Create Mount Point for Oracle Software
9. bash profile configuration
10. Raw Devices Configuration

2.a – Introduction

For Oracle installations, you do not need X desktop running on the Linux Server at all times. Our objective is to allocate as much memory possible to Oracle. We will still need an X server installed as Oracle Universal Installer uses GUI to perform installations. For this we could simply use X11 forwarding feature in SSH and a proper windows X Client (Like Cygwin – You could download Cygwin from http://www.cygwin.com)

 

2.b – Install Linux

Download the ISO images for Linux. You could use these instead of CD ROM’s on your VMWare. If you are using the config files provided then you might get a message like this. You could just select to create a new UUID for the VM

 

 

Click Next

Accept defaults Click Next

Accept Defaults and Click Next

Next

2.b.i – Disk Configuration

Select Automatically Partition on this Screen and Select Next

This is an important Screen. We have 6 drives (sda,sdb,sdc,sdd,sde,sdf as the drives ) but you should use only the 1st drive for the operating system.

Select “YES” only for “sda”. You should select “NO” for the remaining drives

Select Remove all Partitions on the system and click Next

Select Yes here.

Your screen should look similar to the one above. Click Next

Accept Defaults. Click Next

 

 

2.b.ii – Network Configuration

Node Name	Public Interface	Private Interface
Node1	192.168.56.101	10.0.0.101
Node2	192.168.56.102	10.0.0.102

In this installation of VMWare my NAT 3rd octet is 56. It might be different for your installation. Check it by Going into Edit –> Virtual Network Preferences

Check both Network Interfaces to “Activate on Boot”. Click Edit for eth0. This is connected to the NAT interface. So I chose the IP adress 192.168.56.101 (Since this is Node1, For Node2 it should be 192.168.56.102)

 

For the Private interconnect, I chose 10.0.0.101 as the IP address for Node1 and 10.0.0.102 for Node2

Set the Host Name to Node1, If you are installing Node2 set it to Node2. Also set the Gateway and Primary DNS Servers.

 

I chose No Firewall.

Accept defaults and Click Next

I live in Newyork!!

Choose a root password and click Next

Select Customize packages and Click Next

2.b.iii – Select Packages to Install

• Uncheck GNOME desktop Environment
• Uncheck Web Server
• Uncheck Windows File Server
• Check Legacy Nework Server
  • Go into Details
  • Check Only rsh-server (Uncheck everything else
• Check Development Tools
• Check Kernel Development
• Check Legacy Software Development

If you follow the instructions above the total size should be 1979 MB. Click Next

I skipped X configuration as i am not going to use the graphical desktop

Once Installation is done. the machine will be rebooted.

2.c – Disable all services that are not needed

Start the virtual machine you just created and logon as root. Execute the command ntsysv

I chose to uncheck everything except

• crond
• network
• rawdevices
• rsh
• sshd
• syslog
• xinetd

Now reboot the machine. This time the boot time should be reduced considerably.

2.d – Configure Kernel Parameters

Logon to the machine as root and edit the file /etc/sysctl.conf

Add the following lines

kernel.shmall = 2097152 kernel.shmmax = 2147483648 kernel.shmmni = 4096 kernel.sem = 250 32000 100 128 fs.file-max = 65536 net.ipv4.ip_local_port_range = 1024 65000

The resulting file should look something like this

Both nodes ( Location : /etc/sysctl.conf )

# Kernel sysctl configuration file for Red Hat Linux # # For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and # sysctl.conf(5) for more details. # Controls IP packet forwarding net.ipv4.ip_forward = 0 # Controls source route verification net.ipv4.conf.default.rp_filter = 1 # Controls the System Request debugging functionality of the kernel kernel.sysrq = 0 # Controls whether core dumps will append the PID to the core filename. # Useful for debugging multi-threaded applications. kernel.core_uses_pid = 1 kernel.shmall = 2097152 kernel.shmmax = 2147483648 kernel.shmmni = 4096 kernel.sem = 250 32000 100 128 fs.file-max = 65536 net.ipv4.ip_local_port_range = 1024 65000

2.e – /etc/hosts File Configuration

Logon to the machine as root and edit the file /etc/hosts. The file should look like the one below. You should perform this operation on both nodes

Both nodes ( Location : /etc/hosts )

# Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 localhost.localdomain localhost 192.168.107.101 node1 192.168.107.102 node2 10.0.0.101 node1-prv 10.0.0.102 node2-prv 192.168.107.103 node1-vip 192.168.107.104 node2-vip

2.f – rsh Configuration

Oracle universal installer could use either rsh or ssh to copy the files (actually scp or rcp) across multiple nodes. We will use rsh instead of ssh just for simplicity sake. For rsh to work, you need to edit the file /etc/hosts.equiv . If this file does not exist just create it. You should perform this operation on both nodes.

Both nodes ( Location : /etc/hosts.equiv )

+node1 +node2

2.g – Create Oracle Groups/ User

Execute the following commands in that order to create oracle users and groups. You should perform this operation on both nodes.

groupadd oinstall groupadd dba useradd -g oinstall -G dba oracle passwd oracle

The last command will set a password for Oracle.

2.h – Create Mount Point for Oracle Software

Execute the following commands to create an Oracle base for software installation. You should perform this operation on both nodes.

mkdir -p /u01/app/oracle chown -R oracle.oinstall /u01

2.i – Bash Profile configuration

Logon to the machine as oracle and edit the file ~/.bash_profile. You should perform this operation on both nodes. The files should look like the ones below.On the second node make sure the ORACLE_SID is set to RACDB2

Node 1 ( Location : /home/oracle/.bash_profile )

# .bash_profile # Get the aliases and functions if [ -f ~/.bashrc ]; then . ~/.bashrc fi # User specific environment and startup programs export ORACLE_BASE=/u01/app/oracle export ORACLE_HOME=/u01/app/oracle/product/10.1.0/RAC export ORA_CRS_HOME=/u01/app/oracle/product/10.1.0/CRS export ORACLE_SID=RACDB1 PATH=$PATH:$HOME/bin:$ORACLE_HOME/bin:$ORA_CRS_HOME/bin export PATH unset USERNAME

 

Node 2 ( Location : /home/oracle/.bash_profile )

# .bash_profile # Get the aliases and functions if [ -f ~/.bashrc ]; then . ~/.bashrc fi # User specific environment and startup programs export ORACLE_BASE=/u01/app/oracle export ORACLE_HOME=/u01/app/oracle/product/10.1.0/RAC export ORA_CRS_HOME=/u01/app/oracle/product/10.1.0/CRS export ORACLE_SID=RACDB2 PATH=$PATH:$HOME/bin:$ORACLE_HOME/bin:$ORA_CRS_HOME/bin export PATH unset USERNAME

 

2.j – Raw Devices Configuration

Logon to the machine as root and execute the following commands . The file should look like the one below. You should perform this operation on both nodes

raw /dev/raw/raw1 /dev/sdb raw /dev/raw/raw2 /dev/sdc raw /dev/raw/raw3 /dev/sdd raw /dev/raw/raw4 /dev/sde raw /dev/raw/raw5 /dev/sdf raw /dev/raw/raw6 /dev/sdg mv /dev/raw/raw1 /dev/raw/votingdisk mv /dev/raw/raw2 /dev/raw/ocr.dbf mv /dev/raw/raw3 /dev/raw/spfile+ASM.ora mv /dev/raw/raw4 /dev/raw/ASM1 mv /dev/raw/raw5 /dev/raw/ASM2 mv /dev/raw/raw6 /dev/raw/ASM3 chown oracle.dba /dev/raw/votingdisk chown oracle.dba /dev/raw/ocr.dbf chown oracle.dba /dev/raw/spfile+ASM.ora chown oracle.dba /dev/raw/ASM1 chown oracle.dba /dev/raw/ASM2 chown oracle.dba /dev/raw/ASM3 echo "/dev/raw/ocr.dbf /dev/sdb" >> /etc/sysconfig/rawdevices echo "/dev/raw/votingdisk /dev/sdc" >> /etc/sysconfig/rawdevices echo "/dev/raw/spfile+ASM.ora /dev/sdd" >> /etc/sysconfig/rawdevices echo "/dev/raw/ASM1 /dev/sde" >> /etc/sysconfig/rawdevices echo "/dev/raw/ASM2 /dev/sdf" >> /etc/sysconfig/rawdevices echo "/dev/raw/ASM3 /dev/sdg" >> /etc/sysconfig/rawdevices

 

Your /etc/sysconfig/rawdevices should look like this on both nodes

Both nodes ( Location : /etc/sysconfig/rawdevices )

# raw device bindings # format: <rawdev> <major> <minor> # <rawdev> <blockdev> # example: /dev/raw/raw1 /dev/sda1 # /dev/raw/raw2 8 5 /dev/raw/ocr.dbf /dev/sdb /dev/raw/votingdisk /dev/sdc /dev/raw/spfile+ASM.ora /dev/sdd /dev/raw/ASM1 /dev/sde /dev/raw/ASM2 /dev/sdf /dev/raw/ASM3 /dev/sdg