Building an Oracle10g RAC using VMWare Workstation 5.0

  1. Setting up VMWare for Clustering
    1. Introduction
    2. Architecture
    3. Install VmWare 5.0
    4. Create virtual machines
      1. Node 1 Config File
      2. Node 2 Config File
  2. Install Redhat Linux 3.0
    1. Introduction
    2. Install Linux
      1. Disk Configuration
      2. Network Configuration
      3. Select Packages to Install
    3. Disable all services that are not needed
    4. Configure Kernel Parameters
    5. /etc/hosts file configuration
    6. rsh configuration
    7. Create Oracle Groups/ User
    8. Create Mount Point for Oracle Software
    9. bash profile configuration
    10. Raw Devices Configuration
  3. Install Clusterware (Oracle CRS): http://docs.oracle.com/cd/B28359_01/install.111/b28263/crsunix.htm
  4. Install Oracle 10g database software
  5. Create a new database using DBCA
  6. Downloads and additional scripts: …

————————————————————

Detail:

1. Setting up VMWare for Clustering

  1. Introduction
  2. Architecture
  3. Install VmWare 5.0
  4. Create virtual machines
    1. Node 1 Config File
    2. Node 2 Config File

1.a Introduction

Usually Oracle 10g RAC database is installed on multiple machines connected to shared storage (such as an EMC) via fiber channel. We will try to do the same thing using VMWare on one machine. The trick is we could emulate all the components of the cluster using Vmware. The computer which runs the VMWare emulation software is called as the HOST computer and the virtual machines are called GUEST computers. So basically if you are planning to build a RAC on a single workstation, you will end up with 3 operating systems (2 virtual and 1 host) running on the same machine. Needless to say you will need atleast 2 GB of RAM and a good processor to try this setup.

Here are the minimum requirements.

Component Minimm Recommended
Processor 2 GHZ Pentium 4 Dual Core Pentium 3.2 GHZ
Memory 2 GB 4GB
Hard Disk 20 GB Free 60 GB free

 

1.b Architecture

A typical Real application cluster installation would look like the image below. You have multiple nodes each running an instance of the database communicating over a private network (interconnect) and using virtual IP addresses to communicate with the clients. Usually these virtual IP addresses are on the public network. We will try to emulate all of the following components using VMWare

  • Private Network
  • Public Network
  • Nodes
  • Shared Storage
  • Client that is outside of the cluster

 

When using VmWare you could create multiple networks internally by making the nodes connect to virtual switches. Connect both the nodes to the same switch and give them consecutive IP addresses and they will be able to communicate immediately.

For the public network I have chosen to use the NAT feature, although i chose the IP addresses instead of relying on DHCP.

Node Name Public Interface Private Interface
Node1 192.168.107.101 10.0.0.101
Node2 192.168.107.102 10.0.0.102

Your DHCP 3rd octet might be different. I noticed it was changing on different machines with VMWare installed.

 

1.c Install VMWare 5.0

Installing VMWare is straight forward. Download and Install version 5.0 workstation from http://www.vmware.com .

1.d Create Virtual Machines

The best way to organize your virtual machines is as shown in the image below. You will have one Root folder called RAC – Each Node will have its own folder. Remember each node has its own OS / Oracle installation disk. This is named sda.vmdk. You will also need an OCR (Oracle cluster registry) and voting disk , I created both of them around 500 MB. There are advantages in creating them as seperate disks instead of partitions of a bigger disk

  • More Stable
  • Easy to backup the entire disk.

Also create ASM disks as you prefer. Here i have 3 ASM disks each with 10 GB capacity.

Component File Name Size
Entire Structure RAC.zip 24 KB
Empty 10GB Disk 10G.zip 3 KB
Empty 20GB Disk 20G.zip 6 KB
Empty 500M Disk 500M.zip 1 KB

The file sizes are small as VMWare does not allocate disk space(It is an option if you want to preallocate space) when creating the virtual machine. Below is the VMX file for each vmware node (Node 1 and Node 2).

1.d.i Node 1 Configuration File

Node 1 ( Location : C:\RAC\Node1\rhel3.vmx )
config.version = “8”
virtualHW.version = “4”
scsi0.present = “TRUE”
scsi0.virtualDev = “lsilogic”
memsize = “1024”
scsi0:0.present = “TRUE”
scsi0:0.fileName = “sda.vmdk”
ide1:0.present = “TRUE”
ide1:0.fileName = “auto detect”
ide1:0.deviceType = “cdrom-raw”
floppy0.present = “FALSE”
ethernet0.present = “TRUE”
ethernet0.connectionType = “nat”
usb.present = “FALSE”
sound.present = “FALSE”
sound.virtualDev = “es1371”
displayName = “Node1”
guestOS = “rhel3”
nvram = “rhel3.nvram”ethernet1.present = “TRUE”
ethernet1.connectionType = “custom”
ethernet1.vnet = “VMnet2”

scsi1.present = “TRUE”
scsi1.virtualDev = “lsilogic”
scsi1.sharedBus = “VIRTUAL”
scsi1:1.present = “TRUE”
scsi1:1.mode = “independent-persistent”
scsi1:1.fileName = “..\Shared\ocr.vmdk”
scsi1:1.deviceType = “disk”

scsi1:2.present = “TRUE”
scsi1:2.mode = “independent-persistent”
scsi1:2.fileName = “..\Shared\votingdisk.vmdk”
scsi1:2.deviceType = “disk”

scsi1:3.present = “TRUE”
scsi1:3.mode = “independent-persistent”
scsi1:3.fileName = “..\Shared\spfileasm.vmdk”
scsi1:3.deviceType = “disk”

scsi1:4.present = “TRUE”
scsi1:4.mode = “independent-persistent”
scsi1:4.fileName = “..\Shared\ASM1.vmdk”
scsi1:4.deviceType = “disk”

scsi1:5.present = “TRUE”
scsi1:5.mode = “independent-persistent”
scsi1:5.fileName = “..\Shared\ASM2.vmdk”
scsi1:5.deviceType = “disk”

scsi1:6.present = “TRUE”
scsi1:6.mode = “independent-persistent”
scsi1:6.fileName = “..\Shared\ASM3.vmdk”
scsi1:6.deviceType = “disk”

disk.locking = “FALSE”
diskLib.dataCacheMaxSize = “0”
diskLib.dataCacheMaxReadAheadSize = “0”
diskLib.dataCacheMinReadAheadSize = “0”
diskLib.dataCachePageSize = “4096”
diskLib.maxUnsyncedWrites = “0”

scsi0:0.redo = “”
scsi1:1.redo = “”

ethernet0.addressType = “generated”
ethernet1.addressType = “generated”
uuid.location = “56 4d 72 58 f2 c4 75 75-bd 79 8c 2b b4 75 ad ca”
uuid.bios = “56 4d 72 58 f2 c4 75 75-bd 79 8c 2b b4 75 ad ca”
tools.remindInstall = “TRUE”
ethernet0.generatedAddress = “00:0c:29:75:ad:ca”
ethernet0.generatedAddressOffset = “0”
ethernet1.generatedAddress = “00:0c:29:75:ad:d4”
ethernet1.generatedAddressOffset = “10”

ide1:0.startConnected = “TRUE”

tools.syncTime = “FALSE”

scsi1:2.redo = “”
scsi1:3.redo = “”
scsi1:4.redo = “”
scsi1:5.redo = “”
scsi1:6.redo = “”
scsi1:7.redo = “”

1.d.ii Node 2 Configuration File

Node 2 ( Location : C:\RAC\Node2\rhel3.vmx )
config.version = “8”
virtualHW.version = “4”
scsi0.present = “TRUE”
scsi0.virtualDev = “lsilogic”
memsize = “1024”
scsi0:0.present = “TRUE”
scsi0:0.fileName = “sda.vmdk”
ide1:0.present = “TRUE”
ide1:0.fileName = “auto detect”
ide1:0.deviceType = “cdrom-raw”
floppy0.present = “FALSE”
ethernet0.present = “TRUE”
ethernet0.connectionType = “nat”
usb.present = “FALSE”
sound.present = “FALSE”
sound.virtualDev = “es1371”
displayName = “Node2”
guestOS = “rhel3”
nvram = “rhel3.nvram”

ethernet1.present = “TRUE”
ethernet1.connectionType = “custom”
ethernet1.vnet = “VMnet2”

scsi1.present = “TRUE”
scsi1.virtualDev = “lsilogic”
scsi1.sharedBus = “VIRTUAL”
scsi1:1.present = “TRUE”
scsi1:1.mode = “independent-persistent”
scsi1:1.fileName = “..\Shared\ocr.vmdk”
scsi1:1.deviceType = “disk”

scsi1:2.present = “TRUE”
scsi1:2.mode = “independent-persistent”
scsi1:2.fileName = “..\Shared\votingdisk.vmdk”
scsi1:2.deviceType = “disk”

scsi1:3.present = “TRUE”
scsi1:3.mode = “independent-persistent”
scsi1:3.fileName = “..\Shared\spfileasm.vmdk”
scsi1:3.deviceType = “disk”

scsi1:4.present = “TRUE”
scsi1:4.mode = “independent-persistent”
scsi1:4.fileName = “..\Shared\ASM1.vmdk”
scsi1:4.deviceType = “disk”

scsi1:5.present = “TRUE”
scsi1:5.mode = “independent-persistent”
scsi1:5.fileName = “..\Shared\ASM2.vmdk”
scsi1:5.deviceType = “disk”

scsi1:6.present = “TRUE”
scsi1:6.mode = “independent-persistent”
scsi1:6.fileName = “..\Shared\ASM3.vmdk”
scsi1:6.deviceType = “disk”

disk.locking = “FALSE”
diskLib.dataCacheMaxSize = “0”
diskLib.dataCacheMaxReadAheadSize = “0”
diskLib.dataCacheMinReadAheadSize = “0”
diskLib.dataCachePageSize = “4096”
diskLib.maxUnsyncedWrites = “0”

uuid.bios = “56 4d 88 e1 d8 25 00 e0-e3 ae f1 4f b1 65 76 26”

scsi0:0.redo = “”
scsi1:1.redo = “”
ethernet0.addressType = “generated”
ethernet1.addressType = “generated”
uuid.location = “56 4d 88 e1 d8 25 00 e0-e3 ae f1 4f b1 65 76 26”
tools.remindInstall = “TRUE”
ethernet0.generatedAddress = “00:0c:29:65:76:26”
ethernet0.generatedAddressOffset = “0”
ethernet1.generatedAddress = “00:0c:29:65:76:30”
ethernet1.generatedAddressOffset = “10”

tools.syncTime = “FALSE”

scsi1:2.redo = “”
scsi1:3.redo = “”
scsi1:4.redo = “”
scsi1:5.redo = “”
scsi1:6.redo = “”

 

Open the virtual machines now in VmWare. You should see the screens like below.


Freshly opened VM’s in VMWare

All the hardware components in each of your virtual machines. Now both nodes are ready for RHEL 3.0 installations.

2. Installing RHEL 3.0 for Oracle on VMWare 5.0


  1. Introduction
  2. Install Linux
    1. Disk Configuration
    2. Network Configuration
    3. Select Packages to Install
  3. Disable all services that are not needed
  4. Configure Kernel Parameters
  5. /etc/hosts file configuration
  6. rsh configuration
  7. Create Oracle Groups/ User
  8. Create Mount Point for Oracle Software
  9. bash profile configuration
  10. Raw Devices Configuration

2.a – Introduction

For Oracle installations, you do not need X desktop running on the Linux Server at all times. Our objective is to allocate as much memory possible to Oracle. We will still need an X server installed as Oracle Universal Installer uses GUI to perform installations. For this we could simply use X11 forwarding feature in SSH and a proper windows X Client (Like Cygwin – You could download Cygwin from http://www.cygwin.com)

 

2.b – Install Linux

Download the ISO images for Linux. You could use these instead of CD ROM’s on your VMWare. If you are using the config files provided then you might get a message like this. You could just select to create a new UUID for the VM

 

 

Click Next

Accept defaults Click Next

Accept Defaults and Click Next

Next

2.b.i – Disk Configuration

Select Automatically Partition on this Screen and Select Next

This is an important Screen. We have 6 drives (sda,sdb,sdc,sdd,sde,sdf as the drives ) but you should use only the 1st drive for the operating system.

Select “YES” only for “sda”. You should select “NO” for the remaining drives

Select Remove all Partitions on the system and click Next

Select Yes here.

Your screen should look similar to the one above. Click Next

Accept Defaults. Click Next

 

 

2.b.ii – Network Configuration

Node Name Public Interface Private Interface
Node1 192.168.56.101 10.0.0.101
Node2 192.168.56.102 10.0.0.102

In this installation of VMWare my NAT 3rd octet is 56. It might be different for your installation. Check it by Going into Edit –> Virtual Network Preferences

Check both Network Interfaces to “Activate on Boot”. Click Edit for eth0. This is connected to the NAT interface. So I chose the IP adress 192.168.56.101 (Since this is Node1, For Node2 it should be 192.168.56.102)

 

For the Private interconnect, I chose 10.0.0.101 as the IP address for Node1 and 10.0.0.102 for Node2

Set the Host Name to Node1, If you are installing Node2 set it to Node2. Also set the Gateway and Primary DNS Servers.

 

I chose No Firewall.

Accept defaults and Click Next

I live in Newyork!!

Choose a root password and click Next

Select Customize packages and Click Next

2.b.iii – Select Packages to Install

  • Uncheck GNOME desktop Environment
  • Uncheck Web Server
  • Uncheck Windows File Server
  • Check Legacy Nework Server
    • Go into Details
    • Check Only rsh-server (Uncheck everything else
  • Check Development Tools
  • Check Kernel Development
  • Check Legacy Software Development

If you follow the instructions above the total size should be 1979 MB. Click Next

I skipped X configuration as i am not going to use the graphical desktop

Once Installation is done. the machine will be rebooted.

2.c – Disable all services that are not needed

Start the virtual machine you just created and logon as root. Execute the command ntsysv

I chose to uncheck everything except

  • crond
  • network
  • rawdevices
  • rsh
  • sshd
  • syslog
  • xinetd

Now reboot the machine. This time the boot time should be reduced considerably.

2.d – Configure Kernel Parameters

Logon to the machine as root and edit the file /etc/sysctl.conf

Add the following lines

kernel.shmall = 2097152
kernel.shmmax = 2147483648
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000

The resulting file should look something like this

Both nodes ( Location : /etc/sysctl.conf )
# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled.  See sysctl(8) and
# sysctl.conf(5) for more details.

# Controls IP packet forwarding
net.ipv4.ip_forward = 0

# Controls source route verification
net.ipv4.conf.default.rp_filter = 1

# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0

# Controls whether core dumps will append the PID to the core filename.
# Useful for debugging multi-threaded applications.
kernel.core_uses_pid = 1
kernel.shmall = 2097152
kernel.shmmax = 2147483648
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000

2.e – /etc/hosts File Configuration

Logon to the machine as root and edit the file /etc/hosts. The file should look like the one below. You should perform this operation on both nodes

Both nodes ( Location : /etc/hosts )
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1	localhost.localdomain	localhost

192.168.107.101 node1
192.168.107.102 node2
10.0.0.101 node1-prv
10.0.0.102 node2-prv
192.168.107.103 node1-vip
192.168.107.104 node2-vip

2.f – rsh Configuration

Oracle universal installer could use either rsh or ssh to copy the files (actually scp or rcp) across multiple nodes. We will use rsh instead of ssh just for simplicity sake. For rsh to work, you need to edit the file /etc/hosts.equiv . If this file does not exist just create it. You should perform this operation on both nodes.

Both nodes ( Location : /etc/hosts.equiv )
+node1
+node2

2.g – Create Oracle Groups/ User

Execute the following commands in that order to create oracle users and groups. You should perform this operation on both nodes.

groupadd oinstall
groupadd dba
useradd -g oinstall -G dba oracle
passwd oracle

The last command will set a password for Oracle.

2.h – Create Mount Point for Oracle Software

Execute the following commands to create an Oracle base for software installation. You should perform this operation on both nodes.

mkdir -p /u01/app/oracle

chown -R oracle.oinstall /u01

2.i – Bash Profile configuration

Logon to the machine as oracle and edit the file ~/.bash_profile. You should perform this operation on both nodes. The files should look like the ones below.On the second node make sure the ORACLE_SID is set to RACDB2

Node 1 ( Location : /home/oracle/.bash_profile )
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

# User specific environment and startup programs

export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=/u01/app/oracle/product/10.1.0/RAC
export ORA_CRS_HOME=/u01/app/oracle/product/10.1.0/CRS
export ORACLE_SID=RACDB1

PATH=$PATH:$HOME/bin:$ORACLE_HOME/bin:$ORA_CRS_HOME/bin

export PATH
unset USERNAME


 

Node 2 ( Location : /home/oracle/.bash_profile )
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

# User specific environment and startup programs

export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=/u01/app/oracle/product/10.1.0/RAC
export ORA_CRS_HOME=/u01/app/oracle/product/10.1.0/CRS
export ORACLE_SID=RACDB2

PATH=$PATH:$HOME/bin:$ORACLE_HOME/bin:$ORA_CRS_HOME/bin

export PATH
unset USERNAME


 

2.j – Raw Devices Configuration

Logon to the machine as root and execute the following commands . The file should look like the one below. You should perform this operation on both nodes

raw /dev/raw/raw1 /dev/sdb
raw /dev/raw/raw2 /dev/sdc
raw /dev/raw/raw3 /dev/sdd
raw /dev/raw/raw4 /dev/sde
raw /dev/raw/raw5 /dev/sdf
raw /dev/raw/raw6 /dev/sdg
mv /dev/raw/raw1 /dev/raw/votingdisk
mv /dev/raw/raw2 /dev/raw/ocr.dbf
mv /dev/raw/raw3 /dev/raw/spfile+ASM.ora
mv /dev/raw/raw4 /dev/raw/ASM1
mv /dev/raw/raw5 /dev/raw/ASM2
mv /dev/raw/raw6 /dev/raw/ASM3
chown oracle.dba /dev/raw/votingdisk
chown oracle.dba /dev/raw/ocr.dbf
chown oracle.dba /dev/raw/spfile+ASM.ora 
chown oracle.dba /dev/raw/ASM1 
chown oracle.dba /dev/raw/ASM2
chown oracle.dba /dev/raw/ASM3
echo "/dev/raw/ocr.dbf /dev/sdb" >> /etc/sysconfig/rawdevices
echo "/dev/raw/votingdisk /dev/sdc" >> /etc/sysconfig/rawdevices
echo "/dev/raw/spfile+ASM.ora /dev/sdd" >> /etc/sysconfig/rawdevices
echo "/dev/raw/ASM1 /dev/sde" >> /etc/sysconfig/rawdevices
echo "/dev/raw/ASM2 /dev/sdf" >> /etc/sysconfig/rawdevices
echo "/dev/raw/ASM3 /dev/sdg" >> /etc/sysconfig/rawdevices

 

Your /etc/sysconfig/rawdevices should look like this on both nodes

Both nodes ( Location : /etc/sysconfig/rawdevices )
# raw device bindings
# format:  <rawdev> <major> <minor>
#          <rawdev> <blockdev>
# example: /dev/raw/raw1 /dev/sda1
#          /dev/raw/raw2 8 5
/dev/raw/ocr.dbf /dev/sdb
/dev/raw/votingdisk /dev/sdc
/dev/raw/spfile+ASM.ora /dev/sdd
/dev/raw/ASM1 /dev/sde
/dev/raw/ASM2 /dev/sdf
/dev/raw/ASM3 /dev/sdg
          

 

 

 

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s