Learn how to set up and configure an Oracle rac 10

Download 0.57 Mb.
NameLearn how to set up and configure an Oracle rac 10
A typeDocumentation
manual-guide.com > manual > Documentation
1   2   3   4   5   6   7   8   9

Note that the virtual IP addresses only need to be defined in the /etc/hosts file for both nodes. The public virtual IP addresses will be configured automatically by Oracle when you run the Oracle Universal Installer, which starts Oracle's Virtual Internet Protocol Configuration Assistant (VIPCA). All virtual IP addresses will be activated when the srvctl start nodeapps -n command is run. This is the Host Name/IP Address that will be configured in the client(s) tnsnames.ora file (more details later).

In the screenshots below, only node 1 (linux1) is shown. Ensure to make all the proper network settings to both nodes.

Figure 2: Network Configuration Screen, Node 1 (linux1)

Figure 3: Ethernet Device Screen, eth0 (linux1)

Figure 4: Ethernet Device Screen, eth1 (linux1)

Figure 5: Network Configuration Screen, /etc/hosts (linux1)

When the network if configured, you can use the ifconfig command to verify everything is working. The following example is from linux1:

$ /sbin/ifconfig -a

eth0 Link encap:Ethernet HWaddr 00:0C:41:F1:6E:9A

inet addr: Bcast: Mask:


RX packets:421591 errors:0 dropped:0 overruns:0 frame:0

TX packets:403861 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:78398254 (74.7 Mb) TX bytes:51064273 (48.6 Mb)

Interrupt:9 Base address:0x400
eth1 Link encap:Ethernet HWaddr 00:0D:56:FC:39:EC

inet addr: Bcast: Mask:


RX packets:1715352 errors:0 dropped:1 overruns:0 frame:0

TX packets:4257279 errors:0 dropped:0 overruns:0 carrier:4

collisions:0 txqueuelen:1000

RX bytes:802574993 (765.3 Mb) TX bytes:1236087657 (1178.8 Mb)

lo Link encap:Local Loopback

inet addr: Mask:


RX packets:1273787 errors:0 dropped:0 overruns:0 frame:0

TX packets:1273787 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:246580081 (235.1 Mb) TX bytes:246580081 (235.1 Mb)

About Virtual IP

Why do we have a Virtual IP (VIP) in 10g? Why does it just return a dead connection when its primary node fails?

It's all about availability of the application. When a node fails, the VIP associated with it is supposed to be automatically failed over to some other node. When this occurs, two things happen.

  1. The new node re-arps the world indicating a new MAC address for the address. For directly connected clients, this usually causes them to see errors on their connections to the old address.

  2. Subsequent packets sent to the VIP go to the new node, which will send error RST packets back to the clients. This results in the clients getting errors immediately.

This means that when the client issues SQL to the node that is now down, or traverses the address list while connecting, rather than waiting on a very long TCP/IP time-out (~10 minutes), the client receives a TCP reset. In the case of SQL, this is ORA-3113. In the case of connect, the next address in tnsnames is used.

Without using VIPs, clients connected to a node that died will often wait a 10-minute TCP timeout period before getting an error. As a result, you don't really have a good HA solution without using VIPs (Source - Metalink Note 220970.1) .

Confirm the RAC Node Name is Not Listed in Loopback Address

Ensure that none of the node names (linux1 or linux2) are not included for the loopback address in the /etc/hosts file. If the machine name is listed in the in the loopback address entry as below: linux1 localhost.localdomain localhost

it will need to be removed as shown below: localhost.localdomain localhost

If the RAC node name is listed for the loopback address, you will receive the following error during the RAC installation:

ORA-00603: ORACLE server session terminated by fatal error


ORA-29702: error occurred in Cluster Group Service operation

Adjusting Network Settings

With Oracle and later, Oracle makes use of UDP as the default protocol on Linux for inter-process communication (IPC), such as Cache Fusion and Cluster Manager buffer transfers between instances within the RAC cluster.

Oracle strongly suggests to adjust the default and maximum send buffer size (SO_SNDBUF socket option) to 256KB, and the default and maximum receive buffer size (SO_RCVBUF socket option) to 256KB.

The receive buffers are used by TCP and UDP to hold received data until it is read by the application. The receive buffer cannot overflow because the peer is not allowed to send data beyond the buffer size window. This means that datagrams will be discarded if they don't fit in the socket receive buffer, potentially causing the sender to overwhelm the receiver.

The default and maximum window size can be changed in the /proc file system without reboot:

# su - root
# sysctl -w net.core.rmem_default=262144

net.core.rmem_default = 262144
# sysctl -w net.core.wmem_default=262144

net.core.wmem_default = 262144
# sysctl -w net.core.rmem_max=262144

net.core.rmem_max = 262144
# sysctl -w net.core.wmem_max=262144

net.core.wmem_max = 262144

The above commands made the changes to the already running OS. You should now make the above changes permanent (for each reboot) by adding the following lines to the /etc/sysctl.conf file for each node in your RAC cluster:

# Default setting in bytes of the socket receive buffer

# Default setting in bytes of the socket send buffer

# Maximum socket receive buffer size which may be set by using

# the SO_RCVBUF socket option

# Maximum socket send buffer size which may be set by using

# the SO_SNDBUF socket option


Enabling Telnet and FTP Services

Linux is configured to run the Telnet and FTP server, but by default, these services are disabled. To enable the telnet these service, login to the server as the root user account and run the following commands:

# chkconfig telnet on

# service xinetd reload

Reloading configuration: [ OK ]

Starting with the Red Hat Enterprise Linux 3.0 release (and in WBEL), the FTP server (wu-ftpd) is no longer available with xinetd. It has been replaced with vsftp and can be started from /etc/init.d/vsftpd as in the following:

# /etc/init.d/vsftpd start

Starting vsftpd for vsftpd: [ OK ]

If you want the vsftpd service to start and stop when recycling (rebooting) the machine, you can create the following symbolic links:

# ln -s /etc/init.d/vsftpd /etc/rc3.d/S56vsftpd

# ln -s /etc/init.d/vsftpd /etc/rc4.d/S56vsftpd

# ln -s /etc/init.d/vsftpd /etc/rc5.d/S56vsftpd

Allowing Root Logins to Telnet and FTP Services

Before getting into the details of how to configure Red Hat Linux for root logins, keep in mind that this is very poor security. Never configure your production servers for this type of login.

To configure Telnet for root logins, simply edit the file /etc/securetty and add the following to the end of the file:











This will allow up to 10 telnet sessions to the server as root. To configure FTP for root logins, edit the files /etc/vsftpd.ftpusers and /etc/vsftpd.user_list and remove the 'root' line from each file.
8. Obtain and Install a Proper Linux Kernel

Perform the following kernel upgrade on all nodes in the cluster!

The next step is to obtain and install a new Linux kernel that supports the use of IEEE1394 devices with multiple logins. In a previous version of this article, I included the steps to download a patched version of the Linux kernel (source code) and then compile it. Thanks to Oracle's Linux Projects development group, this is no longer a requirement. They provide a pre-compiled kernel for RHEL3 (which also works with WBEL!), that can simply be downloaded and installed. The instructions for downloading and installing the kernel are included in this section. Before going into the details of how to perform these actions, however, lets take a moment to discuss the changes that are required in the new kernel.

While FireWire drivers already exist for Linux, they often do not support shared storage. Typically when you logon to an OS, the OS associates the driver to a specific drive for that machine alone. This implementation simply will not work for our RAC configuration. The shared storage (our FireWire hard drive) needs to be accessed by more than one node. We need to enable the FireWire driver to provide nonexclusive access to the drive so that multiple servers—the nodes that comprise the cluster—will be able to access the same storage. This goal is accomplished by removing the bit mask that identifies the machine during login in the source code, resulting in nonexclusive access to the FireWire hard drive. All other nodes in the cluster login to the same drive during their logon session, using the same modified driver, so they too also have nonexclusive access to the drive.

Our implementation describes a dual node cluster (each with a single processor), each server running WBEL. Keep in mind that the process of installing the patched Linux kernel will need to be performed on both Linux nodes. White Box Enterprise Linux 3.0 (Respin 1) includes kernel 2.4.21-15.EL #1; we will need to download the version hosted at http://oss.oracle.com/projects/firewire/files, 2.4.21-27.0.2.ELorafw1.

Download one of the following files:

kernel-2.4.21-27.0.2.ELorafw1.i686.rpm - (for single processor)


kernel-smp-2.4.21-27.0.2.ELorafw1.i686.rpm - (for multiple processors)

Make a backup of your GRUB configuration file:

In most cases you will be using GRUB for the boot loader. Before actually installing the new kernel, backup a copy of your /etc/grub.conf file:

# cp /etc/grub.conf /etc/grub.conf.original

Install the new kernel, as root:

# rpm -ivh --force kernel-2.4.21-27.0.2.ELorafw1.i686.rpm - (for single processor)


# rpm -ivh --force kernel-smp-2.4.21-27.0.2.ELorafw1.i686.rpm - (for multiple processors)

Note: Installing the new kernel using RPM will also update your GRUB (or lilo) configuration with the appropiate stanza. There is no need to add any new stanza to your boot loader configuration unless you want to have your old kernel image available.

The following is a listing of my /etc/grub.conf file before and then after the kernel install. As you can see, my install put in another stanza for the 2.4.21-27.0.2.ELorafw1 kernel. If you want, you can chance the entry (default) in the new file so that the new kernel will be the default one booted. By default, the installer keeps the default kernel (your original one) by setting it to default=1. You should change the default value to zero (default=0) in order to enable the new kernel to boot by default.

Original File

# grub.conf generated by anaconda


# Note that you do not have to rerun grub after making changes to this file

# NOTICE: You have a /boot partition. This means that

# all kernel and initrd paths are relative to /boot/, eg.

# root (hd0,0)

# kernel /vmlinuz-version ro root=/dev/hda2

# initrd /initrd-version.img





title White Box Enterprise Linux (2.4.21-15.EL)

root (hd0,0)

kernel /vmlinuz-2.4.21-15.EL ro root=LABEL=/

initrd /initrd-2.4.21-15.EL.img

Newly Configured File After Kernel Install

# grub.conf generated by anaconda


# Note that you do not have to rerun grub after making changes to this file

# NOTICE: You have a /boot partition. This means that

# all kernel and initrd paths are relative to /boot/, eg.

# root (hd0,0)

# kernel /vmlinuz-version ro root=/dev/hda2

# initrd /initrd-version.img





title White Box Enterprise Linux (2.4.21-27.0.2.ELorafw1)

root (hd0,0)

kernel /vmlinuz-2.4.21-27.0.2.ELorafw1 ro root=LABEL=/

initrd /initrd-2.4.21-27.0.2.ELorafw1.img

title White Box Enterprise Linux (2.4.21-15.EL)

root (hd0,0)

kernel /vmlinuz-2.4.21-15.EL ro root=LABEL=/

initrd /initrd-2.4.21-15.EL.img

Add module options:

Add the following lines to /etc/modules.conf:

alias ieee1394-controller ohci1394

options sbp2 sbp2_exclusive_login=0

post-install sbp2 insmod sd_mod

post-install sbp2 insmod ohci1394

post-remove sbp2 rmmod sd_mod

It is vital that the parameter sbp2_exclusive_login of the Serial Bus Protocol module (sbp2) be set to zero to allow multiple hosts to login to and access the FireWire disk concurrently. The second line ensures the SCSI disk driver module (sd_mod) is loaded as well since (sbp2) requires the SCSI layer. The core SCSI support module (scsi_mod) will be loaded automatically if (sd_mod) is loaded; no need to make a separate entry for it.

Connect FireWire drive to each machine and boot into the new kernel:

After performing the above tasks on both nodes in the cluster, power down both Linux machines:

# hostname

# init 0
# hostname

# init 0


After both machines are powered down, connect each of them to the back of the FireWire drive. Power on the FireWire drive. Finally, power on each Linux server and ensure to boot each machine into the new kernel.

Loading the FireWire stack:

In most cases, the loading of the FireWire stack will already be configured in the /etc/rc.sysinit file. The commands that are contained within this file that are responsible for loading the FireWire stack are:

# modprobe sbp2

# modprobe ohci1394

In older versions of Red Hat, this was not the case and these commands would have to be manually run or put within a startup file. With Red Hat Enterprise Linux 3 and later, these commands are already put within the /etc/rc.sysinit file and run on each boot.

Check for SCSI Device:

After each machine has rebooted, the kernel should automatically detect the disk as a SCSI device (/dev/sdXX). This section will provide several commands that should be run on all nodes in the cluster to verify the FireWire drive was successfully detected and being shared by all nodes in the cluster.

For this configuration, I was performing the above procedures on both nodes at the same time. When complete, I shutdown both machines, started linux1 first, and then linux2. The following commands and results are from my linux2 machine. Again, make sure that you run the following commands on all nodes to ensure both machine can login to the shared drive.

Let's first check to see that the FireWire adapter was successfully detected:

# lspci

00:00.0 Host bridge: Intel Corp. 82845G/GL[Brookdale-G]/GE/PE DRAM Controller/Host-Hub Interface (rev 01)

00:02.0 VGA compatible controller: Intel Corp. 82845G/GL[Brookdale-G]/GE Chipset Integrated Graphics Device (rev 01)

00:1d.0 USB Controller: Intel Corp. 82801DB (ICH4) USB UHCI #1 (rev 01)

00:1d.1 USB Controller: Intel Corp. 82801DB (ICH4) USB UHCI #2 (rev 01)

00:1d.2 USB Controller: Intel Corp. 82801DB (ICH4) USB UHCI #3 (rev 01)

00:1d.7 USB Controller: Intel Corp. 82801DB (ICH4) USB2 EHCI Controller (rev 01)

00:1e.0 PCI bridge: Intel Corp. 82801BA/CA/DB/EB/ER Hub interface to PCI Bridge (rev 81)

00:1f.0 ISA bridge: Intel Corp. 82801DB (ICH4) LPC Bridge (rev 01)

00:1f.1 IDE interface: Intel Corp. 82801DB (ICH4) Ultra ATA 100 Storage Controller (rev 01)

00:1f.3 SMBus: Intel Corp. 82801DB/DBM (ICH4) SMBus Controller (rev 01)

00:1f.5 Multimedia audio controller: Intel Corp. 82801DB (ICH4) AC'97 Audio Controller (rev 01)

01:04.0 FireWire (IEEE 1394): Texas Instruments TSB43AB23 IEEE-1394a-2000 Controller (PHY/Link)

01:05.0 Modem: Intel Corp.: Unknown device 1080 (rev 04)

01:06.0 Ethernet controller: Linksys NC100 Network Everywhere Fast Ethernet 10/100 (rev 11)

01:09.0 Ethernet controller: Broadcom Corporation BCM4401 100Base-T (rev 01)

Second, let's check to see that the modules are loaded:

# lsmod |egrep "ohci1394|sbp2|ieee1394|sd_mod|scsi_mod"

sd_mod 13744 0

sbp2 19724 0

scsi_mod 106664 3 [sg sd_mod sbp2]

ohci1394 28008 0 (unused)

ieee1394 62884 0 [sbp2 ohci1394]

Third, let's make sure the disk was detected and an entry was made by the kernel:

# cat /proc/scsi/scsi

Attached devices:

Host: scsi0 Channel: 00 Id: 00 Lun: 00

Vendor: Maxtor Model: OneTouch Rev: 0200

Type: Direct-Access ANSI SCSI revision: 06

Now let's verify that the FireWire drive is accessible for multiple logins and shows a valid login:

# dmesg | grep sbp2

ieee1394: sbp2: Query logins to SBP-2 device successful

ieee1394: sbp2: Maximum concurrent logins supported: 3

ieee1394: sbp2: Number of active logins: 1

ieee1394: sbp2: Logged into SBP-2 device

ieee1394: sbp2: Node[01:1023]: Max speed [S400] - Max payload [2048]

From the above output, you can see that the FireWire drive I have can support concurrent logins by up to three servers. It is vital that you have a drive where the chipset supports concurrent access for all nodes within the RAC cluster.

One other test I like to perform is to run a quick fdisk -l from each node in the cluster to verify that it is really being picked up by the OS. It will show that the device does not contain a valid partition table, but this is OK at this point of the RAC configuration.

# fdisk -l

Disk /dev/sda: 203.9 GB, 203927060480 bytes

255 heads, 63 sectors/track, 24792 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sda doesn't contain a valid partition table
Disk /dev/hda: 40.0 GB, 40000000000 bytes

255 heads, 63 sectors/track, 4863 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System

/dev/hda1 * 1 13 104391 83 Linux

/dev/hda2 14 4609 36917370 83 Linux

/dev/hda3 4610 4863 2040255 82 Linux swap

Rescan SCSI bus no longer required:

In older versions of the kernel, I would need to run the rescan-scsi-bus.sh script in order to detect the FireWire drive. The purpose of this script was to create the SCSI entry for the node by using the following command:

echo "scsi add-single-device 0 0 0 0" > /proc/scsi/scsi

With Red Hat Enterprise Linux 3, this step is no longer required and the disk should be detected automatically.

Troubleshooting SCSI Device Detection:

If you are having troubles with any of the procedures (above) in detecting the SCSI device, you can try the following:

# modprobe -r sbp2

# modprobe -r sd_mod

# modprobe -r ohci1394

# modprobe ohci1394

# modprobe sd_mod

# modprobe sbp2

You may also want to unplug any USB devices connected to the server. The system may not be able to recognize your FireWire drive if you have a USB device attached!
9. Create "oracle" User and Directories (both nodes)

Perform the following procedure on all nodes in the cluster!

I will be using the Oracle Cluster File System (OCFS) to store the files required to be shared for the Oracle Cluster Ready Services (CRS). When using OCFS, the UID of the UNIX user oracle and GID of the UNIX group dba must be identical on all machines in the cluster. If either the UID or GID are different, the files on the OCFS file system will show up as "unowned" or may even be owned by a different user. For this article, I will use 175 for the oracle UID and 115 for the dba GID.

Create Group and User for Oracle

Let's continue our example by creating the Unix dba group and oracle user account along with all appropriate directories.

# mkdir -p /u01/app

# groupadd -g 115 dba

# useradd -u 175 -g 115 -d /u01/app/oracle -s /bin/bash -c "Oracle Software Owner" -p oracle oracle

# chown -R oracle:dba /u01

# passwd oracle

# su - oracle

Note: When you are setting the Oracle environment variables for each RAC node, ensure to assign each RAC node a unique Oracle SID! For this example, I used:

  • linux1 : ORACLE_SID=orcl1

  • linux2 : ORACLE_SID=orcl2

After creating the "oracle" UNIX userid on both nodes, ensure that the environment is setup correctly by using the following .bash_profile:


# .bash_profile
# Get the aliases and functions

if [ -f ~/.bashrc ]; then

. ~/.bashrc

alias ls="ls -FA"
# User specific environment and startup programs

export ORACLE_BASE=/u01/app/oracle

export ORACLE_HOME=$ORACLE_BASE/product/10.1.0/db_1

export ORA_CRS_HOME=$ORACLE_BASE/product/10.1.0/crs_1
# Each RAC node must have a unique ORACLE_SID. (i.e. orcl1, orcl2,...)

export ORACLE_SID=orcl1
export PATH=.:${PATH}:$HOME/bin:$ORACLE_HOME/bin

export PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin

export ORACLE_TERM=xterm

export TNS_ADMIN=$ORACLE_HOME/network/admin

export ORA_NLS33=$ORACLE_HOME/ocommon/nls/admin/data



export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib





export THREADS_FLAG=native

export TEMP=/tmp

export TMPDIR=/tmp

export LD_ASSUME_KERNEL=2.4.1

Now, let's create the mount point for the Oracle Cluster File System (OCFS) that will be used to store files for the Oracle Cluster Ready Service (CRS). These commands will need to be run as the "root" user account:

$ su -

# mkdir -p /u02/oradata/orcl

# chown -R oracle:dba /u02

Note: The Oracle Universal Installer (OUI) requires at most 400MB of free space in the /tmp directory.

You can check the available space in /tmp by running the following command:

# df -k /tmp

Filesystem 1K-blocks Used Available Use% Mounted on

/dev/hda2 36337384 4691460 29800056 14% /

If for some reason you do not have enough space in /tmp, you can temporarily create space in another file system and point your TEMP and TMPDIR to it for the duration of the install. Here are the steps to do this:

# su -

# mkdir //tmp

# chown root.root //tmp

# chmod 1777 //tmp

# export TEMP=//tmp # used by Oracle

# export TMPDIR=//tmp # used by Linux programs

# like the linker "ld"

When the installation of Oracle is complete, you can remove the temporary directory using the following:

# su -

# rmdir //tmp

# unset TEMP

# unset TMPDIR
10. Creating Partitions on the Shared FireWire Storage Device

Create the following partitions on only one node in the cluster!

The next step is to create the required partitions on the FireWire (shared) drive. As I mentioned previously, we will use OCFS to store the two files to be shared for CRS. We will then use ASM for all physical database files (data/index files, online redo log files, control files, SPFILE, and archived redo log files).

The following table lists the individual partitions that will be created on the FireWire (shared) drive and what files will be contained on them.

Oracle Shared Drive Configuration

File System Type



Mount Point

File Types





Oracle Cluster Registry File - (~100MB)
CRS Voting Disk - (~20MB)





Oracle Database Files





Oracle Database Files





Oracle Database Files






1   2   3   4   5   6   7   8   9


Learn how to set up and configure an Oracle rac 10 iconSet call options When you first sign in to Lync, you’ll be guided...

Learn how to set up and configure an Oracle rac 10 iconObjectives In this episode you will: Learn about adverbs of frequency....

Learn how to set up and configure an Oracle rac 10 iconPartners in 17 Categories Recognized at Oracle OpenWorld for Excellence...

Learn how to set up and configure an Oracle rac 10 iconFull life cycle development, implementation and support with expertise...

Learn how to set up and configure an Oracle rac 10 iconIdc white Paper sponsored by Oracle Corporation, "Oracle e-business...

Learn how to set up and configure an Oracle rac 10 iconOracle® e-business Suite, Siebel crm, PeopleSoft Enterprise and jd...

Learn how to set up and configure an Oracle rac 10 iconPhp version: 17 set foreign key checks=0; set sql mode="no auto value...

Learn how to set up and configure an Oracle rac 10 iconThe student will learn about reading instruction from birth to secondary...

Learn how to set up and configure an Oracle rac 10 iconDG102sh console Command Set(set coding, set h323, and show)

Learn how to set up and configure an Oracle rac 10 iconResearch into an Intermediate Accounting level practice set was conducted....

Learn how to set up and configure an Oracle rac 10 iconAbstract This integration is a part of Siebel Integrations between...

Learn how to set up and configure an Oracle rac 10 iconTo set up the fw-1884 in Cubase or Nuendo 3 To set up the fw-1884 in Mackie Emulation protocol 3

Learn how to set up and configure an Oracle rac 10 iconDownload and installation: Configure the initial scan

Learn how to set up and configure an Oracle rac 10 iconHow to configure timeGuardian/QuickBooks for direct integration

Learn how to set up and configure an Oracle rac 10 iconConfigure the cics environment for the Debug Tool

Learn how to set up and configure an Oracle rac 10 iconPart 2: Configure ssh access to the Switches

Learn how to set up and configure an Oracle rac 10 iconOracle® SuperCluster T5-8

Learn how to set up and configure an Oracle rac 10 iconOracle® SuperCluster M6-32

Learn how to set up and configure an Oracle rac 10 iconMacBook-Pro-de-Martin-Ortuno: libusb 9 martinortuno$./configure

Learn how to set up and configure an Oracle rac 10 iconLearning to drive -how do you want to learn?


When copying material provide a link © 2017