{"id":4436,"date":"2022-09-02T07:00:00","date_gmt":"2022-09-02T04:00:00","guid":{"rendered":"https:\/\/www.dataplatform.gr\/?p=4436"},"modified":"2023-11-05T21:37:38","modified_gmt":"2023-11-05T18:37:38","slug":"pos-kanoyme-egkatastasi-ena-oracle-real-application-cluster-rac-se-l","status":"publish","type":"post","link":"https:\/\/www.dataplatform.gr\/en\/pos-kanoyme-egkatastasi-ena-oracle-real-application-cluster-rac-se-l\/","title":{"rendered":"How to install an Oracle Real Application Cluster (RAC) on Linux using Direct NFS"},"content":{"rendered":"

In this article we will see step by step how to install an Oracle Database that belongs to Real Application Cluster (RAC)<\/strong> on a Linux operating system (Oracle Linux 7.7) with shared NFS disks without using Oracle ASM (Automatic Storage Management). Connectivity between Oracle Database and NFS Server is done directly through its technology Oracle Direct NFS<\/strong>, improving the performance at the I\/O level as well as the other resources (CPU, RAM).<\/p>\n\n\n\n

In the example we will see we will create a 2-node Oracle RAC that contains two Database Servers with Oracle Linux 7.7 operating system and an NFS Server with Windows Server 2019. We will install the Oracle Grid Infrastructure and Oracle Database software on a shared NFS share although we will be able, due to redundancy, to do them separately, either locally or in NFS Shares.<\/p>\n\n\n\n

What is Oracle Real Application Cluster (RAC)<\/h2>\n\n\n\n

The Oracle RAC<\/strong> is a software that allows us to connect multiple servers (up to 100) in a cluster for Oracle Database <\/strong>which appears as a server to the end user or application. It provides us with the combined power of all servers and we have the ability to Read\/Write from any server-node belonging to this cluster. So it is a very good solution for High Availability and Load Balancing.<\/p>\n\n\n\n

Detailed information about Oracle RAC and how it works is available in the article here<\/a>.<\/p>\n\n\n

\n
\"\"
docs.oracle.com<\/figcaption><\/figure>\n<\/div>\n\n\n

The prerequisites<\/h2>\n\n\n\n

First, let's download the software for the Oracle Database Grid Infrastructure<\/strong> (Oracle Database Grid Infrastructure 12.1.0.2.0 for Linux x86-64) and the software for Oracle Database<\/strong> (Oracle Database 12.1.0.2.0 for Linux x86-64) from the link here<\/a>.<\/p>\n\n\n\n

We need an NFS Server with various NFS shares. At least one for the Grid Infrastructure software and for Oracle Database which for example we will say \u201c\/oracle\/nfs<\/strong>\u201d (unless we install the software locally on both servers). An NFS share for the database files\/oracle\/oradata<\/strong>\u201d and one for fast_recovery_area (FRA) \u201c\/oracle\/fra<\/strong>". How to set up a Windows Server 2019 as an NFS Share and how to mount disks on Linux is described in detail in the article here<\/a>.<\/p>\n\n\n\n

 We should install the required libraries in our operating system. To do this we run the following on each Oracle Linux Server:<\/p>\n\n\n\n

yum install oracle-rdbms-server-12cR1-preinstall -y<\/pre>\n\n\n\n
\"\"
01<\/figcaption><\/figure>\n\n\n\n
If we didn't already have a user on the operating system oracle<\/em><\/strong> we should set the password for it:<\/p>\n\n\n\n
passwd oracle<\/pre>\n\n\n\n
At the operating system level we have defined the names oracledev1 <\/code>and oracledev2 <\/code>as hostname:<\/p>\n\n\n\n
su root\n\nvi \/etc\/hostname<\/pre>\n\n\n\n
oracledev1<\/code><\/pre>\n\n\n\n
Respectively:<\/p>\n\n\n\n
oracledev2<\/code><\/pre>\n\n\n\n
Then disable the firewall on both machines:<\/p>\n\n\n\n
systemctl stop firewalld\nsystemctl disable firewalld<\/pre>\n\n\n\n
We also change the entry for SELINUX to permissive in the following file:<\/p>\n\n\n\n
vi \/etc\/selinux\/config<\/pre>\n\n\n\n
SELINUX=permissive<\/code><\/pre>\n\n\n\n
For the network we have two network adapters one with public ip and one with private ip.<\/p>\n\n\n\n
In the oracledev1 <\/code>we put public ip 172.25.94.41 <\/em>and private 192.168.10.101<\/em>:<\/p>\n\n\n\n
\"\"
02<\/figcaption><\/figure>\n\n\n\n
In the oracledev2 <\/code>we put public ip 172.25.94.42 <\/em>and private 192.168.10.102<\/em>:<\/p>\n\n\n\n
\"\"
03<\/figcaption><\/figure>\n\n\n\n
Then we add to each Linux in the file \/etc\/hosts <\/code>the IP of the NFS Server (win-nfs), the public and private IPs. We also define a VIP (Virtual Internet Protocol) IP for everyone who belongs to the same subnet as the public IP:<\/p>\n\n\n\n
vi \/etc\/hosts<\/pre>\n\n\n\n
#public\n172.25.94.41   oracledev1\n172.25.94.42   oracledev2\n#private\n192.168.10.101   oracledev1-priv\n192.168.10.102   oracledev2-priv\n#vip\n172.25.94.43   oracledev1-vip\n172.25.94.44   oracledev2-vip\n#nfs\n172.25.94.95    win-nfs\n<\/code><\/pre>\n\n\n\n
Then we should define 3 different IPs with one name in the DNS Server in the same subnet as the public IP SCAN <\/strong>(Single Client Access Name) to round-robin<\/strong> for the SCAN listener:<\/p>\n\n\n\n
\n172.25.94.45   oracledev-scan\n172.25.94.46   oracledev-scan\n172.25.94.47   oracledev-scan<\/code><\/pre>\n\n\n\n
Since in the example I am not using a DNS Server, I will add the first of the 3 SCAN IPs and this one to \/etc\/hosts<\/code>:<\/p>\n\n\n\n
\"\"
04<\/figcaption><\/figure>\n\n\n\n
After we have created the NFS Shares on the NFS Server to mount them, we go to each Oracle Linux server and do the following actions.<\/p>\n\n\n\n
We create the folders with the oracle user as owner and the following permissions:<\/p>\n\n\n\n
su root\nmkdir -p \/oracle\/nfs\nmkdir -p \/oracle\/oradata\nmkdir -p \/oracle\/fra\nchown -R oracle:oinstall \/oracle\/nfs\nchown -R oracle:oinstall \/oracle\/oradata\nchown -R oracle:oinstall \/oracle\/fra\nchmod -R 775 \/oracle\/nfs\nchmod -R 775 \/oracle\/oradata\nchmod -R 775 \/oracle\/fra<\/pre>\n\n\n\n
To be sure that the rights to the NFS Server will not be cut, we go to the exports file and add a record for each NFS Share as below:<\/p>\n\n\n\n
vi \/etc\/exports<\/pre>\n\n\n\n
\/oracle\/nfs         *(rw,sync,no_wdelay,insecure_locks,no_root_squash)\n\/oracle\/oradata         *(rw,sync,no_wdelay,insecure_locks,no_root_squash)\n\/oracle\/fra         *(rw,sync,no_wdelay,insecure_locks,no_root_squash)<\/code><\/pre>\n\n\n\n
Then we restart the NFS service:<\/p>\n\n\n\n
chkconfig nfs on\nservice nfs restart<\/code><\/pre>\n\n\n\n
We add to the file \/etc\/fstab<\/code> one entry for each NFS Share so that they mount themselves when each machine starts:<\/p>\n\n\n\n
vi \/etc\/fstab<\/pre>\n\n\n\n
win-nfs:\/nfs \/oracle\/nfs  nfs rw,bg,hard,nointr,tcp,vers=3,timeo=600,rsize=32768,wsize=32768,actimeo=0  0 0\nwin-nfs:\/oradata \/oracle\/oradata  nfs rw,bg,hard,nointr,tcp,vers=3,timeo=600,rsize=32768,wsize=32768,actimeo=0  0 0\nwin-nfs:\/fra \/oracle\/fra  nfs rw,bg,hard,nointr,tcp,vers=3,timeo=600,rsize=32768,wsize=32768,actimeo=0  0 0<\/code><\/pre>\n\n\n\n
\"\"
05<\/figcaption><\/figure>\n\n\n\n
Now we are ready to mount them:<\/p>\n\n\n\n
mount \/oracle\/nfs\nmount \/oracle\/oradata\nmount \/oracle\/fra\n\ndf<\/pre>\n\n\n\n
\"\"
06<\/figcaption><\/figure>\n\n\n\n
After the NFS Shares are mounted, create the following folder locally on all servers:<\/p>\n\n\n\n
mkdir -p \/oracle\/oraInventory\nchown -R oracle:oinstall \/oracle\/oraInventory\nchmod -R 775 \/oracle\/oraInventory<\/pre>\n\n\n\n
And the following once from one of the two servers, as they are shared:<\/p>\n\n\n\n
mkdir -p \/oracle\/nfs\/app\/grid\nmkdir -p \/oracle\/nfs\/app\/oracle\/product\/12.1.0.2\/dbhome_1\nmkdir -p \/oracle\/nfs\/crs_config\n\nchown -R oracle:oinstall \/oracle\/nfs\nchmod -R 775 \/oracle\/nfs<\/code><\/pre>\n\n\n\n
At this point we should fix it bash_profile <\/code>with the parameters for each user's environment.<\/p>\n\n\n\n
To oracledev1 <\/code>we change it bash_profile <\/code>to the oracle user:<\/p>\n\n\n\n
su oracle\ncd $HOME\nvi .bash_profile<\/pre>\n\n\n\n
export ORACLE_BASE=\/oracle\/nfs\/app\/oracle\nexport DB_HOME=\/oracle\/nfs\/app\/oracle\/product\/12.1.0.2\/dbhome_1\nexport GRID_HOME=\/oracle\/nfs\/app\/12.1.0\/grid\nexport ORACLE_HOME=$DB_HOME\nexport ORACLE_SID=racdb1\nexport ORACLE_UNQNAME=racdevdb\nexport NLS_LANG=AMERICAN_AMERICA.AL32UTF8\nexport TNS_ADMIN=$ORACLE_HOME\/network\/admin\nexport BASE_PATH=\/usr\/sbin:$PATH\nexport PATH=$ORACLE_HOME\/bin:$BASE_PATH,\nexport LD_LIBRARY_PATH=$ORACLE_HOME\/lib:\/lib:\/usr\/lib\nexport CLASSPATH=$ORACLE_HOME\/jlib:$ORACLE_HOME\/rdbms\/jlib<\/code><\/pre>\n\n\n\n
\"\"
07<\/figcaption><\/figure>\n\n\n\n
To oracledev1 <\/code>we change it bash_profile <\/code>to the root user:<\/p>\n\n\n\n
su root\ncd $HOME\nvi .bash_profile<\/pre>\n\n\n\n
export ORACLE_BASE=\/oracle\/nfs\/app\/oracle\nexport DB_HOME=\/oracle\/nfs\/app\/oracle\/product\/12.1.0.2\/dbhome_1\nexport GRID_HOME=\/oracle\/nfs\/app\/12.1.0\/grid\nexport ORACLE_HOME=$GRID_HOME\nexport ORACLE_SID=racdb1\nexport ORACLE_UNQNAME=racdevdb\nexport NLS_LANG=AMERICAN_AMERICA.AL32UTF8\nexport TNS_ADMIN=$ORACLE_HOME\/network\/admin\nexport BASE_PATH=\/usr\/sbin:$PATH\nexport PATH=$ORACLE_HOME\/bin:$BASE_PATH,\nexport LD_LIBRARY_PATH=$ORACLE_HOME\/lib:\/lib:\/usr\/lib\nexport CLASSPATH=$ORACLE_HOME\/jlib:$ORACLE_HOME\/rdbms\/jlib\n<\/code><\/pre>\n\n\n\n
We repeat the same process for oracledev2 in both root user and oracle only changing the value to ORACLE_SID <\/code>in racdb2:<\/p>\n\n\n\n
export ORACLE_SID=racdb2<\/code><\/pre>\n\n\n\n
In order to load the changes, we either have to login \/ logout, or we run the following command with the corresponding user we are each time:<\/p>\n\n\n\n
source ~\/.bash_profile<\/code><\/pre>\n\n\n\n
The installation<\/h2>\n\n\n\n
For Oracle Grid Infrastructure<\/h4>\n\n\n\n
To install Oracle Grid Infrastructure, connect to the oracle user (unless we want to create a user with the name grid and define it as an administrator) and go to the folder where we downloaded it and unzip the software:<\/p>\n\n\n\n
su oracle\nsource ~\/.bash_profile\ncd \/home\/oracle\/Downloads\/grid\n\nchmod -R 775 \/home\/oracle\/Downloads\/grid<\/pre>\n\n\n\n
Before running the setup we define as ORACLE_HOME <\/code>the GRID_HOME <\/code>which is the path we want the installation to be:<\/p>\n\n\n\n
export ORACLE_HOME=$GRID_HOME<\/pre>\n\n\n\n
Then we run the setup:<\/p>\n\n\n\n
.\/runInstaller<\/pre>\n\n\n\n
On the first screen we choose Install and Configure Oracle Grid Infrastructure for a Cluster<\/em>:<\/p>\n\n\n\n
\"\"
08<\/figcaption><\/figure>\n\n\n\n
Then as cluster type, Configure a<\/em> Standard clusters:<\/p>\n\n\n\n
\"\"
09<\/figcaption><\/figure>\n\n\n\n
In the next tab we select Advanced Installation<\/em>:<\/p>\n\n\n\n
\"\"
10<\/figcaption><\/figure>\n\n\n\n
Next we define a name for the cluster, as SCAN Name<\/em> the name we had defined at the beginning as SCAN <\/strong>and the door the listener will use. Finally we make the choice Configure GNS<\/em>:<\/p>\n\n\n\n
\"\"
11<\/figcaption><\/figure>\n\n\n\n
On the next screen we choose Add\u2026<\/em> and we add the servers with the public name and the VIP name we defined in \/etc\/hosts<\/code>:<\/p>\n\n\n\n
\"\"
12<\/figcaption><\/figure>\n\n\n\n
On the same screen we choose SSH connectivity<\/em>, put the oracle user code and select Setup<\/em>:<\/p>\n\n\n\n
\"\"
13<\/figcaption><\/figure>\n\n\n\n
In the next tab we define which subnet will be public and which will be private as we defined it in \/etc\/hosts<\/code>:<\/p>\n\n\n\n
\"\"
14<\/figcaption><\/figure>\n\n\n\n
In the tab Storage Option<\/em>, we choose Use Share File System<\/em> as we don't want to use ASM:<\/p>\n\n\n\n
\"\"
15<\/figcaption><\/figure>\n\n\n\n
On the next screen for the OCR Storage <\/em>we choose External Redundancy<\/em> and set the NFS Share path for the file:<\/p>\n\n\n\n
\"\"
16<\/figcaption><\/figure>\n\n\n\n
On the next screen we do the same again, we choose External Redundancy<\/em> and set the NFS Share path for the file:<\/p>\n\n\n\n
\"\"
17<\/figcaption><\/figure>\n\n\n\n
Then we define the ORACLE_BASE <\/code>and GRID_HOME <\/code>as we added them to bash_profile<\/code>, in this case we chose to be Shared NFS for both servers:<\/p>\n\n\n\n
\"\"
18<\/figcaption><\/figure>\n\n\n\n
We don't choose to run the root script automatically:<\/p>\n\n\n\n
\"\"
19<\/figcaption><\/figure>\n\n\n\n
Before the installation starts it will check what needs fixing by selecting Fix & Check Again<\/em> Will try to fix some if possible. In our case it didn't show anything serious other than Warnings so we choose Ignore All <\/em>and Next<\/em>:<\/p>\n\n\n\n
\"\"
20<\/figcaption><\/figure>\n\n\n\n
Then the installation starts:<\/p>\n\n\n\n
\"\"
21<\/figcaption><\/figure>\n\n\n\n
Just before the installation is complete, it asks us to connect as root to each server and run the following scripts, starting first from the server where we started the installation:<\/p>\n\n\n\n
\"\"
22<\/figcaption><\/figure>\n\n\n\n
Running the root.sh script on oracledev1, if everything goes well we will see the following messages:<\/p>\n\n\n\n
\"\"
23<\/figcaption><\/figure>\n\n\n\n
Oracle Grid installation is finally complete:<\/p>\n\n\n\n
\"\"
24<\/figcaption><\/figure>\n\n\n\n
<\/p>\n\n\n\n
For Oracle Database software<\/h4>\n\n\n\n
After the installation of the Grid is completed, we are going to proceed with the installation of the Oracle Database Software. Go to the folder where we downloaded it and unzip the software:<\/p>\n\n\n\n
cd \/home\/oracle\/Downloads\/database\n\nchmod -R 775 \/home\/oracle\/Downloads\/database<\/pre>\n\n\n\n
And we run the setup:<\/p>\n\n\n\n
.\/runInstaller<\/pre>\n\n\n\n
As soon as the setup opens on the first tab, select the option for security updates:<\/p>\n\n\n\n
\"\"
25<\/figcaption><\/figure>\n\n\n\n
We choose to only install the software (we will create the database later):<\/p>\n\n\n\n
\"\"
26<\/figcaption><\/figure>\n\n\n\n
On the next screen we choose Oracle Real Application Clusters database installation<\/em>:<\/p>\n\n\n\n
\"\"
27<\/figcaption><\/figure>\n\n\n\n
We select all the servers we want to participate in the cluster:<\/p>\n\n\n\n
\"\"
28<\/figcaption><\/figure>\n\n\n\n
Then we define the ORACLE_BASE<\/code> <\/em>and DB_HOME <\/code>as we added them to .bash_profile<\/code>, in this case we chose to be Shared NFS for both servers:<\/p>\n\n\n\n
\"\"
29<\/figcaption><\/figure>\n\n\n\n
Before the installation starts it will check what needs fixing by selecting Fix & Check Again<\/em> Will try to fix some if possible. The SCAN message it shows is normal as we didn't use DNS Server with 3 IP round-robin but put only one in \/etc\/hosts<\/code> so we choose Ignore All <\/em>and Next<\/em>:<\/p>\n\n\n\n
\"\"
30<\/figcaption><\/figure>\n\n\n\n
Then the installation starts:<\/p>\n\n\n\n
\"\"
31<\/figcaption><\/figure>\n\n\n\n
Just before the installation is complete, it asks us to connect as root to each server and run the following scripts, starting first from the server where we started the installation:<\/p>\n\n\n\n
\"\"
32<\/figcaption><\/figure>\n\n\n\n
The installation of the Oracle Database software is finally complete:<\/p>\n\n\n\n
\"\"
33<\/figcaption><\/figure>\n\n\n\n
<\/p>\n\n\n\n
To install Direct NFS<\/h4>\n\n\n\n
From Oracle Database version 12c onwards Direct NFS is enabled by default by itself! It gets the information about NFS Shares from \/etc\/fstab<\/code> (mtab).<\/p>\n\n\n\n
To see that it has been activated when we create the database we run the following select, to see if it will bring records:<\/p>\n\n\n\n
select * from v$dnfs_servers;<\/pre>\n\n\n\n
Another way to see if it has been activated is to look at it alert log<\/code> of Oracle Database when it starts if it displays this message:<\/p>\n\n\n\n
tail -1000 $ORACLE_BASE\/diag\/rdbms\/racdb\/racdb1\/trace\/alert*<\/pre>\n\n\n\n
Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 3.0<\/code><\/pre>\n\n\n\n
More about Direct NFS is in the article here<\/a>.<\/p>\n\n\n\n
<\/p>\n\n\n\n
Create the database<\/h4>\n\n\n\n
To build the database, we start the setup with the following:<\/p>\n\n\n\n
su oracle\ncd $HOME\nsource ~\/.bash_profile\n\ndbca<\/pre>\n\n\n\n
We choose Create Database<\/em>:<\/p>\n\n\n\n
\"\"
34<\/figcaption><\/figure>\n\n\n\n
For example we do not go to Advanced Mode but set the name of the database as Storage Type: File System<\/em>, the NFS Shares oradata \/ fra, encoding the database to AL32UTF8 <\/code>(for example), the code and remove the option Create As Container Database<\/em>:<\/p>\n\n\n\n
\"\"
35<\/figcaption><\/figure>\n\n\n\n
Then it will show us what problems the installer has found, the problems in Cluster Validation concern the SCAN name that we have not passed to DNS Server with 3 IP round-robin. So we touch it by choosing Ignore All<\/em> and Next<\/em>:<\/p>\n\n\n\n
\"\"
36<\/figcaption><\/figure>\n\n\n\n
After some time, if everything goes well, the creation of the Oracle Database will be completed:<\/p>\n\n\n\n
\"\"
37<\/figcaption><\/figure>\n\n\n\n
<\/p>\n\n\n\n
Useful Commands<\/h2>\n\n\n\n
For Clusterware (crsctl)<\/h4>\n\n\n\n
To see the status of the Cluster with all Resources, run the following command with the root user:<\/p>\n\n\n\n
su root\ncd $HOME\nsource ~\/.bash_profile\n\ncrsctl status resource -t<\/pre>\n\n\n\n
\"\"
38<\/figcaption><\/figure>\n\n\n\n
We see that the database status is Open in both instances.<\/p>\n\n\n\n
To stop the cluster resources we run on each server with the root user:<\/p>\n\n\n\n
crsctl stop crs<\/pre>\n\n\n\n
Accordingly, to start them:<\/p>\n\n\n\n
crsctl start crs<\/pre>\n\n\n\n
For Oracle Database (srvctl)<\/h4>\n\n\n\n
To see the database configuration, run the following with the oracle user:<\/p>\n\n\n\n
su oracle\ncd $HOME\nsource ~\/.bash_profile\n\nsrvctl config database -d racdb<\/pre>\n\n\n\n
\"\"
39<\/figcaption><\/figure>\n\n\n\n
If we want to stop the database and from the 2 instances we run with the oracle user:<\/p>\n\n\n\n
srvctl stop database -d racdb<\/pre>\n\n\n\n
While to start it:<\/p>\n\n\n\n
srvctl start database -d racdb<\/pre>\n\n\n\n
<\/p>\n\n\n\n
To connect to Oracle Database<\/h4>\n\n\n\n
To connect to the database, go to the client's tnsnames.ora and simply add the SCAN name:<\/p>\n\n\n\n
RACDB =\n  (DESCRIPTION =\n  (LOAD_BALANCE = ON)\n    (CONNECT_TIMEOUT=5) (TRANSPORT_CONNECT_TIMEOUT=3) (RETRY_COUNT=3)\n    (ADDRESS_LIST =    \n      (ADDRESS = (PROTOCOL = TCP)(HOST = oracledev-scan)(PORT = 1521))\n    )\n    (CONNECT_DATA =\n      (SERVICE_NAME = racdb)\n    )\n  )<\/pre>\n\n\n\n
After doing this we can connect to sqlplus<\/code> with tnsname:<\/p>\n\n\n\n
sqlplus \"sys\/Password@racdb as sysdba\"<\/pre>\n\n\n\n
or without using tnsname, with EZconnect<\/code>:<\/p>\n\n\n\n
sqlplus \"sys\/Password@oracledev-scan:1521\/racdb as sysdba\"<\/pre>\n\n\n\n
<\/p>\n\n\n\n
To see the status of the instances with SQL<\/h4>\n\n\n\n
With the following query we can see the status of the instances:<\/p>\n\n\n\n
 select host_name,instance_name,status,sys_context('USERENV', 'CURRENT_USER') \"User\",sys_context('USERENV', 'INSTANCE_NAME') \"Connected to\",startup_time,version,sysdate\n from gv$instance;<\/pre>\n\n\n\n
\"\"
40<\/figcaption><\/figure>\n\n\n\n
For Direct NFS<\/h4>\n\n\n\n
To see if NFS Shares are being used with the increased performance of Direct NFS, the following query must return results:<\/p>\n\n\n\n
select * from v$dnfs_servers;<\/pre>\n\n\n\n
\"\"
41<\/figcaption><\/figure>\n\n\n\n
We can also see the files that use Direct NFS with the following query:<\/p>\n\n\n\n
select * from v$dnfs_files;<\/pre>\n\n\n\n
\"\"
42<\/figcaption><\/figure>\n\n\n\n
<\/p>\n\n\n\n
Sources:<\/h2>\n\n\n\n