Following are the steps to install Oracle 12c Grid Infrastructure for standalone server on Oracle Linux 6.
We need Oracle Grid Infrastructure on standalone box to use ASM, so the very first thing which we need to configure is the disks which we are going to use with ASM to store Oracle files.
Here is how disks are setup on my machine:
The NAME parameter in above file defines the name of the block device which will be created after applying these rules. I am calling it asm-disk1 and asm-disk2, so that these can be easily identified in the device listing.
After creating the new rules, we can test the newly created rule using following command for each disk:
# /sbin/udevadm test /dev/sdb1
# /sbin/udevadm test /dev/sdc1
Since the default diskstring is set to /dev/sd*, it wasn't finding any of our previously configured disks.
Change the discovery string to /dev/asm* because the devices which we created for ASM are /dev/asm-disk1 and /dev/asm-disk2.
[root@orarac01 ~]# fdisk -l | grep /dev
Disk /dev/sda: 128.8 GB, 128849018880 bytes
/dev/sda1 * 1 64 512000 83 Linux
/dev/sda2 64 15666 125316096 8e Linux LVM
Disk /dev/sdb: 12.9 GB, 12884901888 bytes
Disk /dev/sdc: 12.9 GB, 12884901888 bytes
/dev/sdb and /dev/sdc are the disks which I am going to use with ASM. In this exercise, I am going to create 1 ASM diskgroup with NORMAL redundancy and will use these 2 disks for 2 failure groups.
We need to create 1 primary partition on each of the disk, covering the whole disk and will use that partition in our ASM disk creation process.
Here are the steps to create 1 primary partition on disk /dev/sdb:
fdisk /dev/sdb
Type n (for new partition) and press enter
Type p (for primary partition) and press enter
Type 1 (for partition number) and press enter
Type p (to print the partition table) and press enter
Type w (to write the partition table to disk permanently) and press enter
Here is the complete action log of above mentioned steps for disk /dev/sdb:
[root@orarac01 ~]# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x9018e049.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').
Command (m for help): m
Command action
a toggle a bootable flag
b edit bsd disklabel
c toggle the dos compatibility flag
d delete a partition
l list known partition types
m print this menu
n add a new partition
o create a new empty DOS partition table
p print the partition table
q quit without saving changes
s create a new empty Sun disklabel
t change a partition's system id
u change display/entry units
v verify the partition table
w write table to disk and exit
x extra functionality (experts only)
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1566, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-1566, default 1566):
Using default value 1566
Command (m for help): p
Disk /dev/sdb: 12.9 GB, 12884901888 bytes
255 heads, 63 sectors/track, 1566 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x9018e049
Device Boot Start End Blocks Id System
/dev/sdb1 1 1566 12578863+ 83 Linux
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
Repeat the above mentioned steps for all the disks you are planning to use with ASM.
Now we need to make the SATA devices trusted by adding following line in /etc/scsi_id.config file:
[root@orarac01 rules.d]# cat /etc/scsi_id.config
options=-g
If the /etc/scsi_id.config file doesn't exist, then just create one and add only the above line.
Check the SCSI ID of the disks which we are going to use in ASM:
[root@orarac01 rules.d]# /sbin/scsi_id -g -u -d /dev/sdb1
1ATA_VBOX_HARDDISK_VBdda4193b-5e1d9aed
[root@orarac01 rules.d]# /sbin/scsi_id -g -u -d /dev/sdc1
1ATA_VBOX_HARDDISK_VB9d7a8cf7-04f8bb36
We need to make sure that the disks are owned by Oracle software owner and to make these permissions permanent, we need to add udev rules by creating a new rules file for asm disks.
Create a new file called something like 90-asmdisks.rules in /etc/udev/rules.d directory and paste the disk information (SCSI ID which we retrieved in previous step).
Here is what i have created for my environment:
[root@orarac01 rules.d]# cat /etc/udev/rules.d/90-asmdisks.rules
KERNEL=="sdb1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VBdda4193b-5e1d9aed", NAME="asm-disk1", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sdc1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB9d7a8cf7-04f8bb36", NAME="asm-disk2", OWNER="oracle", GROUP="dba", MODE="0660"
The NAME parameter in above file defines the name of the block device which will be created after applying these rules. I am calling it asm-disk1 and asm-disk2, so that these can be easily identified in the device listing.
After creating the new rules, we can test the newly created rule using following command for each disk:
# /sbin/udevadm test /dev/sdb1
# /sbin/udevadm test /dev/sdc1
Now we should be able to see the newly created devices in /dev directory and the permissions of devices should also be owned by oracle/dba:
[root@orarac01 rules.d]# ls -l /dev/asm*
brw-rw----. 1 oracle dba 8, 17 Jul 21 23:23 /dev/asm-disk1
brw-rw----. 1 oracle dba 8, 33 Jul 21 23:23 /dev/asm-disk2
The above output tells that our new rules just worked without any issue and the disk setup completes here and we are ready to launch the OUI to install Grid Infrastructure now.
Go to the Oracle 12c media directory and run the installer from there to launch the OUI:
[oracle@orarac01 grid]$ ./runInstaller
Starting Oracle Universal Installer...
Checking Temp space: must be greater than 120 MB. Actual 43003 MB Passed
Checking swap space: must be greater than 150 MB. Actual 2015 MB Passed
Checking monitor: must be configured to display at least 256 colors. Actual 16777216 Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2013-07-21_11-38-28PM. Please wait ...[oracle@orarac01 grid]$ You can find the log of this install session at:
/u01/app/oraInventory/logs/installActions2013-07-21_11-38-28PM.log
This should bring the following installer 12c Grid Infrastructure Install window:
In the above window you have 3 options:
First is to provide your metalink credentials, so the the installer can pull all the recommended upgrades for your product.
Second to apply already downloaded software updates
And the last to skip the software update completely.
Choose an option as per your requirement, I am going with option 3 to skip the updates completely.
Select the appropriate radio button and hit Next.
In the above window, select "Install and Configure Grid Infrastructure for Standalone Server" and hit Next.
English language is selected by default, so hit Next.
In the above window, enter the diskgroup name you want to create for ASM.
Then select the diskgroup redundancy:
High -> 3 way mirroring
Normal -> 2 way mirroring
External -> No mirroring at all
And the allocation unit size (au_size). If you are not going to host really large database then 1MB is good enough.
Since there are no disks in the available disk list, click "Change Discovery Path" button to change the ASM diskstring.
Since the default diskstring is set to /dev/sd*, it wasn't finding any of our previously configured disks.
Change the discovery string to /dev/asm* because the devices which we created for ASM are /dev/asm-disk1 and /dev/asm-disk2.
Hit Ok.
Now we can see our disks. Select all the disks and hit Next.
In the above window, either specify different passwords for SYS and ASMSNMP users OR use the same password for both of the users and hit Next.
If you chose same password then you will get above warning window, just hit Yes.
In above window, select the OS group for SYSASM, SYSDBA & SYSOPER operational users.
If you have separate team who deals with just storage then you need separate roles setup, otherwise just use the same and hit Next.
If you chose same OS group then you will get above warning window, just hit Yes.
In the above window, specify the location/path of ORACLE_BASE & ORACLE_HOME and hit Next.
In the above window, specify the location/path of Oracle inventory and hit Next.
In the above window, if you want the installer to run the root scripts for you then either provide root password or sudo access credentials to do the same, otherwise just leave it unchecked and hit Next.
In the above window, you can view the results of Oracle pre-req check run. I am going to ignore all of the above failed pre-reqs but it's not advisable for production boxes.
You will get the above warning window, if you ignore any failed pre-req, just hit Yes.
In the above window, it's showing the summary of all the choices we made for the install. Hit the Install button to actually install as per the given instructions.
The above window shows the overall progress & status of all the selected operations.
The above window is asking for the confirmation to run the root specific operations as privileged user (root). Hit Yes to continue.
In the above window, you can see that ASMCA failed for some reason and I just continued the installation. Will fix this manually after the install.
In the above window, install finished successfully. Now we need to fix the Automatic Storage Management Configuration Assistant error, we got earlier while installing.
Troubleshooting
Even though our installation failed on configuring ASM, it still configured HAS and other resources for us. We can query that as oracle user:
[oracle@orarac01 ~]$ crsctl status res
NAME=ora.LISTENER.lsnr
TYPE=ora.listener.type
TARGET=ONLINE
STATE=ONLINE on orarac01
NAME=ora.cssd
TYPE=ora.cssd.type
TARGET=ONLINE
STATE=OFFLINE
NAME=ora.diskmon
TYPE=ora.diskmon.type
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.evmd
TYPE=ora.evm.type
TARGET=ONLINE
STATE=ONLINE on orarac01
NAME=ora.ons
TYPE=ora.ons.type
TARGET=OFFLINE
STATE=OFFLINE
As you can see that the CSSD service is down, we need to start that up before starting ASM:
[oracle@orarac01 dbs]$ crsctl start res ora.cssd
CRS-2672: Attempting to start 'ora.cssd' on 'orarac01'
CRS-2672: Attempting to start 'ora.diskmon' on 'orarac01'
CRS-2676: Start of 'ora.diskmon' on 'orarac01' succeeded
CRS-2676: Start of 'ora.cssd' on 'orarac01' succeeded
Now we need to manually create a pfile in $ORACLE_HOME/dbs directory for ASM and start the instance using that:
This is what I put in my ASM init file:
[oracle@orarac01 dbs]$ cat init+ASM.ora
instance_type=ASM
db_unique_name=+ASM
asm_diskstring='/dev/asm*'
asm_diskgroups='DATA'
sga_target=400M
If you want you can use memory_target as well, it just needs the /dev/shm partition mount point.
SQL> startup pfile=/u01/app/oracle/product/12.1.0/grid/dbs/init+ASM.ora
ASM instance started
Total System Global Area 1135747072 bytes
Fixed Size 2297344 bytes
Variable Size 1108283904 bytes
ASM Cache 25165824 bytes
ORA-15032: not all alterations performed
ORA-15017: diskgroup "DATA" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DATA"
We can ignore the above error because we know that DATA diskgroup hasn't been created yet.
Now, If I try to create spfile from memory or pfile, it will fail with following error:
SQL> create spfile='/u01/app/oracle/product/12.1.0/grid/dbs/spfile+ASM.ora' from pfile='/u01/app/oracle/product/12.1.0/grid/dbs/init+ASM.ora';
create spfile='/u01/app/oracle/product/12.1.0/grid/dbs/spfile+ASM.ora' from pfile='/u01/app/oracle/product/12.1.0/grid/dbs/init+ASM.ora'
*
ERROR at line 1:
ORA-29786: SIHA attribute GET failed with error [Attribute 'SPFILE' sts[200]
lsts[0]]
We are getting the above error because ASM is not registered with OCR yet, we need to register it using SRVCTL and then only we can create spfile.
Here is how to register ASM with OCR:
[oracle@orarac01 dbs]$ srvctl add asm -listener LISTENER -spfile '/u01/app/oracle/product/12.1.0/grid/dbs/spfile+ASM.ora'
Eventhough we don't have spfile yet, just mention the complete path of it while registering the ASM, it will work and we won't have to change it after creating the spfile created.
Now since ASM is registered with OCR, let's try creating the spfile again:
[oracle@orarac01 dbs]$ sqlplus / as sysasm
SQL*Plus: Release 12.1.0.1.0 Production on Mon Jul 22 00:54:17 2013
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production
With the Automatic Storage Management option
SQL> create spfile='/u01/app/oracle/product/12.1.0/grid/dbs/spfile+ASM.ora' from pfile='/u01/app/oracle/product/12.1.0/grid/dbs/init+ASM.ora';
File created.
We have the spfile now and the ASM configuration in OCR is already pointing to it, so that's good as well.
And here are our disks which we configured earlier:
SQL> col path format a30 trunc
SQL> select path, header_status from v$asm_disk;
PATH HEADER_STATU
------------------------------ ------------
/dev/asm-disk2 CANDIDATE
/dev/asm-disk1 CANDIDATE
Everything is setup correctly so far, ASM can see our disks as CANDIDATE, which means we can use them to create new diskgroup.
Now we can create the diskgroup manually:
SQL> create diskgroup DATA normal redundancy failgroup DATA_FG1 disk '/dev/asm-disk1' name DATA_FG1_DISK1 failgroup DATA_FG2 disk '/dev/asm-disk2' name DATA_FG2_DISK1;
Diskgroup created.
SQL> select name,total_mb,free_mb,path,header_status,failgroup from v$asm_disk;
NAME TOTAL_MB FREE_MB PATH HEADER_STATU FAILGROUP
------------------------------ ---------- ---------- ------------------------------ ------------ -------------
DATA_FG2_DISK1 12284 12233 /dev/asm-disk2 MEMBER DATA_FG2
DATA_FG1_DISK1 12284 12233 /dev/asm-disk1 MEMBER DATA_FG1
Now, I am shutting it down, so that we can start it from CRS resource:
SQL> shut immediate
ASM diskgroups volume disabled
ASM diskgroups dismounted
ASM instance shutdown
SQL> quit
Start the ASM using crsctl command:
[oracle@orarac01 dbs]$ crsctl start res ora.asm
CRS-2672: Attempting to start 'ora.asm' on 'orarac01'
CRS-2676: Start of 'ora.asm' on 'orarac01' succeeded
CRS-2672: Attempting to start 'ora.DATA.dg' on 'orarac01'
CRS-2676: Start of 'ora.DATA.dg' on 'orarac01' succeeded
[oracle@orarac01 dbs]$ crsctl status res
NAME=ora.DATA.dg
TYPE=ora.diskgroup.type
TARGET=ONLINE
STATE=ONLINE on orarac01
NAME=ora.LISTENER.lsnr
TYPE=ora.listener.type
TARGET=ONLINE
STATE=ONLINE on orarac01
NAME=ora.asm
TYPE=ora.asm.type
TARGET=ONLINE
STATE=ONLINE on orarac01
NAME=ora.cssd
TYPE=ora.cssd.type
TARGET=ONLINE
STATE=ONLINE on orarac01
NAME=ora.diskmon
TYPE=ora.diskmon.type
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.evmd
TYPE=ora.evm.type
TARGET=ONLINE
STATE=ONLINE on orarac01
NAME=ora.ons
TYPE=ora.ons.type
TARGET=OFFLINE
STATE=OFFLINE
And at the very end, just to check all the resource dependencies, I am bouncing whole HAS stack:
[oracle@orarac01 dbs]$ crsctl stop has
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'orarac01'
CRS-2673: Attempting to stop 'ora.evmd' on 'orarac01'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'orarac01'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'orarac01'
CRS-2677: Stop of 'ora.DATA.dg' on 'orarac01' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'orarac01'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'orarac01' succeeded
CRS-2677: Stop of 'ora.asm' on 'orarac01' succeeded
CRS-2677: Stop of 'ora.evmd' on 'orarac01' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'orarac01'
CRS-2677: Stop of 'ora.cssd' on 'orarac01' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'orarac01' has completed
CRS-4133: Oracle High Availability Services has been stopped.
See how all the services got stopped by HAS in proper order, same applied to startup as well:
[oracle@orarac01 dbs]$ crsctl start has
CRS-4123: Oracle High Availability Services has been started.
[oracle@orarac01 dbs]$
[oracle@orarac01 dbs]$ crsctl status res
NAME=ora.DATA.dg
TYPE=ora.diskgroup.type
TARGET=ONLINE
STATE=ONLINE on orarac01
NAME=ora.LISTENER.lsnr
TYPE=ora.listener.type
TARGET=ONLINE
STATE=ONLINE on orarac01
NAME=ora.asm
TYPE=ora.asm.type
TARGET=ONLINE
STATE=ONLINE on orarac01
NAME=ora.cssd
TYPE=ora.cssd.type
TARGET=ONLINE
STATE=ONLINE on orarac01
NAME=ora.diskmon
TYPE=ora.diskmon.type
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.evmd
TYPE=ora.evm.type
TARGET=ONLINE
STATE=ONLINE on orarac01
NAME=ora.ons
TYPE=ora.ons.type
TARGET=OFFLINE
STATE=OFFLINE
All the registered services are up and everybody seems to be happy now. :)
Thanks
Daljit Singh
2 comments:
This is great information for someone who needs to understand how 12c works. Good work Daljit.
Very nice article Good one Dalit keep it up !!!
Post a Comment