Richmond Cluster (05)
Richmond Cluster (04)
Richmond Cluster (06)

Saturday 03 May, 2008 - 09:45

Set up the ssh user equivalency between central , richmond1 , and richmond2 by going to central to download the ssh public keys from the other servers:
$ su - oracle
$ cd .ssh
$ rm known_hosts (stops conflict over the same IP address)
$ sftp richmond1-mgt
cd .ssh
get id_dsa.pub richmond1_dsa.pub
get id_rsa.pub richmond1_rsa.pub
exit

Repeated for richmond2 .

Create a single repository for all public keys for oracle on central , then copied to the other servers:
$ su - oracle
$ cd .ssh
$ cat *.pub >authorized_keys (overwrites previous version of authorized_keys)
$ scp authorized_keys richmond1-mgt:.ssh/
$ scp authorized_keys richmond2-mgt:.ssh/

On each of central , richmond1 , and richmond2 , establish the bona-fides of the servers to each other (including itself to itself) by executing the following series of commands (the date command is just a simple command to execute and return to the calling host):
ssh-agent $SHELL
ssh-add
ssh central date
ssh central.yaocm.id.au date
ssh richmond1 date
ssh richmond1.yaocm.id.au date
ssh richmond1-mgt date
ssh richmond1-mgt.yaocm.id.au date
ssh richmond2 date
ssh richmond2.yaocm.id.au date
ssh richmond2-mgt date
ssh richmond2-mgt.yaocm.id.au date

Now to load the clusterware installation software on both richmond1 and richmond2 :
$ su - oracle
$ mkdir installation
$ cd installation
$ unzip /mnt/hgfs/OCM/10201_clusterware_linux32.zip

Installed cvuqdisk package on both richmond1 and richmond2 :
$ su -
# cd /home/oracle/installation/clusterware/rpm
# ls
cvuqdisk-1.0.1-1.rpm
# rpmquery cvuqdisk
package cvuqdisk is not installed
# export CVUQDISK_GRP=oinstall
# rpm -iv cvuqdisk-1.0.1-1.rpm
Preparing packages for installation...
cvuqdisk-1.0.1-1
# rpmquery cvuqdisk
cvuqdisk-1.0.1-1

On richmond1 ,
$ su -
# parted /dev/sdb
GNU Parted 1.6.3
...
Using /dev/sdb
Error: Unable to open /dev/sdb - unrecognised disk label.
Information: The operating system thinks the geometry on /dev/sdb is 261/255/63.
(parted) mklabel gpt
(parted) mkpart primary ext3 0 200
(parted) mkpart primary ext3 200 250
(parted) print
Disk geometry for /dev/sdb: 0.000-2048.000 megabytes
Disk label type: gpt
Minor Start End Filesystem Name Flags
1 0.017 200.000
2 200.000 250.000
(parted) select /dev/sdc
Using /dev/sdc
Error: Unable to open /dev/sdc - unrecognised disk label.
Information: The operating system thinks the geometry on /dev/sdc is 261/255/63.
(parted) mklabel gpt
(parted) mkpart primary ext3 0 200
(parted) mkpart primary ext3 200 250
(parted) print
Disk geometry for /dev/sdc: 0.000-2048.000 megabytes
Disk label type: gpt
Minor Start End Filesystem Name Flags
1 0.017 200.000
2 200.000 250.000
(parted) select /dev/sdd
Using /dev/sdd
Error: Unable to open /dev/sdd - unrecognised disk label.
Information: The operating system thinks the geometry on /dev/sdd is 261/255/63.
(parted) mklabel gpt
(parted) mkpart primary ext3 0 200
(parted) mkpart primary ext3 200 250
(parted) print
Disk geometry for /dev/sdd: 0.000-2048.000 megabytes
Disk label type: gpt
Minor Start End Filesystem Name Flags
1 0.017 200.000
2 200.000 250.000
(parted) quit
Information: Don't forget to update /etc/fstab, if necessary.

However when I use parted on richmond2 , I get the following:
$ su -
# parted /dev/sdb
...
Using /dev/sdb
Warning: /dev/sdb contains GPT signatures, indicating that it has a GPT table. However, it does not have a valid fake msdos
partition table, as it should. Perhaps it was corrupted - possibly by a program that doesn't understand GPT partition
tables. Or perhaps you deleted the GPT table, and are now using an msdos partition table. Is this a GPT partition table?
Yes/No? yes
Error: The primary GPT table is corrupt, but the backup appears ok, so that will be used.
OK/Cancel? OK
Information: The operating system thinks the geometry on /dev/sdb is 261/255/63.
(parted) print
Disk geometry for /dev/sdb: 0.000-2048.000 megabytes
Disk label type: gpt
Minor Start End Filesystem Name Flags
1 0.017 200.000
2 200.000 250.000

This was also true for /dev/sdc and /dev/sdd as well. But this occurred only the first time on richmond2 .

At this stage, I should be ready to install the clusterware. So, I run a preinstallation check:
$ su - oracle
$ cd ~/installation/clusterware/cluvfy
$ ./runcluvfy stage -pre crsinst -n richmond1,richmond2 -r 10gR2 -c /dev/sdb1 -q /dev/sdb2 -osdba dba -oinv oinstall -verbose
Performing pre-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "richmond1"
Destination Node Reachable?
------------------------------------ ------------------------
richmond2 yes
richmond1 yes
Result: Node reachability check passed from node "richmond1".


Checking user equivalence...

Check: User equivalence for user "oracle"
Node Name Comment
------------------------------------ ------------------------
richmond2 passed
richmond1 passed
Result: User equivalence check passed for user "oracle".

Checking administrative privileges...

Check: Existence of user "oracle"
Node Name User Exists Comment
------------ ------------------------ ------------------------
richmond2 yes passed
richmond1 yes passed
Result: User existence check passed for "oracle".

Check: Existence of group "oinstall"
Node Name Status Group ID
------------ ------------------------ ------------------------
richmond2 exists 600
richmond1 exists 600
Result: Group existence check passed for "oinstall".

Check: Membership of user "oracle" in group "oinstall" [as Primary]
Node Name User Exists Group Exists User in Group Primary Comment
---------------- ------------ ------------ ------------ ------------ ------------
richmond2 yes yes yes yes passed
richmond1 yes yes yes yes passed
Result: Membership check for user "oracle" in group "oinstall" [as Primary] passed.

Administrative privileges check passed.

Checking node connectivity...


Interface information for node "richmond2"
Interface Name IP Address Subnet
------------------------------ ------------------------------ ----------------
eth0 192.168.100.66 192.168.100.0
eth1 192.168.101.2 192.168.101.0


Interface information for node "richmond1"
Interface Name IP Address Subnet
------------------------------ ------------------------------ ----------------
eth0 192.168.100.65 192.168.100.0
eth1 192.168.101.1 192.168.101.0


Check: Node connectivity of subnet "192.168.100.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
richmond2:eth0 richmond1:eth0 yes
Result: Node connectivity check passed for subnet "192.168.100.0" with node(s) richmond2,richmond1.

Check: Node connectivity of subnet "192.168.101.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
richmond2:eth1 richmond1:eth1 yes
Result: Node connectivity check passed for subnet "192.168.101.0" with node(s) richmond2,richmond1.

Suitable interfaces for the private interconnect on subnet "192.168.100.0":
richmond2 eth0:192.168.100.66
richmond1 eth0:192.168.100.65

Suitable interfaces for the private interconnect on subnet "192.168.101.0":
richmond2 eth1:192.168.101.2
richmond1 eth1:192.168.101.1

ERROR:
Could not find a suitable set of interfaces for VIPs.

Result: Node connectivity check failed.


Checking shared storage accessibility...

ERROR: /dev/sdb1
Unable to determine the sharedness of /dev/sdb on nodes:
richmond2,richmond1

Shared storage check failed on nodes "richmond2,richmond1".

Checking shared storage accessibility...

ERROR: /dev/sdb2
Unable to determine the sharedness of /dev/sdb on nodes:
richmond2,richmond1


Shared storage check failed on nodes "richmond2,richmond1".

Checking system requirements for 'crs'...

Check: Total memory
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
richmond2 1003.38MB (1027460KB) 512MB (524288KB) passed
richmond1 1003.38MB (1027460KB) 512MB (524288KB) passed
Result: Total memory check passed.

Check: Free disk space in "/tmp" dir
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
richmond2 3.22GB (3376644KB) 400MB (409600KB) passed
richmond1 3.14GB (3289632KB) 400MB (409600KB) passed
Result: Free disk space check passed.

Check: Swap space
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
richmond2 1.95GB (2040244KB) 1GB (1048576KB) passed
richmond1 1.95GB (2040244KB) 1GB (1048576KB) passed
Result: Swap space check passed.

Check: System architecture
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
richmond2 i686 i686 passed
richmond1 i686 i686 passed
Result: System architecture check passed.

Check: Kernel version
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
richmond2 2.4.21-40.EL 2.4.21-15EL passed
richmond1 2.4.21-40.EL 2.4.21-15EL passed
Result: Kernel version check passed.

Check: Package existence for "make-3.79"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
richmond2 make-3.79.1-17.1 passed
richmond1 make-3.79.1-17.1 passed
Result: Package existence check passed for "make-3.79".

Check: Package existence for "binutils-2.14"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
richmond2 binutils-2.14.90.0.4-42 passed
richmond1 binutils-2.14.90.0.4-42 passed
Result: Package existence check passed for "binutils-2.14".

Check: Package existence for "gcc-3.2"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
richmond2 gcc-3.2.3-54 passed
richmond1 gcc-3.2.3-54 passed
Result: Package existence check passed for "gcc-3.2".

Check: Package existence for "glibc-2.3.2-95.27"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
richmond2 glibc-2.3.2-95.39 passed
richmond1 glibc-2.3.2-95.39 passed
Result: Package existence check passed for "glibc-2.3.2-95.27".

Check: Package existence for "compat-db-4.0.14-5"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
richmond2 compat-db-4.0.14-5.1 passed
richmond1 compat-db-4.0.14-5.1 passed
Result: Package existence check passed for "compat-db-4.0.14-5".

Check: Package existence for "compat-gcc-7.3-2.96.128"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
richmond2 compat-gcc-7.3-2.96.128 passed
richmond1 compat-gcc-7.3-2.96.128 passed
Result: Package existence check passed for "compat-gcc-7.3-2.96.128".

Check: Package existence for "compat-gcc-c++-7.3-2.96.128"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
richmond2 compat-gcc-c++-7.3-2.96.128 passed
richmond1 compat-gcc-c++-7.3-2.96.128 passed
Result: Package existence check passed for "compat-gcc-c++-7.3-2.96.128".

Check: Package existence for "compat-libstdc++-7.3-2.96.128"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
richmond2 compat-libstdc++-7.3-2.96.128 passed
richmond1 compat-libstdc++-7.3-2.96.128 passed
Result: Package existence check passed for "compat-libstdc++-7.3-2.96.128".

Check: Package existence for "compat-libstdc++-devel-7.3-2.96.128"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
richmond2 compat-libstdc++-devel-7.3-2.96.128 passed
richmond1 compat-libstdc++-devel-7.3-2.96.128 passed
Result: Package existence check passed for "compat-libstdc++-devel-7.3-2.96.128".

Check: Package existence for "openmotif-2.2.3"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
richmond2 openmotif-2.2.3-5.RHEL3.2 passed
richmond1 openmotif-2.2.3-5.RHEL3.2 passed
Result: Package existence check passed for "openmotif-2.2.3".

Check: Package existence for "setarch-1.3-1"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
richmond2 setarch-1.3-1 passed
richmond1 setarch-1.3-1 passed
Result: Package existence check passed for "setarch-1.3-1".

Check: Group existence for "dba"
Node Name Status Comment
------------ ------------------------ ------------------------
richmond2 exists passed
richmond1 exists passed
Result: Group existence check passed for "dba".

Check: Group existence for "oinstall"
Node Name Status Comment
------------ ------------------------ ------------------------
richmond2 exists passed
richmond1 exists passed
Result: Group existence check passed for "oinstall".

Check: User existence for "nobody"
Node Name Status Comment
------------ ------------------------ ------------------------
richmond2 exists passed
richmond1 exists passed
Result: User existence check passed for "nobody".

System requirement passed for 'crs'


Pre-check for cluster services setup was unsuccessful on all the nodes.

The only issues are the VIP problem and the disk shareability which I know about and accept.