Overview
After the failure to use iSCSI with ReadyNAS, I eventually decided to try using XEN with local shared files. To this end, I set up VICTORIA as follows:
References
- Xen Project Wiki
- Michael Kerrisk - man7.org
New Design
There are two (2) changes to the design presented in Planning :
- Operating System will be OEL7 (64-bit) on XEN
- ASM Disk Groups will be on local file system on VICTORIA
ASM Disk Groups
| Disk Group Name | Redundancy | Access | Location | Size (GB) |
|---|---|---|---|---|
| +DATA | External | Local | /OVS/running_pool/REDFERN/shared/DATA_01 | 50 |
| +FRA | External | Local | /OVS/running_pool/REDFERN/shared/FRA_01 | 75 |
| +REDO1 | External | Local | /OVS/running_pool/REDFERN/shared/REDO1_01 | 5 |
| +REDO2 | External | Local | /OVS/running_pool/REDFERN/shared/REDO2_01 | 5 |
| +VOTE | External | Local | /OVS/running_pool/REDFERN/shared/VOTE_01 | 5 |
Procedure
Create XEN Directories
Used the following command to create the directories on VICTORIA :
mkdir -p /OVS/running_pool/REDFERN/shared mkdir -p /OVS/running_pool/REDFERN/redfern1 mkdir -p /OVS/running_pool/REDFERN/redfern2
Create Local ASM Disks
Created the required ASM disks as follows:
dd bs=1G count=50 if=/dev/zero of=/OVS/running_pool/REDFERN/shared/DATA_01 & dd bs=1G count=75 if=/dev/zero of=/OVS/running_pool/REDFERN/shared/FRA_01 & dd bs=1G count=5 if=/dev/zero of=/OVS/running_pool/REDFERN/shared/REDO1_01 & dd bs=1G count=5 if=/dev/zero of=/OVS/running_pool/REDFERN/shared/REDO2_01 & dd bs=1G count=5 if=/dev/zero of=/OVS/running_pool/REDFERN/shared/VOTE_01 &
After a while, I can confirm the files have been created successfully:
[root@victoria network-scripts]# ls -lh /OVS/running_pool/REDFERN/shared total 186G -rw-r--r-- 1 root root 50G Apr 4 13:59 DATA_01 -rw-r--r-- 1 root root 75G Apr 4 14:14 FRA_01 -rw-r--r-- 1 root root 5.0G Apr 4 12:14 REDO1_01 -rw-r--r-- 1 root root 5.0G Apr 4 12:16 REDO2_01 -rw-r--r-- 1 root root 5.0G Apr 4 16:51 VOTE_01
Note: dd did not fail with bs=1G as it did in Set Up Shared Disks for RAC for REVESBY .
Create Local Disk for REDFERN1
To be on the safe side, I created a 50G disk for use by the operating system for REDFERN1 as follows:
dd bs=1G count=50 if=/dev/zero of=/OVS/running_pool/REDFERN/redfern1/root_disk &
Create XEN Configuration File for REDFERN1
I used the examples in 3.2 Attaching virtual devices to the appropriate bridge and the documentation in XL Network Configuration to construct the entries for VIF using the allocated values from Victoria VM Resources .
The description of the entries for the DISK parameter can be found in xl-disk-configuration.txt .
The other options are described in xl.cfg .
cd /OVS/running_pool/REDFERN/redfern1 cat <<DONE >redfern1.hvm # ===================================================================== # HVM guest configuration for REDFERN1 # Generated $(date +"%F %T") # ===================================================================== # # This is a fairly minimal example of what is required for an # HVM guest. For a more complete guide see xl.cfg(5) # This configures an HVM rather than PV guest builder = "hvm" # Guest name name = "redfern1" # 128-bit UUID for the domain as a hexadecimal number. # Use "uuidgen" to generate one if required. uuid = "$(uuidgen)" # Initial memory allocation (MB) memory = 4096 # Number of VCPUS vcpus = 2 # Network devices # A list of 'vifspec' entries as described in # docs/misc/xl-network-configuration.markdown vif = [ 'mac=00:16:3E:00:00:0E,bridge=xenbr0', 'mac=00:16:3E:00:00:0F,bridge=xenbr1', 'mac=00:16:3E:00:00:10,bridge=xenbr2', 'mac=00:16:3E:00:00:11,bridge=xenbr3' ] # Disk Devices # A list of 'diskspec' entries as described in # docs/misc/xl-disk-configuration.txt disk = [ '/OVS/running_pool/REDFERN/redfern1/root_disk,raw,xvda,rw', '/OVS/iso_pool/V46135-01.iso,raw,xvdc,ro,cdrom', '/OVS/running_pool/REDFERN/shared/DATA_01,raw,xvdd,rw', '/OVS/running_pool/REDFERN/shared/FRA_01,raw,xvde,rw', '/OVS/running_pool/REDFERN/shared/REDO1_01,raw,xvdf,rw', '/OVS/running_pool/REDFERN/shared/REDO2_01,raw,xvdg,rw', '/OVS/running_pool/REDFERN/shared/VOTE_01,raw,xvdh,rw' ] # Guest VGA console configuration, either SDL or VNC vnc = 1 vnclisten = "0.0.0.0" vncdisplay = "3" DONE
Start REDFERN1 VM (First Time)
When I try to start the REDFERN1 VM, I get the following error:
[root@victoria redfern1]# xl create redfern1.hvm xend is running, which may cause unpredictable results when using this xl command. Please shut down xend before continuing. (This check can be overridden with the -f option.)
Disable XEND
I used the following commands to disable XEND permanently:
service xend stop chkconfig xend off
Install OEL7 on REDFERN1
Started the REDFERN1 VM again with the following command:
xl create redfern1.hvm
The output is:
Parsing config from redfern1.hvm xc: info: VIRTUAL MEMORY ARRANGEMENT: Loader: 0000000000100000->000000000019cbe4 Modules: 0000000000000000->0000000000000000 TOTAL: 0000000000000000->00000000ff800000 ENTRY ADDRESS: 0000000000100000 xc: info: PHYSICAL MEMORY ALLOCATION: 4KB PAGES: 0x0000000000000200 2MB PAGES: 0x00000000000003fb 1GB PAGES: 0x0000000000000002 Daemon running with PID 9853
Verified the running status as follows:
[root@victoria redfern1]# xl list Name ID Mem VCPUs State Time(s) Domain-0 0 1102 8 r----- 5501.2 redfern1 3 3840 2 -b---- 247.3
Used a VNCViewer (Remmina on Ubuntu) to connect to 192.168.1.100 and installed OEL 7. It was a minimal installation with NO NICs configured.
At the end of the installation, I tried to eject the DVD via xl, but got the following error:
[root@victoria redfern1]# xl cd-eject 3 xvdc libxl: error: libxl_qmp.c:289:qmp_handle_error_response: received an error message from QMP server: Device 'ide-51744' is locked
3 was the current domain id for the REDFERN1 VM.
I had to use the Reboot on the installer to reboot the system. The default boot device search order is disk then cd-rom. A successful installation would then choose the disk.
Once the system came up, I shut the VM down gracefully through:
xl shutdown 4
4 was the current domain id for the REDFERN1 VM.
Remove Installation DVD from VM
I editted the redfern1.hvm to remove the following line:
'/OVS/iso_pool/V46135-01.iso,raw,xvdc,ro,cdrom',
Verify Network Configuration
Used the following command to verify the network configuration of the REDFERN1 VM:
xl network-list 7
7 was the current domain id for the REDFERN1 VM.
The output should be similar to:
Idx BE Mac Addr. handle state evt-ch tx-/rx-ring-ref BE-path 0 0 00:16:3e:00:00:0e 0 4 -1 372/373 /local/domain/0/backend/vif/7/0 1 0 00:16:3e:00:00:0f 1 4 -1 374/375 /local/domain/0/backend/vif/7/1 2 0 00:16:3e:00:00:10 2 4 -1 376/377 /local/domain/0/backend/vif/7/2 3 0 00:16:3e:00:00:11 3 4 -1 378/379 /local/domain/0/backend/vif/7/3
Complete Network Configuration
Using the example in 07 Set Up XEN Network , I ran the following script, on REDFERN1 , to populate the device descriptions for the four (4) Ethernet NICs:
cd /etc/sysconfig/network-scripts
for iface in 0 1 2 3
do
nw=$(((iface + 1)))
nic=$(((14 + iface)))
ifacename=eth${iface}
uuid=$(uuidgen)
hwaddr=$(printf "00:16:3e:00:00:%0.2X" ${nic})
# ==== Create Interface
cat <<DONE >ifcfg-${ifacename}
# -----------------------------------------------------------------------------
# NIC ${ifacename}
# Generated $(date +"%F %T")
# -----------------------------------------------------------------------------
# Device identification
DEVICE="${ifacename}"
UUID="${uuid}"
HWADDR="${hwaddr}"
# Device options
BOOTPROTO="static"
TYPE="Ethernet"
ONBOOT="yes"
DELAY="0"
NM_CONTROLLED="no"
# IPV4 Networking
IPADDR="192.168.${nw}.140"
NETMASK="255.255.255.0"
GATEWAY="192.168.1.1"
# DNS
DNS1="192.168.1.252"
DNS2="192.168.1.1"
PEERDNS="yes"
DOMAIN="yaocm.id.au"
DONE
done
Then I brought this NICs up as follows:
ifup eth0 ifup eth1 ifup eth2 ifup eth3
ping confirmed connectivity on both REDFERN1 and VICTORIA :
ping -c 1 192.168.1.140 ping -c 1 192.168.2.140 ping -c 1 192.168.3.140 ping -c 1 192.168.4.140