Storage Server Patch
Database Server Patch
Infiniband Switches Patch
I will write all the steps and explain why and how.
Critical NOTE: Before starting [patching, it is highly recommended to restart all Storage Servers and DB Nodes. Why? Because, during patching they will automatically restarted and if the restart is not successful, then you will have big headache! So, in my case everything went fine.
In our case we needed to download patch 17816100: "Quarterly Full Stack Download Patch For Oracle Exadata (Jan 2014 - 11.2.0.3.0)". Download from MOS.
Red Color are my comments, black are commands and commands'outputs.
1. Create folder tmpstage in /u01
[root@exa101 ~]# cd /u01/
[root@exa101 u01]# mkdir tmpstage
2. Copy all patch17816100* and p17938410_112330_Linux-x86-64 to /u01/tmpstage
3. The Quarterly Full Stack Download Patch For Oracle Exadata is a 3.7G tar image file when fully extracted. Due to it's size it has been split
into ~1.2G sized zip files for downloading from ARU. To assemble the image, please follow below procedure :
[root@exa101 tmpstage]# ls -l
total 3792432
-rw-r--r-- 1 root root 1258297983 Apr 14 12:21 p17816100_112030_Linux-x86-64_1of4.zip
-rw-r--r-- 1 root root 1258494785 Apr 14 12:11 p17816100_112030_Linux-x86-64_2of4.zip
-rw-r--r-- 1 root root 1246102503 Apr 14 02:56 p17816100_112030_Linux-x86-64_3of4.zip
-rw-r--r-- 1 root root 116728120 Apr 13 23:16 p17816100_112030_Linux-x86-64_4of4.zip
-rw-r--r-- 1 root root 1774 Apr 17 10:54 p17938410_112330_Linux-x86-64.zip
[root@exa101 tmpstage]# unzip p17816100_112030_Linux-x86-64_1of4.zip
Archive: p17816100_112030_Linux-x86-64_1of4.zip
inflating: 17816100.tar.splitaa
inflating: README.html
inflating: README.txt
inflating: PatchSearch.xml
[root@exa101 tmpstage]# unzip p17816100_112030_Linux-x86-64_2of4.zip
Archive: p17816100_112030_Linux-x86-64_2of4.zip
inflating: 17816100.tar.splitab
[root@exa101 tmpstage]# unzip p17816100_112030_Linux-x86-64_2of4.zip
Archive: p17816100_112030_Linux-x86-64_2of4.zip
replace 17816100.tar.splitab? [y]es, [n]o, [A]ll, [N]one, [r]ename: [root@exa101 tmpstage]# unzip p17816100_112030_Linux-x86-64_2of4.zip
[root@exa101 tmpstage]# unzip p17816100_112030_Linux-x86-64_3of4.zip
Archive: p17816100_112030_Linux-x86-64_3of4.zip
inflating: 17816100.tar.splitac
[root@exa101 tmpstage]# unzip p17816100_112030_Linux-x86-64_4of4.zip
Archive: p17816100_112030_Linux-x86-64_4of4.zip
inflating: 17816100.tar.splitad
[root@exa101 tmpstage]# ls -l
total 7598492
-rw-r--r-- 1 root root 1258291200 Jan 14 10:00 17816100.tar.splitaa
-rw-r--r-- 1 root root 1258291200 Jan 14 10:20 17816100.tar.splitab
-rw-r--r-- 1 root root 1258291200 Jan 14 10:39 17816100.tar.splitac
-rw-r--r-- 1 root root 118661120 Jan 14 10:41 17816100.tar.splitad
-rw-r--r-- 1 root root 1258297983 Apr 14 12:21 p17816100_112030_Linux-x86-64_1of4.zip
-rw-r--r-- 1 root root 1258494785 Apr 14 12:11 p17816100_112030_Linux-x86-64_2of4.zip
-rw-r--r-- 1 root root 1246102503 Apr 14 02:56 p17816100_112030_Linux-x86-64_3of4.zip
-rw-r--r-- 1 root root 116728120 Apr 13 23:16 p17816100_112030_Linux-x86-64_4of4.zip
-rw-r--r-- 1 root root 1774 Apr 17 10:54 p17938410_112330_Linux-x86-64.zip
-rw-rw-r-- 1 root root 3602 Jan 15 07:46 PatchSearch.xml
-rw-r--r-- 1 root root 28716 Jan 14 10:56 README.html
-rw-r--r-- 1 root root 9650 Jan 14 10:56 README.txt
[root@exa101 tmpstage]# cat *.tar.* | tar -xvf -
17816100/
17816100/Infrastructure/
17816100/Infrastructure/ExadataDBNodeUpdate/
17816100/Infrastructure/ExadataDBNodeUpdate/2.39/
17816100/Infrastructure/ExadataDBNodeUpdate/2.39/README.txt
17816100/Infrastructure/ExadataDBNodeUpdate/2.39/p16486998_121110_Linux-x86-64.zip
17816100/Infrastructure/ExadataStorageServer/
17816100/Infrastructure/ExadataStorageServer/11.2.3.3.0/
17816100/Infrastructure/ExadataStorageServer/11.2.3.3.0/README-16278923.html
17816100/Infrastructure/ExadataStorageServer/11.2.3.3.0/README-17809253.html
17816100/Infrastructure/ExadataStorageServer/11.2.3.3.0/p17809253_112330_Linux-x86-64.zip
17816100/Infrastructure/ExadataStorageServer/11.2.3.3.0/p16278923_112330_Linux-x86-64.zip
17816100/Infrastructure/SunRackIIPDUMeteringUnitFirmware/
17816100/Infrastructure/SunRackIIPDUMeteringUnitFirmware/1.06/
17816100/Infrastructure/SunRackIIPDUMeteringUnitFirmware/1.06/README.txt
17816100/Infrastructure/SunRackIIPDUMeteringUnitFirmware/1.06/p16523441_106_Generic.zip
17816100/SystemsManagement/
17816100/SystemsManagement/CloudControl12/
17816100/SystemsManagement/CloudControl12/WLS/
17816100/SystemsManagement/CloudControl12/WLS/14603262/
17816100/SystemsManagement/CloudControl12/WLS/14603262/README.txt
17816100/SystemsManagement/CloudControl12/WLS/14603262/p14603262_111160_Generic.zip
17816100/SystemsManagement/CloudControl12/WLS/16420963/
17816100/SystemsManagement/CloudControl12/WLS/16420963/README.txt
17816100/SystemsManagement/CloudControl12/WLS/16420963/p16420963_1036_Generic.zip
17816100/SystemsManagement/CloudControl12/Repository/
17816100/SystemsManagement/CloudControl12/Repository/11061801/
17816100/SystemsManagement/CloudControl12/Repository/11061801/p11061801_112030_Generic.zip
17816100/SystemsManagement/CloudControl12/Repository/11061801/README.txt
17816100/SystemsManagement/CloudControl12/Repository/16869210/
17816100/SystemsManagement/CloudControl12/Repository/16869210/README.txt
17816100/SystemsManagement/CloudControl12/Repository/16869210/p16869210_112030_Linux-x86-64.zip
17816100/SystemsManagement/CloudControl12/EMFramework/
17816100/SystemsManagement/CloudControl12/EMFramework/16822934/
17816100/SystemsManagement/CloudControl12/EMFramework/16822934/README.txt
17816100/SystemsManagement/CloudControl12/EMFramework/16822934/p16822934_121030_Generic.zip
17816100/SystemsManagement/CloudControl12/EMFramework/13983293/
17816100/SystemsManagement/CloudControl12/EMFramework/13983293/README.txt
17816100/SystemsManagement/CloudControl12/EMFramework/13983293/p13983293_111160_Generic.zip
17816100/README.txt
17816100/README.html
17816100/Database/
17816100/Database/OPatch/
17816100/Database/OPatch/11.2.0.3.6/
17816100/Database/OPatch/11.2.0.3.6/README.txt
17816100/Database/OPatch/11.2.0.3.6/p6880880_112000_Linux-x86-64.zip
17816100/Database/11.2.0.3.22_QDPE_Jan2014/
17816100/Database/11.2.0.3.22_QDPE_Jan2014/README.html
17816100/Database/11.2.0.3.22_QDPE_Jan2014/p17747147_112030_Linux-x86-64.zip
17816100/Database/OPlan/
17816100/Database/OPlan/12.1.0.1.5/
17816100/Database/OPlan/12.1.0.1.5/README.txt
17816100/Database/OPlan/12.1.0.1.5/p11846294_121000_Generic.zip
[root@exa101 tmpstage]# ls -l
total 7598496
drwxr-xr-x 5 3849 42424 4096 Jan 14 07:53 17816100
-rw-r--r-- 1 root root 1258291200 Jan 14 10:00 17816100.tar.splitaa
-rw-r--r-- 1 root root 1258291200 Jan 14 10:20 17816100.tar.splitab
-rw-r--r-- 1 root root 1258291200 Jan 14 10:39 17816100.tar.splitac
-rw-r--r-- 1 root root 118661120 Jan 14 10:41 17816100.tar.splitad
-rw-r--r-- 1 root root 1258297983 Apr 14 12:21 p17816100_112030_Linux-x86-64_1of4.zip
-rw-r--r-- 1 root root 1258494785 Apr 14 12:11 p17816100_112030_Linux-x86-64_2of4.zip
-rw-r--r-- 1 root root 1246102503 Apr 14 02:56 p17816100_112030_Linux-x86-64_3of4.zip
-rw-r--r-- 1 root root 116728120 Apr 13 23:16 p17816100_112030_Linux-x86-64_4of4.zip
-rw-r--r-- 1 root root 1774 Apr 17 10:54 p17938410_112330_Linux-x86-64.zip
-rw-rw-r-- 1 root root 3602 Jan 15 07:46 PatchSearch.xml
-rw-r--r-- 1 root root 28716 Jan 14 10:56 README.html
-rw-r--r-- 1 root root 9650 Jan 14 10:56 README.txt
[root@exa101 tmpstage]#
4. Verify the structure of the contents of the Quarterly Full Stack Download Patch For Oracle Exadata. The top-level directory contains this
README document and some sub-directories in the following layout:
17816100
|
|- README.txt
|
|- README.html
|
|
|- Infrastructure/
| --ExadataStorageServer/
| -11.2.3.3.0/
| -p16278923_112330_Linux-x86-64.zip
| -p17809253_112330_Linux-x86-64.zip
| -README-16278923.html
| -README-17809253.html
|
| --ExadataDBNodeUpdate/
| -2.39/
| -p16486998_121110_Linux-x86-64.zip
| -README.txt
|
| --SunRackIIPDUMeteringUnitFirmware/
| -1.06/
| -p16523441_106_Generic.zip
| -README.txt
|
|
|- Database/
| --11.2.0.3.22_QDPE_Jan2014/
| -p17747147_112030_Linux-x86-64.zip
| -README.html
|
| --OPatch/
| -11.2.0.3.6/
| -p6880880_112000_Linux-x86-64.zip
| -README.txt
|
| --OPlan/
| -12.1.0.1.5/
| -p11846294_121000_Generic.zip
| -README.txt
|
|
|- SystemsManagement/
|
| --CloudControl12/
|
| --EMFramework/
| -13983293/
| -p13983293_111160_Generic.zip
| -README.txt
| -16822934/
| -p16822934_121030_Generic.zip
| -README.txt
|
| --Repository/
| -11061801/
| -p11061801_112030_Generic.zip
| -README.txt
| -16869210/
| -p16869210_112030_Linux-x86-64.zip
| -README.txt
|
| --WLS/
| -14603262/
| -p14603262_111160_Generic.zip
| -README.txt
| -16420963/
| -p16420963_1036_Generic.zip
| -README.txt
|
5. Patching Exadata Cells
5.1 Preparing Exadata Cells for Patch Application
All steps must be done under root user.
5.1.1 Set up the root user SSH equivalence database server to all cell servers:
a. Create a file named cell_group that has one cell host name or IP address per line for each cell to be patched:
[root@exa101 ~]# cat cell_group
exa1cel01
exa1cel02
exa1cel03
b. Check for existing root SSH equivalence. The following command should require no password prompts and no interaction. It should return the
list of host names in the cell_group file.
[root@exa101 ~]# dcli -g cell_group -l root 'hostname -i'
exa1cel01: 10.45.33.4
exa1cel02: 10.45.33.5
exa1cel03: 10.45.33.6
Set up SSH root equivalence if not already done so from the launch server. Do not do this step if you already have root SSH equivalence.
Generate root SSH keys as follows:
ssh-keygen -t rsa
Accept the defaults so the SSH keys are created for the root user.
Push the SSH keys to set up SSH equivalence using the following command. Enter the root password when prompted.
dcli -g cell_group -l root -k
5.1.2 We will perform non-rolling upgrade, so Stop all CRS completely:
[root@exa102 ~]# /u01/app/11.2.0.3/grid/bin/crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'exa102'
CRS-2673: Attempting to stop 'ora.crsd' on 'exa102'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'exa102'
CRS-2673: Attempting to stop 'ora.DBFS_DG.dg' on 'exa102'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'exa102'
CRS-2673: Attempting to stop 'ora.rmanst.db' on 'exa102'
CRS-2673: Attempting to stop 'ora.pdmn.dpmn_dg.svc' on 'exa102'
CRS-2673: Attempting to stop 'ora.exa101.vip' on 'exa102'
CRS-2673: Attempting to stop 'ora.cvu' on 'exa102'
CRS-2673: Attempting to stop 'ora.LISTENER_DG.lsnr' on 'exa102'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'exa102'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'exa102'
CRS-2673: Attempting to stop 'ora.exa1db02-dg-vip.vip' on 'exa102'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'exa102'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'exa102'
CRS-2673: Attempting to stop 'ora.oc4j' on 'exa102'
CRS-2677: Stop of 'ora.pdmn.dpmn_dg.svc' on 'exa102' succeeded
CRS-2673: Attempting to stop 'ora.pdmn.db' on 'exa102'
CRS-2677: Stop of 'ora.exa101.vip' on 'exa102' succeeded
CRS-2677: Stop of 'ora.exa1db02-dg-vip.vip' on 'exa102' succeeded
CRS-2677: Stop of 'ora.cvu' on 'exa102' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'exa102' succeeded
CRS-2673: Attempting to stop 'ora.exa102.vip' on 'exa102'
CRS-2677: Stop of 'ora.LISTENER_DG.lsnr' on 'exa102' succeeded
CRS-2673: Attempting to stop 'ora.exa1db01-dg-vip.vip' on 'exa102'
CRS-2677: Stop of 'ora.exa102.vip' on 'exa102' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'exa102' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'exa102'
CRS-2677: Stop of 'ora.exa1db01-dg-vip.vip' on 'exa102' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'exa102' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'exa102' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'exa102'
CRS-2677: Stop of 'ora.scan2.vip' on 'exa102' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'exa102' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'exa102'
CRS-2677: Stop of 'ora.scan3.vip' on 'exa102' succeeded
CRS-2677: Stop of 'ora.registry.acfs' on 'exa102' succeeded
CRS-2677: Stop of 'ora.pdmn.db' on 'exa102' succeeded
CRS-2677: Stop of 'ora.rmanst.db' on 'exa102' succeeded
CRS-2673: Attempting to stop 'ora.DATA_EXA1.dg' on 'exa102'
CRS-2673: Attempting to stop 'ora.RECO_EXA1.dg' on 'exa102'
CRS-2677: Stop of 'ora.RECO_EXA1.dg' on 'exa102' succeeded
CRS-2677: Stop of 'ora.DATA_EXA1.dg' on 'exa102' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'exa102' succeeded
CRS-2677: Stop of 'ora.DBFS_DG.dg' on 'exa102' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'exa102'
CRS-2677: Stop of 'ora.asm' on 'exa102' succeeded
CRS-2673: Attempting to stop 'ora.net2.network' on 'exa102'
CRS-2673: Attempting to stop 'ora.ons' on 'exa102'
CRS-2677: Stop of 'ora.net2.network' on 'exa102' succeeded
CRS-2677: Stop of 'ora.ons' on 'exa102' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'exa102'
CRS-2677: Stop of 'ora.net1.network' on 'exa102' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'exa102' has completed
CRS-2677: Stop of 'ora.crsd' on 'exa102' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'exa102'
CRS-2673: Attempting to stop 'ora.evmd' on 'exa102'
CRS-2673: Attempting to stop 'ora.asm' on 'exa102'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'exa102'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'exa102'
CRS-2677: Stop of 'ora.evmd' on 'exa102' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'exa102' succeeded
CRS-2677: Stop of 'ora.asm' on 'exa102' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'exa102'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'exa102' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'exa102' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'exa102' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'exa102'
CRS-2677: Stop of 'ora.cssd' on 'exa102' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'exa102'
CRS-2673: Attempting to stop 'ora.crf' on 'exa102'
CRS-2677: Stop of 'ora.diskmon' on 'exa102' succeeded
CRS-2677: Stop of 'ora.crf' on 'exa102' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'exa102'
CRS-2677: Stop of 'ora.gipcd' on 'exa102' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'exa102'
CRS-2677: Stop of 'ora.gpnpd' on 'exa102' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'exa102' has completed
CRS-4133: Oracle High Availability Services has been stopped.
5.1.3 Shut down all cell services on all cells to be patched. This may be done by the root user on each cell by running cellcli -e 'alter cell shutdown services all' or by the following dcli command to do all cells at the same time:
[root@exa101 ~]# dcli -g cell_group -l root "cellcli -e alter cell shutdown services all"
exa1cel01:
exa1cel01: Stopping the RS, CELLSRV, and MS services...
exa1cel01: The SHUTDOWN of services was successful.
exa1cel02:
exa1cel02: Stopping the RS, CELLSRV, and MS services...
exa1cel02: The SHUTDOWN of services was successful.
exa1cel03:
exa1cel03: Stopping the RS, CELLSRV, and MS services...
exa1cel03: The SHUTDOWN of services was successful.
[root@exa101 ~]#
5.2 Applying the Patch to Exadata Cells
5.2.1 Note below cases:
-Do not use the serial console, the ILOM or LO web-based console to start the patchmgr utility. You are using the serial console if the output
from the following command is consoletype=serial.
[root@exa101 ~]# echo $consoletype
pty
For Oracle Exadata Database Machine with Sun hardware cells: Use SSH to the ILOM host name or IP address as the root user. Do the following to
start the serial console,
start /SP/console
To stop it press the Escape key (ESC) followed by ( and then stop /SP/console.
-Use a fresh log in session for each patch or rollback procedure. Do not run the rollback procedure from the same login session that the patch
was applied. Do not re-run a patch application from a login session in which the rollback procedure was run.
-Do not interrupt the patching procedure once it is initiated on a cell.
-Cells automatically reboot, as needed, during patch application. Do not reboot or power cycle cells while applying the patch.
-Do not edit any log file or open log files in writable mode. You may use any of the following to view a log: view, less, more or tail. You may
cause the patch application to be interrupted by editing its logs while applying the patch.
-Monitor patch activity using less -rf patchmgr.stdout from another terminal session or window to see raw log details from the patchmgr utility.
5.2.2 Patching Using the patchmgr Utility
Unzip the patch, it will extract into the patch_11.2.3.3.0.131014.1 directory. Change to the patch_11.2.3.3.0.131014.1 directory.
[root@exa101 ~]# unzip p16278923_112330_Linux-x86-64.zip
Archive: p16278923_112330_Linux-x86-64.zip
creating: patch_11.2.3.3.0.131014.1/
inflating: patch_11.2.3.3.0.131014.1/sundcs_36p_repository_2.1.3_4.pkg
inflating: patch_11.2.3.3.0.131014.1/upgradeIBSwitch.sh
creating: patch_11.2.3.3.0.131014.1/linux.db.rpms/
inflating: patch_11.2.3.3.0.131014.1/linux.db.rpms/perl-XML-Parser-2.34-6.1.2.2.1.x86_64.rpm
creating: patch_11.2.3.3.0.131014.1/plugins/
inflating: patch_11.2.3.3.0.131014.1/plugins/000-check_dummy_bash
inflating: patch_11.2.3.3.0.131014.1/plugins/000-check_dummy_perl
inflating: patch_11.2.3.3.0.131014.1/imageLogger
inflating: patch_11.2.3.3.0.131014.1/ExadataSendNotification.pm
inflating: patch_11.2.3.3.0.131014.1/11.2.3.3.0.131014.1.patch.tar
inflating: patch_11.2.3.3.0.131014.1/dostep.sh
inflating: patch_11.2.3.3.0.131014.1/patchmgr
inflating: patch_11.2.3.3.0.131014.1/ExaXMLNode.pm
inflating: patch_11.2.3.3.0.131014.1/README.txt
inflating: patch_11.2.3.3.0.131014.1/dcli
creating: patch_11.2.3.3.0.131014.1/etc/
creating: patch_11.2.3.3.0.131014.1/etc/config/
inflating: patch_11.2.3.3.0.131014.1/etc/config/inventory.xml
inflating: patch_11.2.3.3.0.131014.1/README.html
inflating: patch_11.2.3.3.0.131014.1/sundcs_36p_repository_1.3.3_2.pkg
creating: patch_11.2.3.3.0.131014.1/ibdiagtools/
creating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/Torus.pm
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/QuarterRack.pm
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/FatTree.pm
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/HalfRack.pm
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/fetchTopology.pm
creating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/
creating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/sscHalfRack/
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/sscHalfRack/ib_switches.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/sscHalfRack/ibnetdiscover.map
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/sscHalfRack/ib_hosts.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/sscHalfRack/iblinkinfo.log
creating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/quarterrack/
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/quarterrack/ib_switches.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/quarterrack/ibnetdiscover.map
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/quarterrack/ib_hosts.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/quarterrack/iblinkinfo.log
creating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/halfrack/
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/halfrack/ib_switches.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/halfrack/ibnetdiscover.map
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/halfrack/ib_hosts.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/halfrack/iblinkinfo.log
creating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/fullX2Rack/
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/fullX2Rack/ib_switches.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/fullX2Rack/ibnetdiscover.map
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/fullX2Rack/ib_hosts.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/fullX2Rack/iblinkinfo.log
creating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/quarterrackError/
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/quarterrackError/ib_switches.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/quarterrackError/ibnetdiscover.map
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/quarterrackError/ib_hosts.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/quarterrackError/iblinkinfo.log
creating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/storageOnlyHalf/
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/storageOnlyHalf/ib_switches.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/storageOnlyHalf/ibnetdiscover.map
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/storageOnlyHalf/ib_hosts.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/storageOnlyHalf/iblinkinfo.log
creating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/storageOnlyFull/
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/storageOnlyFull/ib_switches.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/storageOnlyFull/ibnetdiscover.map
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/storageOnlyFull/ib_hosts.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/storageOnlyFull/iblinkinfo.log
creating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/sscFullRack/
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/sscFullRack/ib_switches.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/sscFullRack/ibnetdiscover.map
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/sscFullRack/ib_hosts.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/sscFullRack/iblinkinfo.log
creating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/twoX2racks/
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/twoX2racks/ib_switches.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/twoX2racks/ibnetdiscover.map
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/twoX2racks/ib_hosts.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/twoX2racks/iblinkinfo.log
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/mapCheck.pm
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/fattree.map
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/nm2ports.map
extracting: patch_11.2.3.3.0.131014.1/ibdiagtools/xmonib.sh
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/infinicheck
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/dcli
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/verify-topology
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topology-zfs
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/README
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/setup-ssh
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/SampleOutputs.txt
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/ibping_test
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/monitord
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/tar_ibdiagtools
extracting: patch_11.2.3.3.0.131014.1/ibdiagtools/VERSION_FILE
creating: patch_11.2.3.3.0.131014.1/ibdiagtools/netcheck/
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/netcheck/LinuxAdapter.pm
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/netcheck/CommonUtils.pm
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/netcheck/OSAdapter.pm
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/netcheck/spawnProc.pm
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/netcheck/remoteLauncher.pm
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/netcheck/remoteScriptGenerator.pm
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/netcheck/remoteConfig.pm
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/netcheck/SolarisAdapter.pm
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/netcheck/runDiagnostics.pm
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/checkbadlinks.pl
inflating: patch_11.2.3.3.0.131014.1/11.2.3.3.0.131014.1.iso
inflating: patch_11.2.3.3.0.131014.1/ExadataImageNotification.pl
inflating: patch_11.2.3.3.0.131014.1/exadataLogger.pm
[root@exa101 ~]#
Download any patchmgr plug-ins attached to the My Oracle Support note 1487339.1, and install them as documented in the My Oracle Support note.
Oracle recommends reviewing the My Oracle Support note, issues and workarounds listed in the note just before starting to actually apply the
patch.
In our case it is p17938410_112330_Linux-x86-64.zip.
[root@exa101 ~]# cd /u01/tmpstage/
[root@exa101 tmpstage]# ls -l
total 7598496
drwxr-xr-x 5 3849 42424 4096 Jan 14 07:53 17816100
-rw-r--r-- 1 root root 1258291200 Jan 14 10:00 17816100.tar.splitaa
-rw-r--r-- 1 root root 1258291200 Jan 14 10:20 17816100.tar.splitab
-rw-r--r-- 1 root root 1258291200 Jan 14 10:39 17816100.tar.splitac
-rw-r--r-- 1 root root 118661120 Jan 14 10:41 17816100.tar.splitad
-rw-r--r-- 1 root root 1258297983 Apr 14 12:21 p17816100_112030_Linux-x86-64_1of4.zip
-rw-r--r-- 1 root root 1258494785 Apr 14 12:11 p17816100_112030_Linux-x86-64_2of4.zip
-rw-r--r-- 1 root root 1246102503 Apr 14 02:56 p17816100_112030_Linux-x86-64_3of4.zip
-rw-r--r-- 1 root root 116728120 Apr 13 23:16 p17816100_112030_Linux-x86-64_4of4.zip
-rw-r--r-- 1 root root 1774 Apr 17 10:54 p17938410_112330_Linux-x86-64.zip
-rw-rw-r-- 1 root root 3602 Jan 15 07:46 PatchSearch.xml
-rw-r--r-- 1 root root 28716 Jan 14 10:56 README.html
-rw-r--r-- 1 root root 9650 Jan 14 10:56 README.txt
[root@exa101 tmpstage]# unzip -d /root/patch_11.2.3.3.0.131014.1/plugins/ p17938410_112330_Linux-x86-64.zip -x Readme.txt
Archive: p17938410_112330_Linux-x86-64.zip
inflating: /root/patch_11.2.3.3.0.131014.1/plugins/001-post_11_2_3_3_0
[root@exa101 tmpstage]# cd /root/patch_11.2.3.3.0.131014.1/plugins/
[root@exa101 plugins]# ls -l
total 12
-r-xr-xr-x 1 root root 3986 Oct 15 2013 000-check_dummy_bash
-r-xr-xr-x 1 root root 3370 Oct 15 2013 000-check_dummy_perl
-rw-r--r-- 1 root root 3505 Dec 11 15:16 001-post_11_2_3_3_0
[root@exa101 plugins]# pwd
/root/patch_11.2.3.3.0.131014.1/plugins
[root@exa101 plugins]# chmod +x *
[root@exa101 plugins]# ls -l
total 12
-r-xr-xr-x 1 root root 3986 Oct 15 2013 000-check_dummy_bash
-r-xr-xr-x 1 root root 3370 Oct 15 2013 000-check_dummy_perl
-rwxr-xr-x 1 root root 3505 Dec 11 15:16 001-post_11_2_3_3_0
Recommended: Reset the server to a known state using the following command:
This step is only done the first time the cells are patched to this release. It is not necessary to use the command for subsequent cell patching,
even after rolling back the patch.
[root@exa101]# cd /root/patch_11.2.3.3.0.131014.1/
[root@exa101 patch_11.2.3.3.0.131014.1]# ./patchmgr -cells cell_group -reset_force
2014-04-24 20:42:19 +0400 :DONE: reset_force
Clean up any previous patchmgr utility runs using the following command: (As I told in the beginning, use fresh session for each patchmgr
session)
[root@exa101 patch_11.2.3.3.0.131014.1]# ./patchmgr -cells cell_group -cleanup
2014-04-24 20:44:38 +0400 :DONE: Cleanup
Verify that the cells meet prerequisite checks using the following command. Use the -rolling option if you plan to use rolling updates:
[root@exa101 patch_11.2.3.3.0.131014.1]# ./patchmgr -cells cell_group -patch_check_prereq
date: invalid date `%F %T %z'
2014-04-24 20:47:03 +0400 :Working: DO: Check cells have ssh equivalence for root user. Up to 10 seconds per cell ...
2014-04-24 20:47:05 +0400 :SUCCESS: DONE: Check cells have ssh equivalence for root user.
2014-04-24 20:47:05 +0400 :Working: DO: Initialize files, check space and state of cell services. Up to 1 minute ...
2014-04-24 20:48:05 +0400 :SUCCESS: DONE: Initialize files, check space and state of cell services.
2014-04-24 20:48:05 +0400 :Working: DO: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size
correction. Up to 40 minutes ...
2014-04-24 20:48:19 +0400 Wait correction of degraded md11 due to md partner size mismatch. Up to 30 minutes.
2014-04-24 20:48:20 +0400 :SUCCESS: DONE: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner
size correction.
2014-04-24 20:48:20 +0400 :Working: DO: Check prerequisites on all cells. Up to 2 minutes ...
2014-04-24 20:49:04 +0400 :SUCCESS: DONE: Check prerequisites on all cells.
2014-04-24 20:49:04 +0400 :Working: DO: Execute plugin check for Patch Check Prereq ...
2014-04-24 20:49:05 +0400 :SUCCESS: DONE: Execute plugin check for Patch Check Prereq.
Output from patchmgr:
[root@exa101 patch_11.2.3.3.0.131014.1]# tail -f patchmgr.stdout
2014-04-24 20:42:19 +0400 :DONE: reset_force
================PatchMgr run ended 2014-04-24 20:42:19 +0400 ===========
================PatchMgr run started 2014-04-24 20:43:37 +0400 ===========
2014-04-24 20:44:38 +0400 :DONE: Cleanup
================PatchMgr run ended 2014-04-24 20:44:44 +0400 ===========
================PatchMgr run started 2014-04-24 20:46:01 +0400 ===========
[INFO]: Install rpm /root/patch_11.2.3.3.0.131014.1/linux.db.rpms/perl-XML-Parser-2.34-6.1.2.2.1.x86_64.rpm
Preparing... ##################################################
perl-XML-Parser ##################################################
2014-04-24 20:47:03 +0400 :Working: DO: Check cells have ssh equivalence for root user. Up to 10 seconds per cell ...
2014-04-24 20:47:04 +0400 ++++++++++++++++++ Logs so far begin ++++++++++
2014-04-24 20:47:04 +0400 ++++++++++++++++++ Logs so far end ++++++++++
2014-04-24 20:47:05 +0400 :SUCCESS: DONE: Check cells have ssh equivalence for root user.
2014-04-24 20:47:05 +0400 :Working: DO: Initialize files, check space and state of cell services. Up to 1 minute ...
2014-04-24 20:48:05 +0400 ++++++++++++++++++ Logs so far begin ++++++++++
exa1cel01:
exa1cel01: 2014-04-24 20:47:10 +0400 [INFO]: patchmgr launch attempt from exa101.bog.ge_10.0.0.150_root_patch_11.2.3.3.0.131014.1.
exa1cel01: 2014-04-24 20:47:10 +0400 [INFO]: dostep called: prechk:no:1406 patch_prereq -no-auto-rollback non_rolling 600 3600 900 36000 default
noforce
exa1cel01: 2014-04-24 20:47:10 +0400 [INFO]: patchmgr launched from exa101.bog.ge_10.0.0.150_root_patch_11.2.3.3.0.131014.1
exa1cel01: _EXIT_PASS_Cell exa1cel01 10.45.33.4 2014-04-24 20:48:05 +0400:
exa1cel02:
exa1cel02: 2014-04-24 20:47:09 +0400 [INFO]: patchmgr launch attempt from exa101.bog.ge_10.0.0.150_root_patch_11.2.3.3.0.131014.1.
exa1cel02: 2014-04-24 20:47:09 +0400 [INFO]: dostep called: prechk:no:1406 patch_prereq -no-auto-rollback non_rolling 600 3600 900 36000 default
noforce
exa1cel02: 2014-04-24 20:47:09 +0400 [INFO]: patchmgr launched from exa101.bog.ge_10.0.0.150_root_patch_11.2.3.3.0.131014.1
exa1cel02: _EXIT_PASS_Cell exa1cel02 10.45.33.5 2014-04-24 20:48:04 +0400:
exa1cel03:
exa1cel03: 2014-04-24 20:47:10 +0400 [INFO]: patchmgr launch attempt from exa101.bog.ge_10.0.0.150_root_patch_11.2.3.3.0.131014.1.
exa1cel03: 2014-04-24 20:47:10 +0400 [INFO]: dostep called: prechk:no:1406 patch_prereq -no-auto-rollback non_rolling 600 3600 900 36000 default
noforce
exa1cel03: 2014-04-24 20:47:10 +0400 [INFO]: patchmgr launched from exa101.bog.ge_10.0.0.150_root_patch_11.2.3.3.0.131014.1
exa1cel03: _EXIT_PASS_Cell exa1cel03 10.45.33.6 2014-04-24 20:48:04 +0400:
exa1cel01:
exa1cel01: _EXIT_PASS_Cell exa1cel01 10.45.33.4
exa1cel02:
exa1cel02: _EXIT_PASS_Cell exa1cel02 10.45.33.5
exa1cel03:
exa1cel03: _EXIT_PASS_Cell exa1cel03 10.45.33.6
exa1cel01:
exa1cel01: [INFO] Free space in /boot (/dev/md4) before clean up is 58Mb
exa1cel01: [INFO] Kernel version on runtime: 2.6.32-400.11.1.el5uek
exa1cel01: [INFO] Kernel version in /opt/oracle.cellos/image.id: 2.6.32-400.11.1.el5uek
exa1cel01: [INFO] Free space in /boot (/dev/md4) after clean up is 58Mb
exa1cel01: [INFO] Size for all files for the kernel 2.6.32-400.11.1.el5uek in /boot is 25Mb
exa1cel01: [INFO] Required free space on /boot is 50Mb
exa1cel01: Cell exa1cel01 10.45.33.4
exa1cel01: _EXIT_PASS_Cell exa1cel01 10.45.33.4 2014-04-24 20:48:05 +0400:
exa1cel02:
exa1cel02: [INFO] Free space in /boot (/dev/md4) before clean up is 58Mb
exa1cel02: [INFO] Kernel version on runtime: 2.6.32-400.11.1.el5uek
exa1cel02: [INFO] Kernel version in /opt/oracle.cellos/image.id: 2.6.32-400.11.1.el5uek
exa1cel02: [INFO] Free space in /boot (/dev/md4) after clean up is 58Mb
exa1cel02: [INFO] Size for all files for the kernel 2.6.32-400.11.1.el5uek in /boot is 25Mb
exa1cel02: [INFO] Required free space on /boot is 50Mb
exa1cel02: Cell exa1cel02 10.45.33.5
exa1cel02: _EXIT_PASS_Cell exa1cel02 10.45.33.5 2014-04-24 20:48:04 +0400:
exa1cel03:
exa1cel03: [INFO] Free space in /boot (/dev/md4) before clean up is 58Mb
exa1cel03: [INFO] Kernel version on runtime: 2.6.32-400.11.1.el5uek
exa1cel03: [INFO] Kernel version in /opt/oracle.cellos/image.id: 2.6.32-400.11.1.el5uek
exa1cel03: [INFO] Free space in /boot (/dev/md4) after clean up is 58Mb
exa1cel03: [INFO] Size for all files for the kernel 2.6.32-400.11.1.el5uek in /boot is 25Mb
exa1cel03: [INFO] Required free space on /boot is 50Mb
exa1cel03: Cell exa1cel03 10.45.33.6
exa1cel03: _EXIT_PASS_Cell exa1cel03 10.45.33.6 2014-04-24 20:48:04 +0400:
2014-04-24 20:48:05 +0400 ++++++++++++++++++ Logs so far end ++++++++++
2014-04-24 20:48:05 +0400 :SUCCESS: DONE: Initialize files, check space and state of cell services.
2014-04-24 20:48:05 +0400 :Working: DO: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size
correction. Up to 40 minutes ...
2014-04-24 20:48:19 +0400 ++++++++++++++++++ Logs so far begin ++++++++++
exa1cel01:
exa1cel01: 2014-04-24 20:48:08 +0400 [INFO]: patchmgr launch attempt from exa101.bog.ge_10.0.0.150_root_patch_11.2.3.3.0.131014.1.
exa1cel01: 2014-04-24 20:48:08 +0400 [INFO]: dostep called: 4p patch_prereq -no-auto-rollback non_rolling 600 3600 900 36000 default
exa1cel01: _EXIT_PASS_Cell exa1cel01 10.45.33.4 2014-04-24 20:48:09 +0400:
exa1cel01:
exa1cel01: 2014-04-24 20:48:19 +0400 [INFO]: patchmgr launch attempt from exa101.bog.ge_10.0.0.150_root_patch_11.2.3.3.0.131014.1.
exa1cel01: 2014-04-24 20:48:19 +0400 [INFO]: dostep called: 4p_md11_fix_pw patch_prereq -no-auto-rollback non_rolling 600 3600 900 36000 default
exa1cel01: _EXIT_PASS_Cell exa1cel01 10.45.33.4 2014-04-24 20:48:19 +0400:
exa1cel02:
exa1cel02: 2014-04-24 20:48:08 +0400 [INFO]: patchmgr launch attempt from exa101.bog.ge_10.0.0.150_root_patch_11.2.3.3.0.131014.1.
exa1cel02: 2014-04-24 20:48:08 +0400 [INFO]: dostep called: 4p patch_prereq -no-auto-rollback non_rolling 600 3600 900 36000 default
exa1cel02: _EXIT_PASS_Cell exa1cel02 10.45.33.5 2014-04-24 20:48:08 +0400:
exa1cel02:
exa1cel02: 2014-04-24 20:48:19 +0400 [INFO]: patchmgr launch attempt from exa101.bog.ge_10.0.0.150_root_patch_11.2.3.3.0.131014.1.
exa1cel02: 2014-04-24 20:48:19 +0400 [INFO]: dostep called: 4p_md11_fix_pw patch_prereq -no-auto-rollback non_rolling 600 3600 900 36000 default
exa1cel02: _EXIT_PASS_Cell exa1cel02 10.45.33.5 2014-04-24 20:48:19 +0400:
exa1cel03:
exa1cel03: 2014-04-24 20:48:08 +0400 [INFO]: patchmgr launch attempt from exa101.bog.ge_10.0.0.150_root_patch_11.2.3.3.0.131014.1.
exa1cel03: 2014-04-24 20:48:08 +0400 [INFO]: dostep called: 4p patch_prereq -no-auto-rollback non_rolling 600 3600 900 36000 default
exa1cel03: _EXIT_PASS_Cell exa1cel03 10.45.33.6 2014-04-24 20:48:09 +0400:
exa1cel03:
exa1cel03: 2014-04-24 20:48:19 +0400 [INFO]: patchmgr launch attempt from exa101.bog.ge_10.0.0.150_root_patch_11.2.3.3.0.131014.1.
exa1cel03: 2014-04-24 20:48:19 +0400 [INFO]: dostep called: 4p_md11_fix_pw patch_prereq -no-auto-rollback non_rolling 600 3600 900 36000 default
exa1cel03: _EXIT_PASS_Cell exa1cel03 10.45.33.6 2014-04-24 20:48:19 +0400:
exa1cel01: Cell exa1cel01 10.45.33.4
exa1cel01: _EXIT_PASS_Cell exa1cel01 10.45.33.4 2014-04-24 20:48:19 +0400:
exa1cel02: Cell exa1cel02 10.45.33.5
exa1cel02: _EXIT_PASS_Cell exa1cel02 10.45.33.5 2014-04-24 20:48:19 +0400:
exa1cel03: Cell exa1cel03 10.45.33.6
exa1cel03: _EXIT_PASS_Cell exa1cel03 10.45.33.6 2014-04-24 20:48:19 +0400:
2014-04-24 20:48:20 +0400 ++++++++++++++++++ Logs so far end ++++++++++
2014-04-24 20:48:20 +0400 :SUCCESS: DONE: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner
size correction.
2014-04-24 20:48:20 +0400 :Working: DO: Check prerequisites on all cells. Up to 2 minutes ...
2014-04-24 20:49:03 +0400 ++++++++++++++++++ Logs so far begin ++++++++++
exa1cel01:
exa1cel01: 2014-04-24 20:48:20 +0400 [INFO]: patchmgr launch attempt from exa101.bog.ge_10.0.0.150_root_patch_11.2.3.3.0.131014.1.
exa1cel01: 2014-04-24 20:48:20 +0400 [INFO]: dostep called: 5p patch_prereq -no-auto-rollback non_rolling 600 3600 900 36000 default
exa1cel01: _EXIT_PASS_Cell exa1cel01 10.45.33.4 2014-04-24 20:49:03 +0400: 2014-04-24 20:49:03 +0400 [INFO] Started to patch.
exa1cel02:
exa1cel02: 2014-04-24 20:48:20 +0400 [INFO]: patchmgr launch attempt from exa101.bog.ge_10.0.0.150_root_patch_11.2.3.3.0.131014.1.
exa1cel02: 2014-04-24 20:48:20 +0400 [INFO]: dostep called: 5p patch_prereq -no-auto-rollback non_rolling 600 3600 900 36000 default
exa1cel02: _EXIT_PASS_Cell exa1cel02 10.45.33.5 2014-04-24 20:49:03 +0400: 2014-04-24 20:49:02 +0400 [INFO] Started to patch.
exa1cel03:
exa1cel03: 2014-04-24 20:48:20 +0400 [INFO]: patchmgr launch attempt from exa101.bog.ge_10.0.0.150_root_patch_11.2.3.3.0.131014.1.
exa1cel03: 2014-04-24 20:48:20 +0400 [INFO]: dostep called: 5p patch_prereq -no-auto-rollback non_rolling 600 3600 900 36000 default
exa1cel03: _EXIT_PASS_Cell exa1cel03 10.45.33.6 2014-04-24 20:49:03 +0400: 2014-04-24 20:49:03 +0400 [INFO] Started to patch.
exa1cel01: 2014-04-24 20:48:20 +0400
exa1cel01: 2014-04-24 20:48:20 +0400 ./install.sh started with command line: -patch
exa1cel01: 2014-04-24 20:48:21 +0400 [INFO] Check md devices have clean state ...
exa1cel01: 2014-04-24 20:48:21 +0400 [INFO] Check disks and partitions ...
exa1cel01: Active image info
exa1cel01: Version : 11.2.3.2.1.130109
exa1cel01: System partition on device : /dev/md6
exa1cel01: Software partition on device : /dev/md8
exa1cel01: Cell boot usb version : 11.2.3.2.1.130109
exa1cel01:
exa1cel01: Patch proposal
exa1cel01: Version : 11.2.3.3.0.131014.1
exa1cel01: System partition on device : /dev/md5
exa1cel01: Software partition on device : /dev/md7
exa1cel01: Upgrade mode : patch
exa1cel01: Zero auto rollback : enabled
exa1cel01: Auto rollback : enabled
exa1cel01: Reset SP : yes
exa1cel01: First boot device : usb
exa1cel01: Use usb on upgrade : no
exa1cel01: Last console in hd grub.conf : web
exa1cel01: Stop on zero boot : disabled
exa1cel01: Redirection log file :
exa1cel01:
exa1cel01: 2014-04-24 20:48:27 +0400 [INFO] Shut down MS before doing ipmitool commands.
exa1cel01:
exa1cel01: Stopping MS services...
exa1cel01: The SHUTDOWN of MS services was successful.
exa1cel01: 2014-04-24 20:48:35 +0400 [INFO] Service Processor (SP) is available
exa1cel01: 2014-04-24 20:48:35 +0400 [INFO] Start MS after doing ipmitool commands.
exa1cel01:
exa1cel01: Starting MS services...
exa1cel01: The STARTUP of MS services was successful.
exa1cel01:
exa1cel01: Stopping MS services...
exa1cel01: The SHUTDOWN of MS services was successful.
exa1cel01: 2014-04-24 20:48:50 +0400 [INFO] No running instance of checkdeveachboot found. Safe to proceed
exa1cel01: 2014-04-24 20:48:50 +0400 [INFO] Check healthy states of Bootable or/and MD device(s)
exa1cel01:
exa1cel01: /opt/oracle.cellos/validations/init.d/checkdeveachboot started at 2014-04-24 20:48:50
exa1cel01: Set default check mode
exa1cel01: Set lock /tmp/cellos/locks/4636.checkdeveachboot
exa1cel01: Check mode is in use
exa1cel01: Check system disks for the cell node
exa1cel01: lsi controller has disks that are mapped to: /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj
/dev/sdk /dev/sdl
exa1cel01: The current system disks with valid partition table: /dev/sda /dev/sdb
exa1cel01: System disk /dev/sda is a bootable device
exa1cel01: Partition /dev/sda1 has a proper boot signature
exa1cel01: System disk /dev/sdb is a bootable device
exa1cel01: Partition /dev/sdb1 has a proper boot signature
exa1cel01: Check for degraded or stopped md devices
exa1cel01: Check CELLBOOT USB
exa1cel01: Check CELLBOOT USB
exa1cel01: USB partition /dev/sdm1 has a valid boot signature for CELLBOOT usb
exa1cel01: USB device /dev/sdm is a bootable device
exa1cel01: USB partition /dev/sdm1 is mountable
exa1cel01: USB has version 11.2.3.2.1.130109
exa1cel01: Reset lock /tmp/cellos/locks/4636.checkdeveachboot
exa1cel01: 0:0:0:0
exa1cel01:
exa1cel01: Starting MS services...
exa1cel01: The STARTUP of MS services was successful.
exa1cel01: /root/_patch_hctap_/_p_/biosbootorder started at 2014-04-24_20:48:59
exa1cel01: Check and set BIOS boot order
exa1cel01: 2014-04-24 20:48:59 +0400 CELLBOOT USB partition: /dev/sdm1
exa1cel01: 2014-04-24 20:48:59 +0400 [INFO] CELLBOOT USB block device: /sys/block/sdm
exa1cel01: 2014-04-24 20:48:59 +0400 CELLBOOT USB vendor: ORACLE
exa1cel01: 2014-04-24 20:48:59 +0400 [INFO] CELLBOOT USB model: UNIGEN-UFD
exa1cel01: 2014-04-24 20:48:59 +0400 [INFO] Looking for CELLBOOT USB pattern "(USB:Port0:ORACLE|USB:.*ORACLE .*UNIGEN-UFD.*)" in BIOS boot order
exa1cel01: 2014-04-24 20:48:59 +0400 [INFO] Looking for RAID pattern "RAID:" in BIOS boot order
exa1cel01:
exa1cel01: Copyright (C) SUN Microsystems 2009.
exa1cel01: BIOSconfig Utility Version 2.2.1
exa1cel01: Build Date: Aug 24 2009
exa1cel01: Build Time: 09:01:30
exa1cel01:
exa1cel01: BIOSconfig Specification Version 2.4
exa1cel01:
exa1cel01: Processing Input BIOS Data....
exa1cel01:
exa1cel01: Success
exa1cel01: 2014-04-24 20:49:03 +0400 [INFO] The current BIOS boot order:
exa1cel01:
exa1cel01:
exa1cel01:
exa1cel01:
exa1cel01:
exa1cel01:
exa1cel01:
exa1cel01:
exa1cel01:
exa1cel01:
exa1cel01:
exa1cel01:
exa1cel01:
exa1cel01:
exa1cel01:
exa1cel01:
exa1cel01: 2014-04-24 20:49:03 +0400 [INFO] CELLBOOT USB has proper order 1
exa1cel01:
exa1cel01: 2014-04-24 20:49:03 +0400 [INFO] RAID controller has proper order 2
exa1cel01:
exa1cel01: 2014-04-24 20:49:03 +0400 [INFO] Boot device order is correct.
exa1cel01: Cell exa1cel01 10.45.33.4
exa1cel01: _EXIT_PASS_Cell exa1cel01 10.45.33.4 2014-04-24 20:49:03 +0400: 2014-04-24 20:49:03 +0400 [INFO] Started to patch.
exa1cel02: 2014-04-24 20:48:20 +0400
exa1cel02: 2014-04-24 20:48:20 +0400 ./install.sh started with command line: -patch
exa1cel02: 2014-04-24 20:48:20 +0400 [INFO] Check md devices have clean state ...
exa1cel02: 2014-04-24 20:48:21 +0400 [INFO] Check disks and partitions ...
exa1cel02: Active image info
exa1cel02: Version : 11.2.3.2.1.130109
exa1cel02: System partition on device : /dev/md6
exa1cel02: Software partition on device : /dev/md8
exa1cel02: Cell boot usb version : 11.2.3.2.1.130109
exa1cel02:
exa1cel02: Patch proposal
exa1cel02: Version : 11.2.3.3.0.131014.1
exa1cel02: System partition on device : /dev/md5
exa1cel02: Software partition on device : /dev/md7
exa1cel02: Upgrade mode : patch
exa1cel02: Zero auto rollback : enabled
exa1cel02: Auto rollback : enabled
exa1cel02: Reset SP : yes
exa1cel02: First boot device : usb
exa1cel02: Use usb on upgrade : no
exa1cel02: Last console in hd grub.conf : web
exa1cel02: Stop on zero boot : disabled
exa1cel02: Redirection log file :
exa1cel02:
exa1cel02: 2014-04-24 20:48:27 +0400 [INFO] Shut down MS before doing ipmitool commands.
exa1cel02:
exa1cel02: Stopping MS services...
exa1cel02: The SHUTDOWN of MS services was successful.
exa1cel02: 2014-04-24 20:48:35 +0400 [INFO] Service Processor (SP) is available
exa1cel02: 2014-04-24 20:48:35 +0400 [INFO] Start MS after doing ipmitool commands.
exa1cel02:
exa1cel02: Starting MS services...
exa1cel02: The STARTUP of MS services was successful.
exa1cel02:
exa1cel02: Stopping MS services...
exa1cel02: The SHUTDOWN of MS services was successful.
exa1cel02: 2014-04-24 20:48:50 +0400 [INFO] No running instance of checkdeveachboot found. Safe to proceed
exa1cel02: 2014-04-24 20:48:50 +0400 [INFO] Check healthy states of Bootable or/and MD device(s)
exa1cel02:
exa1cel02: /opt/oracle.cellos/validations/init.d/checkdeveachboot started at 2014-04-24 20:48:50
exa1cel02: Set default check mode
exa1cel02: Set lock /tmp/cellos/locks/32222.checkdeveachboot
exa1cel02: Check mode is in use
exa1cel02: Check system disks for the cell node
exa1cel02: lsi controller has disks that are mapped to: /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj
/dev/sdk /dev/sdl
exa1cel02: The current system disks with valid partition table: /dev/sda /dev/sdb
exa1cel02: System disk /dev/sda is a bootable device
exa1cel02: Partition /dev/sda1 has a proper boot signature
exa1cel02: System disk /dev/sdb is a bootable device
exa1cel02: Partition /dev/sdb1 has a proper boot signature
exa1cel02: Check for degraded or stopped md devices
exa1cel02: Check CELLBOOT USB
exa1cel02: Check CELLBOOT USB
exa1cel02: USB partition /dev/sdm1 has a valid boot signature for CELLBOOT usb
exa1cel02: USB device /dev/sdm is a bootable device
exa1cel02: USB partition /dev/sdm1 is mountable
exa1cel02: USB has version 11.2.3.2.1.130109
exa1cel02: Reset lock /tmp/cellos/locks/32222.checkdeveachboot
exa1cel02: 0:0:0:0
exa1cel02:
exa1cel02: Starting MS services...
exa1cel02: The STARTUP of MS services was successful.
exa1cel02: /root/_patch_hctap_/_p_/biosbootorder started at 2014-04-24_20:48:58
exa1cel02: Check and set BIOS boot order
exa1cel02: 2014-04-24 20:48:59 +0400 CELLBOOT USB partition: /dev/sdm1
exa1cel02: 2014-04-24 20:48:59 +0400 [INFO] CELLBOOT USB block device: /sys/block/sdm
exa1cel02: 2014-04-24 20:48:59 +0400 CELLBOOT USB vendor: ORACLE
exa1cel02: 2014-04-24 20:48:59 +0400 [INFO] CELLBOOT USB model: UNIGEN-UFD
exa1cel02: 2014-04-24 20:48:59 +0400 [INFO] Looking for CELLBOOT USB pattern "(USB:Port0:ORACLE|USB:.*ORACLE .*UNIGEN-UFD.*)" in BIOS boot order
exa1cel02: 2014-04-24 20:48:59 +0400 [INFO] Looking for RAID pattern "RAID:" in BIOS boot order
exa1cel02:
exa1cel02: Copyright (C) SUN Microsystems 2009.
exa1cel02: BIOSconfig Utility Version 2.2.1
exa1cel02: Build Date: Aug 24 2009
exa1cel02: Build Time: 09:01:30
exa1cel02:
exa1cel02: BIOSconfig Specification Version 2.4
exa1cel02:
exa1cel02: Processing Input BIOS Data....
exa1cel02:
exa1cel02: Success
exa1cel02: 2014-04-24 20:49:02 +0400 [INFO] The current BIOS boot order:
exa1cel02:
exa1cel02:
exa1cel02:
exa1cel02:
exa1cel02:
exa1cel02:
exa1cel02:
exa1cel02:
exa1cel02:
exa1cel02:
exa1cel02:
exa1cel02:
exa1cel02:
exa1cel02:
exa1cel02:
exa1cel02:
exa1cel02: 2014-04-24 20:49:02 +0400 [INFO] CELLBOOT USB has proper order 1
exa1cel02:
exa1cel02: 2014-04-24 20:49:02 +0400 [INFO] RAID controller has proper order 2
exa1cel02:
exa1cel02: 2014-04-24 20:49:02 +0400 [INFO] Boot device order is correct.
exa1cel02: Cell exa1cel02 10.45.33.5
exa1cel02: _EXIT_PASS_Cell exa1cel02 10.45.33.5 2014-04-24 20:49:03 +0400: 2014-04-24 20:49:02 +0400 [INFO] Started to patch.
exa1cel03: 2014-04-24 20:48:21 +0400
exa1cel03: 2014-04-24 20:48:21 +0400 ./install.sh started with command line: -patch
exa1cel03: 2014-04-24 20:48:21 +0400 [INFO] Check md devices have clean state ...
exa1cel03: 2014-04-24 20:48:21 +0400 [INFO] Check disks and partitions ...
exa1cel03: Active image info
exa1cel03: Version : 11.2.3.2.1.130109
exa1cel03: System partition on device : /dev/md6
exa1cel03: Software partition on device : /dev/md8
exa1cel03: Cell boot usb version : 11.2.3.2.1.130109
exa1cel03:
exa1cel03: Patch proposal
exa1cel03: Version : 11.2.3.3.0.131014.1
exa1cel03: System partition on device : /dev/md5
exa1cel03: Software partition on device : /dev/md7
exa1cel03: Upgrade mode : patch
exa1cel03: Zero auto rollback : enabled
exa1cel03: Auto rollback : enabled
exa1cel03: Reset SP : yes
exa1cel03: First boot device : usb
exa1cel03: Use usb on upgrade : no
exa1cel03: Last console in hd grub.conf : web
exa1cel03: Stop on zero boot : disabled
exa1cel03: Redirection log file :
exa1cel03:
exa1cel03: 2014-04-24 20:48:27 +0400 [INFO] Shut down MS before doing ipmitool commands.
exa1cel03:
exa1cel03: Stopping MS services...
exa1cel03: The SHUTDOWN of MS services was successful.
exa1cel03: 2014-04-24 20:48:35 +0400 [INFO] Service Processor (SP) is available
exa1cel03: 2014-04-24 20:48:35 +0400 [INFO] Start MS after doing ipmitool commands.
exa1cel03:
exa1cel03: Starting MS services...
exa1cel03: The STARTUP of MS services was successful.
exa1cel03:
exa1cel03: Stopping MS services...
exa1cel03: The SHUTDOWN of MS services was successful.
exa1cel03: 2014-04-24 20:48:50 +0400 [INFO] No running instance of checkdeveachboot found. Safe to proceed
exa1cel03: 2014-04-24 20:48:50 +0400 [INFO] Check healthy states of Bootable or/and MD device(s)
exa1cel03:
exa1cel03: /opt/oracle.cellos/validations/init.d/checkdeveachboot started at 2014-04-24 20:48:50
exa1cel03: Set default check mode
exa1cel03: Set lock /tmp/cellos/locks/28294.checkdeveachboot
exa1cel03: Check mode is in use
exa1cel03: Check system disks for the cell node
exa1cel03: lsi controller has disks that are mapped to: /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj
/dev/sdk /dev/sdl
exa1cel03: The current system disks with valid partition table: /dev/sda /dev/sdb
exa1cel03: System disk /dev/sda is a bootable device
exa1cel03: Partition /dev/sda1 has a proper boot signature
exa1cel03: System disk /dev/sdb is a bootable device
exa1cel03: Partition /dev/sdb1 has a proper boot signature
exa1cel03: Check for degraded or stopped md devices
exa1cel03: Check CELLBOOT USB
exa1cel03: Check CELLBOOT USB
exa1cel03: USB partition /dev/sdm1 has a valid boot signature for CELLBOOT usb
exa1cel03: USB device /dev/sdm is a bootable device
exa1cel03: USB partition /dev/sdm1 is mountable
exa1cel03: USB has version 11.2.3.2.1.130109
exa1cel03: Reset lock /tmp/cellos/locks/28294.checkdeveachboot
exa1cel03: 0:0:0:0
exa1cel03:
exa1cel03: Starting MS services...
exa1cel03: The STARTUP of MS services was successful.
exa1cel03: /root/_patch_hctap_/_p_/biosbootorder started at 2014-04-24_20:48:59
exa1cel03: Check and set BIOS boot order
exa1cel03: 2014-04-24 20:48:59 +0400 CELLBOOT USB partition: /dev/sdm1
exa1cel03: 2014-04-24 20:48:59 +0400 [INFO] CELLBOOT USB block device: /sys/block/sdm
exa1cel03: 2014-04-24 20:48:59 +0400 CELLBOOT USB vendor: ORACLE
exa1cel03: 2014-04-24 20:48:59 +0400 [INFO] CELLBOOT USB model: UNIGEN-UFD
exa1cel03: 2014-04-24 20:48:59 +0400 [INFO] Looking for CELLBOOT USB pattern "(USB:Port0:ORACLE|USB:.*ORACLE .*UNIGEN-UFD.*)" in BIOS boot order
exa1cel03: 2014-04-24 20:48:59 +0400 [INFO] Looking for RAID pattern "RAID:" in BIOS boot order
exa1cel03:
exa1cel03: Copyright (C) SUN Microsystems 2009.
exa1cel03: BIOSconfig Utility Version 2.2.1
exa1cel03: Build Date: Aug 24 2009
exa1cel03: Build Time: 09:01:30
exa1cel03:
exa1cel03: BIOSconfig Specification Version 2.4
exa1cel03:
exa1cel03: Processing Input BIOS Data....
exa1cel03:
exa1cel03: Success
exa1cel03: 2014-04-24 20:49:02 +0400 [INFO] The current BIOS boot order:
exa1cel03:
exa1cel03:
exa1cel03:
exa1cel03:
exa1cel03:
exa1cel03:
exa1cel03:
exa1cel03:
exa1cel03:
exa1cel03:
exa1cel03:
exa1cel03:
exa1cel03:
exa1cel03:
exa1cel03:
exa1cel03:
exa1cel03: 2014-04-24 20:49:02 +0400 [INFO] CELLBOOT USB has proper order 1
exa1cel03:
exa1cel03: 2014-04-24 20:49:03 +0400 [INFO] RAID controller has proper order 2
exa1cel03:
exa1cel03: 2014-04-24 20:49:03 +0400 [INFO] Boot device order is correct.
exa1cel03: Cell exa1cel03 10.45.33.6
exa1cel03: _EXIT_PASS_Cell exa1cel03 10.45.33.6 2014-04-24 20:49:03 +0400: 2014-04-24 20:49:03 +0400 [INFO] Started to patch.
2014-04-24 20:49:04 +0400 ++++++++++++++++++ Logs so far end ++++++++++
2014-04-24 20:49:04 +0400 :SUCCESS: DONE: Check prerequisites on all cells.
2014-04-24 20:49:04 +0400 :Working: DO: Execute plugin check for Patch Check Prereq ...
2014-04-24 20:49:04 +0400 :WARNING: patchmgr plugin /root/patch_11.2.3.3.0.131014.1/plugins/001-post_11_2_3_3_0.
2014-04-24 20:49:04 +0400 :INFO: patch stage:Patch Check Prereq.
2014-04-24 20:49:04 +0400 :INFO: cells:exa1cel01,exa1cel02,exa1cel03.
2014-04-24 20:49:04 +0400 :INFO: patch action:patch_prereq.
2014-04-24 20:49:04 +0400 :INFO: rolling mode:non-rolling.
2014-04-24 20:49:04 +0400 :INFO: label image version:11.2.3.3.0.131014.1.
2014-04-24 20:49:04 +0400 :INFO: patchmgr log:/root/patch_11.2.3.3.0.131014.1/patchmgr.stdout.
2014-04-24 20:49:04 +0400 ++++++++++++++++++ Logs so far begin ++++++++++
2014-04-24 20:49:05 +0400 ++++++++++++++++++ Logs so far end ++++++++++
2014-04-24 20:49:05 +0400 :SUCCESS: DONE: Execute plugin check for Patch Check Prereq.
================PatchMgr run ended 2014-04-24 20:49:05 +0400 ===========
So, we can start patching:
./patchmgr -cells cell_group -patch
Wait until it finishes!
At the end check the image version:
[root@exa1cel01 ~]# imageinfo
Kernel version: 2.6.39-400.126.1.el5uek #1 SMP Fri Sep 20 10:54:38 PDT 2013 x86_64
Cell version: OSS_11.2.3.3.0_LINUX.X64_131014.1
Cell rpm version: cell-11.2.3.3.0_LINUX.X64_131014.1-1
Active image version: 11.2.3.3.0.131014.1
Active image activated: 2014-04-24 22:09:25 +0400
Active image status: success
Active system partition on device: /dev/md5
Active software partition on device: /dev/md7
In partition rollback: Impossible
Cell boot usb partition: /dev/sdm1
Cell boot usb version: 11.2.3.3.0.131014.1
Inactive image version: 11.2.3.2.1.130109
Inactive image activated: 2013-05-25 10:41:59 +0400
Inactive image status: success
Inactive system partition on device: /dev/md6
Inactive software partition on device: /dev/md8
Boot area has rollback archive for the version: 11.2.3.2.1.130109
Rollback to the inactive partitions: Possible
[root@exa1cel01 ~]# imagehistory
Version : 11.2.3.1.1.120607
Image activation date : 2012-08-30 12:58:48 +0400
Imaging mode : fresh
Imaging status : success
Version : 11.2.3.2.1.130109
Image activation date : 2013-05-25 10:41:59 +0400
Imaging mode : out of partition upgrade
Imaging status : success
Version : 11.2.3.3.0.131014.1
Image activation date : 2014-04-24 22:09:25 +0400
Imaging mode : out of partition upgrade
Imaging status : success
[root@exa1cel01 ~]#
5.3 Updating Database Servers on Oracle Exadata Database Machine
Copy p16486998_121110_Linux-x86-64.zip to /root/ and unzip it:
[root@exa101 ~]# unzip p16486998_121110_Linux-x86-64.zip
Archive: p16486998_121110_Linux-x86-64.zip
inflating: dbnodeupdate.sh
inflating: dbupdate-helpers.zip
inflating: README.txt
[root@exa101 ~]# ./dbnodeupdate.sh -u -l /u01/tmpstage/17816100/Infrastructure/ExadataStorageServer/11.2.3.3.0/p17809253_112330_Linux-x86-64.zip
(*) 2014-04-24 22:39:42: Unzipping helpers (/root/dbupdate-helpers.zip) to /opt/oracle.SupportTools/dbnodeupdate_helpers
(*) 2014-04-24 22:39:42: Initializing logfile /var/log/cellos/dbnodeupdate.log
(*) 2014-04-24 22:39:42: Collecting system configuration details, this may take some time...
(*) 2014-04-24 22:39:51: Validating system details (known issues, best practices). This may take a while
(*) 2014-04-24 22:39:51: Checking free space in /u01
(*) 2014-04-24 22:39:51: Unzipping /u01/tmpstage/17816100/Infrastructure/ExadataStorageServer/11.2.3.3.0/p17809253_112330_Linux-x86-64.zip to
/u01/app/oracle/stage.240414223942, this may take a while
(*) 2014-04-24 22:39:59: Backing up /etc/yum.repos.d/Exadata-computenode.repo in /etc/yum.repos.d.bak/240414223942
(*) 2014-04-24 22:39:59: Backing up /etc/yum.repos.d/Exadata-computenode.repo.sample in /etc/yum.repos.d.bak/240414223942
(*) 2014-04-24 22:39:59: Backing up /etc/yum.repos.d/Exadata-computenode.repo.sample.backup in /etc/yum.repos.d.bak/240414223942
Warning: Network routing configuration requires change before updating database server. See MOS 1306154.1
Continue ? [y/N]
y
Active Image version : 11.2.3.2.1.130302
Active Kernel version : 2.6.18-308.24.1.0.1.el5
Active LVM Name : /dev/mapper/VGExaDb-LVDbSys1
Inactive Image version : 11.2.3.1.1.120607
Inactive LVM Name : /dev/mapper/VGExaDb-LVDbSys2
Current user id : root
Action : upgrade
Upgrading to : 11.2.3.3.0.131014.1
Baseurl : file:///var/www/html/yum/unknown/EXADATA/dbserver/240414223942/x86_64/ (iso)
Iso file : /u01/app/oracle/stage.240414223942/repoimage.iso
Create a backup : Yes
Shutdown stack : No (Currently stack is down)
RPM exclusion list : Not in use (add rpms to /etc/exadata/yum/exclusion.lst and restart dbnodeupdate.sh)
RPM obsolete list : /etc/exadata/yum/obsolete.lst (available post-update, lists rpms removed by the update)
Logfile : /var/log/cellos/dbnodeupdate.log (runid: 240414223942)
Diagfile : /var/log/cellos/dbnodeupdate.240414223942.diag
Server model : SUN FIRE X4170 M2 SERVER
dbnodeupdate.sh rel. : 2.39 (always check MOS 1553103.1 for the latest release of dbnodeupdate)
Automatic checks incl. : Known Issue - Database Server upgrades may hit network routing issues after the upgrade
: Issue - Yum rolling update requires fix for 11768055 when Grid Infrastructure is below 11.2.0.2 BP12
: Issue 1.7 - Updating database servers with customized partitions may remove partitions already in use
Note : After upgrading and rebooting run './dbnodeupdate.sh -c' to finish post steps
Continue ? [Y/n]
y
(*) 2014-04-24 22:40:34: Verifying GI and DB's are shutdown
(*) 2014-04-24 22:40:36: Collecting console history for diag purposes
(*) 2014-04-24 22:40:51: Unmount of /boot successful
(*) 2014-04-24 22:40:51: Check for /dev/sda1 successful
(*) 2014-04-24 22:40:51: Mount of /boot successful
(*) 2014-04-24 22:40:51: Disabling stack from starting
(*) 2014-04-24 22:40:51: Performing filesystem backup to /dev/mapper/VGExaDb-LVDbSys2. Avg. 30 minutes (max. 120) depends per
environment.........
(*) 2014-04-24 22:48:52: Backup successful
(*) 2014-04-24 22:48:52: Verifying and updating yum.conf (backup in /etc/yum.conf.240414_223942)
(*) 2014-04-24 22:48:52: Disabling other repositories, generating Exadata repos
(*) 2014-04-24 22:48:52: Generating /etc/yum.repos.d/Exadata-computenode.repo
(*) 2014-04-24 22:48:53: Verifying baseurl
(*) 2014-04-24 22:48:54: OSWatcher stopped successful
(*) 2014-04-24 22:48:54: Cleaning up the yum cache
(*) 2014-04-24 22:48:54: Removing rpm libcxgb3-static.x86_64 (if installed)
(*) 2014-04-24 22:48:54: Removing rpm rpm-build.x86_64 (if installed)
(*) 2014-04-24 22:48:54: Preparing update for releases 11.2.3.3.0 and later
(*) 2014-04-24 22:48:59: Preparation done succesfully
(*) 2014-04-24 22:48:59: Drive /dev/sda has more than two partitions. This is non-standard. If hotspare still exists it will not be reclaimed
(*) 2014-04-24 22:48:59: Performing yum update. Node is expected to reboot when finished
(*) 2014-04-24 22:50:56: Waiting for post rpm script to finish. Sleeping another 60 seconds (60 / 900)
Remote broadcast message (Thu Apr 24 22:51:08 2014):
Exadata post install steps started.
It may take up to 5 minutes.
(*) 2014-04-24 22:51:57: Waiting for post rpm script to finish. Sleeping another 60 seconds (120 / 900)
Remote broadcast message (Thu Apr 24 22:52:24 2014):
Exadata post install steps completed with success
(*) 2014-04-24 22:52:57: Waiting for post rpm script to finish. Sleeping another 60 seconds (180 / 900)
(*) 2014-04-24 22:53:57: All post steps are finished.
(*) 2014-04-24 22:53:57: System will reboot automatically for changes to take effect
(*) 2014-04-24 22:53:57: After reboot run "./dbnodeupdate.sh -c" to complete the upgrade
(*) 2014-04-24 22:53:57: Restoring /etc/yum.repos.d.bak/240414223942/Exadata-computenode.repo to /etc/yum.repos.d
(*) 2014-04-24 22:53:57: Restoring /etc/yum.repos.d.bak/240414223942/Exadata-computenode.repo.sample to /etc/yum.repos.d
(*) 2014-04-24 22:53:57: Restoring /etc/yum.repos.d.bak/240414223942/Exadata-computenode.repo.sample.backup to /etc/yum.repos.d
(*) 2014-04-24 22:54:01: Rebooting now...
Broadcast message from root (pts/2) (Thu Apr 24 22:54:01 2014):
The system is going down for reboot NOW!
Post installation:
[root@exa101 ~]# ./dbnodeupdate.sh -c
(*) 2014-04-24 23:28:48: Unzipping helpers (/root/dbupdate-helpers.zip) to /opt/oracle.SupportTools/dbnodeupdate_helpers
(*) 2014-04-24 23:28:48: Initializing logfile /var/log/cellos/dbnodeupdate.log
(*) 2014-04-24 23:28:49: Collecting system configuration details, this may take some time...
Active Image version : 11.2.3.3.0.131014.1
Active Kernel version : 2.6.39-400.126.1.el5uek
Active LVM Name : /dev/mapper/VGExaDb-LVDbSys1
Inactive Image version : 11.2.3.2.1.130302
Inactive LVM Name : /dev/mapper/VGExaDb-LVDbSys2
Current user id : root
Action : finish-post (perform post steps, relink enable/disable crs)
Shutdown stack : No (Currently stack is down)
Logfile : /var/log/cellos/dbnodeupdate.log (runid: 240414232848)
Diagfile : /var/log/cellos/dbnodeupdate.240414232848.diag
Server model : SUN FIRE X4170 M2 SERVER
dbnodeupdate.sh rel. : 2.39 (always check MOS 1553103.1 for the latest release of dbnodeupdate)
: Issue - Yum rolling update requires fix for 11768055 when Grid Infrastructure is below 11.2.0.2 BP12
: Issue 1.7 - Updating database servers with customized partitions may remove partitions already in use
Continue ? [Y/n]
y
(*) 2014-04-24 23:29:19: Verifying GI and DB's are shutdown
(*) 2014-04-24 23:29:21: Verifying firmware updates/validations. Maximum wait time: 30 minutes.
(*) 2014-04-24 23:29:21: If the node reboots during this execution, re-run './dbnodeupdate.sh -c' after the node restarts..
(*) 2014-04-24 23:29:21: Collecting console history for diag purposes
(*) 2014-04-24 23:30:06: ExaWatcher stopped successful
(*) 2014-04-24 23:30:06: Fix for 17885493 applied successful
(*) 2014-04-24 23:30:06: Starting ExaWatcher
(*) 2014-04-24 23:30:06: Relinking all homes
(*) 2014-04-24 23:30:06: Unlocking /u01/app/11.2.0.3/grid
(*) 2014-04-24 23:30:12: Relinking /u01/app/11.2.0.3/grid as grid (with rds option)
(*) 2014-04-24 23:30:24: Skipping /u01/app/oracle/agent11g/agent11g
(*) 2014-04-24 23:30:24: Relinking /u01/app/oracle/product/11.2.0.3/dbhome_1 as oracle (with rds option)
(*) 2014-04-24 23:30:44: Executing /u01/app/11.2.0.3/grid/crs/install/rootcrs.pl -patch
(*) 2014-04-24 23:31:45: Stack started
(*) 2014-04-24 23:31:45: Enabling stack to start at reboot. Disable this when the stack should not be starting on a next boot
(*) 2014-04-24 23:31:45: All post steps are finished.
Finished 1st DB Node upgrading. Do the same in remaining Nodes.
5.4 Infiniband switch upgrading:
Prepare ibswitches.lst file with the name of 1st switch:
[root@exa101 patch_11.2.3.3.0.131014.1]# cat ibswitches.lst
exa1sw-ib3
[root@exa101 patch_11.2.3.3.0.131014.1]# ./patchmgr -ibswitches ibswitches.lst -upgrade -ibswitch_precheck
2014-04-25 00:12:13 +0400 1 of 1 :SUCCESS: DO: Initiate pre-upgrade validation check on InfiniBand switch(es).
----- InfiniBand switch update process started Fri Apr 25 00:12:13 GET 2014 -----
[NOTE ] Log file at /var/log/cellos/upgradeIBSwitch.log
[INFO ] List of InfiniBand switches for upgrade: ( exa1sw-ib3 )
[PROMPT ] Use the default password for all switches? (y/n) [n]: y
[PROMPT ] Mismatch in the list of switches provided ( exa1sw-ib3 ) and the output of the command ibswitches ( exa1sw-ib3 exa1sw-ib2 ). Are you
sure you want to continue? (y/n) [n]: y
[PROMPT ] Updating only 1 switch(es). Are you sure you want to continue? (y/n) [n]: y
[SUCCESS ] Verifying Network connectivity to exa1sw-ib3
[SUCCESS ] Validating verify-topology output
[INFO ] Master Subnet Manager is set to exa1sw-ib3 in all Switches
[PROMPT ] Found 1 InfiniBand switches with subnet manager running. Are you sure you want to continue? (y/n) [n]: y
[INFO ] Found 1 InfiniBand switches with subnet manager running. Continuing as per user request.
[INFO ] ---------- Starting with IBSwitch exa1sw-ib3
[SUCCESS ] exa1sw-ib3 is at 1.3.3-2. Meets minimal patching level 1.3.3-2
[SUCCESS ] Verifying that /tmp has 120M in exa1sw-ib3, found 249M
[SUCCESS ] Verifying that / has 80M in exa1sw-ib3, found 200M
[SUCCESS ] Verifying that exa1sw-ib3 has 120M free memory, found 411M
[SUCCESS ] Verifying host details in /etc/hosts and /etc/sysconfig/network-scripts/ifcfg-eth0 /etc/sysconfig/network-scripts/ifcfg-eth1 for
exa1sw-ib3
[SUCCESS ] Verifying that exa1sw-ib3 has at least 1 NTP Server, found 1
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2014-04-25 Time:(HH:MM:SS) 04:24:36
[SUCCESS ] Pre-update validation on exa1sw-ib3
[SUCCESS ] Overall status
----- InfiniBand switch update process ended Fri Apr 25 00:13:20 GET 2014 -----
2014-04-25 00:13:20 +0400 1 of 1 :SUCCESS: DONE: Initiate pre-upgrade validation check on InfiniBand switch(es).
[root@exa101 patch_11.2.3.3.0.131014.1]# ./patchmgr -ibswitches ibswitches.lst -upgrade
2014-04-25 00:13:38 +0400 1 of 1 :SUCCESS: DO: Initiate upgrade of InfiniBand switches to 2.1.3-4. Expect up to 15 minutes for each switch
----- InfiniBand switch update process started Fri Apr 25 00:13:38 GET 2014 -----
[NOTE ] Log file at /var/log/cellos/upgradeIBSwitch.log
[INFO ] List of InfiniBand switches for upgrade: ( exa1sw-ib3 )
[PROMPT ] Use the default password for all switches? (y/n) [n]: y
[PROMPT ] Mismatch in the list of switches provided ( exa1sw-ib3 ) and the output of the command ibswitches ( exa1sw-ib3 exa1sw-ib2 ). Are you
sure you want to continue? (y/n) [n]: y
[PROMPT ] Updating only 1 switch(es). Are you sure you want to continue? (y/n) [n]: y
[SUCCESS ] Verifying Network connectivity to exa1sw-ib3
[SUCCESS ] Validating verify-topology output
[INFO ] Proceeding with upgrade of InfiniBand switches to version 2.1.3_4
[INFO ] Master Subnet Manager is set to exa1sw-ib3 in all Switches
[PROMPT ] Found 1 InfiniBand switches with subnet manager running. Are you sure you want to continue? (y/n) [n]: y
[INFO ] Found 1 InfiniBand switches with subnet manager running. Continuing as per user request.
[INFO ] ---------- Starting with IBSwitch exa1sw-ib3
[SUCCESS ] Disable Subnet Manager on exa1sw-ib3
[SUCCESS ] Copy firmware packages to exa1sw-ib3
[SUCCESS ] exa1sw-ib3 is at 1.3.3-2. Meets minimal patching level 1.3.3-2
[SUCCESS ] Verifying that /tmp has 120M in exa1sw-ib3, found 138M
[SUCCESS ] Verifying that / has 80M in exa1sw-ib3, found 200M
[SUCCESS ] Verifying that exa1sw-ib3 has 120M free memory, found 301M
[SUCCESS ] Verifying host details in /etc/hosts and /etc/sysconfig/network-scripts/ifcfg-eth0 /etc/sysconfig/network-scripts/ifcfg-eth1 for
exa1sw-ib3
[SUCCESS ] Verifying that exa1sw-ib3 has at least 1 NTP Server, found 1
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2014-04-25 Time:(HH:MM:SS) 04:25:51
[SUCCESS ] Pre-update validation on exa1sw-ib3
[INFO ] Starting upgrade on exa1sw-ib3 to 2.1.3_4. Please give upto 10 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C
during the upgrade
[SUCCESS ] Load firmware 2.1.3_4 onto exa1sw-ib3
[SUCCESS ] Disable Subnet Manager on exa1sw-ib3
[SUCCESS ] Verify that /conf/configvalid is set to 1 on exa1sw-ib3
[SUCCESS ] Set SMPriority to 5 on exa1sw-ib3
[INFO ] Rebooting exa1sw-ib3. Wait for 240 secs before continuing
[SUCCESS ] Reboot exa1sw-ib3
[SUCCESS ] Restart Subnet Manager on exa1sw-ib3
[INFO ] Starting post-update validation on exa1sw-ib3
[SUCCESS ] Inifiniband switch exa1sw-ib3 is at target patching level
[SUCCESS ] Verifying host details in /etc/hosts and /etc/sysconfig/network-scripts/ifcfg-eth0 /etc/sysconfig/network-scripts/ifcfg-eth1 for
exa1sw-ib3
[SUCCESS ] Verifying that exa1sw-ib3 has at least 1 NTP Server, found 1
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2014-04-25 Time:(HH:MM:SS) 04:39:02
[SUCCESS ] Firmware verification on InfiniBand switch exa1sw-ib3
[INFO ] Post-check validation on IBSwitch exa1sw-ib3
[SUCCESS ] Update switch exa1sw-ib3 to 2.1.3_4
[INFO ] InfiniBand Switches ( exa1sw-ib3 ) updated to 2.1.3_4
[SUCCESS ] Overall status
----- InfiniBand switch update process ended Fri Apr 25 00:31:59 GET 2014 -----
2014-04-25 00:31:59 +0400 1 of 1 :SUCCESS: DONE: Upgrade InfiniBand switch(es) to 2.1.3-4.
Edit ibswitches.lst file and replace with 2nd switch's name:
[root@exa101 patch_11.2.3.3.0.131014.1]# cat ibswitches.lst
exa1sw-ib2
[root@exa101 patch_11.2.3.3.0.131014.1]# ./patchmgr -ibswitches ibswitches.lst -upgrade -ibswitch_precheck
2014-04-25 00:33:44 +0400 1 of 1 :SUCCESS: DO: Initiate pre-upgrade validation check on InfiniBand switch(es).
----- InfiniBand switch update process started Fri Apr 25 00:33:45 GET 2014 -----
[NOTE ] Log file at /var/log/cellos/upgradeIBSwitch.log
[INFO ] List of InfiniBand switches for upgrade: ( exa1sw-ib2 )
[PROMPT ] Use the default password for all switches? (y/n) [n]: y
[PROMPT ] Mismatch in the list of switches provided ( exa1sw-ib2 ) and the output of the command ibswitches ( exa1sw-ib3 exa1sw-ib2 ). Are you
sure you want to continue? (y/n) [n]: y
[PROMPT ] Updating only 1 switch(es). Are you sure you want to continue? (y/n) [n]: y
[SUCCESS ] Verifying Network connectivity to exa1sw-ib2
[SUCCESS ] Validating verify-topology output
[INFO ] Master Subnet Manager is set to exa1sw-ib2 in all Switches
[PROMPT ] Found 1 InfiniBand switches with subnet manager running. Are you sure you want to continue? (y/n) [n]: y
[INFO ] Found 1 InfiniBand switches with subnet manager running. Continuing as per user request.
[INFO ] ---------- Starting with IBSwitch exa1sw-ib2
[SUCCESS ] exa1sw-ib2 is at 1.3.3-2. Meets minimal patching level 1.3.3-2
[SUCCESS ] Verifying that /tmp has 120M in exa1sw-ib2, found 249M
[SUCCESS ] Verifying that / has 80M in exa1sw-ib2, found 199M
[SUCCESS ] Verifying that exa1sw-ib2 has 120M free memory, found 411M
[SUCCESS ] Verifying host details in /etc/hosts and /etc/sysconfig/network-scripts/ifcfg-eth0 /etc/sysconfig/network-scripts/ifcfg-eth1 for
exa1sw-ib2
[SUCCESS ] Verifying that exa1sw-ib2 has at least 1 NTP Server, found 1
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2014-04-25 Time:(HH:MM:SS) 00:34:10
[SUCCESS ] Pre-update validation on exa1sw-ib2
[SUCCESS ] Overall status
----- InfiniBand switch update process ended Fri Apr 25 00:34:11 GET 2014 -----
2014-04-25 00:34:11 +0400 1 of 1 :SUCCESS: DONE: Initiate pre-upgrade validation check on InfiniBand switch(es).
[root@exa101 patch_11.2.3.3.0.131014.1]# ./patchmgr -ibswitches ibswitches.lst -upgrade
2014-04-25 00:34:20 +0400 1 of 1 :SUCCESS: DO: Initiate upgrade of InfiniBand switches to 2.1.3-4. Expect up to 15 minutes for each switch
----- InfiniBand switch update process started Fri Apr 25 00:34:21 GET 2014 -----
[NOTE ] Log file at /var/log/cellos/upgradeIBSwitch.log
[INFO ] List of InfiniBand switches for upgrade: ( exa1sw-ib2 )
[PROMPT ] Use the default password for all switches? (y/n) [n]: y
[PROMPT ] Mismatch in the list of switches provided ( exa1sw-ib2 ) and the output of the command ibswitches ( exa1sw-ib3 exa1sw-ib2 ). Are you
sure you want to continue? (y/n) [n]: y
[PROMPT ] Updating only 1 switch(es). Are you sure you want to continue? (y/n) [n]: y
[SUCCESS ] Verifying Network connectivity to exa1sw-ib2
[SUCCESS ] Validating verify-topology output
[INFO ] Proceeding with upgrade of InfiniBand switches to version 2.1.3_4
[INFO ] Master Subnet Manager is set to exa1sw-ib2 in all Switches
[PROMPT ] Found 1 InfiniBand switches with subnet manager running. Are you sure you want to continue? (y/n) [n]: y
[INFO ] Found 1 InfiniBand switches with subnet manager running. Continuing as per user request.
[INFO ] ---------- Starting with IBSwitch exa1sw-ib2
[SUCCESS ] Disable Subnet Manager on exa1sw-ib2
[SUCCESS ] Copy firmware packages to exa1sw-ib2
[SUCCESS ] exa1sw-ib2 is at 1.3.3-2. Meets minimal patching level 1.3.3-2
[SUCCESS ] Verifying that /tmp has 120M in exa1sw-ib2, found 138M
[SUCCESS ] Verifying that / has 80M in exa1sw-ib2, found 199M
[SUCCESS ] Verifying that exa1sw-ib2 has 120M free memory, found 302M
[SUCCESS ] Verifying host details in /etc/hosts and /etc/sysconfig/network-scripts/ifcfg-eth0 /etc/sysconfig/network-scripts/ifcfg-eth1 for
exa1sw-ib2
[SUCCESS ] Verifying that exa1sw-ib2 has at least 1 NTP Server, found 1
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2014-04-25 Time:(HH:MM:SS) 00:35:14
[SUCCESS ] Pre-update validation on exa1sw-ib2
[INFO ] Starting upgrade on exa1sw-ib2 to 2.1.3_4. Please give upto 10 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C
during the upgrade
[SUCCESS ] Load firmware 2.1.3_4 onto exa1sw-ib2
[SUCCESS ] Disable Subnet Manager on exa1sw-ib2
[SUCCESS ] Verify that /conf/configvalid is set to 1 on exa1sw-ib2
[SUCCESS ] Set SMPriority to 5 on exa1sw-ib2
[INFO ] Rebooting exa1sw-ib2. Wait for 240 secs before continuing
[SUCCESS ] Reboot exa1sw-ib2
[SUCCESS ] Restart Subnet Manager on exa1sw-ib2
[INFO ] Starting post-update validation on exa1sw-ib2
[SUCCESS ] Inifiniband switch exa1sw-ib2 is at target patching level
[SUCCESS ] Verifying host details in /etc/hosts and /etc/sysconfig/network-scripts/ifcfg-eth0 /etc/sysconfig/network-scripts/ifcfg-eth1 for
exa1sw-ib2
[SUCCESS ] Verifying that exa1sw-ib2 has at least 1 NTP Server, found 1
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2014-04-25 Time:(HH:MM:SS) 00:48:18
[SUCCESS ] Firmware verification on InfiniBand switch exa1sw-ib2
[INFO ] Post-check validation on IBSwitch exa1sw-ib2
[SUCCESS ] Update switch exa1sw-ib2 to 2.1.3_4
[INFO ] InfiniBand Switches ( exa1sw-ib2 ) updated to 2.1.3_4
[SUCCESS ] Overall status
----- InfiniBand switch update process ended Fri Apr 25 00:52:30 GET 2014 -----
2014-04-25 00:52:30 +0400 1 of 1 :SUCCESS: DONE: Upgrade InfiniBand switch(es) to 2.1.3-4.
at the end, picture of upgraded EXADATAs:
That is All, Good Luck!
