Exadata Upgrade

📘 Introduction

We recently completed an upgrade of our Exadata system from version 11.2.2.3.2 to 12.1.1.1.2. This post outlines the key steps and practical considerations we followed during the upgrade.

For full reference and latest updates, please refer to Oracle MOS Document ID 888828.1.

🧪 Pre-Upgrade Preparation

Before starting the upgrade, we prepared the environment as follows:

• Created LVM snapshots of / and /u01 on database servers
• Created a Cell Boot Image on an external USB for each storage server
• Backed up InfiniBand Switch configuration files

⚙️ Storage Server Upgrade (12.1.1.1.2)

Step-by-step actions:

On all cell servers (as root):

• Stop the EM Agent
• Shut down Cell Services on the first cell
• Log in to a DB node and check prerequisites:./patchmgr -cells cell_group -patch_check_prereq -rolling
• Adjust disk_repair_time as needed

dcli -g ~/all_group -l root ‘/bin/rm -f /opt/oracle.cellos/DISABLE_HARDWARE_FIRMWARE_CHECKS’
dcli -g ~/all_group -l root ‘touch /.updfrm_exact’

Patch using patchmgr:

./patchmgr -cells ~/cell_group -reset_force  
./patchmgr -cells cell_group -cleanup  
./patchmgr -cells cell_group -patch -rolling

• After ~5 minutes, monitor via ILOM
• Verify patch success:imageinfo
  imagehistory
  /opt/MegaRAID/MegaCli/MegaCli64 -AdpAllInfo -a0 -NoLog | grep ‘FW Package Build’
• Confirm all grid disks are ONLINE:dcli -g ./cell_group -l root cellcli -e “list griddisk attributes name,status,asmmodestatus” | grep -v ONLINE
• Check for errors:dcli -g cell_group -l root grep -i fail /var/log/cellos/validations.log
• Remove exact firmware directive:dcli -g all_group -l root ‘/bin/rm -f /.updfrm_exact’
• Final cleanup:patchmgr -cells ~/cell_group -cleanup

🖥️ Database Server Upgrade

Phase 1 – Apply 11.2.2.4.2

• Stop services:stop osw
  stop nfs
  stop dbfs
  shut down agent
  stop CRS on first node
• Run installation:./install.sh -force

Phase 2 – Apply 12.1.1.1.2

• Create local YUM repository
• Verify repo access:./dbnodeupdate.sh -u -v -l http://my.repository.url/x86_64
• Phase 1 upgrade:./dbnodeupdate.sh -u -l http://my.repository.url/x86_64
• Phase 2 upgrade:./dbnodeupdate.sh -u -p 2

🛠️ Completion Phase – Error and Workaround

We encountered this error:

bashERROR: Unable to unlock home /u01/app/11.2.0/grid_2

Workaround

As root:

bash/usr/bin/perl /u01/patch/oldopatch/OPatch/crs/patch112.pl -unlock -desthome=/u01/app/11.2.0/grid_2

As oracle:

bashcd /u01/app/11.2.0/grid_2/rdbms/lib  ORACLE_HOME=/u01/app/11.2.0/grid_2 make -f ins_rdbms.mk ipc_rds ioracle

cd /u01/app/oracle/product/11.2.0/dbhome_2/rdbms/lib  
ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_2 make -f ins_rdbms.mk ipc_rds ioracle

As root:

bash./dbnodeupdate.sh -c -i  /u01/app/oracle/product/11.2.0/dbhome_2/root.sh  
/usr/bin/perl /u01/patch/oldopatch/OPatch/crs/patch112.pl -patch -desthome=/u01/app/11.2.0/grid_2

🛠️ Completion Phase – Error and Workaround

We encountered this error:

bashERROR: Unable to unlock home /u01/app/11.2.0/grid_2

Workaround

As root:

bash/usr/bin/perl /u01/patch/oldopatch/OPatch/crs/patch112.pl -unlock -desthome=/u01/app/11.2.0/grid_2

As oracle:

bashcd /u01/app/11.2.0/grid_2/rdbms/lib  ORACLE_HOME=/u01/app/11.2.0/grid_2 make -f ins_rdbms.mk ipc_rds ioracle

cd /u01/app/oracle/product/11.2.0/dbhome_2/rdbms/lib  
ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_2 make -f ins_rdbms.mk ipc_rds ioracle

---

🌐 InfiniBand Switch Upgrade

One switch failed initially due to lack of /tmp space. After cleaning space, we succeeded.

We modified the following:

• Script upgradeIBSwitch.sh, changing:bashlocal Cmd=”$PWD/ibdiagtools/verify-topology” to
  local Cmd=”$PWD/ibdiagtools/verify-topology -i”
• Modified md5sum_files.lst as needed

Run upgrade:

bashexport EXADATA_IMAGE_IBSWITCH_ROLLBACK_VERSION=1.3.3-2  ./patchmgr -ibswitches ibswitches.lst -upgrade -ibswitch_precheck  
./patchmgr -ibswitches ibswitches.lst -upgrade

Tags: