Tags

11g (5) 12c (6) 18c (3) 19c (4) ASM (1) Critical Patch (11) Data Pump (1) Dataguard (9) Diverse (3) GRID (7) GitLab (2) Linux (8) OEM (2) ORA Errors (13) Oracle (12) RMAN (4)

Sunday, December 8, 2019

Oracle Grid Upgrade 12.2.0.1 to 18.3.0.0


Upgrading Oracle Grid Infrastructure

Oracle Grid Upgrade 12.2.0.1 to 18c

Upgrading to Oracle Database 18c

Upgrade Grid Infrastructure to 18c


STEP 1 Grid Upgrade:

Oracle Grid upgrade from 12g to 18c

PURPOSE: All documents are provided on this Blog just for educational purposes only.  Please make sure that you run it in your test environment before to move on to production environment. 

Preparation:
Keep following detail for later

srvctl config asm
ASM home: <CRS home>
Password file:
Backup of Password file:
ASM listener: LISTENER
Spfile: +DATA/asm/asmparameterfile/registry.253.1024325155
ASM diskgroup discovery string: /dev/*

We need to download following Files:
LINUX.X64_180000_grid_home.zip
p28096386_180000_Linux-x86-64.zip

Create NEW directory for NEW Oracle Grid home:

$ORACLE_BASE/product/18.0.0/grid_18c
mv /download/LINUX.X64_180000_grid_home.zip $ORACLE_BASE/product/18.0.0/grid_18c
unzip LINUX.X64_180000_grid_home.zip

Stop ASM and DB
SHUTDOWN DB FROM ASM HOME

srvctl status database -d KARDBPRD
Database is running.
srvctl stop database -d KARDBPRD

srvctl status  listener -l LISTENER
srvctl stop listener -l LISTENER

srvctl status asm
ASM is running on kardbprd.us.com

srvctl stop asm -f
PRCR-1065 : Failed to stop resource ora.asm
CRS-2675: Stop of 'ora.DATA.dg' on 'kardbprd.us.com' failed
The workaround is to shutdown ASM using sqlplus(Doc ID 2271669.1)

sqlplus -s / as sysasm
shutdown immediate
OR
Stop complete Restart stack with below command.

$crsctl stop has
crsctl stat res -t

Precheck:

/apps/oracle/product/18.0.0/grid_18c/runcluvfy.sh stage -pre hacfg

This is the output of the above command:
Verifying Physical Memory ...PASSED
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...PASSED
Verifying Free Space: kardbprd.us.com:/usr,kardbprd.us.com:/var,kardbprd.us.com:/etc,kardbprd.us.com:/sbin,kardbprd.us.com:/tmp ...PASSED
Verifying User Existence: oracle ...
  Verifying Users With Same UID: 1001 ...PASSED
Verifying User Existence: oracle ...PASSED
Verifying Group Existence: dba ...PASSED
Verifying Group Membership: dba ...PASSED
Verifying Run Level ...PASSED
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying OS Kernel Parameter: panic_on_oops ...PASSED
Verifying Package: binutils-2.20.51.0.2 ...PASSED
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...PASSED
Verifying Package: libgcc-4.4.7 (x86_64) ...PASSED
Verifying Package: libstdc++-4.4.7 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.4.7 (x86_64) ...PASSED
Verifying Package: sysstat-9.0.4 ...PASSED
Verifying Package: gcc-4.4.7 ...PASSED
Verifying Package: gcc-c++-4.4.7 ...PASSED
Verifying Package: ksh ...PASSED
Verifying Package: make-3.81 ...PASSED
Verifying Package: glibc-2.12 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.12 (x86_64) ...PASSED
Verifying Package: libaio-0.3.107 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.107 (x86_64) ...PASSED
Verifying Package: nfs-utils-1.2.3-15 ...PASSED
Verifying Package: smartmontools-5.43-1 ...PASSED
Verifying Package: net-tools-1.60-110 ...PASSED
Verifying Users With Same UID: 0 ...PASSED
Verifying Current Group ID ...PASSED
Verifying Root user consistency ...PASSED

Pre-check for Oracle Restart configuration was successful.

CVU operation performed:      stage -pre hacfg
Date:                         Nov 25, 2019 10:09:28 AM
CVU home:                     /apps/oracle/product/18.0.0/grid_18c/
User:                         oracle

Unzip the patch which is required for Installation.
unzip p28096386_180000_Linux-x86-64.zip

As root user:
# cd $ORACLE_BASE/product/18.0.0/grid_18c/cv/rpm
# rpm -Uvh cvuqdisk-1.0.10-1.rpm
Preparing...                ########################################### [100%]
   1:cvuqdisk               ########################################### [100%]

STARTING with Upgrade   

I am using MobaXterm for X display
unset ORACLE_BASE
unset ORACLE_HOME
unset ORACLE_SID

DISPLAY=xxx.xx.xxx.xxx:0.0; export DISPLAY
./gridSetup.sh -applyRU /download/28096386
NOTE: I am not going upload the screenshot. Since the steps are pretty straightforward. I chose following options.
=>> Upgrade Orcle Grid Infrastructure
=>> Registration with Enterprise Manager (EM) Cloud Control =>> no need uncheck it
=>> Oracle_base (keep the same location)
=>> Automatically run configuration scripts =>> no need uncheck it

As root user:


/apps/oracle/product/18.0.0/grid_18c/rootupgrade.sh

I am getting following error:
  /apps/oracle/crsdata/kardbprd.us.com/crsconfig/roothas_2019-11-25_10-38-40AM.log
2019/11/25 10:38:40 CLSRSC-595: Executing upgrade step 1 of 12: 'UpgPrechecks'.
2019/11/25 10:38:42 CLSRSC-595: Executing upgrade step 2 of 12: 'GetOldConfig'.
2019/11/25 10:38:58 CLSRSC-318: Failed to start Oracle OHASD service
Died at /apps/oracle/product/18.0.0/grid_18c/crs/install/crsupgrade.pm line 6104.

After debugging I realized I need to start the process manually
crsctl start has
CRS-4123: Oracle High Availability Services has been started.

Executing rootupgrade.sh script again and got another error :

/apps/oracle/product/18.0.0/grid_18c/rootupgrade.sh
CLSRSC-164: ASM upgrade failed
CLSRSC-304: Failed to upgrade ASM for Oracle Restart configuration

I have to start ASM via following command in a different screen.
srvctl start asm

Executing rootupgrade.sh script again:

/apps/oracle/product/18.0.0/grid_18c/rootupgrade.sh
It Works!
I just want to make sure that everything works properly after upgrade. I am going stop everything and will start again.
srvctl status asm
crsctl stat res -t
srvctl stop asm -f
crsctl stop has
crsctl start has
srvctl start asm

Adjust the location and any parameter required in the tnsnames.ora file
srvctl stop listener -l LISTENER
srvctl start listener -l LISTENER
Stop/Start DB from the DB_HOME
srvctl stop database -d KARDBPRD
srvctl start database -d KARDBPRD

Login into DB and in ASM Instance and verify the standard configurations.

$ sqlplus / as sysasm
SQL*Plus: Release 18.0.0.0.0 - Production on Mon Nov 25 13:38:48 2019
Version 18.3.0.0.0
Copyright (c) 1982, 2018, Oracle.  All rights reserved.

Connected to:
Oracle Database 18c Enterprise Edition Release 18.0.0.0.0 - Production
Version 18.3.0.0.0

Change the compatibility parameter:
sqlplus / as sysasm
 col COMPATIBILITY form a10
 col DATABASE_COMPATIBILITY form a10
 col NAME form a20
select group_number, name, compatibility, database_compatibility from v$asm_diskgroup;
GROUP_NUMBER NAME                 COMPATIBIL DATABASE_C
------------ -------------------- ---------- ----------
           1 DATA                 11.2.0.0.0 10.1.0.0.0
asmcmd setattr -G DATA compatible.asm 12.2.0.0.0
or
ALTER DISKGROUP DATA SET ATTRIBUTE 'compatible.asm' = '12.2.0.0.0'
SQL> select group_number, name, compatibility, database_compatibility from v$asm_diskgroup;

GROUP_NUMBER NAME                 COMPATIBIL DATABASE_C
------------ -------------------- ---------- ----------
           1 DATA                 12.2.0.0.0 10.1.0.0.0


Login to the DB:
sqlplus / as sysdba
SQL*Plus: Release 12.2.0.1.0 Production on Mon Nov 25 13:30:02 2019
Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> select name from v$database;
NAME
---------
KARDBPRD

Looks everything good. At this point my Grid ASM is running to the version 18.3.0 and my DB has 12.2.0.1 Version.

Let us upgrade DB now:

STEP 2 Database Upgrade:

Oracle database upgrade from 12g to 18c

1. Login as 12c DB
SQL> create pfile='/download/initDB_12c.ora' from spfile;
NOTE: Do not change this parameter keep as it is. You can do later.
*.compatible
I am using the GUI for Installation. Install Just Software.
We need following files. Download it and move or copy the file to the NEW $ORACLE_HOME
mv LINUX.X64_180000_db_home.zip /apps/oracle/product/18.0.0/db_18c
cd $ORACLE_HOME
unzip LINUX.X64_180000_db_home.zip

2. Install just Software.

export DISPLAY=xx.xx.xx.xx:0.0
unset ORACLE_BASE
unset ORACLE_HOME
unset ORACLE_SID
./runInstaller

Chose following options:
=>> Set Up Software Only
=>> Single Instance database installation
=>> Enterprise Edition
=>> chose the group you have. I chose “dba
=>> Install

Execute the root script as a root user.

/apps/oracle/product/18.0.0/db_18c/root.sh
This is the output of the root.sh script:
The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /apps/oracle/product/18.0.0/db_18c

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Do you want to setup Oracle Trace File Analyzer (TFA) now ? yes|[no] :
no
Oracle Trace File Analyzer (TFA - Non Daemon Mode) is available at :
    /apps/oracle/product/18.0.0/db_18c/suptools/tfa/release/tfa_home/bin/tfactl

Note :
1. tfactl will use TFA Daemon Mode if TFA already running in Daemon Mode and user has access to TFA
2. tfactl will configure TFA Non Daemon Mode only if user has no access to TFA Daemon mode or TFA Daemon mode is not installed
OR
Oracle Trace File Analyzer (TFA - Daemon Mode) can be installed by running this script :
    /apps/oracle/product/18.0.0/db_18c/suptools/tfa/release/tfa_home/install/roottfa.sh
3. Execute preupgrade script as follow.

apps/oracle/product/12.2.0.1/db_12c/jdk/bin/java -jar /apps/oracle/product/18.0.0/db_18c/rdbms/admin/preupgrade.jar FILE TEXT
==================
PREUPGRADE SUMMARY
==================
  /apps/oracle/cfgtoollogs/KARDBPRD/preupgrade/preupgrade.log
  /apps/oracle/cfgtoollogs/KARDBPRD/preupgrade/preupgrade_fixups.sql
  /apps/oracle/cfgtoollogs/KARDBPRD/preupgrade/postupgrade_fixups.sql

Execute fixup scripts as indicated below:

Before upgrade log into the database and execute the preupgrade fixups
@/apps/oracle/cfgtoollogs/KARDBPRD/preupgrade/preupgrade_fixups.sql

After the upgrade:

Log into the database and execute the postupgrade fixups
@/apps/oracle/cfgtoollogs/KARDBPRD/preupgrade/postupgrade_fixups.sql

Preupgrade complete: 2019-11-26T15:14:48
4. Run 12.2.0.1.0 $ORACLE_HOME/rdbms/admin/utlrp.sql to recompile invalid
@/apps/oracle/product/12.2.0.1/db_12c/rdbms/admin/utlrp.sql

5. Run the preupgrade_fixups.sql script
@/oracle/cfgtoollogs/KARDBPRD/preupgrade/preupgrade_fixups.sql
Check invalid Objects:
select comp_id,status from dba_registry;

Stop DB:
srvctl stop database -d KARDBPRD
srvctl stop listener -l LISTENER

Login as 18c:
create spfile from pfile='/download/initDB_12c.ora';

SQL> startup upgrade;
ORACLE instance started.

Total System Global Area 3.0266E+10 bytes
Fixed Size                 12214256 bytes
Variable Size            3623878656 bytes
Database Buffers         2.6575E+10 bytes
Redo Buffers               54894592 bytes
Database mounted.
Database opened.

Execute following command to start the Update:
Start the database upgrade in parallel as follow. Or you can use "dbupgrade" command which is a wrapper for $ORACLE_HOME/perl/bin/perl catctl.pl -n 8 catupgrd.sql:
I am using the command as follow:
cd $ORACLE_HOME/rdbms/admin
$ORACLE_HOME/perl/bin/perl catctl.pl -n 8 catupgrd.sql
...
...
------------------------------------------------------
Phases [0-108]         End Time:[2019_11_26 17:00:36]
------------------------------------------------------

Grand Total Time: 783s

 LOG FILES: (/apps/oracle/product/18.0.0/db_18c/cfgtoollogs/KARDBPRD/upgrade20191126164734/catupgrd*.log)

Upgrade Summary Report Located in:
/apps/oracle/product/18.0.0/db_18c/cfgtoollogs/KARDBPRD/upgrade20191126164734/upg_summary.log

Grand Total Upgrade Time:    [0d:0h:13m:3s]

Add DB to the NEW Oracle_Home via srvctl command:
srvctl add database -d KARDBPRD -o /apps/oracle/product/18.0.0/db_18c
srvctl start database -d KARDBPRD

 Execute the postupgrade script.

@/apps/oracle/cfgtoollogs/KARDBPRD/preupgrade/postupgrade_fixups.sql
EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS;
EXECUTE DBMS_STATS.GATHER_FIXED_OBJECTS_STATS;

Check Components:
col COMP_NAME format a50;
set linesize 300
set pagesize 100
select COMP_NAME, version, status from dba_registry;SQL> SQL> SQL>

COMP_NAME                                          VERSION                        STATUS
-------------------------------------------------- ------------------------------ -----------
Oracle Database Catalog Views                      18.0.0.0.0                     UPGRADED
Oracle Database Packages and Types                 18.0.0.0.0                     UPGRADED
JServer JAVA Virtual Machine                       18.0.0.0.0                     VALID
Oracle XDK                                         18.0.0.0.0                     UPGRADED
Oracle Database Java Packages                      18.0.0.0.0                     UPGRADED
Oracle Real Application Clusters                   18.0.0.0.0                     OPTION OFF
Oracle Workspace Manager                           18.0.0.0.0                     UPGRADED
Oracle XML Database                                18.0.0.0.0                     UPGRADED
Oracle Multimedia                                  18.0.0.0.0                     UPGRADED
Oracle Application Express                         18.2.0.00.12                   INVALID

10 rows selected.

SQL> select VERSION,name,open_mode from v$instance,v$database;

VERSION           NAME      OPEN_MODE
----------------- --------- --------------------
18.0.0.0.0        KURSDB    READ WRITE

Checking TimeZone Version, which is also need to update:
SQL> SELECT * FROM v$timezone_file;

FILENAME                VERSION     CON_ID
-------------------- ---------- ----------
timezlrg_14.dat              14          0

SQL> select TZ_VERSION from registry$database;

TZ_VERSION
----------
        14
====================
Update the timezone
====================
You may need to update the timezone as follow:
You need following commands:
@?/rdbms/admin/utltz_countstar.sql
@?/rdbms/admin/utltz_upg_check.sql
@?/rdbms/admin/utltz_upg_apply.sql

SQL> @utltz_upg_check.sql

Session altered.

INFO: Starting with RDBMS DST update preparation.
INFO: NO actual RDBMS DST update will be done by this script.
INFO: If an ERROR occurs the script will EXIT sqlplus.
INFO: Doing checks for known issues ...
INFO: Database version is 18.0.0.0 .
INFO: Database RDBMS DST version is DSTv14 .
INFO: No known issues detected.
INFO: Now detecting new RDBMS DST version.
A prepare window has been successfully started.
INFO: Newest RDBMS DST version detected is DSTv31 .
INFO: Next step is checking all TSTZ data.
INFO: It might take a while before any further output is seen ...
A prepare window has been successfully ended.
INFO: A newer RDBMS DST version than the one currently used is found.
INFO: Note that NO DST update was yet done.
INFO: Now run utltz_upg_apply.sql to do the actual RDBMS DST update.
INFO: Note that the utltz_upg_apply.sql script will
INFO: restart the database 2 times WITHOUT any confirmation or prompt.

Session altered.

SQL> @utltz_upg_apply.sql

SQL> SELECT * FROM v$timezone_file;

FILENAME                VERSION     CON_ID
-------------------- ---------- ----------
timezlrg_31.dat              31          0

1 row selected.

SQL>  select TZ_VERSION from registry$database;

TZ_VERSION
----------
        31
At this point we are done:

The best person is the one who benefits all human beings. [Prophet Muhammad (PBUH)]

This is the COMPLETE OUTPUT FROM THE rootupgrade.sh Script:
/apps/oracle/product/18.0.0/grid_18c/rootupgrade.sh
Performing root user operation.


The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /apps/oracle/product/18.0.0/grid_18c


Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.


Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /apps/oracle/product/18.0.0/grid_18c/crs/install/crsconfig_params
The log of current session can be found at:
  /apps/oracle/crsdata/kardbprd.us.com/crsconfig/roothas_2019-11-25_12-45-31AM.log
2019/11/25 12:45:32 CLSRSC-595: Executing upgrade step 1 of 12: 'UpgPrechecks'.
2019/11/25 12:45:34 CLSRSC-595: Executing upgrade step 2 of 12: 'GetOldConfig'.
2019/11/25 12:45:35 CLSRSC-595: Executing upgrade step 3 of 12: 'GenSiteGUIDs'.
2019/11/25 12:45:35 CLSRSC-595: Executing upgrade step 4 of 12: 'SetupOSD'.
2019/11/25 12:45:35 CLSRSC-595: Executing upgrade step 5 of 12: 'PreUpgrade'.


ASM has been upgraded and started successfully.


2019/11/25 12:46:13 CLSRSC-595: Executing upgrade step 6 of 12: 'UpgradeAFD'.
2019/11/25 12:46:13 CLSRSC-595: Executing upgrade step 7 of 12: 'UpgradeOLR'.
Creating OCR keys for user 'oracle', privgrp 'dba'..
Operation successful.
2019/11/25 12:46:19 CLSRSC-595: Executing upgrade step 8 of 12: 'UpgradeOCR'.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node kardbprd.us.com successfully pinned.
2019/11/25 12:46:20 CLSRSC-595: Executing upgrade step 9 of 12: 'CreateOHASD'.
2019/11/25 12:46:21 CLSRSC-595: Executing upgrade step 10 of 12: 'ConfigOHASD'.
2019/11/25 12:46:21 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.conf'
2019/11/25 12:46:34 CLSRSC-595: Executing upgrade step 11 of 12: 'UpgradeSIHA'.
CRS-4123: Oracle High Availability Services has been started.




kardbprd.us.com     2019/11/25 12:47:05     /apps/oracle/product/18.0.0/grid_18c/cdata/kardbprd.us.com/backup_20191125_124705.olr     70732493


kardbprd.us.com     2019/11/19 14:47:54     /apps/oracle/product/12.2.0.1/grid_12c/cdata/kardbprd.us.com/backup_20191119_144754.olr     1039292241


kardbprd.us.com     2019/11/14 14:44:41     /apps/oracle/product/11.2.0.3/grid/cdata/kardbprd.us.com/backup_20191114_144441.olr     -
2019/11/25 12:47:06 CLSRSC-595: Executing upgrade step 12 of 12: 'InstallACFS'.
CRS-4123: Oracle High Availability Services has been started.


CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'kardbprd.us.com'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'kardbprd.us.com' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2019/11/25 12:47:53 CLSRSC-327: Successfully configured Oracle Restart for a standalone server




No comments:

Post a Comment

physical standby without duplicate command

physical standby without duplicate command create a physical standby database using RMAN without using duplicate command PURPOSE:   All docu...