AutoUpgrade: Where do you find all the logfiles?

AutoUpgrade: Where do you find all the logfiles?In the previous blog posts you could read about how to create the configuration file, adjust it and tweak the init parameters for the AutoUpgrade tool. And then I described the different modes of it. In this blog post I’d like to give you some insights into AutoUpgrade: Where do you find all the logfiles?

AutoUpgrade: Where do you find all the logfiles?

AutoUpgrade – Step-by-step

  1. The new AutoUpgrade Utility – Download, documentation and supported versions
  2. Create and adjust the config file for AutoUpgrade
  3. Config file for AutoUpgrade – Advanced options
  4. Config file for AutoUpgrade – Tweaking init parameters
  5. AutoUpgrade: ANALYZE, FIXUPS, UPGRADE and DEPLOY modes
  6. AutoUpgrade: Where do you find all the logfiles?
  7. UPG: The AutoUpgrade Command Line Interface
  8. Upgrading Multitenant databases with AutoUpgrade
  9. Moving to a new server with AutoUpgrade
  10. How to tweak the hidden settings in AutoUpgrade
  11. AutoUpgrade and Data Guard, RAC, Restart and non-CDB to PDB
  12. AutoUpgrade and Wallets

The Config File

In the config file you set up earlier, you needed to define a global logging directory. And in addition, you needed to set one for each database.

global.autoupg_log_dir=/home/oracle/logs
upg1.log_dir=/home/oracle/logs/upgr

That’s it.

You need to specify an global.autoup_log_dir and upg1.log_dir Parameters. Without these defined, the config file won’t be accepted by AutoUpgrade.

But you may ask yourself: Which files will the tool write, and where do I find the files I’m looking for? And which files does Support want in case something fails?

The Documentation

The log structure is pretty well described in the Database Upgrade 19c Guide:

But nevertheless, I will explain a bit more about it.

The Directory Structure

First of all, the directory structure unfolds under the global.autoup_log_dir. You could have the log_dir for individual databases in a separate structure but I totally disrecommend this. Let me give you an overview about the entire logging structure – and then I will describe the different directories and point to the most important logs.

In the below tree, I used the database names DB12 and UPGR to avoid placeholders and the recommended approach of running –mode analyze at first followed by -mode deploy.

  • /cfgtoollogs
    • /upgrade/auto
      • /config_files
      • /lock
      • /status
  • /DB12
    • /100
      • /prechecks
    • /102
      • /preupgrade
      • /prechecks
      • /grp
      • /prefixups
      • /drain
      • /dbupgrade
      • /postchecks
      • /postfixups
      • /postupgrade
    • /temp
      • /DB12
        • Timezone
  • /UPGR
    • /101
      • /prechecks
    • /103
      • /preupgrade
      • /prechecks
      • /grp
      • /prefixups
      • /drain
      • /dbupgrade
      • /postchecks
      • /postfixups
      • /postupgrade
    • /temp
      • /UPGR
        • Timezone

/cfgtoollogs

Under this directory you will find a static structure, /upgrade/auto, and then 3 subdirectories. The autoupgrade.jar tool uses this structure for the tool itself, not for individual database information. From the diagnostic perspective, you will find the most important logfiles in the /upgrade/auto subdirectory:

-rwx------. 1 oracle dba    158 Jul 15 22:24 autoupgrade_err.log
drwx------. 2 oracle dba   4096 Jul 15 23:06 status
-rwx------. 1 oracle dba  12376 Jul 15 23:06 state.html
-rwx------. 1 oracle dba    781 Jul 15 23:06 autoupgrade_user.log
-rwx------. 1 oracle dba 683294 Jul 15 23:06 autoupgrade.log
drwx------. 2 oracle dba     22 Jul 15 23:06 config_files
drwx------. 2 oracle dba      6 Jul 15 23:06 lock

The most important, and main log file is the autoupgrade.log. The autoupgrade_user.log contains information from the interaction with you. For instance, when I commented out the global logging directory parameter, the tool complained and exit – and this information from the command prompt gets written to the autoupgrade_user.log.  The other subdirectories contain information about the status of each phase, we handle locking – and we write a state.html with the current status.

AutoUpgrade: Where do you find all the logfiles?

/cfgtoollogs/upgrade/auto/state.html

The ./status subdirectory contains files indication the .success or .failure of a phase with file suffixes. And you will find there also two json files which get updated constantly throughout the upgrade: status.json and progress.json.

If you’d like to monitor the status during an upgrade, please see this blog post:

/<database>

Besides the general logs from the autoupgrade.jar you will find individual logs in the subtree for each individual database. It unfolds into subdirectories for each job, simply named with the job number starting the count at 100, and another one called /temp.

The /temp directory contains temporary initialization parameter files the tool creates for the different phases of the upgrade. It is also used to maintain certain parameters and make sure these are set correctly after the upgrade. cluster_database is a typical example as we need to switch it to FALSE during the upgrade but the tool will revert it to the previous setting after the entire process has been completed. Additionally, it contains also sql scripts the tool uses during the upgrade, such as <sid>_objcompare.sql to compare invalid objects before-after.

More important for you may be the job number subdirectories. It starts with 100. If you follow the best practice of running the -mode analyze at first, then followed by a -mode deploy upgrade afterwards, then you will find two subdirectories, one for each job: 100 and 101.

I made a correction after the initial post: When you run -mode analyze, then you will find only a /prechecks subdirectory for job 100 for DB12, and the same for job 101 for UPGR in my above example. Initially I’ve had the same tree as for jobs 102 and 103 listed.

In the root of the job number, you can identify the individual autoupgrade.log for this database plus an additional autoupgrade_user.log. The latter contains not only some useful information about this particular database but also the upgrade progress:

2019-07-15 22:45:51.087 INFO [Upgrading] is [59%] completed for [db12]
+---------+-------------+
|CONTAINER|   PERCENTAGE|
+---------+-------------+
|     DB12|UPGRADE [59%]|
+---------+-------------+

Even more important, in the subdirectories preupgrade, prechecks, grp, prefixups, drain, dbupgrade, postchecks, postfixups and postupgrade you find detailed information about each phase of the autoupgrade.jar. I would like to point out only the most important ones, especially in relation to a regular command line upgrade.

/<database>/prechecks

In the /prechecks directory you can identify a preupgrade.log – but it looks different than the output of the preupgrade.jar if you call it directly from ?/rdbms/admin. In addition, the tool generates a html file as well:

AutoUpgrade: Where do you find all the logfiles?

/DB12/102/prechecks/db12_preupgrade.html

/<database>/dbupgrade

In the /dbupgrade directory you will find the standard upgrade logs from each worker and – at the end – and upg_summary.log. These files are identical to the ones you would see when you do a regular command line upgrade with dbupgrade or catcl.pl.

In my example these files are in the /dbupgrade subdirectory:

-rwx------. 1 oracle dba    30637 Jul 15 22:30 phase.log
-rwx------. 1 oracle dba      500 Jul 15 22:30 catupgrd20190715222614db12_catcon_7698.lst
-rwx------. 1 oracle dba        0 Jul 15 22:49 catupgrd20190715222614db12_datapatch_upgrade.err
-rwx------. 1 oracle dba     1304 Jul 15 22:51 catupgrd20190715222614db12_datapatch_upgrade.log
-rwx------. 1 oracle dba     4612 Jul 15 22:51 during_upgrade_pfile_catctl.ora
-rwx------. 1 oracle dba  9206559 Jul 15 22:52 catupgrd20190715222614db121.log
-rwx------. 1 oracle dba  4558045 Jul 15 22:52 catupgrd20190715222614db122.log
-rwx------. 1 oracle dba  6854830 Jul 15 22:52 catupgrd20190715222614db123.log
-rwx------. 1 oracle dba    36860 Jul 15 22:53 catupgrd20190715222614db12_stderr.log
-rwx------. 1 oracle dba      516 Jul 15 22:53 db12_autocompile20190715222614db12_catcon_13694.lst
-rwx------. 1 oracle dba     1912 Jul 15 22:59 db12_autocompile20190715222614db12_stderr.log
-rwx------. 1 oracle dba    11801 Jul 15 22:59 autoupgrade20190715222614db12.log
-rwx------. 1 oracle dba    31024 Jul 15 22:59 db12_autocompile20190715222614db120.log
-rwx------. 1 oracle dba      390 Jul 15 22:59 upg_summary_report.pl
-rwx------. 1 oracle dba     1359 Jul 15 22:59 upg_summary.log
-rwx------. 1 oracle dba 32832308 Jul 15 22:59 catupgrd20190715222614db120.log
-rwx------. 1 oracle dba       46 Jul 15 22:59 upg_summary_report.log

I marked the upgrade 4 worker’s logs in red. You can of course tail -f them during the upgrade as well. The worker log with the 0 is the main worker which is active in serial and parallel phases.

Interaction with Oracle Support

In case you open an SR, then the best approach is to zip the entire log directory together and upload it to MyOracle Support (MOS). This way you ensure that all required information is present. The tree contains information about the tool’s activities plus about each individual database.

Of course, if you upgraded 10 databases in one run but only one had issues, then you should include only the /cfgtoollogs and the /<database_with_issues> directories.

Also if you contact us, the Upgrade Development Team, directly please make sure you did open an SR at first. We worked closely together with our Support engineers – and this way the logs are all together.

Further Information and Links

–Mike

Share this: