Hands-On Lab available: Upgrade to Oracle Database 12.2.0.1

Thanks to Roy – our Hands-On Lab “Upgrade, Migrate and Consolidate to Oracle Database 12.2” is now available for download from OTN.

Instructions and technical requirements are clearly documented on the OTN page as well. The lab is based on Virtual Box and can be easily imported upon download and unzip.

You can always access the lab via the blog’s top menu as well:

HOL Link on the Upgrade Blog

Hands-On Lab – Access via the Blog

Inside the lab you’ll find :

  • UPGR – an Oracle 11.2.0.4 database ready to be upgraded to Oracle Database 12.2.0.1
  • CDB2 – an Oracle 12.2.0.1 container database which  adopts the UPGR database after upgrade as a new PDB
  • FTEX – an Oracle 11.2.0.4 database ready to be migrated with Full Transportable Exp/Imp into a PDB in CDB2
  • CDB1 – an Oracle 12.1.0.2 container database – so you can test the unplug/plug/upgrade and many other things

An overview picture showing you always where you are and how to switch environments:Upgrade HOL Oracle 12.2

Upgrade Hands-On Lab – Oracle Database 12.2 including Multitenant

Hope you’ll enjoy it!

–Mike

VBox 5.0.10/12 issues with PERL and Seg Faults – UPDATE

A bit more than two months ago I did hear from several people having issues with our Hands-On Lab environment. And it became clear that only those who use Oracle Virtual Box 5 see such errors.

VBox 5.0.10 crash issues with our Hands-On-Lab

Oracle VirtualBox 5.0.x – Segmentation Fault in PERL

Then I read Danny Bryant‘s blog post (thanks to Deiby Gomez for pointing me to it) about similar issues and a potential solution yesterday:

And interestingly one of my colleagues, our PL/SQL product manager Bryn Llewellyn started an email thread and a test internally yesterday as well. The issue seem to occur only on newer versions of Apple’s MacBooks.

Potential Root Cause

The PERL issues seem to happen only on specific new Intel CPUs with a so called 4th level cache.

The current assumption is that Intel CPUs with Iris Pro graphics are affected. Iris Pro means eDRAM (embedded DRAM) which is reported as 4th level cache in CPUID. We have confirmed that Crystal Well and Broadwell CPUs with Iris Pro are affected. It is likely that the Xeon E3-1200 v4 family is also affected.

It seems to be that there’s a bug in the perl binary. It links against ancient code from the Intel compiler suite doing optimizations according to the CPU features. Very recent Intel CPUs have 4 cache descriptors.

People who encountered this used Virtual Box VBox 5.0.x – and it passes this information to the guest. This leads to a problem within the perl code. You won’t see it on VBox 4.3 as this version does not pass the information to the guest. 

But actually it seems that this issue is independent of Virtual Box or any other virtualization software. It simply happens in this case as many people use VBox on Macs – and some Macs are equipped with this new CPU model. But people run Oracle in VBox environments and therefore see the issue as soon as they upgraded to VBox 5.0.x.

Potential Solutions

If you are using Oracle in VBox there are actually two solutions:

  • Revert to VBox 4.3 as this won’t get you in trouble
    This problem was not triggered on VBox 4.3.x because this version did not  pass the full CPUID cache line information to the guest.
  • Run this sequence of commands in VBox 5.0 to tweak the CPUID bits passed to the guest:
    VBoxManage setextradata VM_NAME "VBoxInternal/CPUM/HostCPUID/Cache/Leaf" "0x4"
    VBoxManage setextradata VM_NAME "VBoxInternal/CPUM/HostCPUID/Cache/SubLeaf" "0x4"
    VBoxManage setextradata VM_NAME "VBoxInternal/CPUM/HostCPUID/Cache/eax"  "0"
    VBoxManage setextradata VM_NAME "VBoxInternal/CPUM/HostCPUID/Cache/ebx" "0" 
    VBoxManage setextradata VM_NAME "VBoxInternal/CPUM/HostCPUID/Cache/ecx" "0" 
    VBoxManage setextradata VM_NAME "VBoxInternal/CPUM/HostCPUID/Cache/edx"  "0"
    VBoxManage setextradata VM_NAME "VBoxInternal/CPUM/HostCPUID/Cache/SubLeafMask" "0xffffffff"
    • Of course you’ll need to replace VM_NAME by the name of your VM.

If the error happens on a bare metal machine meaning it happens not inside a virtual image but on a native environment then the only chance you’ll have (to my knowledge right now) is to exchange the PERL before doing really something such as running root.sh or rootupgrade.sh in your Grid Infrastructure installation or before using the DBCA or the catctl.pl tool to create or upgrade a database.

In this case please refer to the blog post of Laurent Leturgez:

Issues with Oracle PERL causing segmentation faults:
http://laurent-leturgez.com/2015/05/26/oracle-12c-vmware-fusion-and-the-perl-binarys-segmentation-fault

Further Information

This issues is currently tracked internally as bug 22539814: ERRORS INSTALLING GRID INFRASTRUCTURE 12.1.0.2 ON INTEL CPUS WITH 4 CACHE LEVEL.

So far we have not seen reports by people encountering this in a native environment but only by people using VBox 5.0.x or Parallels or VMware on a very modern version of Apple hardware.
–Mike

New version of our Hands-On Lab available for download “Upgrade/Migrate/Consolidate to Oracle 12c” HOL

Yes! Finally …

Roy did all the hard work and patched all our environments with the most recent PSUs. And he managed also to upload our Hands-On Lab not only internally to retriever (Oracle internally) but also to OTN.

What you’ll be able to do?

  • Upgrade an 11.2.0.4 database UPGR to Oracle Database 12.1.0.2
  • Plug in UPGR into the CDB2, an Oracle Database 12.1.0.2 Container database, and make it a PDB
  • Migrate an 11.2.0.4 database FTEX into another PDB using the Full Transportable Export/Import feature
  • Unplug an Oracle Database 12.1.0.1 PDB and plug it into an Oracle Database 12.1.0.2 container database and finally upgrade it
  • And of course you can play around with the entire environment

What we’ve did for your convenience? 

  • Switched to a new PDF viewer which allows better copy&paste – see the instructions document.
    Unfortunately we are not allowed to distribute the relatively old version of Adobe PDF Reader for Linux.
  • Applied the most recent PSUs to the 11.2.0.4 and 12.1.0.2 homes
  • Fixed some minor things
  • Updated the HOL Instructions
  • Added an “aritificial” environment variable $OH12 to shorten typing
  • And some other tiny little things …
  • And the lab is packaged with Oracle VBox 5.0.10 including the matching VBox Extensions which caused us trouble with the previous lab environment which was settled on Vbox 4.3 – so we hope that now all will go well.

Where can you download the lab and the instructions?

All credits to Roy!!!

Thanks
–Mike

VBox 5.0.10 crash issues with our Hands-On-Lab

Milano - Nov 2015 (c) Mike Dietrich

I’ve ran two Hands-On-Workshops with customers and partners in Italy last week in Milano where we used our well known and thousands-of-times proven Hand-On-Lab environment:

But this time some people failed while running the lab with random corruptions either shutting down the entire VM while running – or displaying file corruptions in the spfile – or other issues.

The common thing in all cases: People had VBox 5.0.10 downloaded and installed right before the workshop.

Of course they’ve did it – as I’m tempted too since weeks. Every time I start VBox on my PC Oracle Virtual Box asks me:

Even though the screenshot is German you know what it offers me:
Download and Install Virtual Box 5.0.10.

Actually the current issue reminds me a lot on what I have experienced in 2014 in an Upgrade Hands-On Workshop in Vienna, Austria. 20 Oracle partners came together for two days for a Hands-On Upgrade/Migrate/Consolidate training. And 6 or 7 had random issues with their Virtual Box images. Corruptions. Failing upgrades at random phases. No patterns.

Only until somebody figured out via a Google search that at the same time other people started reporting similar behavior with their own VBox images using the brand new version of Virtual Box. It turned out that this newest version of Oracle Virtual Box 4.3 (I think it was 26) had exactly such issues. Everybody else in our room – including myself – running a version a few weeks older had no issues at all.

When we exchanged the affected installations the next morning replacing it (if I remember correctly: 4.3.24) all went fine for the rest of the workshop.

I won’t say that VBox 5.0.10 is bad as I lack evidence, reproducible test cases, bugs. 

But I follow other people’s Twitter and Facebook messages. And it seems to be that the PERL problem I did report a few days back:

Oracle VirtualBox 5.0.x – Segmentation Fault in PERL

is not he only issue with VBox images build in version 4 – and now running (more or less) on VBox 5.0.10.

Please see also:

–Mike

Oracle VirtualBox 5.0.x – Segmentation Fault in PERL

Please see as well:

VBox 5.0.10 crash issues with our Hands-On-Lab


 

Yesterday and the day before I’ve exchanged several emails with Ana who downloaded our Hands-On-Lab from here:

after OOW15, encountering a SEGMENTATION FAULT when trying to start the database upgrade with catctl.pl:

$ $ORACLE_HOME/perl/bin/perl catctl.pl catupgrd.sql

Segmentation fault 

Very strange thing … 

The database is in upgrade mode (checked this in the alert.log) and there are no strange things mentioned anywhere. Plus hundreds of people have run and completed our lab so far.

Tue Nov 10 20:39:47 2015
MMON started with pid=21, OS id=9828
Starting background process MMNL
Tue Nov 10 20:39:47 2015
MMNL started with pid=22, OS id=9832
Stopping Emon pool
Tue Nov 10 20:39:47 2015
ALTER SYSTEM enable restricted session;
Tue Nov 10 20:39:47 2015
ALTER SYSTEM SET _system_trig_enabled=FALSE
SCOPE=MEMORY;
Autotune of undo retention is turned off.
Tue Nov 10 20:39:47 2015
ALTER SYSTEM SET _undo_autotune=FALSE
SCOPE=MEMORY;
Tue Nov 10 20:39:47 2015
ALTER SYSTEM SET undo_retention=900 SCOPE=MEMORY;
Tue Nov 10 20:39:47 2015
ALTER SYSTEM SET aq_tm_processes=0 SCOPE=MEMORY;
Tue Nov 10 20:39:47 2015
ALTER SYSTEM SET enable_ddl_logging=FALSE
SCOPE=MEMORY;
Resource Manager disabled during database
migration: plan '' not set
Tue Nov 10 20:39:47 2015
ALTER SYSTEM SET resource_manager_plan=
SCOPE=MEMORY;
Tue Nov 10 20:39:47 2015
ALTER SYSTEM SET recyclebin='OFF' DEFERRED
SCOPE=MEMORY;
Resource Manager disabled during database
migration
replication_dependency_tracking turned off (no
async multimaster replication found)
AQ Processes can not start in restrict mode
Starting background process CJQ0
Tue Nov 10 20:39:47 2015
CJQ0 started with pid=27, OS id=9836
Completed: ALTER DATABASE OPEN MIGRATE 

We checked several other things – and then I came across this tweet by Martin Klier yesterday:

and started to search a bit.

I have no 100% proof for the actual reason but several people seem to have issues with SEGMENTATION FAULTs in Oracle’s PERL ($ORACLE_HOME/perl/bin/perl) when using Oracle VirtualBox 5.0.x – and according to VitualBox Forum that seems to happen with the most recent VBox 5.0.10 as well.

The “funny” thing is that all works perfectly well in VBox 4.3.x

It reminds me a lot on the reoccuring VBox bug with my German keyboard not allowing me to type in the | (pipe) character which requires to press “ALT GR” +  “<” keys together.

–Mike
,
.

OOW 2015 – if you’ve missed our Upgrade Hands On Lab

Ready to kick off with the first of our four Hands-On-Labs at OOW15 at the Nikko Hotel. Currently it’s calm before the storm  …

OOW 2015

But just in case you didn’t grab a seat at one of the labs then please come to our talk today (Monday!) at 1:30pm in Moscone South 102 as we’ll show you the core parts of the lab – and you’ll be able to download it later on from the blog and run it by yourself.

Download the lab via this link:

–Mike

Hands-On-Lab “Upgrade, Migrate & Consolidate to Oracle Database 12c” available for DOWNLOAD now!

Wow … that was a hard piece of work. Roy put a lot of effort into getting our Hands-On-Lab on OTN for download. We promised to have it available after OOW – or at least a description how to create it by yourself. And finally it’s there. Find it here:

A few important things to mention before you start the download:

  • It’s a Virtual Box image
  • You will need to install Oracle Virtual Box first – and please install also the VBox Extensions
  • Your PC must have a 64-bit host operating system
  • You need to enable Virtualization options in your computer’s BIOS
  • You PC should have at least 4GB of RAM – having 8GB is very helpful
  • A fast disk (SSD) will speed up things
  • The instructions are available for download but are included in the download as well
  • The lab will guide you through the following tasks:
    1. Upgrade an 11.2.0.4 database to Oracle 12.1.0.2
    2. Plug in this database into a 12.1.0.2 CDB
    3. Migrate an 11.2.0.4 database with Full Transportable Export into another PDB
    4. Unplug an 12.1.0.1 PDB and plug/upgrade it into an 12.1.0.2 CDB

You’ll find a picture as screen background inside the VBox image always giving you guidance about “what to accomplish” and “how to switch environments”.

Enjoy 🙂

–Mike