Well, there are still 80+ comments I need to answer and reply to on the blog. So please be patient – nothing gets deleted or ignored. But it happens that a topic crosses my inbox, and I need to blog about it right now. Today, it is a case where a customer asked me for advice, and mentioned on the side that they are going to 19.6.0 in OCI. In this relation I’d like to explain why Certifying an application on a specific RU- only is wrong.
What’s the story?
At first, I’m fully aware that some of you will completely disagree with me when you read the headline. But before you vent, please read first.
I can tell the story quickly motivating me to write up this blog post. A customer mailed me with some questions regarding a migration to OCI. And on the side he mentioned that they will go to 19.6.0. In my reply I asked whether this was a typo, and the “1” was missing – which would have meant 19.16.0 instead. But he confirmed that it is indeed unfortunately 19.6.0.
So I went to the coffee machine and brewed myself my morning espresso, scratch my head. And it dawned me where this may come from.
The old days …
In the old days of Oracle 10g, 11g and 12.1 (old days refers to my perception since I deal with 19c and future, but I’m aware that some of you just approached your upgrades to 184.108.40.206 as I learned last week from another customer) you certified an application on a patch set. For instance, Oracle EBS did not get certified on 220.127.116.11 as far as I am aware. The same applies to SAP and many other applications. So when an application was certified on, let’s say, 18.104.22.168, this meant it was certified on a full release.
Then, when you moved from 22.214.171.124 to 126.96.36.199, this was a database upgrade involving downtime. You were using catupgrd.sql or DBUA since those were the old days before AutoUpgrade got born. 188.8.131.52 had significant changes compared to 184.108.40.206 as many of you experienced moving this route.
Now we all agree that an application certification was bound to a database patch set release.
Today and in the future …
But all this has changed with Oracle Database 18c when we introduced the new release model, and with 220.127.116.11 already the new patching model with Release Updates. So while in 18.104.22.168 the RUs added an additional longer number including the date to the release (for instance 22.214.171.124.RU20180722) in 18c this became much easier to digest. 18.16.0 clearly points to RU from a given quarter in a given year.
And certainly, you find the same concept with Oracle Database 19c and Oracle Database 21c. In July 2022 we released 19.16.0, in October 2022 we will release 19.17.0 and so on. Much easier for everybody.
So where’s the glitch?
The customer’s statement made me think. Why would one certify an application on 19.6.0? This makes no sense to me.
I know that some of you will disagree but keep in mind that the new features we rolled out such as blockchain tables in 19c RUs require a COMPATIBLE change which we disrecommend to do when you patch for many reasons. Hence, an RU should not involve any functionality change. Of course, there are a lot of fixes in an RU, and we all know IT too well to not being skeptical when you pack 7000+ fixes together that it may not cause one or the other regression. We all know by experience that this happens from time to time. And no discussion, this is frustrating for you but for us as well despite the efforts to improve quality.
Still, in my opinion, you should never certify an application on a given patch level only.
When you compare the old and the new release model, it boils down to me to:
- 126.96.36.199 => 188.8.131.52 … this was a patch set upgrade and you must re-certify your application … it is the same as in the new release model when you move from 18.13.0 => 19.16.0 – then you upgrade, you have downtime, you will re-certify your application certainly
- 184.108.40.206 PSU 20180119 => 220.127.116.11 PSU 20181021 … this is a patch operation, not an upgrade – and you never certified your application again for the new PSU. Of course, you did basic functional testing most likely, and you certainly did some performance testing and checks. But you didn’t ask the application to certify again for the new PSU. I guess you don’t certify again when you apply the 19.15.0 RU to your 19.13.0 database environment, right?!
I think this is where the glitch happens – the “new” release model which is now around for more than 4 years leads to this misunderstanding.
If I’m wrong with my assumption, or if you solidly disagree with me, please use the comment option and be patient (see above – 80+ comments to work on before keeping me busy).
But when I look at our own applications such as Siebel or EBS or PeopleSoft, they are certified on 19c or 18.104.22.168 – but not on a given RU or PSU. The same applies to SAP and others. Of course, vendors may give recommendations for certain patches. This makes sense. Or decide that a given patch is a requirement to be applied.
Certify your application on the base release such as 19c. Or if you intend, do the first certification on a given RU and make this the minimum RU such as “Our application requires at least Oracle Database 19.12.0”. But don’t make this an exclusive statement such as “Our application requires Oracle Database 19.12.0” since this makes no sense to me.
Certify on a base patch level, make this the entry requirement – but allow RUs to be applied on top by saying or writing something such as “Our application requires at least Oracle Database 19.12.0 – or newer”.
And don’t go live in 2022 on 19.6.0 – you will regret it. I bet a good decent bottle of LBV Port that you’ll find yourself patching to a much newer RU just a few days after go-live. Trust me. You miss 6721 fixes, no joke!
This is a problem we have against our vendors as well. As we have hundreds of different applications running in our company I have been quite frustrated when reputable vendors are saying they do not support Oracle 19.15 as they at the moment support only 19.4. This version is way back and for me as a DBA this represent both a security concern as well as a concern when I am in the need of support from Oracle. But still – seems like many peoples and vendors lives in the old days, unfortunately
I know – and this is a reason for my blog post. We have seen similar issues before but mainly caused by ourselves when vendors only certified 18c but refused to certify 19c. I hope that vendors will understand.
oracle produces new bugs with every ru and every software must find the ru that works best. if you are on windows it’s even worse. last stable ru was 19.11 – since then only showstoppers.
you cannot simply patch a database and expect that everything works…
I think you and I are both in agreement that this should not happen. Of course, with 500 and more fixes, there is always a risk for a new regression since software is complex. But you can just stay away from patching because there is a risk and potential for a new issue. And you can’t just freeze an app for months and years without applying the fixes being available.
And one day they come back with the idea of using Autonomous and their idea of certifying against a specific (sub)version will be gone. 🙂 One week it’s 19.x, next week 19.(x+1), the week after the DST changes, all “on its own”. Not mentioning the routine weekly patching.
This is a decades old problem. I work for a pharmaceutical company and FDA/GxP validation is a constant effort and concern. Vendors in this space want a software stack installed and then for the customer to do nothing. Don’t patch anything and keep the software stack frozen. But that’s not possible as we must patch security / stability issues in O/S, database and apps. Regardless, some ISVs push back stating we might then create an unsupportable situation for them. It’s so frustrating but what I’ve seen over the years is that rarely does an O/S or database patch break an application. We have no option – our Security teams & SOPs make us patch which we adhere to but that’s in direct conflict with many ISVs/third-party vendors. It’s been that way forever in this field. I don’t see it changing. ISV’s are covering their butts by stating only a defined software stack is fully certified.
I agree with you, Mike. And especially in the area you work in I’ve had numerous discussions since such software got embedded also in hardware which makes it extremely hard to update and patch – but all these devises are “online” … sometimes I’m frightened and anxious …
And my guess is that we’ll never have a “solution” as such the true best way.
Think you’re wrong on the EBS part.
These are the posts on the EBS technology blog:
Well, what should I say?
At first, thanks for the hint. And second … it sounds odd to me.
HEnce, thanks for the pointers – and I can’t comment further.
Hay Oracle Database 19c supports on IBM Power Linux ?? when it will be updated at CERIFICATION page??
Oracle certifies always the OS. And AIX is certified with many versions for 19c.
Oracle does never certify for specific hardware.
If you refer to the processor architecture, and refer to zLinux, then certainly Oracle 19c is certified on zLinux.
How do you feel about vendors who lock in database software to a specific OS release? I recently tried installing 19.3.0 on OEL 8.6 and found I had to set CV_ASSUME_DISTID=8.1 before it would install. I can understand not allowing an earlier major OS release due to potential incompatibilities, but I have not seen runInstaller require a specific patch release until now.
we only certify against a minimum release, such as OL8.1 – and from there on, all OS patches and bundles are supported generally. It was always the case with Oracle, and as far as I am aware this has never changed. You need the workaround with the variable to “tweak” the install process. This has to do with the fact that when 19c got released, it was not certified on OL8. This certification came later. I would assume that a newer RU does not require this tweak anymore. And you should not use 19.3.0 vanilla but apply the most recent RU on top:
==> A very good reason for vendors to certify only on an specific RU!
Hahaha … you’ve caught me … but seriously, THIS should not happen, especially not:
a) undocumented and
b) without a documented workaround or prevention
So I still insist on my statement – just “somebody” tries to undermine my takes 🙂