It’s time for my Monday morning rant. I guess, I’m turning slowly into a grumpy old man. And today, it is about something which came on my radar some weeks ago. After understanding what it means, I declare Virtual Patching – the biggest nonsense I’ve ever heard about.
Photo by Trym Nilsen on Unsplash
Virtual Patching??
If you read this term for the first time, you may scratch your eyes at first as I did weeks ago when I read “Virtual Patching” in a brochure from a company offering Oracle services. I did ask the person who shared it with me, and we both were more or less wondering. How can you patch “virtually“? Or is this a new marketing gimmick, such as a patch flying in from outer space?
How can a patch be “virtual“? Especially since I’d expect a patch in reality “non-virtual” of course.
Or is it a new magic technology I may have missed so far??
As some of you know, in our team we patch quite a bit. Patching is closely related to database upgrades and migrations. In our reference projects we assist customers in having their environments setup up correctly before going live. In addition, we closely work together with our Advanced Customer Support teams across the globe synchronizing our patching knowledge.
But even there in our regular calls and email exchanges, nobody has ever mentioned “virtual patching“.
I discussed this with Daniel, and he sent me a link to a blog post from Pete Finnigan, a top Oracle security expert. Let me quote Pete with his permission (thank you, Pete!):
http://www.petefinnigan.com/weblog/archives/00001481.htm
Another option that started to appear around 15 years ago is the idea of virtual patching. This is the idea that you cannot or don’t want to patch with Oracle’s security patch so you deploy network software that is a special version of an application firewall or intrusion detection / prevention system. The way this works is that network packets are sniffed (or shared memory attached and parsed) and attacks that could exploit issues fixed in Oracle’s security patch are detected. This is complex and prone to error or hacker bypass and requires the vendors of the virtual patch to reverse engineer (or guess!) what Oracle has fixed; then work out how that fix could be exploited and then how an exploit that could hack the database software can be detected.As I stated above Oracle do not release details of what is fixed so this virtual patching is not perfect and involves a lot of work. Yes, I can see that a product such as this maybe be a vert short term barrier to when a patch of a particular system cannot be applied quickly but its not a perfect fix.
So at first, the idea seems to be around much longer than I thought. It is nothing new.
But is a “virtual patch” now a real patch?
This is complete nonsense
I’m shocked – and scared. Seriously.
There is absolutely NO patching in “virtual patching”.
Zero.
It is like putting my healing hands on your system and promising that it will be protected from now on. If you trust me, please send me a DM, and in reverse I will send you the number of my secret Swiss bank account (which does not exist yet). I wonder how many systems I could “protect” in parallel. I have just two hands unfortunately, but since the patching is “virtual”, my hands could protect your systems “virtually” as well. I should call it “Mike’s Magic Virtual Patching (MVP)” and have a price list attached. [Be aware: this paragraph my contain serious pieces of sarcasm and irony]
At 10:50h on Monday morning, I’m still venting. I really saw and read a lot.
But credible customers with large infrastructures, business critical environments – putting their bets on a broken promise called “virtual patching“?
Let me explain why this is complete nonsense. And if you don’t trust me, please read Pete Finnigan’s blog post again.
The idea of this broken promise is that an external person having no access to the Oracle code would be able to setup up a surrounding protection fence for an Oracle database environment, like a firewall. And maybe it includes also reverse engineering of security patches. Which by the way – in my understanding – would be a violation of license agreements. I just can’t foresee what would happen in terms of an audit.
But even worse, a security patch often isn’t just a new version of a PL/SQL package, or a removed GRANT. It is often a code fix as well. And you’d trust a piece of software erected by somebody who has no code access promising that this will protect your environment?
Good luck with that.
There is no way around security patching
I’m not sure how often I repeated this truth in the past weeks to customers, especially to those who think that they can stay on Oracle 11g for the next years.
You must patch your environments on a regular basis. The fact that your current release may not be mentioned anymore in the risk matrix associated with the quarterly patch bundles just means that the release is not under bug fixing support anymore. But of course it doesn’t mean that your release may not be affected.
The reverse usually is true.
Oracle Database risk matrix – July 2022 – see: https://www.oracle.com/security-alerts/cpujul2022.html#AppendixDB
See the rightmost column. The reason why 10g, 11g, 12.2.0.1 and 18c are not listed is NOT that these releases may not be affected. It is simply that none of it is under any sort of regular bug fixing support anymore.
You should always keep in mind that at the day Oracle publishes security fixes, the information about the issue itself may circulate somewhere already when the issue had been found by an external person. Internally we find security issues as well, and fix them as quickly as possible.
Patch quarterly to protect your environments
We all know what can happen if you don’t patch on a regular basis, especially for security reasons. Since we all are in IT, each of us knows numerous examples of missed and postponed patching sessions, and the disastrous results. I could name a few as well, at least one which caused a multi-billion dollar damage where somebody refused to upgrade the 9i databases for years.
And one of the most impacting ones even made it into Larry Ellison’s keynote at OOW 2017: Equifax.
I cut out the important scenes – so please take these 5 minutes and listen carefully:
For the complete keynote video please see this link.
I think there is nothing more to add here.
Do you think “virtual patching” will protect you against such incidents?
If you trust a myth called “virtual patching“, you also believe in fairy tales, unicorns and my magically healing hands.
There is NO patching in “virtual patching“.
Absolutely ZERO.
Virtual Patching is clearly the biggest nonsense I’ve ever heard about – for a long time.
Further Links and Information
- Pete Finnigan: Should we security patch Oracle databases?
- Why you can’t upgrade from Oracle 11g and 12c to Oracle 23c
- Why you can’t stay on Oracle Database 11g forever
- Larry Ellison: Keynote OOW 2017
- MOS Note: 742060.1 – Release Strategy – Single Source of Truth
- Market Driven Support for Oracle Database 11.2.0.4
- Multiple-hop upgrades
- Long-term support vs innovation release
- AutoUpgrade – a new version is available
- Equifax Data Breach on wikipedia
- Feb 2022: FTC Equifax Breach Settlement
- Feb 2020: weird.com – How 4 hackers took down Equifax
- Feb 2020: LA Times – Equifax left unencrypted data open to Chinese hackers. Most big U.S. companies are just as negligent
–Mike
Hey Mike,
Thank you for one more nice article. I could feel how upset you were writing this. 😀
Keep going 🙂
Mike, I think you are correct about virtual patching not being a solution.
However, people are willing to try an alternative to Oracle Support since they can no longer see the value for their money. We hear the price will rise soon. I get very poor support since the pandemic started (even with SR escalated), and have always ended up solving my problems by troubleshooting myself actually!
People are willing to give a chance to another company that has good support, knowledge base, and some solution to security for cheaper.
Hi Erik,
I understand and know what you mean.
Still, I think putting the future of your database environments which usually are mission critical into a company which promises great things which don’t work is a real risk. And it may end up terrible.
Cheers
Mike
Great article! The obvious usually seems to be self-evident, but some of us will always fall for it. Patches do exist and are released on a regular schedule. Virtual patches do not exist. They are mere fiction.
I’m always wondering, what could be the so famous ROOT CAUSE of having so many bugs and issues in most enterprise-like products, which is requiring quarterly patching over and over. And always just with the same outcome each and every time – after the patching is before the patching …
I’m strongly believe there is something going fundamentally completely wrong in Software Industries.
It is the result of the chase of being the first in delivering of the next “hottest stuff”, which will never be used by 99,9% of users anyway. Overloading products with functionality, just to state “We have this feature too!” etc. etc.
All this makes products highly complex and error-prone.
Unfortunately, there is no real option for the customer left – all enterprise-like product of all vendors are hit by the same issue …
And vendors are in bad position too – What if they would invest more time in software quality and not take part into the “we need to add this currently hyped feature to our product too” chase ?
Thay MAYBE would be able to deliver a better, more stable product – but they WILL be definitively loose market shares too their competitors … vicious cycle …
Just my personal opinion, of course … 🙂
Hi Silvio,
Enterprise software has so many lines of code – and new features get added, in our case even an architectural change (Multitenant).
If you ever have the chance to attend our “From SR to Patch” talk at a conference or virtually, please do so – and you’ll see that things aren’t just black/white. And actually, this applies to all sorts of software, regardless whether it’s Enterprise or not.
Thanks for your feedback!!
Cheers,
Mike
I had the pleasure of dealing with a client that decided to save a few bucks by going with a 3rd party “Oracle Support” provider…probably the one you are referring to. As far as I am concerned that company is a sham operation that poorly rebrands a McAfee database SQL firewall product and Waratek’s “Virtual Patching” for Java applications. (Aka, a Java class file injected into the CLASSPATH to prevent certain classes from being executed.)
I am fairly certain that the customers are making the 3rd party support decision based on one or more of the following:
(1) The customer is tired of Oracle’s strong-arm sales and legal tactics and would prefer not using any Oracle products. (This is generally what I see. Oracle burned too many bridges in recent years, and it’s going to take a very long time to regain customer trust…if ever.)
(2) The buyer did not have the technical background to know any better, and they declare the cost savings as a win.
(3) The company is going broke and has no other option. In this scenario they might as well stop paying anyone for support.
The half-baked 3rd party support solutions will not fix vulnerabilities in other parts of the web/application/database/OS stack. Much less keep the tech stack in a supportable configuration over the years.
Good luck trying to explain that to a non-technical CIO/CFO/CTO who already made the decision and signed the contract primarily based on cost-savings or a grudge…
Thanks a lot for your insights!
And I know how hard it is to explain this to a non-tech person.
It just sounds too sweet, and marketing and sales are spreading tons of “make up” and “ice cream” to make this sound too good (to be true).
Cheers
Mike
Hi Mike,
Could please write a blog pots of how to patch a read only oracle home in RAC environment. I followed few steps provided by oracle support engineer and I have encountered an issue. There are not a lot of blog posts that describes this, so it would really good to have some clarity as this might save a lot of time during patching.
Regards,
Sandeep
Whilst not a simple fix and not virtual patching, SELinux might offer a way to sandbox an application. It works by running an app in permissive mode and capturing all the files and processes used by an application. Then converting these into a set of rules and setting to enforcing mode that limit the app to those.
There are still some challenges with this.
You have to ensure you run all the things the app needs to do, otherwise it will stop valid functionality. That could be a lot of work
Also not clear if you did this on a non production environment, are the rules transferrable, since the file and process IDs may differ from one env to another.
If it works, in theory, the exploit of a vulnerability would be limited because, no matter what the vulnerability, ultimately the hacker needs to call home, download a malicious payload or execute a command that would not be in the rules so would be blocked.
So not a panacea and a lot of work. Patching to remain tech current is still the best option to mitigate risk, but in certain limited cases, SELinux or equivalents on AiX can be worth considering.