Some of you may have seen the alert already released yesterday. And thanks to Peter Lehmann again for alerting me. When you logon to MyOracle Support, you will find it simply in the alert section to the left. On Jan 25, 2023, this Important alert for Oracle Database and GI RU 19.18.0 on Linux has been released.

Photo by Gary Bendig on Unsplash
What is the situation?
Due to an issue described in MOS Note: 2923428.1 – LMON trace file with repeated entries of “Submitting asynchronized dump request” the Release Updates (RU) 19.18.0 for the Oracle Database and Grid Infrastructure on Linux have been removed from the download section temporarily. As of yesterday, Jan 26, the MOS note has been precised quite a bit. So please read it to get more information.
You can find this information in MOS Note: 19202301.9 – Oracle Database 19c Release Update & Release Update Revision January 2023 Known Issues where it says:
The issue is known as Bug 35001966 – LMON TRACE FILE WITH REPEATED ENTRIES OF “SUBMITTING ASYNCHRONIZED DUMP REQUEST”.
This will affect only customers with GI, RAC and ASM since LMON does not run on non-RAC systems (thanks Anil for the clarification). And it has happened on Linux only since the bundles on other platforms weren’t released by the time this got discovered.
What you will find is LMON trace files being written with this information:
2023-01-25 13:11:12.039656 :kjzddmp(): Submitting asynchronized dump request [1c]. summary=[ges process stack dump (kjfcTask_chkCloud)].
Some people reported already massive trace file generation on each node of their RACs.
Download the Patch Bundle?
When you tried to download the patch bundles for 19.18.0 on Linux from Jan 25-30 it was not available.
Reason: We removed it and recut it with the fix included.
As of Jan 31, 2023, the RU 19.18.0 for Linux is available.
Fix available?
The re-release of the RU 19.18.0 on Linux is planned to happen on Tuesday, Jan 31, 2023. It should be available on MOS then again. For those of you who installed 19.18.0 already, a one-off patch will be available and accessible via MOS Note: 2923428.1. And of course, you can also go on with the below workaround.
Workaround available?
At first, from all I read and understand, this issue hits you only when you are working with RAC/GI/ASM – which will be a significant number of you certainly. But as far as my understanding goes, in case you use a standard single instance setup like I have it in our Hands-On Lab environment, I won’t be affected.
Furthermore, MOS Note: 2923428.1 – LMON trace file with repeated entries of “Submitting asynchronized dump request” describes clear workarounds.
Either execute on all instances:
alter system set events='trace[KJZD] { process: pname=LMON } disk=disable' ; alter system set events='trace_suppress[+] { process: pname=DIAG } regexp="kjdg.*_cb.*"' ;
or place this event and the below underscore directly into your init.ora/spfile on all instances:
event='trace[KJZD] { process: pname=LMON } disk=disable' event='trace_suppress[+] { process: pname=DIAG } regexp="kjdg.*_cb.*"' _diag_dump_request_debug_level=0
The idea of rebooting the node before 48 days are reached may be not realistic I’d guess.
Do you have to download it again?
Now a few people asked whether they still can use the original cut of 19.18.0 on Linux in case they have no RAC/GI/ASM. Clear answer: Yes, of course – since the issue being delivered with the original RU does not affect you. You don’t need the one-off patch either in this case.
Further Links and Information
- MOS Note: 2923428.1 – LMON trace file with repeated entries of “Submitting asynchronized dump request”
- MOS Note: 19202301.9 – Oracle Database 19c Release Update & Release Update Revision January 2023 Known Issues
- Bug 35001966 – LMON TRACE FILE WITH REPEATED ENTRIES OF “SUBMITTING ASYNCHRONIZED DUMP REQUEST”.
–Mike
Thanks for the article, Can you please share the same will applies to POWER systems(64-bit) AIX as well?
Hi Mohsin,
as far as I am aware (and this is what the bug says as well), this is a Linux-only issue.
But please check with Support as well.
Cheers,
Mike
Thanks for making sense of this. Compared to SQL Server, patching Oracle is by far more complex and time consuming. Thank you by making sense of this and future directions.
Well, Oracle is way more powerful and complex than SQL Server. But I agree with you. Patching must be made easier. And more reliable. Hence, I can understand your statement. We are pushing some very good enhancements which should make patching Oracle as simple as updating an app on your phone. Watch out for news in a few months.
Mike
Good enhancements? For Windows as well ? Or is it still the crappy way of bundling patches each quarter because development needs to compile a new exe instead of using dll’s ?
You should come to our talk “From SR to Patch – insights into the development process” since we explain the windows topic.
There is a reason for it, and they way they get delivered. On every unix system you have a compiler, normally at no extra cost or covered by the OS license. But not on Windows. Ask MS for a compiler license for all your windows DB servers. You’ll be not amused. Seriously.
If you have a compiler license on your DB servers, then you can request one-offs much easier. But trust me, not many spend this money.
So in this case you are blaming the wrong company. It’s not us.
Cheers,
Mike
Very interesting….every company developing on Windows is capable of delivering customer specific patches. The reason: they choose the correct way of developing on a Windows platform. Which is with the use of dll’s. The idea of compiling pieces of software on a target system at the customer is very 1970. The idea of using make, cc and ld on a Windows platform was a stupid idea. At the time when Oracle ported there software to Windows they made the wrong choices. These choices have been kept alive for to long. And now the software is so big and complex that it is impossible to change direction. It is always easy to blame someone else for your own mistake. Currently Oracle is does a tango with Microsoft and is pushing there products in the Microsoft cloud. As more Oracle rdbms software will be deployed on Windows, be sure the patching issue will become more of a pain.
Sorry to say that but the Oracle concept with one-offs is differently.
I highly recommend that you take the chance to attend one of our “From SR to Patch” sessions when possible. Things aren’t as simple as you think, and it’s not only about exchanging a dll in this case unfortunately.
Cheers
Mike
Hi Mike. Thanks for the heads up. It appears that the non-GI/ASM patch is being held back as well – just tried to download patch 34765931 for the 19.18 DB-RU and get “Details for Patch 34765831 not found”
Hi Devin,
yes, this is what I wrote (and had the screenshot for). As soon as a re-cut is done, the patch will appear again. They are fixing it right now, and I guess it will go through a regression test run as well before being reuploaded.
Cheers,
Mike
every 48 days?
[oracle@oraracn1 trace]$ grep -i ‘asynchronized dump request’ XXXXXX_1_lmon_15436.trc
2023-01-21 14:03:56.855 :kjzddmp(): Submitting asynchronized dump request [1c]. summary=[ges process stack dump (kjfcTask_chkCloud)].
2023-01-22 14:03:56.909 :kjzddmp(): Submitting asynchronized dump request [1c]. summary=[ges process stack dump (kjfcTask_chkCloud)].
2023-01-23 14:03:57.004 :kjzddmp(): Submitting asynchronized dump request [1c]. summary=[ges process stack dump (kjfcTask_chkCloud)].
2023-01-24 14:03:57.034 :kjzddmp(): Submitting asynchronized dump request [1c]. summary=[ges process stack dump (kjfcTask_chkCloud)].
2023-01-25 14:03:57.034 :kjzddmp(): Submitting asynchronized dump request [1c]. summary=[ges process stack dump (kjfcTask_chkCloud)].
[oracle@oraracn1 trace]$ uptime
17:20:41 up 4 days, 3:17, 1 user, load average: 1.37, 1.88, 3.05
RU19.18 was installed on 21st january…
RAC with GI and ASM…
Hi Manfred,
I am with you. I don’t understand these 48 days either – the bug is no help as it does not give more details.
But in the bug I see the same you copy/pasted above, recurring messages. I hope that the MOS notes gets more clarification soon.
Cheers,
Mike
what I currently don’t understand:
is it just the repeating log entries that may cause the trace file getting unusal big? MOS note says something about 200MB.
200MB every 48 days?
Or can the root cause of this messages do some harm to the database?
MOS note says “To prevent impact to Database systems, …”
Hi Manfred,
that is not clear to me either. I wait for some confirmation from the owners but haven’t received any. The MOS note immediately triggers 8 questions for me. Good to see that I’m not the only one. Bad to see that customers have the same questions.
As soon as I get a useful reply I will update the blog post with it.
Cheers
Mike
There are more issues with this retracted patch bundle. Datapatch gives an error when running against cdb/ pdb when open in migration mode. Seems we lack a spatial patch which was available in 19.17 and should be included into 19.18. And other not yet discovered issues.
Can you please share more information about the SPATIAL issue? I know that we’ve had one in 19.17.0 but hoped it will be cured in 19.18.0.
Cheers,
Mike
Bug 34830056 upgrade to 19.17 causes ORA-13199 “spatial index commit failure” during commit. I have at least 1 customer which upgraded from 19.17+patch to 19.18 and got the old spatial commit error.
The fix is clearly included in 19.18.0. Please check again.
Thanks,
Mike
Hi P. Musolf,
we also ran into ORA-13199: Spatial Index Commit Failure after RU17.
Regarding our SR I’ve installed Oracle Spatial and the RU17 Spatial Merge Patch. The error was resolved.
After applying RU 19.18 the error occurs once again.
Today, I’ve tried to resolve this and applied
Patch 34996737: MERGE ON DATABASE RU 19.18.0.0.0 OF 33726590 33899867 34049216 34132182 34162419
But the error remains. I’ve reopen my old SR from RU17. It’s very frustrating!
Cheers Peter
Clearly this is some sort of humble brag about the installations with uptime greater than 42! days, and the 48 is just trying to divert us from the answer to the universe.
Mark,
I’m pretty certain that you are right 🙂
🙂 🙂 🙂
Cheers,
Mike
work-around not working for me.
SQL>alter system set events=”trace[KJZD] { process: pname=LMON } disk=disable”;
alter system set events=”trace[KJZD] { process: pname=LMON } disk=disable”
*
ERROR at line 1:
ORA-02246: missing EVENTS text
SQL>alter system set event=”trace[KJZD] { process: pname=LMON } disk=disable”_diag_dump_request_debug_level=0 comment=’sb1_ru_19.8_fix’;alter system set events=”trace[KJZD] { process: pname=LMON } disk=disable”;
alter system set event=”trace[KJZD] { process: pname=LMON } disk=disable”_diag_dump_request_debug_level=0 comment=’sb1_ru_19.8_fix’;alter system set events=”trace[KJZD] { process: pname=LMON } disk=disable”
*
ERROR at line 1:
ORA-00911: invalid character
It had been updated – sorry for the inconvenience.
Cheers,
Mike
Hi Mike, I patched a test RAC cluster to 19.18. Got hit by the bug one day later (database trace folders filling like crazy). Created a ticket (but found your post before the support team got a chance to act). Oracle support suggests to roll back the patch, wait for the re-release of 19.18 (ETA sometime in February) and then redo the patch. I’m not very enthousiatic about rolling back – will there be a one-off to patch 19.18 version “bad” to 19.18 version “somewhat better”?
Hi Jan,
when you use the events (I updated the blog post now with the corrected ones) I see no reason to rollback the RU. The re-cut 19.18.0 will have a fix for it, and I guess with 19.19.0 this will be fixed by default. So you should be able to safely continue with the events surpressing the dumps, and then remove them as soon as you go to 19.19.0.
Cheers,
Mike
Hi Mike. Please disregard yesterday’s question. see oracle now has updated the metalink note for the work-aorund.
Hi AJ,
actually your message was good since it helped to alert some people that their code is not complete. I think the problem was ” vs ‘ initially. But as you can see now, they have another event to be set to surpress the tracing.
Thanks again, and sorry for all the inconvenience,
Mike
Hi, from oracle support note 2923428.1: Replacement patches to address this issue are being expedited and estimated availability date is Tuesday 31-Jan-2023.
Thanks for the heads-up Andreas!
Cheers,
Mike
Hi Mike,
the new patches are now available for download.
Greetings,
Andreas
Thanks Andreas, I updated the blog post.
Cheers
Mike
Mike,
I’ve tried twice now building a new database with 19.18 and it failed both times. We had created a database with DBCA 19.3 before doing the upgrade with no issues. It looks okay until it tries to start the instance and fails to identify the spfile (in ASM). Have you seen this?
Hi Edward,
no, I haven’t seen this. What error message did you see?
Did you create a CUSTOM database or one of the prebuilt ones?
Thanks,
Mike
Hi Mike,
we installed the original 19.18 RU (downloaded 1/19/2023) in all our non-production environments and patched all non-production databases to this level last Saturday.
I have read the current versions of the relevant documents, also that Oracle recommends uninstalling the original 19.18 RU and installing the current one.
But we don’t use RAC, GI, or ASM and if possible we don’t want different versions in production and non-production and our production patch date is already next week.
Therefore we plan to install the original 19.18 RU in the productive environments as well.
Do you think there is something that speaks against our planned course of action?
Greetings and thank you in advance for your much appreciated assessment, Hans-Martin
Hi Hans-Martin,
no, you can safely use it. The issue which lead to the recut of 19.18.0 won’t affect you if you have no RAC/GI/ASM, and hence you have no reason to download it again.
Cheers,
Mike
Am I the only one where download of the “new” RU19.18 doesn’t work?
The link in the MOS note directs me to https://updates.oracle.com/download/34762026.html
But there the dropdown box to select “Platform or Language” is empty, hence no download link gets presented 🙁
Tried with 2.5 different browsers (Firefox, Edge, Chrome)
Hi Manfred,
no, it is neither you nor your browser. The drop down boxes are empty, and hence you can’t download anything.
I alerted the people in charge …
This is frustrating to me as well.
Thanks for the heads-up, and sorry for the inconvenience again 🙁
Mike
Now It´s becoming really strange….
I´ve been able to download 34762026 yesterday around 13:30 CET
The patch description was as expected: Patch 34762026: GI RELEASE UPDATE 19.18.0.0.0 (REL-JAN230131)
But now it´s not available anymore. Also the README URL https://updates.oracle.com/Orion/Services/download?type=readme&aru=25093922 gives a “File not found”
I´m loosing a bit trust in 19.18
Hi Andreas,
see below – I have no idea what is going wrong there but the download did not work but I haven’t received any answer yet on what has happened or what is going on internally. I see just customers commenting with lots of curiosity.
Cheers,
Mike
I´ve also seen that DB RU (34765931) got re-released today again.
inventory.xml from yesterday showed patch_description as “DATABASE RELEASE UPDATE : 19.18.0.0.0 (REL-JAN230131) (34765931)” while the one from today “DATABASE RELEASE UPDATE : 19.18.0.0.230117 (REL-JAN230131) (34765931)” also the unique_patch_id changed from 25093776 to 25098466.
Due to this unique_patch_id change the diff of actions.xml shows a lot of differences obviously.
I see that Andreas – I just wait for a lot of answers from the owners …
Cheers,
Mike
https://support.oracle.com/epmos/faces/PatchDetail?patchId=34773504 is still there – but that’s thh kitchen-sink solution (Patch 34773504: COMBO OF OJVM RU COMPONENT 19.18.0.0.230117 + GI RU 19.18.0.0.230117)
I know – an ACS colleague has asked the exact same concern a minute ago.
I try to find out more.
Thanks
Mike
Strange!
Yesterday it was no problem to download the patches.
How can I find out which release I got yesterday?
Hi Andre,
I try to find out as well.
Thanks, and sorry for the inconvenience,
Mike
Hi Andre,
exact same software – only different entry in the inventory. Hence, no worries.
Cheers
Mike
https://updates.oracle.com/download/34765931.html is back. Dated Feb 1, though.
Looks like they released for RDBMS but they have change some of the “information” of the patch which is a bit messier for some automatic process you may have when checking for current versions.
The new 19.18 RDBMS version has this information when you run lspatches (19.18.0.0.0)
34765931;DATABASE RELEASE UPDATE : 19.18.0.0.0 (REL-JAN230131) (34765931)
When the previous one, had the “correct” (in my opinion) version (19.18.0.0.230117)
34765931;Database Release Update : 19.18.0.0.230117 (34765931)
Don’t really understand why they broke the “date rule”…
Thanks for this hint, Victor – I will try to find out.
Cheers
Mike
Good article! Thanks!
Hi,
Maybe a bit late but I would like to say that I agree with George Lavan that patching/updating Oracle products is way more complex the Mssql. I’ve been patching Oracle dbs and products like weblogic and the number op patch tools you need is scaring: opatch, datapatch, omspatcher, agentpatcher. And probably I’m forgetting some. Sometimes I’m lucky and are able to patch some compenents using OEM.
And CPU, RU’s, GI,etc adds more complexity.
I’m glad Mike says Oracle is working on it (to make it simpler).
regards,
Ivan
Hi Ivan,
I won’t dispute the arguments since I fully agree that we have to improve in this area massively.
We as a team try to improve this, and your feedback is seen internally too.
I hope that sometime later this year we can relief this part a bit.
Cheers,
Mike
Hi Mike,
I’ve been patching today 2 databases from 19.11 to 19.18 using the patch 19.18.0.0.230117.
What bothers me is that the datapatch does run a first time with errors and a second time without. The datapatch does even retry it a second time himself.
On top I have two interim patches (30978304, 35012866) installed, so the datapatch does pick them up too. However, these remain “WITH ERRORS” even after the implicit second run by datapatch itself.
The good news is, it all seems to get solved by running the utlrp.sql (validating the objects)
So, my question is, does anyone have experience with the datapatch parameter recomp_threshold ?
I would have hoped the invalidation would get solved by using this parameter, however setting it to a value=5 did not do the trick.
Cheers,
Ivan
Hi Ivan,
1. sorry to hear that
2. please have an SR and share the SR number with me the next time – makes it much easier for me to assist you
3. recomp_threshold does number the object count upper limit to whether datapatch does compile or doesn’t. So for instance, if you set it to “1”, datapatch will recompile only when it has 1 invalid object. From 2 on it won’t, and leave it to you then.
Cheers
Mike
Hi Mike,
Thanks for your feedback.
The Oracle SR 3-32377294551 has been closed in the meantime.
However , my main concern remains and wasn’t fully understood by the service engineer.
The first time, in some cases/on some databases, the first datapatch run returns errors on KUPV$FT_INT, which results in an overall datapatch returncode (rc !=0) different from zero.
Running a consecutive datapatch does not encounter those type of errors and ends with rc=0.
It’s a bit strange to encounter this behavior, whithout having a root cause solution for the first runtime.
However since I have a workaround, running it a second time, I did close the Service Request.
Kind regards,
Ivan
Mike,
I have created a dbcs system which has 19.18 and I have had performance issues compared to the 19.17 test system. I have seen this article https://support.oracle.com/epmos/faces/DocumentDisplay?id=2940472.1&displayIndex=1 and implemented the workaround but still see some direct path reads rather than reading from buffers. Is this something you are aware of?
Hi Phil,
you please need to walk forward with Oracle Support.
Thanks,
Mike
Thanks Mike – I have an SR open with support.
regards,
Phil
Hi Phil,
thanks – saw it in your email and answered it yesterday after I commented on the blog 🙂
Thanks!
Mike