You like unexpected changes and surprises, don’t you? And especially those which aren’t in the patch notes or the docs. I blogged about such changes a few weeks ago. And thanks to the people reading this blog, I learned now about another change with Oracle 19.10.0 on the Windows platform. You may receive now an ORA-12638 on Windows only from Oracle 19.10.0 onwards.
Photo by Fanny Rascle on Unsplash
What has been changed?
So at first, thanks to Ernst and Marcus for bringing this to my attention. This is an issue which happens on MS Windows only.
When you have SQLNET.AUTHENTICATION_SERVICES=NTS in your sqlnet.ora, then you may see now ORA-12638: Credential retrieval failed when you for instance select over a database link. Or it happens simply when you try to connect from the client using sqlplus user/pw@servicename.
When you’d do a SQLnet trace, you would spot this message: NO_NTLM set, not local conn, not actively falling back to NTLM even if server might not support kerberos. And this points to the solution. NTLM is the Windows Networks LAN Manager authentication service. Access to it has been disabled by default due to a security fix. So NO_NTLM=TRUE is now the standard from 19.10.0 on for Windows systems.
How to solve it?
I hope you don’t have too many clients.
You will need to set this in your client-side sqlnet.ora:
SQLNET.NO_NTLM=FALSE
Then everything should work as it has been worked before.
Further Questions?
As many of you commented already, I have not more information about this issue – and especially no further advice. I would like to ask you to open SRs and check with Oracle Support for further guidance as I don’t have the chance to test with all the different client options and possibilities. Thanks!
Further Links and Information
- MOS Note: 2757734.1 – Windows: While Connect to Database Getting ORA-12638 After Applying Jan 2021 WINDBBP 19.10.0.0.210119
- New Parameters in Oracle 19.10.0 – and a default change
–Mike
Glad to have helped point this one out, Mike! Whew!!
Ein Prosit!
-Ernst.
I have to thank you, Ernst!
Cheers,
Mike
Thanks to bring that information now on your blog. Maybe others have find this information before they stumble over it.
Whereby since 6.March there is now also a Doc ID 2757734.1 pointing to that behaviour.
Offtopic, but, hope the DOAG takes place this year … would like to talk to you in “REAL”, not only in chat or comments.
Prost 😉
Marcus
Hi Marcus,
I linked it from the blog already – see the last paragraph 🙂
Thanks,
Mike
Mike, I have a SR open with Oracle support on this very subject.
IF ORACLE_HOME is not set we get the ORA-12638, if it is set, we do not see the ORA-12638.
It worked fine on 12.2, but is broken for us with 19c.
Hi Ted,
the Support Engineer will file a bug for this behavior.
Cheers,
Mike
Thanks Mike. Huge detail for those of us on MS Windows platform
Hi Mike,
I was facing this issue recently while connecting to my 19.10. 0.0 DB on Windows server.
It was in my task list to troubleshand and your post came in like a charm this morning 🙂
Thanks much!!
Regards,
Dabir
Hi Dabir,
hope the workaround works for you.
Cheers,
Mike
Hi Mike,
I think my situation is a bit different now. I am encountering ORA-12638 only from Toad (with or without applying the fix of NO_NTLM=FALSE).
Normal sqlplus connectivity is okay even without the fix.
But I started to see it only after 19.10 patching as I use this server everyday to connect my databases.
Regards,
Dabir
Hi Dabir,
I fear you may need to check with Oracle Support.
Cheers,
Mike
Moin Mike,
would this issue be relevant for ODBC database links accessing MS-SQL-Servers using Oracle DG4ODBC from an Oracle 19.10 database?
It is a specific use case I am asking about … but the 11.2.0.4-environments where this is happening right now will upgraded to 19c in the near future …
Hi Christian,
I can’t tell you – you please may need to raise an SR.
Cheers,
Mike
Wow, thank you so much for sharing this information. Having spent the last couple of days, after rolling out 19.10, with production stop in our Windows duplicate database jobs (which we do a lot of every day), your post here came as a ray of sunshine over my morning coffee. I checked our sqlnet traces and found the message you highlighted. Changed the sqlnet.ora to contain SQLNET.NO_NTLM=FALSE in the auxiliary database 19.10 home and Bob’s your uncle, we are again able to duplicate databases. Kudos to Ernst and Marcus!
Btw, besides the lack of information in the patch notes for 19.10, I neither could find NO_NTLM documented in the Database Net Services Reference.
Neither could I, Jeannette – and as you wrote, kudos to Marcus and Ernst!
Sorry for the inconvenience 🙁
Take care!
Mike
Hey Mike. It looks like Doc 2757734.1 has been pulled (or made non-public). Any idea why this was? I have a colleague that has run into this issue and was trying to help him resolve. Thanks
Hi Bryan,
sometime MOS notes go into review state when somebody adds useful information.
Please check again, and if it persists, open an SR and check with Support please.
Cheers,
Mike
Thank you so much for posting this article. Your fix worked, except I had to restart the database after I changed sqlnet.ora.
Glad it helped 🙂
Thanks,
Mike
Mike,
This means that you have add SQLNET.NO_NTLM=FALSE in every client.
You could modify on the RBMS database servers the sqlnet.ora and set :
SQLNET.AUTHENTICATION_SERVICES = (NTS) to NONE.
But that will prevent you from logging in on the server with sqlplus / as sysdba.
I didn’t find any other solution even on the Oracle side how to authenticate with Windows credentials now they disabled NTLM. Any suggestions ?
Regards,
Paul
Hi Paul,
this is not really my area of expertise – you may please need to clarify this with Oracle Support.
Cheers,
Mike
Same issue is raised in 19c oracle rac. ASM fail to start. Issue is resolved using SQLNET.NO_NTLM=FALSE. This works for me. Thanks a lot and really appreciate.
The Oracle 19c RU 19.12 Rac configured in Windows server 2019 standard. Sorry, I forgot to mention in my previous comment
Mike, THANK YOU!
thank you so much, you have no idea how much work you saved me. I just upgraded to 19c and after a few hours of troubleshooting I was getting ready to go back to 12c and your post came in like a charm
Kudos to Marcus and Ernst!
Regards,
Jorge
Hi Jorge,
glad it helped – and all Kudos go to Markus and Ernst 🙂
Cheers,
Mike
Hi, I thought I’d share my experience with Oracle 19.13 on Windows 2019 and having group policies configured per CIS OS hardening guide. The CIS guide suggested defining settings for computer configuration\windows settings\security settings\security options\Network security” Configure encryption types allowed for Kerberos by checking certain protocols. Once I enabled these protocols I could no longer login to the db as sys in a single line even after adding the NO_NTLM=FALSE instruction to my sqlnet file. The fix was to check ALL check boxes (or having this setting undefined completely) for this setting in the local group policy on the db server.
Thanks for sharing, Sergei!
Mike
Sergei,
Thanks for the information but I am not sure I follow (but I am running into the same issue). Can you please
explain further (we made no such hardening changes)?
Thanks in advance.
Nir
I am working a service request with Oracle Support but am not getting very far. I have two workstations where the older one connects successfully (with SQLNET.AUTHENTICATION_SERVICES=(NTS)) while the other/newer one does not. Obviously the DB server is configured the same way the difference must be on the client (some Windows configuration since both Windows and Oracle are patched to the same level on both workstations).
Here is the kicker, on the newer workstation that does not work, it does not matter if I add the “SQLNET.NO_NTLM=FALSE” setting and in fact, I temporarily reverted back from 19.14 to 19.3 and still experienced the same error so this is not related to the change in behavior introduced in 19.10.
We have other people experiencing the same issue (working on some but not others) and there must be some setting on the client that is responsible (as the Oracle clients we tested were identically patched).
Any ideas?
Nir
Hi Nir,
unfortunately not. But please:
1. Raise the severity of the SR to Sev.1 (non 24×7)
2. If you don’t see a response quickly, please escalate the SR verbally.
3. Call the HUB (your countries’ Oracle Support telephone number) – and have the HUB people escalate it
Unfortunately I have no idea 🙁
Thanks,
Mike
OK, thanks Mike, and I have a Zoom session with Oracle Support Friday morning to demonstrate the problem.
The initial response was that the TNS-12638 is expected behavior under Windows with SQLNET.AUTHENTICATION_SERVICES=NTS across domains (or a VPN connection) but that does not explain why one machine connects successfully and the other does not when the Oracle Net setup is identical (this was also never an issue with an 11gR2 client).
I was further told there is an enhancement request (ER 25087191) to allow both NTS and NONE to be set together so remote access with a password and local with NTS would still work but I am not sure it’s applicable here and I also don’t think it would help in the short term.
To be continued tomorrow hopefully…
Nir
Thanks for the update, Nir – I hope you can sort this our together with Support.
Cheers,
Mike
I added the SQLNET.NO_NTLM=FALSE to the sqlnet.ora file
However we are still getting the same ORA-12638
Hi Matt,
then you please need to check with Oracle Support.
Thanks,
Mike
Thanks! These days, when I have a problem, I always end up on your posts, and then my problem is solved 🙂
Thanks for the kind feedback, Sverker!!!
Cheers
Mike
Same issue here (still unresolved) and I have had an open SR with Oracle on it since March 8, 2022…
No resolution yet with Oracle Development looking into it (“SR 3-28856086231 : Getting ORA-12638 from Oracle 19.14 client (even with SQLNET.NO_NTLM = FALSE)”.
Nir
Interesting, Nir.
I checked your SR and I see that it points to the new bug:
BUG 34830121 – CONNECTION ERROR ORA-12638: CREDENTIAL RETRIEVAL FAILED ON WINDOWS 19C CLIENT
Now the bug has a comment that it may be a duplicate of:
BUG 34564005 – DBLINK CONNECTION FAILS WITH ORA-12638: CREDENTIAL RETRIEVAL FAILEDSev 1 SR
which sits in Status:30 (more information needed from filer) – the development owner is the same for both bugs, yours and the potential one it is a duplicate of.
But the 2nd one is actively worked on more or less.
You may need to update the SR with the 2nd bug I gave you since I can’t update the SR on your behalf (I have only read privs).
Ask your engineer to ESCALATE the bug and get in touch with the developer it has been assigned to to check whether this is really a duplicate and what can be done to solve your issue.
Cheers
Mike
PS: I dropped the engineer an email as well
Thank you Mike and done.
Mike,
Oracle Support is pointing to an old Microsoft note (that no longer exists) indicating it’s a Microsoft problem (documented in the following MOS note):
Ora-12638 When Connecting Or Reconnecting On Windows XP Or Windows 2003 (Doc ID 437895.1)
It’s been a frustrating ride as I never had this issue with older versions of Oracle (e.g. 11g/12c) with the same Windows 10 client… Asking me to work with Microsoft on a problem that only affects a 19c client (but no other software) seems like shifting the blame elsewhere.
I don’t doubt the problem is with a Windows security setting, but I need more than pointing into a no-longer existing Microsoft note as the solution…
Any thoughts or comments?
Thanks.
Nir
Hi Nir,
thanks for the update and the pointer to:
Oracle Support Document 437895.1 (Ora-12638 When Connecting Or Reconnecting On Windows XP Or Windows 2003)
https://support.oracle.com/epmos/faces/DocumentDisplay?id=437895.1
From my own Support days I can tell you that it took a lot of effort to stop the finger-pointing and have an escalation manager from MS getting involved and taking care on an issue we were certain is caused by Win, and not by Oracle. These days, I fear that such mechanisms aren’t even in place anymore. And I fully see how frustrating this is.
The only thing you can do as a customer is escalating this to the Support management within Oracle. You can’t involve MS, the Support team has to and figure this out if you’d ask me, no matter how painful this may be. But we have similar cases with RHEL from time to time where we quickly add a fix to OL but it takes sometimes much much longer for RHEL to fix something.
Cheers,
Mike
Mike,
After months of back-and-forth on this SR, I am back to Oracle Development telling me this is a Microsoft issue and that I should open a ticket with them… Keep in mind the only problem I am having is with the Oracle client (and nothing else) but I am not sure what I can do about it at this point.
Any thoughts or suggestions (other than dropping the issue)?
Thanks.
Nir
Hi Nir,
can you please refresh my memory, especially about the SR number?
thanks
Mike