Oracle 19c on Windows may flood your trace file directory

Oh … it’s Windows week here. And all this even though since I didn’t install Oracle on Windows for quite a while. But of course I’m fully aware that many of you out there operate Oracle on Windows. In this particular case thanks to Joël for the pointer to this issue. Oracle 19c on Windows may flood your trace file directory.

Oracle 19c on Windows may flood your trace file directory

Photo by Daniel von Appen on Unsplash

What happens?

In every release of Oracle 19c, at least until 19.10.0 BP, you may find out that every few minutes a trace file gets written into the %ORACLE_BASE%\diag\..\..\trace directory. And all of these files contain messages like these:

Required IPC RDMAV_FORK_SAFE environment not set
Required IPC RDMAV_HUGEPAGES_SAFE environment not set

Of course, you could easily delete them.

Why is this happening?

This happens because Linux environment variable checks have slipped into the generic codeline – and hence will error out under certain circumstances on Windows. This is not dangerous – but if you don’t pay attention, you may end up with 10-thousands of trace files.

You can find more information in MOS Note: 2588705.1 – 19c Database On Windows – ADR Is Flooded By Linux-related Trace Files.

Is there a Workaround?

The MOS Note: 2588705.1 proposes a workaround. You should set RDMAV_FORK_SAFE and RDMAV_HUGEPAGES_SAFE to 0. I guess, you should set it in addition to the user’s environment which runs the Oracle Service.

Another workaround coming to my mind and not in the note may be to change the ADRCI retention for less important traces.

Enter ADRCI:

$ adrci

ADRCI: Release 19.0.0.0.0 - Production on Thu Mar 11 16:17:49 2021

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

ADR base = "/u01/app/oracle"

adrci> show homes
ADR Homes: 
diag/rdbms/upgr/UPGR
diag/rdbms/cdb1/CDB1
diag/rdbms/db12/DB12
diag/rdbms/cdb2/CDB2
diag/rdbms/ftex/FTEX
diag/clients/user_oracle/host_3378310444_107
diag/clients/user_oracle/host_3378310444_80
diag/clients/user_oracle/host_3378310444_110
diag/tnslsnr/hol/listener

adrci> set home diag/rdbms/cdb2/CDB2

adrci> show control

ADR Home = /u01/app/oracle/diag/rdbms/cdb2/CDB2:
*************************************************************************
ADRID                SHORTP_POLICY        LONGP_POLICY         LAST_MOD_TIME                            LAST_AUTOPRG_TIME                        LAST_MANUPRG_TIME                        ADRDIR_VERSION       ADRSCHM_VERSION      ADRSCHMV_SUMMARY     ADRALERT_VERSION     CREATE_TIME                              SIZEP_POLICY         PURGE_PERIOD         FLAGS                PURGE_THRESHOLD      
-------------------- -------------------- -------------------- ---------------------------------------- ---------------------------------------- ---------------------------------------- -------------------- -------------------- -------------------- -------------------- ---------------------------------------- -------------------- -------------------- -------------------- -------------------- 
2334187521           720                  8760                 2019-04-28 00:18:35.957176 +02:00        2019-05-20 21:33:48.508472 +02:00                                                 1                    2                    110                  1                    2019-04-28 00:18:35.957176 +02:00        18446744073709551615 0                    0                    95                  
1 row fetched

adrci> set control (SHORTP_POLICY = 72)

I changed the retention for trace files which are not incident related to 72 hours (3 days). The default is 720 hours (30 days).

Of course you could do an immediate rescue as well and trigger a purge:

adrci> purge -age 4320 -type trace

This will cleanup all traces older than 72 hours or 3 days. Well, now I remember how strange I found the idea of having the control setting in hours but the purging on minutes. Actually 4320 minutes are 72 hours are 3 days. In case you wonder about the strange number.

Is there a fix available yet?

You won’t get one-off or interim fixes on Windows. And to make things a bit worse, Bug 30284751 – Trace File With Messages “Required IPC RDMAV_FORK_SAFE / RDMAV_HUGEPAGES_SAFE” Is Generated In Windows Platform has been fixed already for the next release but has been filed as a P4 bug. This is a low priority issue. And hence, the request for inclusion has been slipped through the cracks so far. Now the right people are checking it. No guarantee that it will be in the next Windows BP but we are trying our best.

Further Links and Information

–Mike

Tags: 19c bug Flood oracle 19c RDMAV_FORK_SAFE RDMAV_HUGEPAGES_SAFE Trace Files Traces Windows Workaround

P. Musolf says:

March 11, 2021 at 17:26

Mike,

I would like to invite you to run a Oracle RDBMS 19.x on a Windows environment. Instead of testing and migrating between Linux to Linux, try a migration between Linux and Windows and vice versa.
Just run a Windows RDBMS 19.x for 6 month’s and discover the pain customers have when running on a Windows platform.

– Windows patches are always a month later. The release of the Windows OJVM patch for jan 2021 took even 2 months.
– Unplug a Windows PDB into a pdb (archive) file and try to plug it in on a Linux…no way. SR 3-21269776311
– catupgrade on a Windows platform will randomly hang. Bad catcon.pm with bugs…Asking for a patch takes ages..

There a lot of issues solved in the Linux version, and not in the Windows version.

I can continue, the list is endless.

Regards,
Paul

Reply
- Mike.Dietrich says:
  
  March 11, 2021 at 21:40
  
  Paul,
  
  please don’t shot the messenger.
  
  Actually I grew up on “Oracle on Windows” when I started in Support in 1996. And as I was the newbie and had Windows experience, I was the one who got selected to do the 7.3.4 calls on WinNT. I’m as unhappy as you with what I see these days.
  
  But you need to raise this please to the people in charge. You have a voice as a customer!
  
  Cheers,
  Mike
  
  Reply
  - P. Musolf says:
    
    March 12, 2021 at 09:37
    
    Mike,
    
    For Oracle the concept of customers…is : I dont know. Escalated multiple SR’s and still no help or it takes weeks before it is solved. The catcon.pm problem took me 2 month’s: calling, making a video of the problem, calling again…..I don’t blame you. More as 10 satisfaction inqueries I descibed the problems…no respons ever.
    
    Reply
    - Mike.Dietrich says:
      
      March 25, 2021 at 18:36
      
      You should ask for management callbacks in a given time frame. This is what I recommend to my customers.
      And it is the quickest way to bring this to the process owner’s attention.
      
      And sorry for all the inconvenience 🙁
      
      Cheers,
      Mike
      
      Reply
- fateh says:
  
  May 11, 2021 at 08:56
  
  100% agreed. The support is super poor on Windows. Even I reported a critical bug in Oracle Wallet 19c since last year and never got fixed. So, we cannot upgrade to 19.10
  
  Reply
fateh says:

March 11, 2021 at 18:06

No one should install Oracle on Win.
First poor and slow support.
Second, 19.3c has a critical bug in SSL/Wallet reported 9 months, and not fixed till now; so, we had to downgrade back to 18.3, and since 18.3 already has bugs >>> we have to migrate to Linux 🙂

Reply
- Mike.Dietrich says:
  
  March 11, 2021 at 21:41
  
  Hi Fateh,
  
  sorry to hear that.
  
  Cheers,
  Mike
  
  Reply
ERNST KALWA says:

March 11, 2021 at 19:06

Thanks so much for this one, Mike! Yes, I had noticed all those trc files too. So on my customers’ Windows servers running 19c, I’ve set SYSTEM-level environment variables:
RDMAV_FORK_SAFE=0
RDMAV_HUGEPAGES_SAFE=0

I had to restart the OracleService for each non-CDB and CDB from the control-panel->services for it to take effect. I no longer see any new .trc files with those annoying warnings getting generated every minute!

Vielen danke!
-Ernst (Ernie)

Reply
- Mike.Dietrich says:
  
  March 11, 2021 at 21:42
  
  Thank you, Ernst!
  As you see, I “live” from the information customers like you, Marcus and Joel send me.
  
  I wouldn’t have known. But from the feedback and comments, I read that many were affected.
  
  Cheers,
  Mike
  
  Reply
Martin J S says:

March 11, 2021 at 21:29

Thank you for drawing your attention this very annoying problem and I am hopeful that this bug will be fixed soon!

I already pointed this out many months ago:
https://mikedietrichde.com/2020/04/14/patching-all-my-environments-with-the-april-2020-patch-bundles/#comment-21331

Be aware that there existed a similar annoying bug which was finally fixed in 19.9:
– Bug 30352715 – Job processes fill the filesystem with trace files after installing bug 29541742 (Doc ID 30352715.8)

Unfortunately we are stuck on Windows operating system for several of our customers. That does not make life easier in many aspects …

Keep up the fantastic work you are doing for the whole Oracle community!

Best regards,
Martin

Reply
JOEL PERAN says:

March 12, 2021 at 06:03

Thanks alot for your feedback Ernst !
By the way, though it is very true that patches and bug fixes are not always available at the same time for Windows, dealing with Oracle databases on Windows is not the mess it used to be (memory consumption on Windows 32 bits ? remember that :)). It’s far better now. Not perfect, but far better.
Thanks also Mike !

Reply
- Mike.Dietrich says:
  
  March 25, 2021 at 18:22
  
  Thanks Joel!
  
  Reply
Martin Decker says:

March 12, 2021 at 09:21

Hello Mike,

thank you for this heads-up. Your team´s work is very much appreciated (with the content you provide) and you in particular make a big difference for customers.

However, I am very disappointed in the organization, especially 2 areas: Support and Patch Delivery.
– support engineers only motivation is to close the SRs as soon as possible.

– they are not interested in improving the product or getting bugs fixed as soon as possible

– support engineer expertise is a big concern

– Your initiative to have one central MOS note for recommeded patches for 19c is very much appreciated but it demonstrates that there is something severely wrong with the whole product patching process.

– In the year 2021, I would expect the possibility to have a one-step approach to patch a system with ALL the recommended patches there are on a quarterly basis. Currently I have to constantly scan these notes:

Oracle Database 19c Important Recommended One-off Patches (Doc ID 2720807.1)
Mandatory patches for 19.8 and 19.9 Grid Infrastructure (GI) [2730332.1]
Things to Consider to Avoid Prominent Wrong Result Problems on 19C Proactively (Doc ID 2606585.1)
Master Note for Database Proactive Patch Program (Doc ID 756671.1)
Oracle Text Mandatory Patches (Doc ID 2644957.1)
Checklist For Slow Performance Of DataPump Export (expdp) And Import (impdp) (Doc ID 453895.1)
Disable Statistics Advisor Task after applying 26749785: exec dbms_stats.set_global_prefs(‘AUTO_STATS_ADVISOR_TASK’,’FALSE’);
Oracle Database 19c Release Update & Release Update Revision October 2020 Known Issues (Doc ID 2694903.1) (or later RUs)

– I can rarely make it happen that I get all the merge patches for a new RU by the time the next RU is available.

– Customers can NOT accept that they run into availability (e.g. 29780459) or WR issues on mission critical systems only realizing that the corresponding patch has been available for quite some time but was not included in RU. This reactive approach of patching interim bugfixes only AFTER an outage occurred is not appreciated by the customers. If you contact Support and ask for a list of patches for known availability issues in order to avoid this reactive approach, they come up empty handed.

Regards,
Martin

Reply
- Mike.Dietrich says:
  
  March 30, 2021 at 21:57
  
  Hi Martin,
  
  I see all your points.
  
  However, even patching may look as a simple process which can be optimised easily, it often is not as simple.
  For an RU, the given requirement is the “RAC rolling” ability. This excludes patches which change the API per se (e.g. Data Pump).
  
  Believe me, we are working on improving the process. As a result, you may have recognized that 19.10.0 contained A LOT of fixes compared to previous RUs.
  And 19.11.0 won’t be much different in the number of inclusions.
  
  I won’t make promises but be aware that all this has visibility to our Exec Management – and there are a lot of pushes in the direction you propose and expect.
  
  Cheers,
  Mike
  
  Reply
satya says:

March 23, 2021 at 10:43

Oracle version: Oracle 19.3.0
Platform: Microsoft Windows Server 2012 Standard
We have recently upgraded the DB from 12.2.0.1 to 19.3.0

==We are hitting the following errors in alert log:

Errors in file D:\ORACLE\app\diag\rdbms\xyz\xyz\trace\xyz_m001_3743.trc:
ORA-27037: unable to obtain file status
OSD-04011: GetFileInformationByHandle() failure, unable to obtain file info
O/S-Error: (OS 5) Access is denied.

===Trace file xyz_m001_3743.trc contains:

Error in computing freespace for file D:\ORACLE\ORADATA\XYZ\SYSTEM01.DBF
ORA-27037: unable to obtain file status
OSD-04011: GetFileInformationByHandle() failure, unable to obtain file info
O/S-Error: (OS 5) Access is denied.
Error in computing freespace for file D:\ORACLE\ORADATA\XYZ\SYSAUX01.DBF
ORA-27037: unable to obtain file status
OSD-04011: GetFileInformationByHandle() failure, unable to obtain file info
O/S-Error: (OS 5) Access is denied.
Error in computing freespace for file D:\ORACLE\ORADATA\XYZ\UNDOTBS01.DBF
ORA-27037: unable to obtain file status
OSD-04011: GetFileInformationByHandle() failure, unable to obtain file info
O/S-Error: (OS 5) Access is denied.
Error in computing freespace for file D:\ORACLE\ORADATA\XYZ\USERS01.DBF
ORA-27037: unable to obtain file status
OSD-04011: GetFileInformationByHandle() failure, unable to obtain file info
O/S-Error: (OS 5) Access is denied.
Error in computing freespace for file D:\ORACLE\ORADATA\XYZ\TEMP01.DBF
ORA-27037: unable to obtain file status
OSD-04011: GetFileInformationByHandle() failure, unable to obtain file info
O/S-Error: (OS 5) Access is denied.

Actually Oracle instance is able read, write , modify, update, delete data from user defined tables(DML).
But how come the messages “ORA-27037: unable to obtain file status” and “O/S-Error: (OS 5) Access is denied” appearing over and over again in alert log and mmon and mmonl traces.

Anywork around or fix will be available to suppress these messages

Reply
- Mike.Dietrich says:
  
  March 30, 2021 at 22:41
  
  Please open an SR for such issues.
  I’d recommend you using 19.10.0, and not 19.3.0 without any RUs as you miss several thousand fixes at this point.
  
  Cheers,
  Mike
  
  Reply
satya says:

March 26, 2021 at 18:27

Hi Mike,

We are seeing the following errors in alert log and mmon, mmnl trace files on Oracle 19.3.0 (Windows).
Recently we upgraded from 12.2.0.1 to 19.3.0.
Following entries are flooding alert log and traces.

Error in computing freespace for file D:\ORACLE\ORADATA\XYZ\SYSTEM01.DBF
ORA-27037: unable to obtain file status
OSD-04011: GetFileInformationByHandle() failure, unable to obtain file info
O/S-Error: (OS 5) Access is denied.
Error in computing freespace for file D:\ORACLE\ORADATA\XYZ\SYSAUX01.DBF
ORA-27037: unable to obtain file status
OSD-04011: GetFileInformationByHandle() failure, unable to obtain file info
O/S-Error: (OS 5) Access is denied.
Error in computing freespace for file D:\ORACLE\ORADATA\XYZ\UNDOTBS01.DBF
ORA-27037: unable to obtain file status
OSD-04011: GetFileInformationByHandle() failure, unable to obtain file info
O/S-Error: (OS 5) Access is denied.
Error in computing freespace for file D:\ORACLE\ORADATA\XYZ\USERS01.DBF
ORA-27037: unable to obtain file status
OSD-04011: GetFileInformationByHandle() failure, unable to obtain file info
O/S-Error: (OS 5) Access is denied.
Error in computing freespace for file D:\ORACLE\ORADATA\XYZ\SYSTEM01.DBF
ORA-27037: unable to obtain file status
OSD-04011: GetFileInformationByHandle() failure, unable to obtain file info
O/S-Error: (OS 5) Access is denied.
Error in computing freespace for file D:\ORACLE\ORADATA\XYZ\SYSAUX01.DBF
ORA-27037: unable to obtain file status
OSD-04011: GetFileInformationByHandle() failure, unable to obtain file info
O/S-Error: (OS 5) Access is denied.
Error in computing freespace for file D:\ORACLE\ORADATA\XYZ\UNDOTBS01.DBF
ORA-27037: unable to obtain file status
OSD-04011: GetFileInformationByHandle() failure, unable to obtain file info
O/S-Error: (OS 5) Access is denied.
Error in computing freespace for file D:\ORACLE\ORADATA\XYZ\USERS01.DBF
ORA-27037: unable to obtain file status
OSD-04011: GetFileInformationByHandle() failure, unable to obtain file info
O/S-Error: (OS 5) Access is denied.
Error in computing freespace for file D:\ORACLE\ORADATA\XYZ\TEMP01.DBF
ORA-27037: unable to obtain file status
OSD-04011: GetFileInformationByHandle() failure, unable to obtain file info
O/S-Error: (OS 5) Access is denied. D:\ORACLE\ORADATA\XYZ\UNDOTBS01.DBF
ORA-27037: unable to obtain file status
OSD-04011: GetFileInformationByHandle() failure, unable to obtain file info
O/S-Error: (OS 5) Access is denied.
*** MODULE NAME:(MMON_SLAVE) 2021-03-23T01:04:55.778683-07:00
*** ACTION NAME:(ADR Container Space Management Statistics Flush) 2021-03-23T01:04:55.778683-07:00

But we are able to create tables in those tablespaces and do all DML. That means we are able to access them. 🙂

but why ?? O/S-Error: (OS 5) Access is denied.

I did not find any bug so far with the above symptoms.

We applied 19.10.0 BP as well but no luck.

Reply
- Mike.Dietrich says:
  
  March 30, 2021 at 23:07
  
  Then please open an SR and check with Oracle Support.
  Unfortunately I don’t have a Windows system to check this and dig deeper.
  
  Cheers,
  Mike
  
  Reply
alles belegt says:

April 7, 2021 at 07:09

Out of curiosity: When was the last time you tried Oracle on Windows?

Reply
- Mike.Dietrich says:
  
  April 12, 2021 at 12:22
  
  Not so long ago 😉
  But I’m 99% on Linux 😉
  
  Cheers,
  Mike
  
  Reply
satya says:

April 8, 2021 at 22:12

Thank you Mike.

Reply