Currently we see a lot of issues with the Multiple Logwriter feature in Oracle Database 12.1.0.2, especially on the IBM AIX platform.
What are Multiple LGWRs?
You will see multiple LGnn (Log Writer Worker) processes on multiprocessor systems, The LGWR (Log Writer) creates worker processes (slaves) to improve the performance of writing to the redo log. LGnn (Log Writer Worker) processes are not used when there is a SYNC standby destination. Possible processes include LG00–LG99.
Here is a list of new background processes in Oracle Database 12.1:
MOS Note 1625912.1 – New Background Processes In 12c
For deep dive information about how to trace the LGWR (and DBWR) and the differences 11.2/12.1 and wait events and much more, please see this Deep Dive PDF from an UKOUG talk from Frits Hoogland.
Known Symptoms:
- This note has been removed without notice – I don’t know the reasons … sorry for that …
MOS Note: 1968563.1
Hang Waiting on ‘LGWR worker group ordering’ with Deadlock Between LGWR (Log Writer) Slaves (LG0n) when Using Multiple LGWR Processes - MOS Note:1957710.1
ALERT: Database hang or may fail to OPEN in 12c IBM AIX – ORA-742, DEADLOCK or ORA-600 [kcrfrgv_nextlwn_scn] ORA-600 [krr_process_read_error_2]
Solution:
Turn off multiple logwriters on IBM AIX and HP-UX – and I turn off multiple logwriters on Linux as well. We are seeing the issues mainly on IBM AIX and HP-UX.
Set in your spfile:
_use_single_log_writer=true
See also:
- Patches:
https://support.oracle.com/epmos/faces/BugMatrix?id=20122715 - Apply Patch 21915719 or April DB PSU to prevent bug 21915719 from happening. The patch does not repair the inconsistencies in the online redo log causing the restart/open issue but prevents it from happening
–Mike
Mike, do you know if there is a date for a patch that will make this feature work as designed on AIX platforms?
We just hit that bug five days before going live with our ERP upgrade. Our database got corrupted and it wouldnt come up not even after applying the _use_single_log_writer=true parameter.
Scary stuff.
Sorry for not coming back earlier …
https://support.oracle.com/epmos/faces/BugMatrix?id=20122715
But you may double check with Support if this fixes all the known issues.
Cheers
Mike
Roberto,
really sorry to hear that. The patch listening seems to say that the patch should be included into the most recent PSU. Did you apply this upfront to your home?
Cheers
Mike
It looks like they just issued another patch to address this. Apparently 20122715 didn’t address the problem.
This could be the case – but I lost track a bit.
My current recommendation to all customers on AIX (and potentially on Linux as well):
_use_single_log_writer=true
Mike
Hi Mike ,
We encounter the same issue.
Impossible to open database !!!
Any suggestion please to avoid restore …
Best regards.
Kais
SQL> alter database open ;
alter database open
*
ERROR at line 1:
ORA-00600: internal error code, arguments: [kcrfrgv_nextlwn_scn], [1771859328],
[26], [1771859330], [26], [], [], [], [], [], [], []
Kais,
please open an SEV 1 (Prod Down) SR with Oracle Support.
Thanks and sorry to hear that it hit you. It won’t help you know that I recommended to disable it:
https://blogs.oracle.com/UPGRADE/entry/parameter_recommendations_for_oracle_database
Cheers
Mike
hi,
i’m installing 12C on AIX 7.2
I came across all of the problem that you mention in this Blog and now it fixed.
i have one more big issue ,
i can’t create a new db with the DBCA i got error
ORA-00449: background process ‘MMON’ unexpectedly terminated with error 448
this is from the instillation log :
File Validations Successful.
Copying database files
DBCA_PROGRESS : 1%
DBCA_PROGRESS : 2%
DBCA_PROGRESS : 8%
DBCA_PROGRESS : 13
DBCA_PROGRESS : 27
Creating and starting Oracle instance
ORA-00449: background process ‘MMON’ unexpectedly terminated with error 448
ORA-01012: not logged on
can you advice ?
Hi Tomer,
glad to hear that the blog helped you.
Even though I’m not "Support" I did quickly check MOS and there are several issues reported on AIX (mostly during shutdown).
Please check: patch 23089357 (or Oracle Support Document 23089357.8 (Bug 23089357 – Shutdown fails with ORA-449 "background process MMNL unexpectedly) can be found at: https://support.oracle.com/epmos/faces/DocumentDisplay?id=23089357.8) – there seem to be several issues on AIX 7.1.4 and newer with MMON and its little brother MMNL (MMon Light).
No guarantee that it fixes it. But worth a try.
Did you see any other errors in the alert.log?
Do you let DBCA create one of the default databases (Data Warehouse or OLTP or whatever)? Then please try instead to create a CUSTOM database with DBCA, something I’d recommend you generally as you can decide which options you’d like to have created. The so called SEED databases have all options in it which may stretch your future upgrade timings.
Cheers
Mike