You may need to run root.sh again after RU patching

Posted on August 7, 2023 by Mike.Dietrich Patch Recommendation 21

With the Oracle Critical Patch Update Advisory – July 2023 – you may have spotted a change in the Database Release Update READMEs: You may need to run root.sh again after RU patching. Why is that?

You may need to run root.sh again after RU patching

Photo by Mario Häfliger on Unsplash

Let us have a quick look

I verified this with the 19.20.0 README on Linux. But I guess this applies to the other READMEs as well. Please bare in mind that this applies to the Database RUs only, not to Grid Infrastructure. You can find this new paragraph in it:

3.3.5 Update Permissions for extjob
Database release update patch installation involves relinking of libraries and executables to be updated. This can sometimes result in permissions changing and needing to be updated. Execute the following as root user:
# chown root $ORACLE_HOME/bin/extjob
# chmod 4750 $ORACLE_HOME/bin/extjob

Now I think to remember that Oracle Support has given this recommendation in the past as well. And if my memory doesn’t fool me, customers commented on the blog as well that there may be occasional issues with the permissions and the execution of external jobs.

So adjusting permissions may be a good advice, especially when you rely on the external job interface.

Of course, you’ll achieve the same result when you invoke root.sh after patching. So either solution is fine, changing permission as show above or running root.sh after patching.

How does it look like after patching?

At the moment, I have not had time to apply 19.20.0 – it is on my to-do list. But I read several other blog posts where people demonstrated that the extjob interface is owned by oracle:oinstall after patching – so this needs to be changed back to root‘s ownership with the correct permissions.

Especially in cases where you patch with Ansible playbooks or other tools of automation, make sure to add the call as root or sudo. Just to avoid issues.

And that’s not the entire story

Thanks to you, the readers, and in particular to Christian Pfundtner from Austria who pointed me to MOS Note: 1555453.1 – Executing “relink all” resets permission of extjob, jssu, oradism, externaljob.ora:

Now see that the note recommends to run:

# chown root $ORACLE_HOME/bin/oradism
# chmod 4750 $ORACLE_HOME/bin/oradism

# chown root $ORACLE_HOME/bin/extjob
# chmod 4750 $ORACLE_HOME/bin/extjob

# chown root $ORACLE_HOME/rdbms/admin/externaljob.ora
# chmod 640 $ORACLE_HOME/rdbms/admin/externaljob.ora

# chown root $ORACLE_HOME/bin/jssu
# chmod 4750 $ORACLE_HOME/bin/jssu

# chown root $ORACLE_HOME/bin/nmb
# chmod 4710 $ORACLE_HOME/bin/nmb

# chown root $ORACLE_HOME/bin/nmhs
# chmod 4710 $ORACLE_HOME/bin/nmhs

# chown root $ORACLE_HOME/bin/nmo
# chmod 4710 $ORACLE_HOME/bin/nmo

when you do a relink_all which shouldn’t be the norm. So generally, there should be no need for additional relinks. But it could be the case. In such cases, it seems to be more wise to me to run root.sh instead of correcting permissions manually. I see the risk of missing something.

Update Oct 19, 2023

Opatch 12.2.0.1.40 includes the fix for Enh 35153024 – ER – OPATCHAUTO TO INCLUDE STEPS TO SET CORRECT OWNERSHIP AND PERMISSIONS ON EXTJOB BINARY UNDER GI HOME

So, please use at least this (or any newer) opatch version.

Update Oct 7, 2024

This query actually may help you (thanks to Alex Zaballa) to verify whether your database is relying on these external call-out possibilities:

SELECT *
  FROM CDB_SCHEDULER_JOBS
 WHERE JOB_TYPE in ('EXTERNAL_SCRIPT','EXECUTABLE','BACKUP_SCRIPT','SQL_SCRIPT');

Further Link and Information

–Mike

Tags: 19c Ansible external job extjob Oracle Patching permissions Release Update Root root.sh RU sudo

Patrick says:

August 7, 2023 at 12:24

I just noticed that sqlpatch was not set to executable.

Reply
- Mike.Dietrich says:
  
  August 8, 2023 at 17:31
  
  Thanks Patrick!
  
  Cheers,
  Mike
  
  Reply
Nelson says:

August 7, 2023 at 14:56

I installed the GI Release Update 19.20.0.0.230718 (which includes the DB RU) but there is no mention of “updating permissions for extjob” in the README. Do I still need to update the permissions?

Reply
- Mike.Dietrich says:
  
  August 8, 2023 at 17:30
  
  This is for the database RU. I doubt that GI has an extjob interface.
  And you install GI with root anyway.
  
  Cheers,
  Mike
  
  Reply
  - Mane says:
    
    August 28, 2023 at 08:31
    
    think there was a misunderstanding: patching of DB HOME with GI RU also seems to set the permissions to oracle:oinstall…
    But the README of GI RU (patch 35319490) doesn’t mention this…
    
    Reply
Neil says:

August 9, 2023 at 00:02

When you open the link you provided ( https://www.oracle.com/security-alerts/cpujul2023.html)

In the Patch availability column, where it has the link “Database” alongside Oracle Database Server, versions 19.3-19.19, 21.3-21.10 – the link points to https://support.oracle.com/rs?type=doc&id=2946185.1 but this doesn’t exist or is not available?

Reply
- Mike.Dietrich says:
  
  January 9, 2024 at 19:05
  
  Hi Neil,
  
  sometimes notes are “under review”, and then not accessible. But they come back quickly, especially when it’s such central notes.
  
  Cheers,
  Mike
  
  Reply
Mustafa KALAYCI says:

August 9, 2023 at 09:53

so, it is another reason to use out-of-place upgrade/patch, am I wrong? it prevent so many unexpected situations like this.

Reply
- Mike.Dietrich says:
  
  January 9, 2024 at 19:05
  
  Absolutely!
  
  Cheers,
  Mike
  
  Reply
Ivan Saez Saez Scheihing says:

August 10, 2023 at 09:14

Mike,

Thanks Mike. I did miss this thing about the permissions of extjob and have corrected it.

regards,

Ivan

Reply
Ivan Saez Saez Scheihing says:

August 10, 2023 at 09:21

Mike,

Just one quick question about the 19c_20 patches. I’ve installed p35320081_190000_Linux-x86-64.zip (RU 19c_20) and p35354406_190000_Linux-x86-64.zip (OJVM). And I do this sequentially:
unzip p35320081; opatch apply; unzip p35354406; opatch apply
Can this be done parallel? thus two opatch apply’s running at the same time. If so it could imply about 10 minutes less down time.

regards,

Ivan

Reply
- Mike.Dietrich says:
  
  August 10, 2023 at 11:03
  
  Yes, you can – please see here:
  
  https://mikedietrichde.com/2023/05/23/patching-to-oracle-19-19-0-out-of-place-with-additional-patches/
  
  Cheers,
  Mike
  
  Reply
  - Ivan Saez Saez Scheihing says:
    
    August 11, 2023 at 14:33
    
    thanks Mike! I’ll give it a try.
    
    regards,
    
    Ivan
    
    Reply
Emad Al-Mousa says:

October 19, 2023 at 18:30

Hi Mike,

I used OPatch version 12.2.0.1.40 to apply 19.21 RU [i was patching single node database] and still after patching …$ORACLE_HOME/bin/extjob ownership changed to oracle user account. I needed to run root.sh script again

Thanks
Emad Al-Mousa

Reply
- Mike.Dietrich says:
  
  October 19, 2023 at 18:32
  
  The fix is in opatchauto, not in opatch itself as far as I see.
  
  Cheers,
  Mike
  
  Reply
Ivan Saez Saez Scheihing says:

October 27, 2023 at 07:00

Hi,

I just have installed 19.21 and noticed one (nice) thing: ./datapatch -sanity_checks (optional)
And I did run it and it many checks (all OK in my case but this one got my attention:
…
Check: Dictionary statistics gathering – OK
Check: Scheduled Jobs – WARNING
Execution of scheduler jobs while database patching is running may lead to failures and/or performance issues.
There are jobs currently running or scheduled to be executed during next hour.
Consider patching the database when jobs are not running and will not be scheduled to run during patching.
To check for jobs that are running or scheduled to run:
SELECT owner as schema_name, job_name, state, next_run_date
FROM sys.all_scheduler_jobs
WHERE state = ‘RUNNING’
UNION
SELECT owner as schema_name, job_name, state, next_run_date
FROM sys.all_scheduler_jobs
WHERE state = ‘SCHEDULED’
and cast(next_run_date as date) > sysdate
and cast(next_run_date as date) < sysdate + 1/24;
xxx1:
| JOB_NAME | NEXT_RUN_DATE | SCHEMA_NAME | STATE |
|————————–+————————————-+————-+———–|
| CLEANUP_ONLINE_IND_BUILD | 27-OCT-23 08.37.59.750277 AM +02:00 | SYS | SCHEDULED |
|————————–+————————————-+————-+———–|
| CLEANUP_ONLINE_PMO | 27-OCT-23 08.38.39.811880 AM +02:00 | SYS | SCHEDULED |
|————————–+————————————-+————-+———–|
| CLEANUP_TAB_IOT_PMO | 27-OCT-23 08.38.09.751347 AM +02:00 | SYS | SCHEDULED |
|————————–+————————————-+————-+———–|
CDB$ROOT:
| JOB_NAME | NEXT_RUN_DATE | SCHEMA_NAME | STATE |
|———————–+————————————-+————-+———–|
| CLEANUP_TRANSIENT_PKG | 27-OCT-23 08.21.22.000000 AM +01:00 | SYS | SCHEDULED |
|———————–+————————————-+————-+———–|
xxx2:
| JOB_NAME | NEXT_RUN_DATE | SCHEMA_NAME | STATE |
|————————–+————————————-+————-+———–|
| CLEANUP_ONLINE_IND_BUILD | 27-OCT-23 08.14.36.695755 AM +02:00 | SYS | SCHEDULED |
|————————–+————————————-+————-+———–|
| CLEANUP_ONLINE_PMO | 27-OCT-23 08.15.16.817499 AM +02:00 | SYS | SCHEDULED |
|————————–+————————————-+————-+———–|
| CLEANUP_TAB_IOT_PMO | 27-OCT-23 08.14.46.697926 AM +02:00 | SYS | SCHEDULED |
|————————–+————————————-+————-+———–|
xxx3:
| JOB_NAME | NEXT_RUN_DATE | SCHEMA_NAME | STATE |
|————————–+————————————-+————-+———–|
| CLEANUP_ONLINE_IND_BUILD | 27-OCT-23 07.54.49.114596 AM +02:00 | SYS | SCHEDULED |
|————————–+————————————-+————-+———–|
| CLEANUP_ONLINE_PMO | 27-OCT-23 07.55.29.095238 AM +02:00 | SYS | SCHEDULED |
|————————–+————————————-+————-+———–|
| CLEANUP_TAB_IOT_PMO | 27-OCT-23 07.54.59.494084 AM +02:00 | SYS | SCHEDULED |
|————————–+————————————-+————-+———–|

Why this warning?? To avoid remote connections to the database while running datapatch I stop the listener.
I remember that in previous versions the instance had to be started in upgrade mode. Is it safer/better to start in upgrade mode again??

regards,

Ivan

Reply
- Mike.Dietrich says:
  
  January 9, 2024 at 20:09
  
  Hi Ivan,
  
  good question, I was stumbling across this as well already in my quick test.
  Let me check with the developers.
  
  Cheers,
  Mike
  
  Reply
Ivan Saez Saez Scheihing says:

November 2, 2023 at 08:08

hi,

I did install 19.21 and read that datapatch has a ‘-sanity_checks’ option. I did run it and it returned a warning about scheduled_tasks which were about to run. And that running jobs while datapatch-ing could lead to problems. I did not know that. Maybe it is safer to start the db in upgrade mode before datapatchi-ing??

regards,

Ivan

Reply
- Mike.Dietrich says:
  
  November 13, 2023 at 09:13
  
  Hi Ivan,
  
  this is just a pre-caution.
  It depends a lot on what is happening in the job. And datapatches’ sanity_checks would just like to highlight a potential thing you should be aware about.
  
  Cheers,
  Mike
  
  Reply
Ivan Saez Saez Scheihing says:

November 14, 2023 at 11:14

Mike,

Thanks for replying. But it seems to me that a “startup upgrade” before datapatch would avoid possible issues with jobs. We have so many Oracle db’s (around 45 production databases) and it would be very cumbersome to inventorize alle jobs. And startup upgrade would need a reboot after datapatch but that takes just a few minutes extra time.
And some additional thoughts: I always start with a pre-req check (opatch prereq CheckConflictAgainstOHWithDetail -ph ./) mostly a day before the actual patch. But opatch apply does execute another prereq check and that takes around 15 minutes. Is there a way to skip a prereq check when executing opatch apply?

regards,

Ivan

Reply
- Mike.Dietrich says:
  
  January 9, 2024 at 20:19
  
  Hi Ivan,
  
  datapatch has an embedded timeout of 15 mins in case it can’t access something it is supposed to be.
  And you can always restart it. Hence, I see no need for “startup upgrade” but instead would handle via a shell script to rerun datapatch in case it could not finish successfully. (and you could even tweak the threshold).
  
  Cheers,
  Mike
  
  Reply