Today, we have continuously hitting "Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x0] [PC:0x2FE4812, ioc_pin_shared_executable_object()+882] [flags: 0x0, count: 1] Mon Jul 14 11:21:45 2014" in the Alert log of 1st Node of our Production EBS RAC Database. (Version is 11.2.0.3)
After some troubleshooting, I got reason behind this issue, the error is most often related to an inconsistency that has been detected between the java shared object loaded in memory and the backing store image stored on disk as a result of calling java code and having the JIT compiler enabled.
I.E. parameter java_jit_enabled=true (this is the default).
These backing store images are created wherever shm_open() puts them. This can vary on different platforms.
For Example:
On Solaris these files are written to /tmp and have names like .SHMDJOXSHM_EXT_...
On Linux these files are written to /dev/shm and have names like JOXSHM_EXT_...
In our case somebody has by mistaken deleted files from /dev/shm and after which we started facing this issue.
Troubleshooting:
The JIT compiler runs as an MMON slave. You can use a system tracer such as TUSC to see what it is doing as follows:
1) Connect as sys
2) Run the following command to get MMON process id for the instance
ps -ef | grep -i MMON
TIP: if several instances are running on the server then use the ORACLE_SID to identify the correct MMON process
3) Run tusc on the mmon process pid
i.e attach to a running process. Send output to /tmp/tusc.out
% tusc -faepo /tmp/tusc.out -p <pid>
4) From the session in step 1 run, SELECT SYS.DBMS_JAVA.LONGNAME('FOO') FROM DUAL; that fails with the error.
If ORA-7445 [ioc_pin_shared_executable_object] is encountered, one should look into the following:
1) Is there any user or job that may have tried to clean up files in the directory where the backing store images are located?
2) Is the location where the backing store files are located out of space?
3) Is the location where the backing store files are located accessible?
After some troubleshooting, I got reason behind this issue, the error is most often related to an inconsistency that has been detected between the java shared object loaded in memory and the backing store image stored on disk as a result of calling java code and having the JIT compiler enabled.
I.E. parameter java_jit_enabled=true (this is the default).
These backing store images are created wherever shm_open() puts them. This can vary on different platforms.
For Example:
On Solaris these files are written to /tmp and have names like .SHMDJOXSHM_EXT_...
On Linux these files are written to /dev/shm and have names like JOXSHM_EXT_...
In our case somebody has by mistaken deleted files from /dev/shm and after which we started facing this issue.
Troubleshooting:
The JIT compiler runs as an MMON slave. You can use a system tracer such as TUSC to see what it is doing as follows:
1) Connect as sys
2) Run the following command to get MMON process id for the instance
ps -ef | grep -i MMON
TIP: if several instances are running on the server then use the ORACLE_SID to identify the correct MMON process
3) Run tusc on the mmon process pid
i.e attach to a running process. Send output to /tmp/tusc.out
% tusc -faepo /tmp/tusc.out -p <pid>
4) From the session in step 1 run, SELECT SYS.DBMS_JAVA.LONGNAME('FOO') FROM DUAL; that fails with the error.
If ORA-7445 [ioc_pin_shared_executable_object] is encountered, one should look into the following:
1) Is there any user or job that may have tried to clean up files in the directory where the backing store images are located?
2) Is the location where the backing store files are located out of space?
3) Is the location where the backing store files are located accessible?
Workarounds:
1) Disable the just-in-time(JIT) compiler for Oracle JVM environment by setting the following initialization parameter as shown: java_jit_enabled = false
2) Cleanly shutdown and startup the database. This should clean out all old backing store images associated with this instance and start creating new ones as java is accessed again.
1) Disable the just-in-time(JIT) compiler for Oracle JVM environment by setting the following initialization parameter as shown: java_jit_enabled = false
2) Cleanly shutdown and startup the database. This should clean out all old backing store images associated with this instance and start creating new ones as java is accessed again.
Additional Information:
It is sometimes asked if there is a way to redirect these backing store images to another location.
This functionality is available with the following bug fix:
This functionality is available with the following bug fix:
Patch 13574534 - JAVA JIT COMPILER GENERATING LOT OF STALE FILES (IE) *JOXSHM* IN \TMP DIRECTORY
After the fix is applied, set the parameter _ncomp_shared_objects_dir to the location where you want to redirect the backing store images.
After the fix is applied, set the parameter _ncomp_shared_objects_dir to the location where you want to redirect the backing store images.
Reference: Ora-7445 [Ioc_pin_shared_executable_object()] (Doc ID 1316906.1)
Applies to: Oracle Database 11.1.0.7 and later
No comments:
Post a Comment