Question & Answer
Question
IBM Java for AIX HowTo: Exempting Java Applications from SED
Answer
This document provides step-by-step instructions for disabling Buffer Overflow Protection (BOP) using Stack/heap Execution Disable (SED) on AIX Java.
Overview
What is Stack Execution Disable Protection (SED) ?
AIX implements buffer overflow protection using Stack/heap Execution Disable (SED) to prevent exploits of buffer overflows. "Buffer overflow attacks occur when an internal program buffer is overwritten because data was not properly validated (such as command line, environmental variable, disk or terminal I/O). Attack code is inserted into a running process through the buffer overflow, changing the execution path of the running process. The return address is overwritten and redirected to the inserted-code location." SED prevents buffer overflow attacks by not executing code in data areas of memory.
Just-In-Time (JIT) compilers and SED
Exempting applications from SED
SED sets flags in the header of executable files to control the level of stack execution. By default, all Java launchers have the appropriate bit set to indicate this file does stack/heap based execution. You can verify this with:
sedmgr -d ExecutableName
For example:
sedmgr -d /usr/java7_64/bin/java
/usr/java7_64/bin/java : exempt
Applications that use their own Java launchers and create JVM instances using JNI may not be exempt.
For example,
sedmgr -d ./SampleProgram
./SampleProgram: system
These applications must be explicitly patched to exempt them from SED.
To turn off the SED request bit:
sedmgr -c exempt ExecutableName
sedmgr -d ExecutableName
For example:
sedmgr -d /usr/java7_64/bin/java
/usr/java7_64/bin/java : exempt
Applications that use their own Java launchers and create JVM instances using JNI may not be exempt.
For example,
sedmgr -d ./SampleProgram
./SampleProgram: system
These applications must be explicitly patched to exempt them from SED.
To turn off the SED request bit:
sedmgr -c exempt ExecutableName
Configuring sedmgr for the entire system
The stack execution disable (SED) mechanism in AIX is implemented through system wide mode flags, as well as individual executable file-based header flags. You can choose to turn off SED system wide.
Use the command: sedmgr -m value to change the system wide settings where value is:
off - The SED mechanism is turned off and no process is marked for SED protection.
select - Only a select set of files are enabled and monitored for SED protection.
all - All executable programs loaded on the system are SED protected except for the files requesting an exemption from SED mode.
To turn off SED use:
sedmgr -m off
and reboot the system
Use the command: sedmgr -m value to change the system wide settings where value is:
off - The SED mechanism is turned off and no process is marked for SED protection.
select - Only a select set of files are enabled and monitored for SED protection.
all - All executable programs loaded on the system are SED protected except for the files requesting an exemption from SED mode.
To turn off SED use:
sedmgr -m off
and reboot the system
Section 5
Section 6
Section 7
Section 8
Section 9
Section 10
Section 11
Section 12
Section 13
Section 14
Section 15
Section 16
Section 17
Section 18
Section 19
Section 20
Contact IBM Support
If, after reading and following the above instructions, further assistance is required, please complete the following steps:
1. Confirm that you have review and completed all of the above steps.
2. Contact IBM and open a new IBM service request (i.e., a new IBM PMR).
3. Collect and upload data as per the data collection procedures noted in the above sections or package and upload the current data and details by following the instructions on this web page:
IBM Java for AIX MustGather: How to upload diagnostic data and testcases to IBM
Document Type: | Technical Document |
Content Type: | General |
Hardware: | all Power |
Operating System: | all AIX Versions |
IBM Java: | all Java Versions |
Author(s): | John Carver |
Reviewer(s): | John Carver |
Click here to submit feedback for this document.
[{"Product":{"code":"SG9NGS","label":"IBM Java"},"Business Unit":{"code":null,"label":null},"Component":"Not Applicable","Platform":[{"code":"PF002","label":"AIX"}],"Version":"Version Independent","Edition":"","Line of Business":{"code":"LOB08","label":"Cognitive Systems"}}]
Was this topic helpful?
Document Information
Modified date:
17 June 2018
UID
isg3T1025240