A common method of computer attack is to place malicious code in memory and then jump to that code. Such attacks rely on segments that are both writable and executable. You can use the nxheap and nxstack security extensions to systematically make the stack and heap of all Oracle Solaris processes non-executable. The nxstack security extension replaces the noexec_user_stack system variable.
Programs read and write data on the stack. Typically, they execute from read-only portions of memory that are specifically designated for code. Some attacks that cause buffers on the stack to overflow try to insert new code on the stack and cause the program to execute it. Removing execute permission from the stack memory prevents these attacks from succeeding. Most programs can function correctly without using executable stacks.
64-bit processes always have non-executable stacks. By default, 32-bit SPARC processes have executable stacks. The nxstack security extension, which is enabled by default, prevents the stacks of 32-bit processes from being executable. Programs that attempt to execute code on their stack are sent a SIGSEGV signal. This signal usually results in the program terminating with a core dump.
A log is written by default. The log is useful for identifying valid programs that depend upon executable stacks that have been prevented from correct operation by setting the nxstack security extension. Even when messages are not being logged, the SIGSEGV signal can continue to cause the executing program to terminate with a core dump. See How to Prevent the Execution of Malicious Code From the Process Stack and Process Heap and the sxadm(1M) man page.
Programs can explicitly mark or prevent stack execution. The mprotect() function in programs explicitly marks the stack as executable. For more information, see the mprotect(2) man page. A program compiled with -z nxstack=enable makes the stack non-executable regardless of the system-wide setting.
The heap is memory set aside for dynamic allocation. It is reclaimed when the application, that is, the process, exits. Removing execute permission from the process heap prevents the storing of malicious code in the heap. Most programs function correctly without executing code on the heap.
The nxheap security extension is enabled by default, as are logs. For examples and information, see How to Prevent the Execution of Malicious Code From the Process Stack and Process Heap and the sxadm(1M) man page.