Large SGA On Linux

On Linux the total SGA size is limited to around 2Gig. There are several ways to allocate more memory than this, but the most robust and scalable method requires the use of a shared memory file system (shmfs). The procedure presented in this article assumes you are using RedHat Advanced Server (AS) 2.1 with the enterprise kernel which supports Page Address Extension (PAE). Create a shared memory file system (shmfs)
The shmfs is a memory file system so it can be as large as the maximum allowable virtual memory supported by Red Hat Linux AS2.1, currently 16 GB, although the enterprise kernel theoretically supports up to 64 GB of RAM.

The shmfs is created using the following command as the root user:
mount -t shm shmfs -o size=3g /dev/shm
The shared memory file system can be mounted automatically by adding the following line into /etc/fstab file:
shmfs /dev/shm shm size=3g 0 0
In the above example I've created the shmfs with a size of 3G as that is the size of the buffer cache I am planning to use. The other elements of the SGA are placed in regular memory, not this shared memory file system, so they should not be included when deciding on the size of the shmfs. It is advisable to size this slightly bigger than the actual size needed, but in this example I've used a 3G shmfs for a 3G buffer cache.

Enabling big pages
Big pages are enabled by adding the bigpages=xMB to the relevant kernel entry in the boot loader file /boot/grub/grub.conf file, where "x" is calculated as follows: 
(Total SGA size in Gig) x 1024
Then round this value to the nearest hundredth. So for a 4G SGA we would do the following:
4 x 1024 = 4096 = 4100
So the /boot/grub/grub.conf file entry might look like this:
kernel /vmlinuz-2.4.9-e.40enterprise ro root=/dev/cciss/c0d0p2 bigpages=4100MB
With this entry saved the system should be rebooted. Once the system is available you must perform the following command as the root user:
echo 2 > /proc/sys/kernel/shm-use-bigpages
Alternatively you can add the following entry into the /etc/sysctl.conf file so this value persists between reboots:
kernel.shm-use-bigpages = 2

Setting the SHMMAX value
The shmmax value should be set at half the physical memory up to a maximum of 4294967295. For a server with 6G of memory we can set this value to 3G (half physical memory) using the following command as the root user:
echo 3221225472 > /proc/sys/kernel/shmmax
Alternatively it can be set in the /etc/sysctl.conf file with the following entry:
kernel.shmmax = 3221225472
The contents of your /etc/sysctl.conf file may look something like this:
kernel.shmmax = 3221225472
kernel.shmmni = 4096
kernel.shmall = 2097152
kernel.sem = 1000 32000 100 150
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000
kernel.shm-use-bigpages = 2
Alterations to the /etc/sysctl.conf file can be applied without a reboot by issuing the following command as root:
/sbin/sysctl -p
Instance Parameters
Some instance parameter changes are necessary to allow the Oracle instance to use the shared memory file system. The spfile parameters can be manipulated using the ALTER SYSTEM SET command in a running instance, or by modifying the spfile contents offline:
-- Change the parameter value in the spfile directly.
ALTER SYSTEM SET parameter = value SCOPE=spfile

-- Create a pfile with the contents of the current spfile.
CREATE PFILE='/tmp/pfile' FROM SPFILE;

-- Manually manipulate the contents of the pfile.

-- Recreate the spfile from the amended pfile.
CREATE SPFILE FROM PFILE='/tmp/pfile';
The following parameter should be added to the spfile or pfile:
use_indirect_data_buffers=true
Also, any references to db_cache_size and db_xK_cache_size parameters should be removed and replaced with the old style db_block_buffers parameter entry:
# 3Gig for an 8K db_block_size.
db_block_buffers = 393216
This means that the multiple block size feature is not available when using this method. Remember that the buffer cache is only one part of the SGA.

For further information see: