Configuring RHAS 30 for Oracle and Increasing Memory Size


Applicable installations
This installation instruction has been checked for:
- Oracle Enterprise Manager (9i and 10g)
- Red Hat Advance Server developer version

Following requirements need to be met for a successful installation of  Oracle 9.2.0.6 (a 3 CD set which includes the base release and all fixes from prior patchsets including 9.2.0.4) on Red Hat Enterprise Linux AS/ES 3.0 (RHEL3) on platform Linux x86 .

Software
* Linux kernel version should be 2.4.21-4.EL
* Glibc Version Number glibc 2.3.2-95.3
* Required OS Components for 9i
- compat-db-4.0.14.5
- compat-gcc-7.3-2.96.122
- compat-gcc-c++-7.3-2.96.122
- compat-libstdc++-7.3-2.96.122
- compat-libstdc++-devel-7.3-2.96.122
- openmotif21-2.1.30-8.i386.rpm
- setarch-1.3-1

* Required OS Components for 10g
+ compat-db-4.1.25-9
+ compat-gcc-32-3.2.3-47.3
+ compat-gcc-32-c++-3.2.3-47.3
+ compat-oracle-rhel4-1.0-3
+ compat-libcwait-2.0-1
+ compat-libgcc-296-2.96-132.7.2
+ compat-libstdc++-296-2.96-132.7.2
+ compat-libstdc++-33-3.2.3-47.3
+ xorg-x11-deprecated-libs-devel-6.8.1-23.EL
+ xorg-x11-deprecated-libs-6.8.1-23.EL

In addition, install the following rpm's (available on Disk3 of the RHEL4.0 media)
+ openmotif21-2.1.30-11
+ libaio-0.3.102-1
+ libaio-devel-0.3.102-1

Installing RHAS 30 (we required to install the developer version)
In this section the OS is installed.
- Insert the first cdrom (autoboot)
- Hit Enter to get going (no need to rescue)
- Skip the media test
- On the installation screen, hit Next
- On the language selection, choose your language
- On the keyboard selection, choose your keyboard
- On the mouse selection, choose your mouse
- On the disk setup, choose to partition with DiskDruid
- create a root (/) partition at least 2GB, preferrably bigger than 2.5GB and make it type ext3
- create a swap partition at least 2GB, recommended double the size of the internal physical memory
- create an oracle partition (/oracle) size depending on the software to be installed, recommended the remaining disk space
- On the bootloader configuration, choose the bootloader of your choice, remmended to go with the default "Grub"
- On the network configuration, configure your network. If you are unsure, leave it to the defaults (DHCP)
- On the Firewall configuration, set your firewall options. If you are unsure, leave it to the defaults (Enable firewall).
- On additional Language support, choose your preferred langauge.
- On the timezone selection, choose your home country/city.
- Set the root password.
- On the package selection, select "customize the set of packages to be installed"
- Select at least the following packagesets:
- X Window System
- Gnome Desktop Environment
- Graphical Internet
- Server Configuration Tools
- Development Tools
- Legacy Software Development
- Administrative Tools
- X Software Development
- At the Graphical Interface configuration, choose the suggested videocard and memory size. If you are unsure, choose the Other > Generic VGA compatible
- At the monitor configuration, choose the suggested monitor. If you are unsure, choose the Generic CRT > Monitor 640x480
- At the Customize Graphics Configuration, choose the the "Text" login type and choose the screen resolution and depth. If you are unsure, choose 640x480 at 8bit.
- Let the system reboot

Installing additional packages (not needed if everything was installed properly)
In this section, additional packages are installed required by oracle.
To see if these development packages are installed on your server, run the following command: 
rpm -q gcc cpp compat-libstdc++ glibc-devel glibc-headers glibc-kernheaders binutils

- After system bootup, login as root and insert the Disk2:
mount /mnt/cdrom
rpm -iv /mnt/cdrom/RedHat/RPMS/sysstat-4.0.7-4.i386.rpm
eject
- insert Disk3:
mount /mnt/cdrom
rpm -iv /mnt/cdrom/RedHat/RPMS/compat-db-4.0.14-5.i386.rpm
rpm -iv /mnt/cdrom/RedHat/RPMS/ORBit-0.5.17-10.4.i386.rpm
rpm -iv /mnt/cdrom/RedHat/RPMS/libpng10-1.0.13-8.i386.rpm
rpm -iv /mnt/cdrom/RedHat/RPMS/gnome-libs-1.4.1.2.90-34.1.i386.rpm
eject

Environment
* Put gcc296 and g++296 first in $PATH variable by creating the following symbolic links (Metalink Note 252217.1), as root perform:
These are required for the relinking of some Oracle binaries (as root)
mv /usr/bin/gcc /usr/bin/gcc323
mv /usr/bin/g++ /usr/bin/g++323
ln -sf /usr/bin/gcc296 /usr/bin/gcc
ln -sf /usr/bin/g++296 /usr/bin/g++
* hostname command should return the fully qualified hostname as shown below:
% hostname
hostname.domainname
* If any Java packages are installed on the system, unset the Java environment variables, for example JAVA_HOME.
* The oracle account used to install Oracle 9.2.0.1, should not have the Oracle install related variables set by default. For example setting ORACLE_HOME, PATH, LD_LIBRARY_PATH to include Oracle binaries in .profile, .login file and /etc/profile.d shouldbe completely avoided.

Configure the kernel parameters
In this section, kernel parameters are configured.

- Configuring the kernel parameters, replace "machine123" and "oracle.com" with your own settings.
echo "" >> /etc/sysctl.conf
echo "# Oracle specific settings" >> /etc/sysctl.conf
echo "kernel.hostname = machine123.oracle.com" >> /etc/sysctl.conf
echo "# semaphores in kernel.sem: semmsl semmns semopm semmni" >> /etc/sysctl.conf
echo "kernel.sem = 256 32000 100 142" >> /etc/sysctl.conf
echo "kernel.shmmax = 2147483648" >> /etc/sysctl.conf
echo "kernel.shmmni = 4096" >> /etc/sysctl.conf
echo "kernel.shmall = 3279547" >> /etc/sysctl.conf
echo "fs.file-max = 327679" >> /etc/sysctl.conf
# Disables packet forwarding
echo "net.ipv4.ip_forward = 0" >> /etc/sysctl.conf
# Enables source route verification
echo "net.ipv4.conf.default.rp_filter = 1" >> /etc/sysctl.conf
echo "net.ipv4.ip_local_port_range = 1024 65000" >> /etc/sysctl.conf
echo "kernel.msgmni = 2878" >> /etc/sysctl.conf
echo "kernel.msgmnb = 65535" >> /etc/sysctl.conf
echo "" >> /etc/security/limits.conf
echo "# Oracle specific settings" >> /etc/security/limits.conf
echo "oracle soft nofile 1024" >> /etc/security/limits.conf
echo "oracle hard nofile 65536" >> /etc/security/limits.conf
echo "oracle soft nproc 2047" >> /etc/security/limits.conf
echo "oracle hard nproc 16384" >> /etc/security/limits.conf

Configure the hostname
In this section, the hostname of the machine is set to the hosts file.

- Add as the first line the ip address, fully qualified name of your machine and alias. Replace the "127.0.0.2" and "machine123.oracle.com" with your own settings.
cd /etc
mv hosts hosts.org
touch hosts
echo "# Do not remove the following line, or various programs" >> /etc/hosts
echo "# that require network functionality will fail." >> /etc/hosts
echo "127.0.0.2 machine123.oracle.com machine123" >> /etc/hosts
echo "127.0.0.1 localhost.localdomain localhost" >> /etc/hosts

Patching the OS for oracle related issues
In this section, postfixes are applied to the OS.

- Apply patch 3006854,INSTALLER INTEROPERABILITY PATCH FOR RHEL3.0
This patch must be installed on systems running Red Hat Enterprise Linux 3.0 before running the Oracle Universal Installer (for any Oracle product). To resolve libcwait symbol issue before invoking Oracle installer. Download from metalink patch
3006854 or download from updates.oracle.com
wget ftp://metalink_user:password@updates.oracle.com/3006854/p3006854_9204_LINUX.zip
unzip p3006854_9204_LINUX.zip
cd 3006854
sh rhel3_pre_install.sh

Shutting down unused programs
We are not using all of the started options. E.g. on your server, you do not have the pcmcia and probably don't use power management. A lot of services are started, but not configured and therefore not running.

- Finding the services configured to start:
chkconfig --list | more
- Finding the services actually running:
ls /var/lock/subsys
- Getting a description of the services
head /etc/init.d/<service>
e.g. "head /etc/init.d/sendmail"
- stopping a service
service <service> stop
e.g. "service sendmail stop"
- configuring a service not to start
chkconfig --level 345 <service> off
e.g. "chkconfig --level 345 sendmail off"

Creating the user oracle
In this section, OS oracle and its group are created.

- create the group oinstall and dba
groupadd dba          # group granted with SYSDBA system privilege
groupadd oinstall     # group owner of Oracle files

- create the user oracle
useradd -c "Oracle software owner" -g oinstall -G dba oracle
passwd oracle

- Create the Oracle Directories
su - root
mkdir /opt/oracle
mkdir /opt/oracle/product
mkdir /opt/oracle/product/9.2.0
chown -R oracle.oinstall /opt/oracle                             
mkdir /var/opt/oracle
chown oracle.dba /var/opt/oracle
chmod 755 /var/opt/oracle

Modify /.bash_profile Adding:

# Set the LD_ASSUME_KERNEL environment variable only for Red Hat 9 and
# for Red Hat Enterprise Linux Advanced Server 3 (RHEL AS 3) !!
export LD_ASSUME_KERNEL=2.4.19
# Oracle Environment
export ORACLE_BASE=/opt/oracle
export ORACLE_HOME=/opt/oracle/product/9.2.0
export ORACLE_SID=FGUARD
export ORACLE_TERM=xterm
# export TNS_ADMIN= Set if sqlnet.ora, tnsnames.ora, etc.
# are not in $ORACLE_HOME/network/admin
export NLS_LANG=AMERICAN;
export ORA_NLS33=$ORACLE_HOME/ocommon/nls/admin/data
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib:/usr/local/lib
# Set shell search paths
export PATH=$PATH:$ORACLE_HOME/bin


Configuring VNC server
VNC server allowes you to run a very small graphical subsystem, without using a console. The Oracle Universal Installer will require a graphical display, but probably you do not want to do an installation on console.

- Configure service vncserver to be started at boottime
chkconfig --level 345 vncserver on
- Configure vncserver to start vncserver as user oracle
echo "VNCSERVERS=\"1:oracle\"" >> /etc/sysconfig/vncservers
- Start vncserver manually for the first time. You will be prompted for a password for the vncserver.
su - oracle
vncserver :1
- As user oracle, configure the vncserver session environment
cd .vnc
mv xstartup xstartup.org
touch xstartup
chmod 755 xstartup
echo "#"\!"/bin/sh" >> xstartup
echo "" >> xstartup
echo "# Uncomment the following two lines for normal desktop:" >> xstartup
echo "# unset SESSION_MANAGER" >> xstartup
echo "# exec /etc/X11/xinit/xinitrc" >> xstartup
echo "" >> xstartup
echo "[ -r \$HOME/.Xresources ] && xrdb \$HOME/.Xresources" >> xstartup
echo "xsetroot -solid grey" >> xstartup
echo "vncconfig -iconic &" >> xstartup
echo "xhost localhost" >> xstartup
echo "PATH=\$PATH:/usr/local/bin" >> xstartup
echo "export PATH" >> xstartup
echo "xterm -geometry 80x24+10+10 -ls -title \"\$VNCDESKTOP Desktop\" &" >> xstartup
echo "twm &" >> xstartup
- As user oracle, stop the vncserver
vncserver -kill :1
exit

Setting ldap ports
If you want to install the Oracle Internet Directory on the ldap standard ports, you have to erase all lines containing "ldap" or "ldaps" out of the /etc/services file.

Reboot the system
- to active all changes, reboot the system
reboot
After reboot, verify all kernel parameters have been set correct, by issuing "sysctl -a"
This because the kernel.semmax is not always accepted on 4GB and then ignored.

Oracle Installation
* Define and export the environment variable LD_ASSUME_KERNEL before starting the install process:
export LD_ASSUME_KERNEL=2.4.19

Additional Installation Steps
* Open a new terminal window
* Apply patch number 3006854 to provide libcwait symbol (as we mentioned before)
* After completing above steps start ./runInstaller from 9iR2 CD 1 from shell where you have defined above variable. You may get the following errors:
- relink error for ins_oemagent.mk - Click on Continue and complete the install
- apply the patch for Bug: 3119415 after the install is completed
- ctx relinking error in ins_ctx.mk: Click ignore. This is fixed by applying 9.2.0.4 or 9.2.0.5 patchset.

NOTE: This step will not be necessary to perform if you are installing from the new release 9.2.0.4 (Part #'s B13496-01, B13497-01, and B13498-01 which is the complete distribution of 9.2.0.1 with the 9.2.0.4 already installed). Do not confuse this with the 9.2.0.4.0 patchset ( Bug: 3095277 ) which only contains the 9.2.0.4 fixes and is not a complete package. Also, it should be considered that if you have installed the 9.2.0.1.0 base release to then apply the 9.2.0.5.0 patchset ( Bug: 3501955 ) instead of 9.2.0.4.0 patchset.

ADDITIONAL NOTES
* For customers using asynch_io , apply bug: 3208258 (Fixed In Ver: 9.2.0.4 or 9.2.0.5)

RELATED DOCUMENTS
[NOTE:252217.1] Requirements for Installing Oracle 9iR2 on RHEL3


Implement Asynchronous I/O
Many times there is a requirement to check if Asynchronous I/O is working on Linux Platform, so we can try to use it for our datafiles access inside database.
slabinfo maintains statistics about objects in memory. Some of the structs used by Asynchronous I/O are threated as objects in the virtual memory, so we can
look for those structs on slabinfo. The ones related to AIO are named kio*.

$ cat /proc/slabinfo | grep kio

for example: output with async io enabled.
$ cat /proc/slabinfo | grep kio
kioctx  270   270 128    9    9 1 : 252 126
kiocb 66080 66080  96 1652 1652 1 : 252 126
kiobuf  236   236  64    4    4 1 : 252 126

output with async io disabled.
$ cat /proc/slabinfo | grep kio
kioctx 0 0 128 0 0 1 : 252 126
kiocb  0 0 96  0 0 1 : 252 126
kiobuf 0 0 64  0 0 1 : 252 126

There are 3 caches involved. The kioctx and kiocb are Async I/O data structures that are defined in aio.h. If it shows a non zero value that means async io is  nabled.
If you have the source code loaded, you can review it at file aio.h. This file is located under: /usr/src/linux-<version>/include/linux/aio.h
These data structures are using to track the I/O requests, and are allocated as part of the __init aio_setup() call in aio.c.


Implement Advanced memory management techniques
Increasing Usable Address Space for Oracle on 32-bit Linux
The current shipping version of Oracle is able to use about 1.7GB of address space for its SGA. To increase this size, Oracle needs to be relinked with a lower SGA base and Linux needs to have the mapped base lowered for processes running Oracle. Increasing the address space allows for more database buffers or a larger indirect data buffer window to be used.
There are changes that need to be made to the Oracle binary and the Linux environment (requiring root access), so the appropriate privileges are needed.
Currently, a solution exists only when running Oracle 9iR2 on Red Hat 2.1 Advanced Server. Red Hat provides an adjustable parameter in the /proc filesystem to allow more useable address space in processes. First, the SGA base address that Oracle uses must be lowered by relinking Oracle. Currently, Oracle ships with this base address set at 0x50000000 so that it is compatible with the defaults set by most distributions of Linux. Lowering this address allows Oracle to use more of the address space in the process, but it is important to note that the newly relinked Oracle binary will no longer work unless a corresponding modification is also made to Linux (Red Hat 2.1AS provides a way to do this at runtime).

Follow these steps to complete the first part of the solution:
1. Shutdown all instances of Oracle
2. cd $ORACLE_HOME/lib
3. cp -a libserver9.a libserver9.a.org (to make a backup copy)
4. cd $ORACLE_HOME/bin
5. cp -a oracle oracle.org (to make a backup copy)
6. cd $ORACLE_HOME/rdbms/lib
7. genksms -s 0x15000000 >ksms.s (lower SGA base to 0x15000000)
8. make -f ins_rdbms.mk ksms.o (compile in new SGA base address)
9. make -f ins_rdbms.mk ioracle (relink)

The relinked Oracle binary now has a lower SGA base and is now able to use about 2.65GB of address space if Linux is also modified to support this. Next, the Linux kernel's mapped base needs to be lowered below Oracle's new SGA base. Red Hat 2.1AS has a parameter in /proc that lowers the kernel's mapped base for each process. This parameter is not a system-wide parameter. It is a perprocess parameter, but it is inherited by child processes. This parameter can only be modified by root. The following steps document how to lower the mapped base for a single bash terminal session. The default mapped base is 0x40000000. Once this session has been modified with the lower mapped base, this session (terminal window) will need to be used for all Oracle commands so that Oracle processes use the inherited (lower) mapped base:
1. Shutdown the instance of Oracle.
2. Open a terminal session (Oracle session), and get the process id using "echo $$”.
3. Open a second terminal session and su to root (root session).
4. Now, from the root session,
echo 268435456 >/proc/<pid>/mapped_base,
where <pid> is the process id determined in step 2. This lowers the mapped base for the Oracle session to 0x10000000.
5. Again, from the root session,
echo 3000000000 >/proc/sys/kernel/shmmax
Ths increases the value of shmmax so that Oracle will allocate the SGA in one segment.

6. From the Oracle terminal session, startup the Oracle instance.
The SGA now begins at a lower address, so more of the address space can be used by Oracle. Now you can increase the init.ora values of db_cache_size or db_block_buffers to increase the size of the database buffer cache. You can also write a small program that uses setuid() to set the /proc/<pid>/mapped_base. It would look something like this:
int main(int argc, char *argv[]) {
pid_t ppid;
char buf[256];
unsigned long mapped_base;
int ret;
#define NEW_MAPPED_BASE 0x10000000
ppid = getppid();
mapped_base = NEW_MAPPED_BASE;
sprintf(buf, "echo %lu >/proc/%u/mapped_base", mapped_base, ppid);
setuid(0);
ret = system(buf);
if (ret == 0)
printf("Lowering mapped base of pid=%u to 0x%X\n", ppid,
mapped_base);
else
printf("unable to lower mapped base. You might need to:\n" "
chmod 4711 lowermap\n chown root.root lowermap\n");
exit(-ret);
}

If you are running with the init.ora parameter 'use_indirect_data_buffers=true' and already have a large buffer cache, you can use the above solution to increase the indirect buffer window size. The default is 512MB and should be fine for most applications. Increasing the window size may increase performance slightly under certain conditions because a larger indirect window reduces the overhead of mapping an indirect buffer into Oracle's address space.
To increase the indirect window size, set the environment variable VLM_WINDOW_SIZE to the window size in bytes before starting up the Oracle instance. For example: export VLM_WINDOW_SIZE=1073741824 to set the indirect window size to 1GB. Any value set should be a multiple of 64KB.
Notes:
1. Increasing the buffer cache size (or the indirect window size) too high can cause Oracle attach errors while starting up.
2. If you try to use an Oracle binary that has a lower SGA base but did lower the /proc/<pid>/mapped_base value, you will experience unpredictable results ranging from ORA-3113 errors, attach errors, etc while starting up.
3. If you don't increase the shmmax value, you could get attach errors while starting up.
4. The address space is limited. So if you lower the SGA base and consume most of the address space with a larger SGA, there will be less room available for PGA memory. If your application uses a lot of PGA memory, you could get ORA-4030 errors (out of process memory). In this case, setting the SGA base to a higher value (and lowering the SGA size) will reserve more space for PGA memory.
5. If you lower the SGA base and your SGA size is below around 800MB, you may get attach errors. Lowering the SGA base is mainly a way to allocate a large SGA area. Sizes below 800MB should work without having to lower the SGA base.
6. It doesn't always help to increase VLM_WINDOW_SIZE. Also, keep in mind that increasing VLM_WINDOW_SIZE reduces the amount of SGA that can be allocated for other memory areas that might be needed (e.g. locks on RAC). It is best to raise this value as the very last step. This value could be increased once you know how much available address space is left after adjusting init.ora parameters.
7. If you get attach errors while starting up, you will probably need to clean up the shared memory segments by running 'ipcs' and then removing segments via 'ipcrm shm XXX' or 'ipcrm sem XXX'.

Page Address Extensions
In order to get above 4GB virtual memory on IA-32 architecture a technique known as PAE (Page Address Extensions) is used. It is a method that translates 32-bit linear addresses to 36-bit physical addresses. In the linux kernel, the support is provided through a compile time option that produces two separate kernels - the SMP kernel which supports only upto 4GB VM and the enterprise kernel which can go up to 64GB VM (also called VLM capable). This means applications like oracle can make use of the large memory and scale up to a large number of users without loss of performance or reliability.

Shared memory file-system (shmfs) support
It is a memory-based file system optimized for shared memory operations and for larger SGA size.
The shmfs (/dev/shm based) is used by oracle to memory map the dynamic portions of the SGA. This can theoretically allow an SGA up to the size of the shmfs file system that is created. Since shmfs is a memory file system, its size can be as high as the maximum allowable VM size which is 64GB.
1. Mount the shmfs file system as root using command:
      mount -t shm shmfs -o nr_blocks=8388608 /dev/shm
2. Set the shmmax parameter to half of RAM size
      echo 3000000000 >/proc/sys/kernel/shmmax
3. Set the init.ora parameter use_indirect_data_buffers=true.
4. Startup oracle.

Bigpages feature
Page frame of size 4MB as opposed to the regular 4KB. Oracle uses a large contiguous area in the VM for mapping the VLM window. These are used for the dynamic part of the SGA the size of which is specified by the db_block_buffers parameter. The pages corresponding to this area in the VM can easily be of a larger size than the default 4KB and yet there would not be any of the problems like granularity associated with using large page size. A page size of 4MB for these pages would reduce the number of pte-s thus reducing the kernel overhead considerably. The number of TLBs used are also fewer thus reducing TLB thrashing. The result is better scalability in terms of the number of oracle users. Better performance is also achieved because the big pages are not swapped out which means the entire db_block_buffers are in physical memory. The system performance increases as a result of kswapd not having to ‘think’ about swapping out these pages. Since swap space is not pre-allocated for these pages, there is more swap area available and less pagecache complexity.
Use the following steps to set Bigpages feature:
1. Calculate bigpages value for your system with the following formula:

                HighTotal – Max memory required by user connections in KB
Bigpages = -------------------------------------------------------------- * 0.8 MB
                             1024

where
• HighTotal is value in Kbytes and obtained from /proc/meminfo.
• Assuming that 20 % memory is reserved for kernel bookkeeping.

For example, assume that a machine with 8 GB memory and HighTotal of 7208944 KB, is estimated to have 2000 concurrent users, each occupying a memory of 3KB.

Now,
                        7208944 - 2000*30
Bigpages = ---------------------------------- * 0.8 MB = 5585 MB
                             1024


There is a trade-off between the number of users and the bigpages value because, if the value for bigpages is set to a very high value, the memory available for user connections would be low. Hence, always estimate a high value for the maximum number of user connections and the memory that each will consume.

2. In the kernel boot options, add the following line to the boot loader file (e.g. /etc/lilo.conf):
      bigpages=<size>MB
where size is a value in MB calculated in the previous step.
3. Set the /proc/sys/kernel/shm-use-bigpages file to contain the value 2. The other possible values are 0 for no bigpages and 1 for bigpages using sysV shared memory (as opposed to shmfs).