Configuring
RHAS 30 for Oracle and Increasing Memory Size
Applicable
installations
This installation instruction has been checked for:
- Oracle Enterprise Manager (9i and 10g)
- Red Hat Advance Server developer version
Following requirements need to be met for a successful installation
of Oracle 9.2.0.6 (a 3 CD set
which includes the base release and all fixes from prior patchsets
including 9.2.0.4) on Red Hat Enterprise Linux AS/ES 3.0 (RHEL3) on
platform Linux x86 .
Software
* Linux kernel version should be 2.4.21-4.EL
* Glibc Version Number glibc 2.3.2-95.3
* Required OS Components for 9i
- compat-db-4.0.14.5
- compat-gcc-7.3-2.96.122
- compat-gcc-c++-7.3-2.96.122
- compat-libstdc++-7.3-2.96.122
- compat-libstdc++-devel-7.3-2.96.122
- openmotif21-2.1.30-8.i386.rpm
- setarch-1.3-1
* Required OS Components for 10g
+ compat-db-4.1.25-9
+ compat-gcc-32-3.2.3-47.3
+ compat-gcc-32-c++-3.2.3-47.3
+ compat-oracle-rhel4-1.0-3
+ compat-libcwait-2.0-1
+ compat-libgcc-296-2.96-132.7.2
+ compat-libstdc++-296-2.96-132.7.2
+ compat-libstdc++-33-3.2.3-47.3
+ xorg-x11-deprecated-libs-devel-6.8.1-23.EL
+ xorg-x11-deprecated-libs-6.8.1-23.EL
In addition, install the following rpm's (available on Disk3 of the
RHEL4.0 media)
+ openmotif21-2.1.30-11
+ libaio-0.3.102-1
+ libaio-devel-0.3.102-1
Installing
RHAS 30 (we required to install the developer version)
In this section the OS is installed.
- Insert the first cdrom (autoboot)
- Hit Enter to get going (no need to rescue)
- Skip the media test
- On the installation screen, hit Next
- On the language selection, choose your language
- On the keyboard selection, choose your keyboard
- On the mouse selection, choose your mouse
- On the disk setup, choose to partition with DiskDruid
- create a root (/) partition at least 2GB, preferrably bigger than
2.5GB and make it type ext3
- create a swap partition at least 2GB, recommended double the size of
the internal physical memory
- create an oracle partition (/oracle) size depending on the software
to be installed, recommended the remaining disk space
- On the bootloader configuration, choose the bootloader of your
choice, remmended to go with the default "Grub"
- On the network configuration, configure your network. If you are
unsure, leave it to the defaults (DHCP)
- On the Firewall configuration, set your firewall options. If you are
unsure, leave it to the defaults (Enable firewall).
- On additional Language support, choose your preferred langauge.
- On the timezone selection, choose your home country/city.
- Set the root password.
- On the package selection, select "customize the set of packages to be
installed"
- Select at least the following packagesets:
- X
Window System
- Gnome Desktop Environment
- Graphical Internet
- Server Configuration Tools
- Development Tools
- Legacy Software Development
- Administrative Tools
- X Software Development
- At the Graphical Interface configuration, choose the suggested
videocard and memory size. If you are unsure, choose the Other >
Generic VGA compatible
- At the monitor configuration, choose the suggested monitor. If you
are unsure, choose the Generic CRT > Monitor 640x480
- At the Customize Graphics Configuration, choose the the "Text" login
type and choose the screen resolution and depth. If you are unsure,
choose 640x480 at 8bit.
- Let the system reboot
Installing
additional packages (not needed if everything was installed properly)
In this section, additional packages are installed required by oracle.
To see if these development packages are installed on your server, run
the following command:
rpm -q gcc cpp compat-libstdc++ glibc-devel glibc-headers glibc-kernheaders binutils
- After system bootup, login as root and insert the Disk2:
mount /mnt/cdrom
rpm -iv /mnt/cdrom/RedHat/RPMS/sysstat-4.0.7-4.i386.rpm
eject
- insert Disk3:
mount /mnt/cdrom
rpm -iv /mnt/cdrom/RedHat/RPMS/compat-db-4.0.14-5.i386.rpm
rpm -iv /mnt/cdrom/RedHat/RPMS/ORBit-0.5.17-10.4.i386.rpm
rpm -iv /mnt/cdrom/RedHat/RPMS/libpng10-1.0.13-8.i386.rpm
rpm -iv /mnt/cdrom/RedHat/RPMS/gnome-libs-1.4.1.2.90-34.1.i386.rpm
eject
Environment
* Put gcc296 and g++296 first in $PATH variable by creating the
following symbolic links (Metalink Note 252217.1), as root perform:
These are required for the relinking of
some Oracle binaries (as root)
mv /usr/bin/gcc /usr/bin/gcc323
mv /usr/bin/g++ /usr/bin/g++323
ln -sf /usr/bin/gcc296 /usr/bin/gcc
ln -sf /usr/bin/g++296 /usr/bin/g++
* hostname command should return the fully qualified hostname as shown
below:
% hostname
hostname.domainname
* If any Java packages are installed on the system, unset the Java
environment variables, for example JAVA_HOME.
* The oracle account used to install Oracle 9.2.0.1, should not have
the Oracle install related variables set by default. For example
setting ORACLE_HOME, PATH, LD_LIBRARY_PATH to include Oracle binaries
in .profile, .login file and /etc/profile.d shouldbe completely avoided.
Configure
the kernel parameters
In this section, kernel parameters are configured.
- Configuring the kernel parameters, replace "machine123" and
"oracle.com" with your own settings.
echo "" >> /etc/sysctl.conf
echo "# Oracle specific settings" >> /etc/sysctl.conf
echo "kernel.hostname =
machine123.oracle.com" >> /etc/sysctl.conf
echo "# semaphores in kernel.sem: semmsl semmns semopm semmni" >>
/etc/sysctl.conf
echo "kernel.sem = 256 32000 100 142" >> /etc/sysctl.conf
echo "kernel.shmmax = 2147483648"
>> /etc/sysctl.conf
echo "kernel.shmmni = 4096" >> /etc/sysctl.conf
echo "kernel.shmall = 3279547" >> /etc/sysctl.conf
echo "fs.file-max = 327679" >> /etc/sysctl.conf
# Disables packet forwarding
echo "net.ipv4.ip_forward = 0" >> /etc/sysctl.conf
# Enables source route verification
echo "net.ipv4.conf.default.rp_filter = 1" >> /etc/sysctl.conf
echo "net.ipv4.ip_local_port_range = 1024 65000" >>
/etc/sysctl.conf
echo "kernel.msgmni = 2878" >> /etc/sysctl.conf
echo "kernel.msgmnb = 65535" >> /etc/sysctl.conf
echo "" >> /etc/security/limits.conf
echo "# Oracle specific settings" >> /etc/security/limits.conf
echo "oracle soft nofile 1024" >> /etc/security/limits.conf
echo "oracle hard nofile 65536" >> /etc/security/limits.conf
echo "oracle soft nproc 2047" >> /etc/security/limits.conf
echo "oracle hard nproc 16384" >> /etc/security/limits.conf
Configure
the hostname
In this section, the hostname of the machine is set to the hosts file.
- Add as the first line the ip address, fully qualified name of your
machine and alias. Replace the "127.0.0.2" and "machine123.oracle.com"
with your own settings.
cd /etc
mv hosts hosts.org
touch hosts
echo "# Do not remove the following line, or various programs" >>
/etc/hosts
echo "# that require network functionality will fail." >>
/etc/hosts
echo "127.0.0.2 machine123.oracle.com machine123" >> /etc/hosts
echo "127.0.0.1 localhost.localdomain localhost" >> /etc/hosts
Patching
the OS for oracle related issues
In this section, postfixes are applied to the OS.
- Apply patch 3006854,INSTALLER INTEROPERABILITY PATCH FOR RHEL3.0
This patch must be installed on systems
running Red Hat Enterprise Linux 3.0 before running the Oracle
Universal Installer (for any Oracle product). To resolve libcwait
symbol issue before invoking Oracle installer. Download from metalink
patch
3006854 or download from updates.oracle.com
wget
ftp://metalink_user:password@updates.oracle.com/3006854/p3006854_9204_LINUX.zip
unzip p3006854_9204_LINUX.zip
cd 3006854
sh rhel3_pre_install.sh
Shutting
down unused programs
We are not using all of the started options. E.g. on your server, you
do not have the pcmcia and probably don't use power management. A lot
of services are started, but not configured and therefore not running.
- Finding the services configured to start:
chkconfig --list | more
- Finding the services actually running:
ls /var/lock/subsys
- Getting a description of the services
head /etc/init.d/<service>
e.g. "head /etc/init.d/sendmail"
- stopping a service
service <service> stop
e.g. "service sendmail stop"
- configuring a service not to start
chkconfig --level 345 <service>
off
e.g. "chkconfig --level 345 sendmail off"
Creating
the user oracle
In this section, OS oracle and its group are created.
- create the group oinstall and dba
groupadd dba
# group granted with SYSDBA system privilege
groupadd oinstall #
group owner of Oracle files
- create the user oracle
useradd -c
"Oracle software owner" -g oinstall -G dba oracle
passwd oracle
- Create the Oracle Directories
su - root
mkdir /opt/oracle
mkdir /opt/oracle/product
mkdir /opt/oracle/product/9.2.0
chown -R oracle.oinstall /opt/oracle
mkdir /var/opt/oracle
chown oracle.dba /var/opt/oracle
chmod
755 /var/opt/oracle
Modify /.bash_profile
Adding:
# Set
the
LD_ASSUME_KERNEL environment variable only for Red Hat 9 and
# for
Red Hat Enterprise
Linux Advanced
Server 3 (RHEL AS 3) !!
export
LD_ASSUME_KERNEL=2.4.19
#
Oracle Environment
export
ORACLE_BASE=/opt/oracle
export
ORACLE_HOME=/opt/oracle/product/9.2.0
export
ORACLE_SID=FGUARD
export
ORACLE_TERM=xterm
#
export
TNS_ADMIN= Set if sqlnet.ora, tnsnames.ora, etc.
#
are not in
$ORACLE_HOME/network/admin
export
NLS_LANG=AMERICAN;
export
ORA_NLS33=$ORACLE_HOME/ocommon/nls/admin/data
export
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib:/usr/local/lib
#
Set shell
search paths
export
PATH=$PATH:$ORACLE_HOME/bin
Configuring
VNC server
VNC server allowes you to run a very small graphical subsystem, without
using a console. The Oracle Universal Installer will require a
graphical display, but probably you do not want to do an installation
on console.
- Configure service vncserver to be started at boottime
chkconfig --level 345 vncserver on
- Configure vncserver to start vncserver as user oracle
echo "VNCSERVERS=\"1:oracle\"" >>
/etc/sysconfig/vncservers
- Start vncserver manually for the first time. You will be prompted for
a password for the vncserver.
su - oracle
vncserver :1
- As user oracle, configure the vncserver session environment
cd .vnc
mv xstartup xstartup.org
touch xstartup
chmod 755 xstartup
echo "#"\!"/bin/sh" >> xstartup
echo "" >> xstartup
echo "# Uncomment the following two lines for normal desktop:" >>
xstartup
echo "# unset SESSION_MANAGER" >> xstartup
echo "# exec /etc/X11/xinit/xinitrc" >> xstartup
echo "" >> xstartup
echo "[ -r \$HOME/.Xresources ] && xrdb \$HOME/.Xresources"
>> xstartup
echo "xsetroot -solid grey" >> xstartup
echo "vncconfig -iconic &" >> xstartup
echo "xhost localhost" >> xstartup
echo "PATH=\$PATH:/usr/local/bin" >> xstartup
echo "export PATH" >> xstartup
echo "xterm -geometry 80x24+10+10 -ls -title \"\$VNCDESKTOP Desktop\"
&" >> xstartup
echo "twm &" >> xstartup
- As user oracle, stop the vncserver
vncserver -kill :1
exit
Setting
ldap ports
If you want to install the Oracle Internet Directory on the ldap
standard ports, you have to erase all lines containing "ldap" or
"ldaps" out of the /etc/services file.
Reboot the
system
- to active all changes, reboot the system
reboot
After reboot, verify all kernel parameters have been set correct, by
issuing "sysctl -a"
This because the kernel.semmax is not always accepted on 4GB and then
ignored.
Oracle
Installation
* Define and export the environment variable LD_ASSUME_KERNEL before
starting the install process:
export LD_ASSUME_KERNEL=2.4.19
Additional
Installation Steps
* Open a new terminal window
* Apply patch number 3006854 to provide libcwait symbol (as we
mentioned before)
* After completing above steps start ./runInstaller from 9iR2 CD 1 from
shell where you have defined above variable. You may get the following
errors:
- relink error for ins_oemagent.mk -
Click on Continue and complete the install
- apply the patch for Bug: 3119415 after the install is completed
- ctx relinking error in ins_ctx.mk: Click ignore. This is fixed by
applying 9.2.0.4 or 9.2.0.5 patchset.
NOTE: This step will not be necessary to perform if you are installing
from the new release 9.2.0.4 (Part #'s B13496-01, B13497-01, and
B13498-01 which is the complete distribution of 9.2.0.1 with the
9.2.0.4 already installed). Do not confuse this with the 9.2.0.4.0
patchset ( Bug: 3095277 ) which only contains the 9.2.0.4 fixes and is
not a complete package. Also, it should be considered that if you have
installed the 9.2.0.1.0 base release to then apply the 9.2.0.5.0
patchset ( Bug: 3501955 ) instead of 9.2.0.4.0 patchset.
ADDITIONAL
NOTES
* For customers using asynch_io , apply bug: 3208258 (Fixed In Ver:
9.2.0.4 or 9.2.0.5)
RELATED
DOCUMENTS
[NOTE:252217.1] Requirements for Installing Oracle 9iR2 on RHEL3
Implement
Asynchronous I/O
Many times there is a requirement to check if Asynchronous I/O is
working on Linux Platform, so we can try to use it for our datafiles
access inside database.
slabinfo maintains statistics about objects in memory. Some of the
structs used by Asynchronous I/O are threated as objects in the virtual
memory, so we can
look for those structs on slabinfo. The ones related to AIO are named
kio*.
$ cat /proc/slabinfo | grep kio
for example: output with async io enabled.
$ cat /proc/slabinfo | grep kio
kioctx 270 270
128 9 9 1 : 252 126
kiocb 66080 66080 96 1652
1652 1 : 252 126
kiobuf 236
236 64 4 4 1 : 252 126
output with async io disabled.
$ cat /proc/slabinfo | grep kio
kioctx 0 0 128 0 0 1 : 252 126
kiocb 0 0 96 0 0 1 :
252 126
kiobuf 0 0 64 0 0 1 : 252
126
There are 3 caches involved. The kioctx and kiocb are Async I/O
data structures that are defined in aio.h. If it shows a non zero value
that means async io is nabled.
If you have the source code loaded, you can review it at file aio.h.
This file is located under:
/usr/src/linux-<version>/include/linux/aio.h
These data structures are using to track the I/O requests, and are
allocated as part of the __init aio_setup() call in aio.c.
Implement
Advanced memory management techniques
Increasing Usable Address
Space for Oracle on 32-bit Linux
The current shipping version of Oracle is able to use about 1.7GB of
address space for its SGA. To increase this size, Oracle needs to be
relinked with a lower SGA base and Linux needs to have the mapped base
lowered for processes running Oracle. Increasing the address space
allows for more database buffers or a larger indirect data buffer
window to be used.
There are changes that need to be made to the Oracle binary and the
Linux environment (requiring root access), so the appropriate
privileges are needed.
Currently, a solution exists only when running Oracle 9iR2 on Red Hat
2.1 Advanced Server. Red Hat provides an adjustable parameter in the
/proc filesystem to allow more useable address space in processes.
First, the SGA base address that Oracle uses must be lowered by
relinking Oracle. Currently, Oracle ships with this base address set at
0x50000000 so that it is compatible with the defaults set by most
distributions of Linux. Lowering this address allows Oracle to use more
of the address space in the process, but it is important to note that
the newly relinked Oracle binary will no longer work unless a
corresponding modification is also made to Linux (Red Hat 2.1AS
provides a way to do this at runtime).
Follow these steps to complete the first part of the solution:
1. Shutdown all instances of Oracle
2. cd $ORACLE_HOME/lib
3. cp -a libserver9.a libserver9.a.org (to make a backup copy)
4. cd $ORACLE_HOME/bin
5. cp -a oracle oracle.org (to make a backup copy)
6. cd $ORACLE_HOME/rdbms/lib
7. genksms -s 0x15000000 >ksms.s (lower SGA base to 0x15000000)
8. make -f ins_rdbms.mk ksms.o (compile in new SGA base address)
9. make -f ins_rdbms.mk ioracle (relink)
The relinked Oracle binary now has a lower SGA base and is now able to
use about 2.65GB of address space if Linux is also modified to support
this. Next, the Linux kernel's mapped base needs to be lowered below
Oracle's new SGA base. Red Hat 2.1AS has a parameter in /proc that
lowers the kernel's mapped base for each process. This parameter is not
a system-wide parameter. It is a perprocess parameter, but it is
inherited by child processes. This parameter can only be modified by
root. The following steps document how to lower the mapped base for a
single bash terminal session. The default mapped base is 0x40000000.
Once this session has been modified with the lower mapped base, this
session (terminal window) will need to be used for all Oracle commands
so that Oracle processes use the inherited (lower) mapped base:
1. Shutdown the instance of Oracle.
2. Open a terminal session (Oracle session), and get the process id
using "echo $$”.
3. Open a second terminal session and su to root (root session).
4. Now, from the root session,
echo 268435456 >/proc/<pid>/mapped_base,
where <pid> is the process id determined in step 2. This lowers
the mapped base for the Oracle session to 0x10000000.
5. Again, from the root session,
echo 3000000000 >/proc/sys/kernel/shmmax
Ths increases the value of shmmax so that Oracle will allocate the SGA
in one segment.
6. From the Oracle terminal session, startup the Oracle instance.
The SGA now begins at a lower address, so more of the address space can
be used by Oracle. Now you can increase the init.ora values of
db_cache_size or db_block_buffers to increase the size of the database
buffer cache. You can also write a small program that uses setuid() to
set the /proc/<pid>/mapped_base. It would look something like
this:
int main(int argc, char *argv[]) {
pid_t ppid;
char buf[256];
unsigned long mapped_base;
int ret;
#define NEW_MAPPED_BASE 0x10000000
ppid = getppid();
mapped_base = NEW_MAPPED_BASE;
sprintf(buf, "echo %lu >/proc/%u/mapped_base", mapped_base, ppid);
setuid(0);
ret = system(buf);
if (ret == 0)
printf("Lowering mapped base of pid=%u to 0x%X\n", ppid,
mapped_base);
else
printf("unable to lower mapped base. You might need to:\n" "
chmod 4711 lowermap\n chown root.root lowermap\n");
exit(-ret);
}
If you are running with the init.ora parameter
'use_indirect_data_buffers=true' and already have a large buffer cache,
you can use the above solution to increase the indirect buffer window
size. The default is 512MB and should be fine for most applications.
Increasing the window size may increase performance slightly under
certain conditions because a larger indirect window reduces the
overhead of mapping an indirect buffer into Oracle's address space.
To increase the indirect window size, set the environment variable
VLM_WINDOW_SIZE to the window size in bytes before starting up the
Oracle instance. For example: export VLM_WINDOW_SIZE=1073741824 to set
the indirect window size to 1GB. Any value set should be a multiple of
64KB.
Notes:
1. Increasing the buffer cache size (or the indirect window size) too
high can cause Oracle attach errors while starting up.
2. If you try to use an Oracle binary that has a lower SGA base but did
lower the /proc/<pid>/mapped_base value, you will experience
unpredictable results ranging from ORA-3113 errors, attach errors, etc
while starting up.
3. If you don't increase the shmmax value, you could get attach errors
while starting up.
4. The address space is limited. So if you lower the SGA base and
consume most of the address space with a larger SGA, there will be less
room available for PGA memory. If your application uses a lot of PGA
memory, you could get ORA-4030 errors (out of process memory). In this
case, setting the SGA base to a higher value (and lowering the SGA
size) will reserve more space for PGA memory.
5. If you lower the SGA base and your SGA size is below around 800MB,
you may get attach errors. Lowering the SGA base is mainly a way to
allocate a large SGA area. Sizes below 800MB should work without having
to lower the SGA base.
6. It doesn't always help to increase VLM_WINDOW_SIZE. Also, keep in
mind that increasing VLM_WINDOW_SIZE reduces the amount of SGA that can
be allocated for other memory areas that might be needed (e.g. locks on
RAC). It is best to raise this value as the very last step. This value
could be increased once you know how much available address space is
left after adjusting init.ora parameters.
7. If you get attach errors while starting up, you will probably need
to clean up the shared memory segments by running 'ipcs' and then
removing segments via 'ipcrm shm XXX' or 'ipcrm sem XXX'.
Page
Address Extensions
In order to get above 4GB virtual memory on IA-32 architecture a
technique known as PAE (Page Address Extensions) is used. It is a
method that translates 32-bit linear addresses to 36-bit physical
addresses. In the linux kernel, the support is provided through a
compile time option that produces two separate kernels - the SMP kernel
which supports only upto 4GB VM and the enterprise kernel which can go
up to 64GB VM (also called VLM capable). This means applications like
oracle can make use of the large memory and scale up to a large number
of users without loss of performance or reliability.
Shared
memory file-system (shmfs) support
It is a memory-based file system optimized for shared memory operations
and for larger SGA size.
The shmfs (/dev/shm based) is used by oracle to memory map the dynamic
portions of the SGA. This can theoretically allow an SGA up to the size
of the shmfs file system that is created. Since shmfs is a memory file
system, its size can be as high as the maximum allowable VM size which
is 64GB.
1. Mount the shmfs file system as root using command:
mount -t shm shmfs -o nr_blocks=8388608
/dev/shm
2. Set the shmmax parameter to half of RAM size
echo 3000000000
>/proc/sys/kernel/shmmax
3. Set the init.ora parameter use_indirect_data_buffers=true.
4. Startup oracle.
Bigpages
feature
Page frame of size 4MB as opposed to the regular 4KB. Oracle uses a
large contiguous area in the VM for mapping the VLM window. These are
used for the dynamic part of the SGA the size of which is specified by
the db_block_buffers parameter. The pages corresponding to this area in
the VM can easily be of a larger size than the default 4KB and yet
there would not be any of the problems like granularity associated with
using large page size. A page size of 4MB for these pages would reduce
the number of pte-s thus reducing the kernel overhead considerably. The
number of TLBs used are also fewer thus reducing TLB thrashing. The
result is better scalability in terms of the number of oracle users.
Better performance is also achieved because the big pages are not
swapped out which means the entire db_block_buffers are in physical
memory. The system performance increases as a result of kswapd not
having to ‘think’ about swapping out these pages. Since swap space is
not pre-allocated for these pages, there is more swap area available
and less pagecache complexity.
Use the following steps to set Bigpages feature:
1. Calculate bigpages value for your system with the following formula:
HighTotal – Max memory required by user connections in KB
Bigpages =
-------------------------------------------------------------- * 0.8 MB
1024
where
• HighTotal is value in Kbytes and obtained from /proc/meminfo.
• Assuming that 20 % memory is reserved for kernel bookkeeping.
For example, assume that a machine with 8 GB memory and HighTotal of
7208944 KB, is estimated to have 2000 concurrent users, each occupying
a memory of 3KB.
Now,
7208944 - 2000*30
Bigpages = ---------------------------------- * 0.8 MB = 5585 MB
1024
There is a trade-off between the number of users and the bigpages value
because, if the value for bigpages is set to a very high value, the
memory available for user connections would be low. Hence, always
estimate a high value for the maximum number of user connections and
the memory that each will consume.
2. In the kernel boot options, add the following line to the boot
loader file (e.g. /etc/lilo.conf):
bigpages=<size>MB
where size is a value in MB calculated in the previous step.
3. Set the /proc/sys/kernel/shm-use-bigpages file to contain the value
2. The other possible values are 0 for no bigpages and 1 for bigpages
using sysV shared memory (as opposed to shmfs).