Linux Administration Commands

Top = Load average should be less than 1, otherwise the processes are fully burdened

The Server Load Average gives the sum of the average number of jobs in the queue over the last 1, 5, and 15 minutes. Load average is not a UNIX command - it is an embedded metric that appears in the output of other UNIX commands such as uptime and procinfo.

The uptime command ($ uptime) is used to display the system load average. The top facility displays the uptime line at the top of its display. The uptime command displays 1 line of output:
The current time, how long the system has been running, how many users are currently logged on, and the system load averages for the past 1, 5 and 15 minutes. There are other commands that display the Server Load Average: procinfo, w and others. Nevertheless, the most informative command by far is the 'top' command. Viewing ALL running processes on your machine using ps -aux is also another good command.

The Top command has lots of command-line options to help you manipulate the way the processes are presented. View the man top pages to view them all. 'Top' is also a great way to view sleeping, running, zombie and stopped processes.

Amount of RAM, hardware config

cat /proc/meminfo
cat /proc/cpuinfo
/bin/dmesg
cat /var/log/dmesg
How to Read '/proc/meminfo' Output (<note:233753.1>)

Check swap activity (Oracle Note 225451.1)

/sbin/swapon -s
free -t
xosview
cat /proc/swaps
Swap Space on RedHat Advanced Server - FAQ
sar -W 5 5

The recommended SWap size is two to three times the amount of Physical Memory for Swap space (unless the system exceeds 1 GB of Physical Memory, where two times the amount of Physical Memory for Swap space is sufficient)
Swap space in Linux is used when the amount of physical memory (RAM) is full.If the system needs more memory resources and the physical memory is full, inactive pages in memory are moved to the swap space. While swap space can help machines with a small amount of RAM, it should not be considered a replacement for more RAM. Swap space is located on hard drives, which have a slower access time than physical memory.
Swapping is one of the Unix mechanisms to accommodate the size limitation of memory by moving entire processes to disk to reclaim memory.
Paging is another Unix machanism to manage the limitations of memory. Unlike swapping, where entire processes are moved in and out of memory, paging moves only individual pages of processes to disk. Paging is not as serious a problem as swapping, as the entire program does not have to reside in memory to run. A small amount of paging may not noticeably affect the performance of a system. However, the performance of a system may degraderapidly as paging activity increases.
Swap space on disk is used to hold pages of memory that have been paged or swapped out. A shortage of swap space may cause symptoms such as system hanging, poor response times, and unsuccessful spawning of new processes.
Swap space can be a dedicated swap partition (recommended), a swap file, or a combination of swap partitions and swap files.
When analyzing your UNIX machine, make sure that the machine is not swapping at all and at worst paging lightly. This indicates a system with a healthy amount of memory available.

How can I enable Swap ?
First check is Swap is enabled:
/sbin/swapon -s

Filename Type Size Used Priority
/dev/hda3 partition 514072 2816 -1

To enable swap, check for swap entries in your /etc/fstab
grep swap /etc/fstab
/dev/hda3 swap swap defaults 0 0

and use the '/sbin/swapon -a' command to enable all Swap partitions listed in /etc/fstab.

How to add a swapfile?

A: Determine the size of the new swap file and multiple by 1024 to determine the block size.
For example, the block size of a 64 MB swap file is 65536.
At a shell prompt as root, type the following command with count being equal to the desired block size:
% dd if=/dev/zero of=/data2/swapfile1 bs=1024 count=65536

Setup the swap file with the command:
% /sbin/mkswap /data2/swapfile1

To enable the swap file immediately but not automatically at boot time:
% /sbin/swapon /data2/swapfile

To enable it at boot time, edit /etc/fstab to include:
/data2/swapfile swap swap defaults 0 0

The next time the system boots, it will enable the new swap file.

Check Services Running and stop them if not used
netstat
netstat -lt
netstat -m     Shows ports where they are running
netstat -tap | grep LISTEN   --> Shows currently running processes with PID and Program Name

Services that should be removed: r* (shell or rsh, login or rlogin, exec or rexec, rcp), telnet, ftp, sendmail, exim, postfix, printer, qmail, http, portmap, SMBD (Samba)

chkconfig --list        --> Show services running and its level
chkconfig --del servicename       --> Stop that service
chkconfig --level 345 servicename off      --> Stop that service for level 3,4,5

Also it could be necessary to check the file /etc/inetd.conf because it has references to some services, if any service that I want to stop is there, comment that line and reboot the server or run
/etc/init.d/inetd restart

Enable FTP and TELNET Services
cd to /etc/xinetd.d
vi wu-ftpd
Change the disable field from "yes" to "no".
Save changes.
vi telnet
Change the disable field from "yes" to "no"
Save changes

OS version

/usr/bin/uname -s -r
cat /proc/version
cat /etc/issue

OS kernel parameters files (<note:68862.1>)

/sbin/sysctl -a
more /etc/sysctl.conf Parameter kernel.shmmax shows Shared Space, must be less than REAL Memory.
/usr/src/linux/include/*
/usr/src/linux/include/linux/shm.h for shared memory
/usr/src/linux/include/linux/sem.h for semaphores
How to permanently set kernel parameters on Linux (<note:242529.1>)

Max number of semaphores sets (SEMMNI)

/usr/bin/ipcs -ls (max number of arrays)
/sbin/sysctl kernel.sem (4th & last value)
awk '{print $4}' /proc/sys/kernel/sem
Linux: How to Check Current Shared Memory, Semaphore Values (<note:226209.1>)

Max number of semaphores systemwide (SEMMNS)

/usr/bin/ipcs -ls (max semaphores system wide)
/sbin/sysctl kernel.sem (2nd value)
awk '{print $2}' /proc/sys/kernel/sem

Max number of shared segments

/sbin/sysctl kernel.shmmni
/usr/bin/ipcs -lm (max number of segments)
cat /proc/sys/kernel/shmmni

Max shared segment size

/sbin/sysctl kernel.shmmax
/usr/bin/ipcs -lm (max seg size (kbytes))
cat /proc/sys/kernel/shmmax (max value=4Gb)

Max number of open files per user/system wide = more /proc/sys/fs/file-nr
The three values in file-nr denote the number of allocated file handles, the number of free file handles and the maximum number of file handles. The number of free file handles is counted out of the number of allocated file handles. This means, that the number of used file handles is (allocated-free) and at any given time, one can open up to (maximum-(allocated-free)) additional file handles. If this number gets close to zero, you probably want to increase the maximum, by writing to file-max, or you'll start getting errors when programs try to open files.

Using PMAP to determine the memory size of background processes
First, to determine the memory size, the process id (PID) of the Oracle background process must be found. This is done by issuing the following command:
% ps -ef |grep SID
oracle 18523     1 0 00:07:03 ?        0:13 oracleDEVSOL (LOCAL=NO)
oracle 15237     1 0 09:21:27 ?        0:12 oracleDEVSOL (LOCAL=NO)
oracle   682     1 0   Jun 25 ?        0:06 oracleDEVSOL (LOCAL=NO)
oracle 16544     1 0 14:25:53 ?        0:00 oracleDEVSOL (LOCAL=NO)
oracle 19326     1 0 06:49:13 ?        0:00 oracleDEVSOL (LOCAL=NO)
oracle   532     1 0   Jun 25 ?        0:01 ora_pmon_DEVSOL
oracle   534     1 0   Jun 25 ?       27:33 ora_dbw0_DEVSOL
oracle   536     1 0   Jun 25 ?       47:12 ora_lgwr_DEVSOL
oracle   538     1 0   Jun 25 ?        3:39 ora_ckpt_DEVSOL
oracle   540     1 0   Jun 25 ?        1:55 ora_smon_DEVSOL
oracle   542     1 0   Jun 25 ?        0:00 ora_reco_DEVSOL
oracle   544     1 0   Jun 25 ?        0:20 ora_cjq0_DEVSOL
oracle   546     1 0   Jun 25 ?        2:40 ora_qmn0_DEVSOL
oracle   548     1 0   Jun 25 ?        0:00 ora_s000_DEVSOL
oracle   550     1 0   Jun 25 ?        0:00 ora_d000_DEVSOL
oracle   555     1 0   Jun 25 ?        0:44 ora_p000_DEVSOL
oracle   557     1 0   Jun 25 ?        0:40 ora_p001_DEVSOL
oracle   559     1 0   Jun 25 ?        1:04 ora_p002_DEVSOL
oracle   561     1 0   Jun 25 ?        2:53 ora_p003_DEVSOL

Second, enter the following command:
pmap -x 540 (540 is the PID for the SMON process)
         Address   Kbytes Resident Shared Private Permissions       Mapped File
0000000100000000   50472   23640   21336    2304 read/exec         oracle
0000000103248000     712     512     368     144 read/write/exec   oracle
00000001032FA000     392     208       -     208 read/write/exec     [ heap ]
0000000380000000 1462272 1462272       - 1462272 read/write/exec/shared [ ism shmid=0x65 ]
FFFFFFFF7CE70000      72      72       -      72 read/write          [ anon ]
FFFFFFFF7CE88000      32      16       -      16 read/write          [ anon ]
FFFFFFFF7CF00000       8       8       -       8 read/write          [ anon ]
FFFFFFFF7CF10000       8       8       -       8 read/write          [ anon ]
FFFFFFFF7CF50000     136     128       -     128 read/write          [ anon ]
FFFFFFFF7CF74000      48      40       -      40 read/write          [ anon ]
FFFFFFFF7D000000       8       -       -       - read/write/exec     [ anon ]
FFFFFFFF7D100000      16      16       8       8 read/exec         libc_psr.so.1
FFFFFFFF7D200000      16      16       8       8 read/exec         libmp.so.2
FFFFFFFF7D304000       8       8       -       8 read/write/exec   libmp.so.2
FFFFFFFF7D400000      88      72      64       8 read/exec         libm.so.1
FFFFFFFF7D516000       8       8       -       8 read/write/exec   libm.so.1
FFFFFFFF7D600000       8       8       -       8 read/write/exec     [ anon ]
FFFFFFFF7D700000       8       8       -       8 read/exec         libkstat.so.1
FFFFFFFF7D802000       8       8       -       8 read/write/exec   libkstat.so.1
FFFFFFFF7D900000      32      32      24       8 read/exec         librt.so.1
FFFFFFFF7DA08000       8       8       -       8 read/write/exec   librt.so.1
FFFFFFFF7DB00000      32      32      24       8 read/exec         libaio.so.1
FFFFFFFF7DC08000       8       8       -       8 read/write/exec   libaio.so.1
FFFFFFFF7DD00000     704     600     504      96 read/exec         libc.so.1
FFFFFFFF7DEB0000      56      56       -      56 read/write/exec   libc.so.1
FFFFFFFF7DEBE000       8       8       -       8 read/write/exec   libc.so.1
FFFFFFFF7E000000      32      24       8      16 read/exec         libgen.so.1
FFFFFFFF7E108000       8       8       -       8 read/write/exec   libgen.so.1
FFFFFFFF7E200000      56      40      32       8 read/exec         libsocket.so.1
FFFFFFFF7E30E000      16      16       -      16 read/write/exec   libsocket.so.1
FFFFFFFF7E400000    5328    1864    1736     128 read/exec         libjox9.so
FFFFFFFF7EA32000     384     288       -     288 read/write/exec   libjox9.so
FFFFFFFF7EA92000       8       -       -       - read/write/exec   libjox9.so
FFFFFFFF7EB00000       8       8       -       8 read/write/exec     [ anon ]
FFFFFFFF7EC00000     656     224     216       8 read/exec         libnsl.so.1
FFFFFFFF7EDA4000      56      56       -      56 read/write/exec   libnsl.so.1
FFFFFFFF7EDB2000      40       -       -       - read/write/exec   libnsl.so.1
FFFFFFFF7EE00000      32      24       8      16 read/exec         libskgxn9.so
FFFFFFFF7EF06000       8       8       -       8 read/write/exec   libskgxn9.so
FFFFFFFF7F000000       8       8       -       8 read/write/exec     [ anon ]
FFFFFFFF7F100000       8       8       -       8 read/exec         libskgxp9.so
FFFFFFFF7F200000       8       8       8       - read/write/exec   libskgxp9.so
FFFFFFFF7F300000       8       8       -       8 read/exec         libodmd9.so
FFFFFFFF7F400000       8       8       8       - read/write/exec   libodmd9.so
FFFFFFFF7F500000       8       8       -       8 read/exec         libdl.so.1
FFFFFFFF7F600000     128     128     120       8 read/exec         ld.so.1
FFFFFFFF7F71E000       8       8       -       8 read/write/exec   ld.so.1
FFFFFFFF7F720000       8       8       -       8 read/write/exec   ld.so.1
FFFFFFFF7FFDC000     144     120       -     120 read/write          [ stack ]
---------------- ------ ------ ------ ------
        total Kb 1522136 1490664   24472 1466192

The private memory of this SMON process is 1466192K minus the SGA size which is the line marked with 'shmid=' above. In this case it is 1462272K.
The calculation is as follows: 1466192K minus 1462272K is 3920K.
So, the process memory for SMON is 3920K.

TCP keepalive parameters

net.ipv4.tcp_keepalive_time (seconds)
net.ipv4.tcp_keepalive_intvl (seconds)
net.ipv4.tcp_keepalive_probes

TCP tuning parameters

send buffer size: net.core.wmem_default/net.core.wmem_max (128 to 256k recommended)
receive buffer size: net.core.rmem_default/net.core.rmem_max (128 to 256k recommended)
Autotuning mode
- min/default/max send buffer size: net.ipv4.tcp_wmem
- min/default/max receive buffer size: net.ipv4.tcp_rmem
- low/default/max system memory for TCP stack (in pages): net.ipv4.tcp_mem

Process Limits (<note:188149.1>) ulimit -a

Display system/error messages = dmesg

OS kernel parameters values = ipcs -l

Am I on a 32 or 64 bit OS? = more /proc/cpuinfo

General Performance

vmstat 5 3 ( = This utility provides a report that covers process activity, paging, memory usage, disk I/O, and CPU usage (also you can use xosview). When analyzing your UNIX machine, make sure that the machine is not swapping at all and at worst paging lightly.
If US + SY near 100 operating at full capacity

procs                      memory      swap          io     system         cpu
r b   swpd   free   buff cache   si   so    bi    bo   in    cs us sy id wa
0 0 329476 54880 91600 613852    0    1     4     2    0     0 1 1 3 1
0 0 329476 54560 91600 613852    0    0     0    36 118   128 25 0 74 0
0 0 329476 54564 91600 613860    0    0     1    48 127   143 25 0 74 1

Here there are NO pageouts (po or so) occurring on this system. It is OK and normal to have page out (po or so) activity. You should get worried when the number of page ins (pi or si) starts rising. This indicates that you system is starting to page
There are no processes that are waiting to be run (r), blocked (b), or waiting for IO (w) in the RUN QUEUE (When a process is ready to be processed by a CPU it will be placed on the waiting line or RUN-QUEUE). You want to keep the RUN-QUEUE under 5-6 for a single CPU machine.

Having any processes in the b or w columns is a sign of a problem system.
Having an id of 0 is a sign that the cpu is overburdoned.
Having high values in pi and po show excessive paging.

procs (Reports the number of processes in each of the following states)
- r : in run queue. Shows # of tasks executing and waiting for CPU Resources, if > than # of CPU’s a bottleneck exists
- b : blocked for resources (I/O, paging etc.)
- w : runnable but swapped
memory (Reports on usage of virtual and real memory)
- swap : swap space currently available (Kbytes)
- free : size of free list (Kbytes)
page (Reports information about page faults and paging activity (units per second)
- re : page reclaims
- mf : minor faults
- pi : Kbytes paged in
- po : Kbytes paged out
- fr : Kbytes freed
- de : anticipated short-term memory shortfall (Kbytes)
- sr : pages scanned by clock algorith
disk (Reports the number of disk operations per second for up to 4 disks
faults (Reports the trap/interupt rates (per second)
- in : (non clock) device interupts
- si : system calls. Page IN operation, swapping, short RAM
- cs : CPU context switches
cpu (Reports the breakdown of percentage usage of CPU time (averaged across all CPUs)
- us : user time. Amount of CPU that is servicing user tasks
- si : system time. % of CPU being used to service SYSTEM Tasks
- cs : idle time
- id : % of CPU that is idle, That means that the CPU is not being heavily used at all

free = To check amount of memory and if exists or not swapping activity

sar -b = To check physical disk I/O activity or buffer Activity

CPU Usage

sar -u 10 5 (10 seconds apart; 8 times):= Reports CPU Utilization. Nice column is the priority of that process, bigger numbers less priority
13:10:56 %usr %sys %wio %idle
13:11:02 51   5        3       42
13:11:07 45   7        16     32
13:11:12 51   4         1      43
13:11:18 51   8         7      34
13:11:23 52   13       3      32
%usr: Percent of CPU in user mode
%sys: Percent of CPU in system mode
%wio: Percent of CPU running idle with a process waiting for block I/O
%idle: Percent of CPU that is idle

Once it is established that the system has high CPU usage, the next step is to find out who is using the CPU.
Ps -fe | grep smon
or
ps -e -o pcpu -o pid -o user -o args | sort -k 1 | tail -21r
Displays the top 10 CPU users on the system.

UID      PID   PPID   C STIME TTY TIME CMD <-- Label added for clarity.
usupport 28180 1      0 Oct 31 - 0:48 ora_smon_V734
usupport 30262 1      0 Nov 01 - 0:00 ora_smon_VKHILL
usupport 30900 1      0 Oct 14 - 9:03 ora_smon_V806
usupport 31958 1    111 Oct 24 - 3:31 ora_smon_V815 <-- Notice the C column
usupport 37986 1      0 Nov 06 - 14:00 ora_smon_V805

Here we can see a smon of the database V815 using a lot of CPU by looking at the C column which reflects the CPU units of processing that are being used.
There are 100 units per CPU so The reason why this number is above 100 is that this machine has 2 cpus.

% /usr/sbin/bindprocessor -q
The available processors are: 0 1

PC XStation Configuration

- Download the CygWin setup.exe from http://www.cygwin.com.
- Install, making sure to select all the XFree86 optional packages.
- If you need root access add the following entry into the /etc/securettys file on each server:
    <client-name>:0
- From the command promot on the PC do the following:
     set PATH=PATH;c:\cygwin\bin;c:\cygwin\usr\X11R6\bin
          XWin.exe :0 -query <server-name>
- The X environment should start in a new window.

Many Linux distributions do not start XDMCP by default. To allow XDMCP access from Cygwin edit the "/etc/X11/gdm/gdm.conf" file. Under the "[xdmcp]" section set "Enable=true".
Also edit the file /etc/X11/xdm/xdm-config and comment the line DisplayManager.requestPort: 0 as !DisplayManager.requestPort:    0

If you are starting any X applications during the session you will need to set the DISPLAY environment variable. Remember, you are acting as an XStation, not the server itself, so this variable must be set as follows:

DISPLAY=<client-name>:0.0; export DISPLAY

sysctl
Configurar los paràmetros del kernel en tiempo de ejuecución.
Ejemplos: sysctl -a

adduser
añadir usuario de sistema.
Ejemplos: adduser pepe, adduser -s /bin/false pepe

userdel
eliminar usuario de sistema
Ejemplos: userdel pepe

usermod
modificar usuario de sistema
Ejemplos: usermod -s /bin/bash pepe

df
disk free. espacio en disco disponible. Muy util.
Ejemplos: df, df -h

uname
Informacion sobre el tipo de unix en el que estamos, kernel, etc.
Ejemplos: uname, uname -a

netstat
la informacion sobre las conexiones de red activas.
Ejemplos: netstat, netstat -ln, netstat -l, netstat -a

traceroute

herramienta de red que nos muestra el camino que se necesita para llegar a otra maquina.
Ejemplos: traceroute www.rediris.es

du
disk use. uso de disco. Muestra el espacio que esta ocupado en disco.
Ejemplos: du *, du -sH /*, du -sH /etc

ifconfig
On Linux systems, the ethernet device is typically called eth0. In order to find the MAC address of the ethernet device, you must first become root, through the use of su. Then, type ifconfig -a and look up the relevant info.
For example:
# ifconfig -a eth0
Link encap:Ethernet HWaddr 00:60:08:C4:99:AA
inet addr:131.225.84.67 Bcast:131.225.87.255 Mask:255.255.248.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:15647904 errors:0 dropped:0 overruns:0
TX packets:69559 errors:0 dropped:0 overruns:0
Interrupt:10 Base address:0x300
The MAC address is the HWaddr listed on the first line.
In the case of this machine, it is 00:60:08:C4:99:AA.

route
gestiona las rutas a otras redes.
Ejemplos: route, route -n

iptraf
muestra en una aplicacion de consola TODO el trafico de red IP, UDP, ICMP.
Permite utilizar filtros, y es SUMAMENTE UTIL para diagnostico y depuracion de firewalls
Ejemplos: iptraf

tcpdump
vuelca el contenido del trafico de red.
Ejemplos: tcpdump, tcpdump -u

lsof
muestra los ficheros(librerias, conexiones) que utiliza cada proceso
Ejemplos: lsof, lsof -i, lsof | grep fichero

lsmod
Muestra los modulos de kernel que estan cargados.
Ejemplos: lsmod

modprobe
Trata de instalar un modulo, si lo encuentra lo instala pero de forma temporal.
Ejemplos: modprobe ip_tables, modprobe eepro100

rmmod
Elimina modulos del kernel que estan cargados
Ejemplos: rmmod <nombre de modulo>

sniffit
Sniffer o husmeador de todo el trafico de red. No suele venir instalado por defecto.
Ejemplos: sniffit -i

who
El comando "who" informa de los usuarios que se hallan presentes en el sistema

Get OS File System Block Size

For ext2 and ext3 file systems, the command is
/sbin/tune2fs -l <device_name>,
which returns a whole bunch of info on the file system, including block size. For example:
/sbin/tune2fs -l /dev/hdb1 | grep 'Block size'
Block size: 4096

Another way to do it could be to create a file with only 1 character and then perform:
du -b filename

Another one:
If you are using ext2 filesystem, 'dumpe2fs <device>' will do it

Managing Packages

See the list of installed packages
rpm -qa

How do I remove the old kernel and keep only the latest kernel?
# rpm -q kernel

To see what packages are installed. Then e.g. remove one (or several at once):
# rpm -e kernel-2.4.18-10 kernel-2.4.18-14

Metalink Notes on Linux

note:265262.1       Oracle on Linux Full Library
note:270683.1       Pre Install checks for the Oracle Application Server 10g (9.0.4) on Linux Platforms
note:263715.1       Configuring RHAS 30 for Usage with Oracle
note:184821.1       Step-By-Step Installation of 9.2.0.4 RAC on Linux
bug:3016968         Asyncio Functionality Is Not Working On RHEL 3 With 10g
note:1037322.6     WHAT IS THE DB_FILE_MULTIBLOCK_READ_COUNT PARAMETER?
note:68633.1         Init.ora Parameter "SORT_MULTIBLOCK_READ_COUNT" Reference
note:39023.1         Init.ora Parameter "HASH_MULTIBLOCK_IO_COUNT" Reference Note
note:47324.1         PARAMETER:DB_FILE_DIRECT_IO_COUNT
note:225751.1       Asynchronous I/O (aio) on RedHat Advanced Server - FAQ