Issue #11 September 2005

Tips & tricks

Red Hat's customer service team receives technical support questions from users all over the world. As they are received, Red Hat technicians add the questions and answers to the Red Hat Knowledgebase on a daily basis. Individuals with a redhat.com login are granted access. Every month, Red Hat Magazine offers a preview into the Red Hat Knowledgebase by highlighting some of the most recent entries.

Tips from RHCEs

Readline is your friend

Long-time Linux users often have a bag of tricks which can help them work more quickly and effectively at the command line. These include things like the screen terminal multiplexer, and the use of functions and aliases in ~/.bashrc.

Another great timesaver is the GNU Readline library. It provides a set of keyboard shortcuts which can be used to navigate and edit lines of input in the shell and several other programs.

The bash man page contains some good documentation—you can jump to the relevant section by executing man bash, then typing /^READ<enter>, or just page down until you come to it.

Here are a selection of handy shortcuts to get started:

Shortcut What it does
Ctrl-A Move to beginning of line
Ctrl-E Move to end of line
Alt-F Move forward one word
Alt-B Move backward one word
Ctrl-K Cut to end of line
Alt-Backspace Cut backwards current word
Ctrl-Y Paste from clipboard
Alt-. Paste last argument from previous command
Ctrl-R Reverse search through history
Note:
To use the Alt-F and Alt-B shortcuts in GNOME terminal, you need to disable its keyboard shortcuts. To disable them, select Edit -> Keyboard Shortcuts... from the menu and check both disable boxes.

You may notice that the navigation/cut'n'paste commands have a distinctly Emacs flavor. If you prefer Vi, execute set -o vi. To make this permanent, drop it into your ~/.bashrc file.

See the bash man page for the full story—you can even provide a configuration file to customize the options, here's a brief example:

# ~/.inputrc
$include /etc/inputrc
# this shadows transpose-chars in the default emacs mode
# works around the default C-s being shadowed by linux's
# C-s scroll-lock behavior
C-t:   forward-search-history
C-f:   'realias\n'
"\C-x\C-r": re-read-init-file

Realias is a shell function which reloads .bashrc:

function realias () { . ~/.bashrc ; }

So if you've edited it, there's no need to close and re-start the terminal to get the changes applied—just type Ctrl-F.

What is the relation between I/O wait and load average?

Linux follows the standard of traditional UNIX and computes its load average as the average number of runnable or running processes (R state), and the number of processes in un-interruptible sleep (D state) over the specified interval.

Some other operating systems calculate their load averages simply by looking at processes in R state. On those systems, load average is synonymous with the run queue -- high load averages mean that the box is CPU bound.

This is not the case with Linux. On Linux the load average is a measurement of the amount of "work" being done by the machine (without being specific as to what that work is). This "work" could reflect a CPU intensive application (compiling a program or encrypting a file), or something I/O intensive (copying a file from disk to disk, or doing a database full table scan), or a combination of the two.

What are the basic tuning settings for Red Hat Directory Server?

There are 3 basic kernel tuning parameters available for Red Hat Directory Server. The tuning schemes involves NFS mount settings, number of TCP ports, and number of open files.

NFS mount settings

If the Red Hat Directory Server needs to write to a NFS mounted drive, the following changes in the /etc/rc.d/init.d/autofs file are recommended:

+localoptions='rsize=8192,wsize=8192,vers=3,tcp'

NFS tuning parameters:

     rsize=8192,wsize=8192 - nfs connection is faster than with the default buffer size of 4096.  

     vers=3 - Specifies which version of NFS protocol to use.  

     tcp - NFS mount will use TCP protocol rather than UDP.

Number of TCP ports

Having enough available local system ports for Red Hat Directory Server requests are also important. TCP tuning can be done by increasing the number of available system ports in the /etc/sysctl.conf file:

net.ipv4.ip_local_port_range = 1024 65000

Execute the command below for the changes to take effect:

sysctl -p

The modifications above will eventually change the value in the /proc/sys/net/ipv4/ip_local_port_range file.

Number of open files

File tuning is required if the current maximum number of files that can be stored on the Red Hat Directory Server is less than 64000.

cat /proc/sys/fs/file-max

Edit /etc/sysctl.conf file and add the fields listed below to adjust these values:

fs.file-max = 64000

Afterwards, increase the maximum number of open files by adding the following line to the /etc/security/limits.conf file:

*-nofile8192

Then, Pluggable Authentication Modules (PAM) pam_limits.so is needed in the /etc/pam.d/system-auth file:

#%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
auth        required      /lib/security/$ISA/pam_env.so
auth        sufficient    /lib/security/$ISA/pam_unix.so likeauth nullok
auth        required      /lib/security/$ISA/pam_deny.so

account     required      /lib/security/$ISA/pam_unix.so

password    required      /lib/security/$ISA/pam_cracklib.so retry=3 type=
password    sufficient    /lib/security/$ISA/pam_unix.so nullok use_authtok md5 shadow
password    required      /lib/security/$ISA/pam_deny.so

session     required      /lib/security/$ISA/pam_limits.so
session     required      /lib/security/$ISA/pam_unix.so

Execute the command below so the changes in /etc/sysctl.conf file will take effect:

sysctl -p

Meanwhile, the users will need to log out and then log back in order for the modifications in the limits.conf file take effect.

How do I limit the number of simultaneous web connections from a client system via the Squid proxy server?

Using Squid's maxconn Access Control List (acl) element it is possible to limit simultaneous web connections coming from client systems that use a Squid proxy server. To use the maxconn element, the "client_db on" directive should be set. By default this is activated in the /etc/squid/squid.conf configuration file.

Edit the /etc/squid/squid.conf file and look for the fields listed below to be able put the maxconn parameters afterwards:

# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS

# Example rule allowing access from your local networks. Adapt
# to list your (internal) IP networks from where browsing should
# be allowed
#acl our_networks src 192.168.1.0/24 192.168.2.0/24
#http_access allow our_networks

Here is an example of limiting 5 simultaneous web access from the same client system:

acl STUDENTS 192.168.3.0/24
acl numbercon maxconn 5
http_access deny STUDENTS numbercon

Restart the Squid service for the changes to take effect:

service squid restart

How do I configure a client system to automatically drop an idle LDAP user connection to the LDAP server based on idle time restrictions?

Using the authconfig tool, it is possible to configure LDAP authentication for a client system. The authconfig tool reads and modifies the /etc/nsswitch.conf, /etc/ldap.conf, /etc/pam.d/system-auth and /etc/openldap/ldap.conf files for LDAP settings.

With the /etc/ldap.conf idle_timelimit directive, the allowed number of seconds an idle LDAP user have access to the LDAP server can be set. Once the idle_timelimit is reached, the connection is automatically dropped.

See the example /etc/ldap.conf setting:


# @(#)$Id: index.html,v 1.6 2005/09/21 16:26:50 tfox Exp $
#
# This is the configuration file for the LDAP nameservice
# switch library and the LDAP PAM module.
#
# PADL Software
# http://www.padl.com
#

# Your LDAP server. Must be resolvable without using LDAP.
# Multiple hosts may be specified, each separated by a
# space. How long nss_ldap takes to failover depends on
# whether your LDAP client library supports configurable
# network or connect timeouts (see bind_timelimit).

host 192.168.0.1
#LDAP Servers
 
base dc=feu,dc=example,dc=org
#base object of the server

idle_timelimit 3600
#sets the user idle time the connection is automatically drop

The changes will take effect once the LDAP user reconnects to the LDAP server.

What is the best way to monitor my system's performance over a long period of time versus short time frames?

The most sophisticated tool available to you for resource monitoring is the Sysstat tool. Sysstat contains the following tools: iostat, mpstat, sadc, and sar. The iostat tool displays an overview of CPU utilization, along with I/O statistics for one or more disk drives. The mpstat tool displays more in-depth CPU statistics. The sadc tool collects system resource utilization information and writes it to a file. The sar tool produces reports created by sadc.

How can I monitor memory utilization over time without having to look at scrolling output?

Use the watch free command. The watch command issues the free command every two seconds, after first clearing the screen. This makes it much easier to determine how memory utilization changes over time, as it is not necessary to scan continually scrolling output. Two seconds is the default display interval for the watch command.

What are Reservations for the ext3 file system in Red Hat Enterprise Linux 4 and can Reservations improve performance issues?

Note:
For the purpose of clarity, this tip simplifies a few technical details on the file system side. The fundamental concept is still the same.

Before going into how Reservations can fix performance issues, an explanation of what the problem is that this technology fixes would be necessary.

What Reservations technology solves

ext3 has a so-called "bitmap" that keeps track of which parts of the file system are free and which parts are in use. Liken this to a big checkers board where a dark square indicates "in use" and a light square indicates "free." For each 4 kilobytes of data (a "block" in ext3 terms) there is one such square and those 4Kb are either entirely "free" or "in use." For this tip, visualize free blocks with dots and occupied blocks with capital letters as shown below:

........................

would be an entirely free (and thus empty) disk and:

AAAAAAAAAAAAAA..........
would be a file system that is mostly occupied with file A.

When creating or growing a file, the file system looks at this bitmap for free blocks to put the data in (at which point the file system will mark those blocks as in use, this is called "allocating"). Normally, the kernel will search for free blocks starting from the end of the existing file, with the goal of finding a free block close to the rest of the file's data.

There would be no problems if only a single file on the system is growing. However, when there are 2 files growing, say file A and file B, the following can happen: File A grows a bit, so the kernel takes the first block of the disk:

A..........................

If file B then grows a bit, the kernel again takes the first free block of the disk, which is the second block:

AB.........................

If file A and B alternatively grows again and again, this happens:

ABAB.......................

The end results could be:

ABABABABABABABABABABAB.....

This may sound like a theoretical scenario that is rare in practice, however, in multi-threading applications such as web servers or databases this is a very typical scenario.

An ABAB pattern is actually very problematic: To read the information in file A, the file system needs to skip ("seek" in technical terms) over all the B data. As a result, the disk is constantly skipping over data. Modern disks are very fast if they can read data without skipping (50 Megabytes per second is no exception). However, if a disk needs to skip a lot, this performance goes down to typically 1 Megabyte per second or less. The conclusion: Skipping is bad for performance and we are skipping a lot in this ABAB pattern, so this ABAB pattern is bad for performance.

What are Reservations?

The idea behind reservations is quite simple: when allocating a block for data, the file system will actually reserve the next blocks on the disk for this same file, in anticipation of future data. In the examples that follow, lowercase letters are used for reserved-but-not-in-use blocks.

File System using Reservations

So in the scenario used before with files A and B growing together, when the first information of file A gets written, the bitmap looks like this:

Aaaaaaaaaaaa...............

Now when file B gets extended, the file system notices the reservations made for file A and will start file B outside of the reserved area. The file system will also reserve some space for future data for file B.

AaaaaaaaaaaaBbbbbbbbbb.....

Now file A gets extra data, and for that, the file system will use up some of the reservation previously made:

AAaaaaaaaaaaBbbbbbbbbb.....

When file B gets more data, the reservation for file B gets used up for that as well. After a few pieces of data from both files (just like before), the bitmap will look like:

AAAAAAAAAAAABBBBBBBBBB.....

When reading back the contents of file A, the file system does not need to skip any information, all the information is stored in consecutive blocks on the disk. The disk will then operate at full speed which is typically 50 Megabytes per second.

Reserved blocks are not really in use, but reserved in memory. If the system gets rebooted or the file system is unmounted, all the information about reservations is forgotten.

If the disk is filling up and there are no free blocks left save for reserved blocks, the file system will reclaim some of the reserved blocks for new data.

Another challenging problem for the file system is the amount of space to reserve for a file. If it reserves too much, there will be small gaps between the files. The disk would also run out of unreserved free blocks faster. If it reserves too little, there might be a case where you get an:

 AAAABBBBAAAABBBBAAAABBBB... 

pattern which, while better than ABABAB, is still not optimal.

How Reservations work

The reservation code in Red Hat Enterprise Linux 4 uses a heuristic for this size: the more data that gets added to a file, the bigger the reservation for future additions will be. Heuristics are never perfect and therefore applications can give the kernel hints about how big the file will become so that the optimal reservation can be made.

Conclusion: With well working reservations, the file system is able to prevent skipping when reading back the data and since skipping costs a lot of performance, reservations enhance performance.

The information provided in this article is for your information only. The origin of this information may be internal or external to Red Hat. While Red Hat attempts to verify the validity of this information before it is posted, Red Hat makes no express or implied claims to its validity.

This article is protected by the Open Publication License, V1.0 or later. Copyright © 2004 by Red Hat, Inc.