Problems with logfiles

1. How do I add browsers and referrers to my logs?

Apache provides a couple of different ways of doing this. The recommended method is to compile the modlogconfig module into your configuration and use the CustomLog directive. You can either log the additional information in files other than your normal transfer log, or you can add them to the records already being written. For example:

CustomLog logs/access_log "%h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-Agent}i\""

This will add the values of the User-agent: and Referer: headers, which indicate the client and the referring page, respectively, to the end of each line in the access log.

2. What is Common Log Format?

The common logfile format is as follows:

remotehost rfc931 authuser [date] "request" status bytes

remotehost

Remote hostname (or IP number if DNS hostname is not available, or if DNSLookup is Off.
rfc931

The remote logname of the user.
authuser

The username as which the user has authenticated himself. [date]

Date and time of the request.
"request"

The request line exactly as it came from the client. status

The HTTP status code returned to the client. bytes

The content-length of the document transferred.

* from http://www.w3.org/Daemon/User/Config/Logging.html#common-logfile-format

3. Can I create multiple custom logs for my server?

Yes you can. Here's an example:

The interface to all this is via a single, simple directive: CustomLog. This directive takes both a file name to log to, and a custom format. For example, to log user-agents to a file called agents in the logs directory, you would use:

CustomLog logs/agent "%{user-agent}i"

Other useful log files can also be created. This next two directives create a referrer log and a log of language preferences of your clients:

             CustomLog   logs/referer  "%{referer}i -> %U"
             CustomLog   logs/language "%{accept-language}i"

4. Can I set up independent logs for my virtual hosted domains?

Yes you can. Here's how:

The logging directives, TransferLog, LogFormat and CustomLog can be used inside virtual hosts just like you would use them for the primary domain. The way they interact with the logs set up outside the virtual hosts is like this:

If there are no TransferLog or CustomLog directives inside the virtual host, log requests for this host to the logs defined in the main server. Otherwise log requests to the log files defined in this virtual host and do not use any of the log files defined in the main server.

If Logformat is used in a virtual host, the format it defines is used for all TransferLog files defined inside that virtual host.

Otherwise the log format defined outside the virtual host is used by the TransferLogs defined inside the host, defaulting to the common log format if no LogFormat is defined in the main server.

5. What configuration sequences are allowed for Apache log Directives?

Here are all the % sequences allowed in the configurable log format in Apache.

%b = bytes sent, excluding HTTP headers

%f = filename

%h = remote host

%{Header}i = The contents of Header: header line(s) in the request sent from the

client

%l = remote username (from identd, if supplied)

%{Note}n = The contents of note "Note" from another module

%{Header}o = The contents of Header: header line(s) in the reply

%p = the port the request was served to

%P = the process ID of the child that serviced the request

%r = first line of request

%s = response status. For requests that got internally redirected, this is

status of the original request: use %>s for the returned status

%t = time, in common log format time format

%{format}t = The time, in the form given by format, which should be in strftime format

%T = the time taken to serve the request, in seconds

%u = remote user (from auth; may be bogus if return status (%s) is 401)

%U = the URL path requested

%v = the name of the server (i.e. the virtual host)

6. How do I reset my SWS server's log files?

Sooner or later, you'll want to reset your log files (accesslog and errorlog) because they are too big, or full of old information you don't need.

access.log typically grows by 1Mb for each 10,000 requests.

Most people's first attempt at replacing the logfile is to just move the logfile or remove the logfile. This doesn't work.

Apache will continue writing to the logfile at the same offset as before the logfile moved. This results in a new logfile being created which is just as big as the old one, but it now contains thousands (or millions) of null characters.

The correct procedure is to move the logfile, then signal Apache to tell it to reopen the logfiles.

Apache is signaled using the SIGHUP (-1) signal. e.g.

mv accesslog accesslog.old
kill -1 `cat httpd.pid`

Note: httpd.pid is a file containing the process id of the Apache httpd daemon, Apache saves this in /var/run/.

Many people use this method to replace (and backup) their logfiles on a nightly or weekly basis.