Your Daily Source for Apache News and Information  
Breaking News Preferences Contribute Triggers Link Us Search About
Apache Today [Your Apache News Source] To internet.com

Apache HTTPD Links
Apache-Related Projects
The Apache Software Foundation
Apache Module Registry
The Jakarta Project
Apache XML Project
Apache-Perl Integration Project
The Java Apache Project
PHP Server Side Scripting
The Apache FAQ
ApacheCon
Apache Project

  internet.com

Internet News
Internet Investing
Internet Technology
Windows Internet Tech.
Linux/Open Source
Web Developer
ECommerce/Marketing
ISP Resources
ASP Resources
Wireless Internet
Downloads
Internet Resources
Internet Lists
International
EarthWeb
Career Resources

Search internet.com
Advertising Info
Corporate Info
Apache Guide: Logging, Part 3 -- Custom Logs
Sep 5, 2000, 17 :37 UTC (12 Talkback[s]) (12945 reads) (Other stories by Rich Bowen)

By

Long ago, log files came in one format. It was called the common format, and you were pretty much stuck with it. Then came custom log file format, and it turned out to be such a good idea that even the common format was reimplemented as a custom log file format.

In this article, you'll find out how to make your log files look like whatever you want, and have whatever information in them that you want.

Overview

Here's the lightning overview for those of you that just want to get something working, and don't care about all the details. You'll need to look at the LogFormat and CustomLog directives. There are several examples in your default httpd.conf file.

LogFormat sets up a format and gives it a nickname by which you can refer to it. CustomLog sets up an actual log file, and indicates the format (by nickname, usually) that file will use.

LogFormat

The LogFormat directive sets up a log format, and a nickname by which you can refer to that format.

For example, in your default httpd.conf file, you'll find the following line:

     LogFormat "%h %l %u %t \"%r\" %>s %b" common

This directive creates a log format called "common", which is in the format specified in quotes. Each one of those letters means a particular piece of information, which is put into the log file in the order indicated.

The available variables, and their meanings, are listed in the documentation, and are reproduced below:

     %...a:          Remote IP-address
     %...A:          Local IP-address
     %...B:          Bytes sent, excluding HTTP headers.
     %...b:          Bytes sent, excluding HTTP headers. In CLF format
                     i.e. a '-' rather than a 0 when no bytes are sent.
     %...{FOOBAR}e:  The contents of the environment variable FOOBAR
     %...f:          Filename
     %...h:          Remote host
     %...H           The request protocol
     %...{Foobar}i:  The contents of Foobar: header line(s) in the request
                     sent to the server.
     %...l:          Remote logname (from identd, if supplied)
     %...m           The request method
     %...{Foobar}n:  The contents of note "Foobar" from another module.
     %...{Foobar}o:  The contents of Foobar: header line(s) in the reply.
     %...p:          The canonical Port of the server serving the request
     %...P:          The process ID of the child that serviced the request.
     %...q           The query string (prepended with a ? if a query string exists,
                     otherwise an empty string)
     %...r:          First line of request
     %...s:          Status.  For requests that got internally redirected, this is
                     the status of the *original* request --- %...>s for the last.
     %...t:          Time, in common log format time format (standard english format)
     %...{format}t:  The time, in the form given by format, which should
                     be in strftime(3) format. (potentially localised)
     %...T:          The time taken to serve the request, in seconds.
     %...u:          Remote user (from auth; may be bogus if return status (%s) is 401)
     %...U:          The URL path requested.
     %...v:          The canonical ServerName of the server serving the request.
     %...V:          The server name according to the UseCanonicalName setting.

In each case, the "..." indicates an (optional) condition. If the condition is met, then the particular. If the condition is ommitted, then the variable will be replaced with a "-" if it is not defined. I'll give some examples of this in a minute.

So, the LogFormat line above, from the default httpd.conf file, creates a log format called common, which contains the remote host, remote logname, remote user, the time of the transaction, the first line of the request, the status of the request, and the number of bytes sent. Which is pretty much what I went through in my last article.

Now, sometimes you'll only want a particlar piece of information logged if it is defined. These is what the "...", referred to above, provides for. If, between the % and the variable, you put one or more HTTP status codes, the variable will only be logged in the event that the request returns one of those status codes. So, if you're trying to keep a log of all the broken links on your site, you might have the following:

     LogFormat %404{Referer}i BrokenLinks

Conversely, if you want to log requests that don't match a particular code, put a ! in there:

     LogFormat %!200U SomethingWrong

CustomLog

Once you have set up one or more LogFormats, you just have to apply them to a particular log file. This is done with the CustomLog directive. You can set up as many log files as you like (well, not really, but you can set up a lot of them). Each one needs to specify a lof file location, and which LogFormat you want to use:

     CustomLog /var/log/httpd/bogus_log SomethingWrong
     CustomLog /usr/local/apache/logs/broken BrokenLinks
     CustomLog /usr/local/apache/logs/access_log common

Conclusion

That's really all there is to it. You can put just about any information in a log file, and format it just about any way you like.

The only disadvantage to doing this is that if you get some off the shelf log analysis application, it will assume that you are using common or combined log format, since those are the ones that are most widely in use.

Next Time

That's all 'til next time. In the next article, I'll talk about parsing your log files -- running some sort of analysis tool on them to get useful information out. That is the thing that your boss really wants. How many people looked at the web site yesterday? Where did they find out about us? Are they coming back, or just one-time visitors? And so on.

Send me a note at if you have any questions, or suggestions for another article. Or make a comment in the Talkback area.

Want to discuss logfiles with other Apache Today readers? Then check out the discussions at Apache Today Discussions.

  Current Newswire:
WDVL: Perl for Web Site Management: Part 3

Retro web application framework V1.1.0 release

Leveraging open standards such as Java, JSP, XML,J2EE, Expresso and Struts.

Netcraft Web Server Survey for November is available

FoxServ 2.0 Released

Ace's Hardware: Building a Better Webserver in the 21st Century

Web Techniques: Customer Number One

Apache-Frontpage RPM project updated

CNet: Open-source approach fades in tough times

NewsForge: VA spin-off releases first product, aims for profit

 Talkback(s) Name  Date
  tried it... didn't work quite right
I tried this, however all I get in my broken link log file is a "-" whenever I test with a bad URL. Possibly this is because there is no referrer. The suggestion is still valid and I'll dig at it some more, but if the author has a suggestion I would be glad to hear it.   
  Sep 6, 2000, 15:17:58
  Analysis Application
If I create a custom log for broken links, these broken links will not be logged in access_log or not to be logged in error_log?   
  Sep 6, 2000, 23:13:16
  different logs
Hi Rich,

The article is very usefull, but I need some help!!!
In my company the log needs to be separeted by day of the month. For exemple:
acess_log.sep012000, acess_log.sep022000 and so on. How can I do this??!

Thanks a lot, Dante   
  Sep 9, 2000, 22:00:46
   Re: tried it... didn't work quite right
> I tried this, however all I get in my broken link log file is a "-" whenever I test with a bad URL. Possibly this is because there is no referrer. The suggestion is still valid and I'll dig at it some more, but if the author has a suggestion I would be glad to hear it.

referer is only set if you followed a link from one page to another. If you type in a URL, there is no referer.
  
  Sep 11, 2000, 18:03:53
  Have u missed %c for cookie...
Hi Rich,
The article was really good and found it quite informative...
I am completely into Log analysis and so the info. was very useful for me.
If I am not mistaken there is also an variable called as %{Cookie}i
which stores the cookie information. Can somebody if has any time guide me on how can I count unique visitors using a cookie. I even made a javascript for the same but could not actually make the logic flow properly...

anybody home!!!   
  Sep 15, 2000, 10:51:28
  %T - time taken to serve request ?
I would like to be able to tell from my access log how long a request took, I have tried adding the '%T' directive to the log format but it only reports in seconds. My question is two-fold

1. is there any directive that will report the time taken in milliseconds ?

2. if my URI is really a java servlet, does this time include any backend processing, i.e. the elapsed time between receiving the request and sending the response back to the browser.

If the answer to the above is no and yes, then I am looking at writing a couple of request handlers using mod_perl and using Time::HiRes to calculate the elapsed time.

tia

Rob.   
  Sep 24, 2000, 14:14:19
  cookie tracking
Hi all,
I am trying to set up the Apache Server to track cookies, and I have tried many things, including setting %{Cookie}o in the LogFormat command, but I haven't tried %{Set_Cookie}i so I will try that soon, although I want a record of the cookies sent to the client, not received from the server. Any help on this matter would be greatly appreciated.
Thanks,
Rob Greathouse   
  Dec 11, 2000, 18:35:30
  LogFormat
Hi,

I cann't use %q variable. I test it with the following format:

1) LogFormat "%h %l %u %t \"%r\" %q %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
2) LogFormat "%h %l %u %t \"%r\" \"%q\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
3) LogFormat "%h %l %u %t \"%r\" %...q %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

But I get the following result:
p> ./apachectl configtest
Syntax error on line 530 of /usr/local/apache/conf/httpd.conf:
Unrecognized LogFormat directive %q
p>

What is wrong? Do I miss somthing?

Best regards,

Hishmat   
  Jul 4, 2001, 08:22:51
  Apache logs
Is there a way to rotate my logs based on size or date? For example, every time the log reaches 1.5Mb or every day, it would roll into a new log. Further, I want to limit how much memory the logs are allowed to take up. For instance, I want all of my logs to only be allowed 100Mb. After 100Mb, the oldest logs are deleted.   
  Aug 3, 2001, 21:24:06
  CustomLog problem
Hi Everybody.

I have a Linux Red Hat 6.0 with Apache Web Server 1.3.19.

I'm trying to avoid the annoying loging of Code-Red requests using conditional logs, i'm using a Enviroment Variable but the web server alway return me an error message like this:

Syntax error on line XXXX of /etc/httpd/conf/httpd.conf:
CustomLog takes two arguments, a file name and a custom log format string or format name

Following i put an excerpt of my configuration file.

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combin
ed
CustomLog /home/usr/teoria/logs/teoria-access_log combined env=!code_red

Where code_red is my Environment Variable.

What's wrong ?
Any tip will be helpful
Thank's in advance   
  Sep 21, 2001, 21:42:08
   Re: CustomLog problem
Very interesting! I on the otherhand am attempting to setup a way to log and notify admins of hosts which connect and check for the exploit.

Any lock so far on your flip side of the coin? I think it is a good idea and I would like to help maybe we can work together on this.

Sean   
  Oct 3, 2001, 16:36:03
   Re: Have u missed %c for cookie...
Yeah, you can get anything out of the http headers with the form:

%{http-var}i

Where 'http-var' is the appropriate http header name, e.g. 'host', 'cookie', etc.

We have a lot of stuff in our cookie header which takes a format similar to:

"rtnuser=7859; JSESSIONID=somebiglongstring"

How can I just log the first part of that - e.g. the value for rtnuser?   
  Oct 19, 2001, 20:15:17
Enter your comments below.
Your Name: Your Email Address:


Subject: CC: [will also send this talkback to an E-Mail address]
Comments:

See our talkback-policy for or guidelines on talkback content.

About Triggers Media Kit Security Triggers Login


All times are recorded in UTC.
Linux is a trademark of Linus Torvalds.
Powered by Linux 2.4, Apache 1.3, and PHP 4
Copyright INT Media Group, Incorporated All Rights Reserved.
Legal Notices,  Licensing, Reprints, & Permissions,  Privacy Policy.
http://www.internet.com/