I recently had to investigate a field issue for our large enterprise application. I was horrified by the logs that I had to comb through in an attempt to find the problem and at the end of the day the logs did not help at all identifying/isolating the bug. 

Note: I understand not all bugs are discoverable through logs. This does not change the fact that the logs are horrible. 

There are some obvious problems with our logging that we can already attempt to fix. I do not want to list those here and I cannot simply show you our log files so you can give advice on what to do. 

Instead, in order to assess how bad we are doing on the logging front, I would like to know: 


What are some guidelines, if any, when it comes to logging for an application, especially large application. 
Are there any patterns we should follow or anti-patterns we should be aware of? 
Is this an important thing to fix or can it even be fixed or all log files are simply huge and you need supplemental scripts to analyze them? 


Side note: we use log4j. 

A few points that my practice proved useful:


Keep all logging code in your production code. Have an ability to enable more/less detailed logging in production, preferably per subsystem and without restarting your program.
Make logs easy to parse by grep and by eye. Stick to several common fields at the beginning of each line. Identify time, severity, and subsystem in every line. Clearly formulate the message. Make every log message easy to map to its source code line.
If an error happens, try to collect and log as much information as possible. It may take long but it's OK because normal processing has failed anyway. Not having to wait when the same condition happens in production with a debugger attached is priceless.


Logs are mostly needed for monitoring and troubleshooting. Put yourself in a troubleshooter's shoes and think what kind of logs you'd like to have when something wrong is happening or has happened in the dead of night. 
