A new month nearly upon us, and the only thing I can think to blog is about the difficulty I had configuring the logging in piece of in-house software recently. The software was written in Java, and used the very configurable log4j library for logging. In an almost textbook use case of the library, I was tasked with converting the software from logging to stdout, to logging to a file which would be more permanent. There are, of course, several things to consider when logging to file, not least security, but another major concern was the size of the log file created. These are not new concerns however, and I decided to make use of the standard syslog facility used in conjunction with the logrotate program, both of which were available on the machine in question.
Of course, log4j supported outputting to syslog, which meant that with a simple configuration change based on any good example on the Web, I could get syslog logging instead of (or as well as) stdout logging. I could even exploit syslog’s network-transparency to log to another host, but I didn’t need that. In theory everything was now ready to go, but when I restarted the Java program, the problems started.
The obvious sign that things weren’t working was that the log file wasn’t created in the directory I had specified. Assuming this was a simple case of permissions, I created the file myself with touch and set it writeable by the group that the program ran as. This didn’t work, so on the advice of a colleague I went one step further and ran the program as root but still nothing appeared in the empty file I had touched. One obvious source of potential error was, of course, my editing of the file /etc/syslog.conf, but when I put in obviously wrong data, the Java program gave errors (generated by the logging component). I even tried not specifying a file to syslog to, on the understanding that it would default to logging to just /var/log/syslog, but this didn’t have the desired effect. Another colleague went so far as to run tail -f simultaneously on all log files in /var/log but the screen settled into silence.
At this point the famous Latin phrase “Quis custodiet ipsos custodes?” came to mind, a phrase which, by my saying in public, had prompted someone to say that I “know too much Latin” and which, in this case, I extended to mean “Who logs the logging software?”. Well, of course there are always logs, and /var/log/syslog should have contained any messages about using syslog incorrectly, but there was nothing there. The fact that my syslog was working at all, though, was confirmed by using the logger command which I found out about.
By this time I had been hitting the Google frantically, only to find out that my problem was not unique. At least one person had reported the problem, without remedy, to the relevant Apache mailing list, and another person had reported it, equally without remedy, to a Sun mailing list. That is not to say I had lost faith in Google, however, as, to my relief, I eventually found a page that, despite being 5 years old, had hidden at the bottom
“If you are trying to log to the Unix syslog under Linux using the SyslogAppender, then the Linux syslog daemon must be configured to accept log input from the network. Otherwise, you will get an IOException: connection refused.”
In fact I wasn’t getting “connection refused”, but gratuitously enabling network connections to the syslog daemon suddenly made everything work. Obviously this had to be followed up with firewall rules that disallow remote connections on UDP port 514 (or something stronger). If your setup does not currently enforce this, you may find this advice to be useful.
Time for a “bugs” category on my blog?
