Nuclear Rooster

Forwarding rsyslog to Graylog2

Graylog2

Graylog2 is centralized log collector that can collect Syslog and structured log entries. It's a little pink, and I have no idea why the Yeti is wearing a birthday hat (other than the obvious), but I think it will be a critical tool for getting a handle on errant Java webapp and Flash player log entries.

Graylog2 and Chef

It's got a nice stack: Java collector, MongoDB storage and a Rails UI. With the help of a pretty complete Chef recipe by JBZ, I got Graylog2 up-and-running in a matter of minutes. The Chef recipe had one or two quirks, but nothing I haven't come to expect with Chef, and the end-product is definitely worth a few minutes of debugging. Another quick chef recipe, and 50 servers are sending their syslogs into Graylog2.

Graylog2 with Rsyslog

I've had my eye on Graylog2, but the whole syslog thing scared me away a little. Syslog definitely falls under the "they don't teach you that in school" banner, and I had no idea how powerful it is. I was a little sad I couldn't find details on how to run Graylog2 with syslog, but once I googled for the right thing, I found this simple video by the creator of rsyslog, the Syslog implementation in use on most of my servers. All you need is to add something like the following into a new file in /etc/rsyslog.d and you'll be forwarding syslog entries via UDP to Graylog2:

# Forward all logs to Graylog2
# Replace 0.0.0.0 with your own Graylog2 server IP
*.*     @0.0.0.0 #udp forwarding

and with Syslog-ng

I've also got some servers that use syslog-ng, which isn't a one-liner but is pretty close:

# Forward all logs to Graylog2
# Replace 0.0.0.0 with your own Graylog2 server IP
destination graylog2 { udp(0.0.0.0); };
log {
        source(s_all);
        filter(f_at_least_info);
        destination(graylog2);
};

It's that easy.

UDP vs TCP

I'm sticking with UDP forwarding for now. It's easy to send syslogs over TCP, and I think Graylog should accept them fine. More to play with! Make sure you understand the tradeoffs between the different protocols, and see which one works for your needs.

Can't wait to see more improvements to Graylog2, and see how it will sooth some of our distributed logging woes.