Sending Munin Alerts to Jabber/XMPP
What
Have munin send alerts to a Jabber server.
Why
Email is great for interpersonal communication, but it doesn't make sense as the medium for every communication that crosses your path. It is confusing to interleave emails from people (that need responses) with email alerts sent from servers (which never need an email response). While both of these messages are important to know about, they are different forms of communication, and should be treated a such.
An XMPP server is a good way to handle alerts because 1) it is sandboxed from other communication (i.e. email), 2) it integrates easily with human workflows (IM), 3) other people and programs can easily listen in.
How
Here's the gameplan:
- Setup Jabber server (PartyChatApp)
- Setup Jabber client (Gmail)
- Create ruby Jabber script
- Configure Munin
- Test Munin
1. Setup Jabber server (PartyChatApp)
The first step is to setup a Jabber server that you can send messages to. Party Chat is a simple and free way to create a Jabber enabled chat room. Party Chat has several cool and silly features. Most importantly, Party Chat will persist the messages, so even if you log out for the night, the full transcript will be available to you when you log back on. This is essential for keeping track of whats going on.
Anyway, so create a PartyChat room with a name you'll remember.
2. Setup Jabber client (Gmail)
Next, you need a Jabber account somewhere. Gmail is easy to setup and free. All you need is a Gmail username and password. Check.
3. Create ruby Jabber script
Up next, we'll create a ruby script that will take input from STDIN and send it to a Jabber server. Using the 'xmpp4r-simple' gem, this is indeed simple. I'm also going to prepend the timestamp to the message, and strip out newlines so it fits nicely on a single line.
#!/usr/bin/env ruby
require 'timeout'
require 'rubygems'
require 'xmpp4r-simple'
message = ""
# Timeout in case script is called without anything on STDIN
Timeout::timeout(5) do
STDIN.each_line do |line|
message << "#{line}\n"
end
end
im = Jabber::Simple.new("some_user@gmail.com", "password")
q = im.deliver("my_munin_alerts@partychapp.appspotchat.com", "#{Time.now}: #{message.gsub("\n","")}")
# wait a little for relay to complete
sleep(1)
4. Configure Munin
Configuring Munin is straight-forward. You can continue to send any email alerts as you are and simply add the Jabber notifications as well. Edit the Munin config file (/etc/munin/munin.conf) with the following:
contact.jabber_alert.command /etc/munin/jabber.rb contact.jabber_alert.always_send warning critical
Here are some additional resources about Munin alerts:
5. Test Munin
In order to test munin, you can make a dummy plugin that will always fail. Put this in with your other plugins at /etc/munin/plugins, or wherever else. Make sure it is executable, and then restart munin-node with '/etc/init.d/munin-node restart'.
#!/usr/bin/ruby if ARGV[0] == 'config' puts "value.label Some Label" puts "value.warning 100" puts "value.critical 150" else puts "value.value 200" end
Now, wait a few minutes, and you should have those Munin alerts popping up in the PartyChat window.
Bonus Round
See how you can use some of PartyChat's features to add some more value to your alert system. Maybe check the messages for 'Critical' or 'Warning' and use PartyChat's plusplusbot to keep track of how many warnings/criticals have been issued.
Or, pass Munin variables to the ruby script in order to make the message prettier.
