I found a nice set of plugins on github from ehamiter to get started with. It's pretty easy to get the swing of things, just be prepared for a few infinite loops and force closing your editor!

Here's the whitespace plugin I came up with to handle three issues vital to my day-to-day development: 1) removing trailing whitespace, 2) expanding tabs, and 3) adding a single newline at the end of the file.

There is an out-of-the-box Sublime Text 2 command for expanding tabs, which I wanted to run on every save, so the ExpandTabs class is a simple pre-save handler that calls the command. All three pre-save handlers check for a configuration settings (set via JSON preferences file), so if you don't use one it will still be light-weight.

This works for me, but a slightly cleaner model would be to create TextCommand actions, in addition to EventListeners that call the command, so you get the best of both worlds (call commands with key bindings, or enable event listeners).

To install, I simply clone to the repo into my packages directory, i.e. git clone git@github.com:nstielau/Sublime-Text-2-Plugins.git /Users/nstielau/Library/Application Support/Sublime\ Text\ 2/Packages/nstielau, and then can easily sync the plugins across workstations via github.

Two features high on my list were trending alerts ("It seems like these damn alerts are going off CONSTANTLY!") and adding Jabber notifications for alerts I didn't care about quite enough to land in my inbox.

Here's a small Sinatra app that makes these possible:

Run the Sinatra app somewhere (bug me and I'll release a chef cookbook for it), and add the URL as a new webhook 'Address' in Cloudkick, and then select it for all applicable 'Monitors'.

To test the webhook, save the example payload to a file:

And use CURL to test:

Here's what the graphite graph starts to look like after a few days (using Graphite's drawNonZeroAsInfinite() data function).

Enjoy!

"Got tight last night on absinthe and did knife tricks." -Hemingway

Taking some inspiration from good ol' Ernest, I got tight on Tecate last night and did knife tricks. Check 'em out.

Grep remote logs across multiple machines

Tail and selectively grep logs across multiple nodes, streaming to your console:

# Look for 502's in Nginx
knife ssh 'role:load_balancer' 'tail -f /var/log/nginx/*.log | grep 502' 

# Tail the web app logs across all servers.
knife ssh 'role:web_app' 'tail -f /opt/web_app/logs/production.log' 

#

Sort nodes across a role

Get a glance at the state of servers with a particular role:

# Sort by 5min load average
knife ssh 'role:transcoder' 'cat /proc/loadavg | awk "{print $2}"' | sort -n -k 2

# Sort by uptime
knife ssh 'role:web_app' 'uptime' | sort -n -k 4

#

The other day I had the pleasure of asking Adam Jacob and Chris Brown a few questions about Chef. Chef as a tool is great, but technical changes are easier than cultural changes, since a single person can implement technical change while the entire team has to buy in to cultural change. Handing off Chef to the dev team, I had a question for Adam and Chris: "Should everybody on the team learn knife?". Their answer was "Yes." Nobody wants to learn a new tool just for the sake of it, but knife has some pretty awesome capabilities, and learning a little might go a long way.

What knife tricks do you have?

Chef Cookbooks are a great way to define server resources. You can abstract configuration across Linux distributions, creates a standardized format for defining resources, and has plenty of flexibility for configuration. Chef is pretty new on the scene, and a lot of best-practices are still being defined and refined. Here are a reason I end up tweaking all but the most trivial cookbooks.

1) Appropriate abstraction
The sweet-spot of abstraction for Chef cookbooks is a best-practice that is still unknown. The official Opscode cookbooks are meant to work for everyone, across distros, which means they are full of case statements for Fedora, RHEL, Debian, Ubuntu, etc. This cross-distro abstraction serves a purpose, but in my world it only adds clutter.

Take the Opscode Java cookbook for example, which had known bugs and oddities relating to installing OpenJDK vs. the Sun JDK. For most circumstances, you'll only run one JDK across your entire system, and dealing with bugs to configure the Sun JDK over the default is a waste of time. Take joy in excising unnecessary recipe abstraction.

Many third-party cookbooks aren't abstract enough, intermingling configuration of dependencies that you have already pulled into a separate, configurable cookbook.

2) They run as root
Additionally, you should really take a good look at anything that will run as root on your system.

3) My inner sysadmin is getting particular
Configure enough systems, and you start to get particular. I, for example, dislike creating non-system users (users with home directories) for applications. /home is for humans, if you ask me. Anyway, right or wrong, I browse through new cookbooks, editing away things that would work, but don't suit to my style.

Bare-bones Jenkins

I wanted to install Jenkins, but the only cookbook I could find violated all three of these principles (extra clutter, runs-as-root, and did stuff I didn't like). Granted the other cookbook has more features, but this is a good place to get started.

Deploy scripts are just one place where a concise and safe way to record a metric/event in important. From Etsy's blog, using vertical lines to represent distinct events (code deployments) to give more context to the login trends:

Sending a metric from the command line, with netcat or curl, is just one bit of 'glue' that is essential for pulling together a complete metrics solution. This is essentially the base-case, the lowest-common-denominator that all metrics collection can be built upon.

While short-and-sweet, this snippit contains two measures of protection. First, it limits the run duration to a single second so it don't have to worry about it hanging, and secondly, it uses UDP, so you don't need to worry about the remote server being performant or even up. True fire-and-forget, so you can recklessly drop this in scripts left and right.

And the bash one-liner:

What metrics can you collect with a one-liner? Add some in the comments.

With Chef's flexible reporting handlers and knife plugins, it's a breeze. Kallistec pointed me towards Nom Nom Nom, a report-handler and redis-backed-sinatra server to collect information about a Chef-client runs. It looks pretty nice, but Redis and Sinatra seem extraneous when Chef is already running CouchDB and a web app.

Report Handler

The report handler gets any resources that were updated in that Chef-client run (i.e., Chef executed commands to converge these resource), and persists some information about those resources under the node[:log] attribute. By default it stores info about the last 10 updated resources.

Example client.rb

Here's an example client to use the new report handler. Check out the Opscode wiki page for more info.

Knife Plugin

Now that our nodes are persisting the data, we can use Chef 0.10.0's handle Knife plguins to create a simple command line tool to query a node's update log.

Ta da!

And, we're logging node updates, for easy debugging. The command line output is nicely formatted with Highline.

I'm also playing around with using a similar report handler to Jabber the Ops chatroom to notify about executed actions, but I'd like a little more control over what resources report changes, and haven't found a good way yet.

Enjoy!

Graylog2 is centralized log collector that can collect Syslog and structured log entries. It's a little pink, and I have no idea why the Yeti is wearing a birthday hat (other than the obvious), but I think it will be a critical tool for getting a handle on errant Java webapp and Flash player log entries.

Graylog2 and Chef

It's got a nice stack: Java collector, MongoDB storage and a Rails UI. With the help of a pretty complete Chef recipe by JBZ, I got Graylog2 up-and-running in a matter of minutes. The Chef recipe had one or two quirks, but nothing I haven't come to expect with Chef, and the end-product is definitely worth a few minutes of debugging. Another quick chef recipe, and 50 servers are sending their syslogs into Graylog2.

Graylog2 with Rsyslog

I've had my eye on Graylog2, but the whole syslog thing scared me away a little. Syslog definitely falls under the "they don't teach you that in school" banner, and I had no idea how powerful it is. I was a little sad I couldn't find details on how to run Graylog2 with syslog, but once I googled for the right thing, I found this simple video by the creator of rsyslog, the Syslog implementation in use on most of my servers. All you need is to add something like the following into a new file in /etc/rsyslog.d and you'll be forwarding syslog entries via UDP to Graylog2:

# Forward all logs to Graylog2
# Replace 0.0.0.0 with your own Graylog2 server IP
*.*     @0.0.0.0 #udp forwarding

and with Syslog-ng

I've also got some servers that use syslog-ng, which isn't a one-liner but is pretty close:

# Forward all logs to Graylog2
# Replace 0.0.0.0 with your own Graylog2 server IP
destination graylog2 { udp(0.0.0.0); };
log {
        source(s_all);
        filter(f_at_least_info);
        destination(graylog2);
};

It's that easy.

UDP vs TCP

I'm sticking with UDP forwarding for now. It's easy to send syslogs over TCP, and I think Graylog should accept them fine. More to play with! Make sure you understand the tradeoffs between the different protocols, and see which one works for your needs.

Can't wait to see more improvements to Graylog2, and see how it will sooth some of our distributed logging woes.

Gabemc just released a revamped version of jabber-tee, jabberified version of your standard linux tee. And it rocks.

Jabber-tee in short provides a dead-simple way to send Jabber alerts via the command line. I'm a huge fan of PartyChat, and after reading Dziuba's monitoring theory post, I feel even stronger that Jabber has a place in an alerting system; it's the right mix of transience and effectiveness for certain notices.

Jabber-tee with Reptile for MySQL replication alerts

A gem a wrote (in working but messy state), Reptile, for MySQL replication monitoring, can spit out information about replication status, or errors if the replication delays are above defined thresholds. When everything is working, the gem's executable outputs something like:

nstielau@s34:~$ replication_status --status --heartbeat
INFO: Checking slave status.
INFO: 'master1_slave1' is 'running'
INFO: Checking heartbeats.
INFO: The slave delay for 'master1_slave1' is 1 seconds

That's all fine, but things start to get interesting when the delays get large, and we need to send some alerts. This is a perfect example of an 'informational but not actionable' event; replication delay often fixes itself and there is nothing you can do to speed it up. Thus, it doesn't make sense to email or page about it, but it is nice to know about. Aha! Jabber.

Run 'replication_status' with the '-l error' flag (only output replication errors) and it makes a great crontab-able check. Define a MAILTO var in your crontab, and you've got email alerts about replication status. But we don't want emails, we want Jabbers. Aha! Jabber tee. Write your config file (/etc/jabber-tee.yml) and update your crontab:

replication_status --status --heartbeat | jabber-tee

and presto! Jabber'ed replication alerts.

Conclusion

Jabber-tee is a great, lightweight utility that can help you achieve the Zen of knowing what's going on, without being woken up at night or fattening up your inbox. What can you do with it?

Other ideas:

• Jabber alerts from munin
• Jabber alerts from certain init scripts ("Service X was just restarted on box11.myservers.com")
]]> http://dev.nuclearrooster.com/2011/04/05/jabber-tee-hotnes/feed/ 0 http://dev.nuclearrooster.com/2011/03/20/real-world-applications-of-monitoring-theory/ http://dev.nuclearrooster.com/2011/03/20/real-world-applications-of-monitoring-theory/#comments Sun, 20 Mar 2011 19:40:33 +0000 nick.stielau http://dev.nuclearrooster.com/?p=574 Monitor Classifications

I recently read an interesting post on monitoring theory by Ted Dziuba. The post colorfully classifies different monitors by their required reaction.

Dziuba's monitor classifications:

• Neither Actionable nor Informative
• Informative, not Actionable
• Actionable and Informative

I'd add a tentative fourth, to the complete the permutations: Actionable, not Informative. This is the worst of all, the equivalent of your boss screaming "It's DOWN! It's DOWN! Stuff if BROKEN!"

Why monitoring theory is new in 2011

This perspective on monitoring theory comes out of read-world experience dealing with software systems. The first instinct when systems start failing is to add monitors to every circumstantial piece of evidence. Load average spiked? Monitor and alert. Replication delay? Monitor and alert. This naive monitoring and alerting will keep your system up, but creates an unsustainable environment for developers and system administrators. With so many alerts flying around, pretty soon your team is either ignoring them or severely sleep-deprived and grumpy.

Dziuba's theory then comes into play here, identifying reactionary measures, classifying monitors, and alerting/persisting with the correct tools such that your system gets the correct amount of attention at the correct time. Monitoring is not new, and disgruntled sysadmins are not new, but there is something about the devops movement that is changing monitoring.

As development and operations align, it is easier to picture sysadmins as bonafide users, and users have the right of an optimal user-experience.

What you need to do, when

Here's a little more detail about each level of monitor: the use-case for investigating, and likely frequency. Like user-features, monitoring should be approached with use-cases. Developers and Sysadmins are users, and their use-cases and experiences should be defined before solutions are implemented.

Neither Actionable nor Informative
Frequency: monthly, quarterly
Purpose: Look at for capacity planning, roadmapping

Informative, not Actionable
Frequency: Daily, weekly
Purpose: Take preventative actions

(Actionable, not Informative)
Frequency: NOW!
Purpose: Figure out what is wrong, and then fix it (suck)

Actionable and Informative
Frequency: NOW!
Purpose: Execute the predefined steps to correct the error

Communicating and persisting alerts

There are dozens of tools to help with monitoring, and almost as many different forms of communication. Here are few methods for recording monitor alerts. They key distinction is how much (or how strongly) a specific medium will demand attention. These are listed in order from least attention-demanding to most.

• Log to file
• Graph (munin, collectd, graphite)
• Record (dashboard, for later viewing)
• Jabber/IM
• Twitter feed
• Digest Email
• Email
• Page/SMS
• Acknowledged Page/SMS

High-level monitoring objectives

As Dziuba notes, a lot of this is subjective. In one system, maybe a queue with 10 items is a problem. In another, maybe 1000 items is fine. The high-level monitoring use-cases give a good picture of the high-level monitoring objectives:

• What do you want to keep running?
• How do you deterministically monitor the health of those systems?
• What secondary and tertiary monitors might give some insight into the system, and how can those be effectively monitored without going crazy?

Monitor specific questions

Here are a few questions that might narrow-done the correct alerting tool for a particular monitor.

• If X happens, is user experience necessarily broken?
• If X happens, is user experience necessarily degraded?
• If X happens, doing Y will fix the problem
• If you go back to sleep, will X likely return to normal state?

If user experience is significantly degraded, and there are concrete actions that will correct the issue, go ahead and demand a reaction.

Test Cases

Here are a few concrete scenarios we can put to test:

• Replication outside delay
  Probably will fix itself, this time, but you should know in general what your delay is. Let's graph this in general, maybe jabber if it is 15 minutes behind, and email if it is 1 hour behind.
• Chef run failed
  Thanks to convergence we can be pretty sure this will sort itself out. In specific circumstances, this could be an issue, so maybe Jabber the error so it's on the radar.
• NTP Time Update failed
  In all but the most finicky systems this should be fine, it'll correct itself soon. No alerting.
• Backend processor CPU/Load Average
  This is what processors do. Graph, no alerts.
• Digest email of processing errors for the past day
  These may be actionable or they may not. The digest is a pretty good way to reduce inbox traffic, but monitors are either they are actionable or they are not. If they are actionable, demand more attention. If they are merely informational, a web dashboard is probably better.
]]> http://dev.nuclearrooster.com/2011/03/20/real-world-applications-of-monitoring-theory/feed/ 0 http://dev.nuclearrooster.com/2011/02/27/escape-json-is-rails-3-views/ http://dev.nuclearrooster.com/2011/02/27/escape-json-is-rails-3-views/#comments Sun, 27 Feb 2011 17:13:27 +0000 nick.stielau http://dev.nuclearrooster.com/?p=568 It's convenient to build up options in your server-side language (Ruby!) even if you are going to output them at JavaScript options for some JavaScript library. This way, you can merge in different options, use server-side logic to specify certain options, etc.

The trick is that Rails 3 automatically escapes HTML, so a simple options.to_json doesn't quite cut it. A this StackOverflow post concludes, you can use the html_safe string method to effectively unescape the json before the template escapes it, giving you just what you are looking for.

<%
options = {
:name => 'foo',
:color => 'bar'
}
%>
<script type="text/javascript">
console.log("My Options are:" + <%= options.to_json.html_safe -%>);
</script>

]]> http://dev.nuclearrooster.com/2011/02/27/escape-json-is-rails-3-views/feed/ 2