On the spectrum of cloud offerings, Rackspace is more of a dense low-hanging San Francisco
fog than a whispy sirrus-stratus cloud formation: the Rackspace Cloud is very approachable
to those coming from tradition server infrastructure, but it lacks many of the features
essential to a true cloud-based deployment of non-trivial system infrastructre.

One example of this is Rackspace Cloud's faculties for interacting with Configuration
Management tools like Chef, Puppet, and CFEngine. Declarative configuration management
is an essential part of large-scale deployments, wherein your systems rely on introspection
and interaction, attempting to achieve the desired configuration state (even if they
have to try a few times to overcome intermittant network outages or other issues).

One thing tools like Chef help with is to abstract away some of the differences between
different cloud providers, public or private. Rackspace cloud, for example, offers
both a public and private network interface whereas Amazon's EC2 offers only a private
interface along with a DNS name that routes to the private interface. Without a tool
to abstract these differences, it is painful to manage systems that span multiple cloud
providers ('cloud agnostic' or 'hybrid cloud' setups).

In order to acheive these abstractions, Chef probes the server and network to determine
which cloud provider is hosting the system. EC2 'plays nice' with Chef, making it easy
to indentify an EC2 server accuratly, tested over hundreds of thousands of server launches.
Rackspace Cloud, does not make things quite so easy, and tools like Chef have to 'guess' and
do not always have enough information to guess correctly, which cracks the cloud-provider
abstractions and can prevent Chef from working at all.

It is 100% in Rackspace's interest to work with the Chef maintainers to define a better
way to deterministically identify Rackspace Cloud servers. For one thing, anybody using
Chef is likely launching many servers, and more servers means more money. And while
Rackspace is probably pulling in some significant dough with their cloud services, they
have some negative brand associations created by their high-touch, support-driven
hierarchies. In my mind, following up directly on THIS TICKET or elsewhere would speak volumes,
indicating that they are actually interesting in providing services for next-generations
application and platform deployments, not just bringing buzzword-compliant virtualization
to clients who can't afford dedicated hardware.

Rackspace is a major player in the public cloud arena, and thousands of customers run
thousands of servers on their platform. However, if Rackspace doesn't address the client-base
that is trying to push the limits of cloud deployments, they will lose customers and thought-leadership
in the long run.

Graylog2, as of version 0.9.6, uses the awesome Elastic Search to index messages, along with MongoDB for stream definitions, config and counters. The messages are the bulk of of the data, and indexing gobs of data takes disk space processing power. Elastic Search's TTL functionality makes it easy to expire messages after a predefined period. Here's a quick rundown of how to set a default TTL for an Elastic Search 'mapping':

I know that it's best to monitor what the user interacts with, but in an attempt proactively precent downtime I also want to check on some lower-level things that may precede user-facing outages. Take a standard HTTP service, with multiple processes running on multiple machines. Do you monitor the service through the load balancer (i.e. what the user sees), per machine, per port? My answer is 'yes!', which works pretty well except for one thing.

My sanity and sleep, and that of my team.

This means we're generally alerted long before the user sees any issues, but it introduces noise into the communication channels and sleepytime pager rotation.

As much as I find false dichotomies distasteful ("You're either with us, or against us"), I had settled on one for monitoring: "You can either have false positive alerts, or false negative alerts, and to keep the customers happy we need to side with false positives at the cost of some pain on our side". Good in theory, until you are generally disgruntled, bleary eyed, tired, unproductive and your heart stutters at the sound of your cellphone (even when it's probably just your mom calling).

The reason false dichotomies are so insidious is that they slowly reduce your perspective to a single dimension, and it's hard to break back out to the multi-dimensional perspective required to innovate. I got a chance to meet with an experienced fellow the other day, and I presented my monitoring false dichotomy. He's response was, yeah, sometimes you're alerts will be going off like a carnival bell, but it doesn't have to stay that way. Imagine a knob, The Knob of Pain, that you can dial up or down. Team pissed off and burnt out on alerts? dial it back (a bit). Team hungry, ready for some risks? Dial it up a bit.

Building software and infrastructure, we make tradeoffs daily. Speed or throughput? Availability or consistency? Data granularity or disk space? The first issue is to understand the tradeoffs. The next issue is to take the tradeoffs under your control, and decrease the effort and time to adjust them, as your costs, customers, resources, or appetite for risk dictate.

I found a nice set of plugins on github from ehamiter to get started with. It's pretty easy to get the swing of things, just be prepared for a few infinite loops and force closing your editor!

Here's the whitespace plugin I came up with to handle three issues vital to my day-to-day development: 1) removing trailing whitespace, 2) expanding tabs, and 3) adding a single newline at the end of the file.

There is an out-of-the-box Sublime Text 2 command for expanding tabs, which I wanted to run on every save, so the ExpandTabs class is a simple pre-save handler that calls the command. All three pre-save handlers check for a configuration settings (set via JSON preferences file), so if you don't use one it will still be light-weight.

This works for me, but a slightly cleaner model would be to create TextCommand actions, in addition to EventListeners that call the command, so you get the best of both worlds (call commands with key bindings, or enable event listeners).

To install, I simply clone to the repo into my packages directory, i.e. git clone git@github.com:nstielau/Sublime-Text-2-Plugins.git /Users/nstielau/Library/Application Support/Sublime\ Text\ 2/Packages/nstielau, and then can easily sync the plugins across workstations via github.

Two features high on my list were trending alerts ("It seems like these damn alerts are going off CONSTANTLY!") and adding Jabber notifications for alerts I didn't care about quite enough to land in my inbox.

Here's a small Sinatra app that makes these possible:

Run the Sinatra app somewhere (bug me and I'll release a chef cookbook for it), and add the URL as a new webhook 'Address' in Cloudkick, and then select it for all applicable 'Monitors'.

To test the webhook, save the example payload to a file:

And use CURL to test:

Here's what the graphite graph starts to look like after a few days (using Graphite's drawNonZeroAsInfinite() data function).

Enjoy!

"Got tight last night on absinthe and did knife tricks." -Hemingway

Taking some inspiration from good ol' Ernest, I got tight on Tecate last night and did knife tricks. Check 'em out.

Grep remote logs across multiple machines

Tail and selectively grep logs across multiple nodes, streaming to your console:

# Look for 502's in Nginx
knife ssh 'role:load_balancer' 'tail -f /var/log/nginx/*.log | grep 502' 

# Tail the web app logs across all servers.
knife ssh 'role:web_app' 'tail -f /opt/web_app/logs/production.log' 

#

Sort nodes across a role

Get a glance at the state of servers with a particular role:

# Sort by 5min load average
knife ssh 'role:transcoder' 'cat /proc/loadavg | awk "{print $2}"' | sort -n -k 2

# Sort by uptime
knife ssh 'role:web_app' 'uptime' | sort -n -k 4

#

The other day I had the pleasure of asking Adam Jacob and Chris Brown a few questions about Chef. Chef as a tool is great, but technical changes are easier than cultural changes, since a single person can implement technical change while the entire team has to buy in to cultural change. Handing off Chef to the dev team, I had a question for Adam and Chris: "Should everybody on the team learn knife?". Their answer was "Yes." Nobody wants to learn a new tool just for the sake of it, but knife has some pretty awesome capabilities, and learning a little might go a long way.

What knife tricks do you have?

Chef Cookbooks are a great way to define server resources. You can abstract configuration across Linux distributions, creates a standardized format for defining resources, and has plenty of flexibility for configuration. Chef is pretty new on the scene, and a lot of best-practices are still being defined and refined. Here are a reason I end up tweaking all but the most trivial cookbooks.

1) Appropriate abstraction
The sweet-spot of abstraction for Chef cookbooks is a best-practice that is still unknown. The official Opscode cookbooks are meant to work for everyone, across distros, which means they are full of case statements for Fedora, RHEL, Debian, Ubuntu, etc. This cross-distro abstraction serves a purpose, but in my world it only adds clutter.

Take the Opscode Java cookbook for example, which had known bugs and oddities relating to installing OpenJDK vs. the Sun JDK. For most circumstances, you'll only run one JDK across your entire system, and dealing with bugs to configure the Sun JDK over the default is a waste of time. Take joy in excising unnecessary recipe abstraction.

Many third-party cookbooks aren't abstract enough, intermingling configuration of dependencies that you have already pulled into a separate, configurable cookbook.

2) They run as root
Additionally, you should really take a good look at anything that will run as root on your system.

3) My inner sysadmin is getting particular
Configure enough systems, and you start to get particular. I, for example, dislike creating non-system users (users with home directories) for applications. /home is for humans, if you ask me. Anyway, right or wrong, I browse through new cookbooks, editing away things that would work, but don't suit to my style.

Bare-bones Jenkins

I wanted to install Jenkins, but the only cookbook I could find violated all three of these principles (extra clutter, runs-as-root, and did stuff I didn't like). Granted the other cookbook has more features, but this is a good place to get started.

Deploy scripts are just one place where a concise and safe way to record a metric/event in important. From Etsy's blog, using vertical lines to represent distinct events (code deployments) to give more context to the login trends:

Sending a metric from the command line, with netcat or curl, is just one bit of 'glue' that is essential for pulling together a complete metrics solution. This is essentially the base-case, the lowest-common-denominator that all metrics collection can be built upon.

While short-and-sweet, this snippit contains two measures of protection. First, it limits the run duration to a single second so it don't have to worry about it hanging, and secondly, it uses UDP, so you don't need to worry about the remote server being performant or even up. True fire-and-forget, so you can recklessly drop this in scripts left and right.

And the bash one-liner:

What metrics can you collect with a one-liner? Add some in the comments.

With Chef's flexible reporting handlers and knife plugins, it's a breeze. Kallistec pointed me towards Nom Nom Nom, a report-handler and redis-backed-sinatra server to collect information about a Chef-client runs. It looks pretty nice, but Redis and Sinatra seem extraneous when Chef is already running CouchDB and a web app.

Report Handler

The report handler gets any resources that were updated in that Chef-client run (i.e., Chef executed commands to converge these resource), and persists some information about those resources under the node[:log] attribute. By default it stores info about the last 10 updated resources.

Example client.rb

Here's an example client to use the new report handler. Check out the Opscode wiki page for more info.

Knife Plugin

Now that our nodes are persisting the data, we can use Chef 0.10.0's handle Knife plguins to create a simple command line tool to query a node's update log.

Ta da!

And, we're logging node updates, for easy debugging. The command line output is nicely formatted with Highline.

I'm also playing around with using a similar report handler to Jabber the Ops chatroom to notify about executed actions, but I'd like a little more control over what resources report changes, and haven't found a good way yet.

Enjoy!

Graylog2 consists of a tastefully designed Rails web-interface and a Java-based TCP/UDP log collector, using MongoDB to store the log messages and application data. Rails is a clear choice for a convenient, easy-to-extend web interface, and Java is efficient for high-traffic message collection and forwarding. The real magic (aside from the core team) is MongoDB, which elegantly ties everything together .

The polyglot approach, another DevOps tactic, utilizes the best tool for the job-at-hand. Graylog2 chooses Ruby and Rails for rapid web UI development, and Java where speed and memory profile are important. With that flexibility in mind, Graylog2 accepts messages over TCP or UDP, whichever protocol suits your needs. MongoDB makes ployglot easy, with a collection of a dozen supported drivers for languages from C to Scala. For Graylog2, this mean Rails and Java can talk to the same data store using updated, documented libraries. Good clean drivers help downright speedy development.

As of version 0.9.5 Graylog2 has ditched MySQL entirely in favor MongoDB, streamlining the installation process and slimming the code base. Reducing the friction during installation is key to driving adoption for any product, and Graylog2's got a few good options for a speedy install. MongoDB is a snap to install, too, and has a user-friendly shell, which makes it familiar for users coming from a SQL background (no required map-reduce or erlang stacktraces, Bob). If you really don't want to install MongoDB, you don't even have to thanks to multiple hosting providers.

The Graylog Extended Log Format (GELF) improves on the standard syslog entry (a 1024 byte string) by adding the ability to specify arbitrary metadata with the log message. This data could be the application environment, server details, debugging data, or anything else. Mongo's document storage handles storing and indexing this data without any fuss (or awkward code forcing a schema upon arbitrary data). No rigid schema means fewer development roadblocks and simpler upgrades, while structured data allows for advanced querying (and indexing). It's the best of both worlds. Check out how Graylog2's 'Quick Filter' exposes these queries:

The collection keeps a fixed number of log entries around, a text-book use-case of MongoDB's capped collection. This means you don't have to worry about cleaning up old entries or overflowing disks. Sure, it isn't that hard write a script to cleanup old entries, but the fewer moving parts, the fewer things to remember, the better (not to mention that capped-collection writes are blazing fast). Another DevOpsy tenant: "Don't use the braintrust". That is, if you are relying on individuals remembering issues and edge-cases, you're headed for trouble. With Graylog2, I have to remember a whole lot less (ssh passwords, log file location, log rotation) to know the state of my apps and servers. Graylog2's use of capped-collections gives me even less to remember (soon, there might even be room left for remembering my childhood).

A true log-management solution, Graylog2 organizes messages into user-defined 'streams', effectively blacklists noisy messages, forwards messages to external resources, and generally makes your logs actionable. Don't send your logs to files so they can die, send them to Graylog2 so they can come to life.

Check out the Graylog2 website or dig into the code. Hope on IRC (#graylog2 on freenode) or the mailing list (http://groups.google.com/group/graylog2?hl=en) if you need a hand getting Graylog2 configured.

YETI PARTY!