Chef already logs a quite a bit of information, but it’s not that useful to me, since it is difficult to wade through, and is located on each individual node. You could log to syslog and collect it centrally, but that still doesn’t have the fine grain I’m looking for. Ideally, I could have an easy way to look at watch actions were actually executed on which resources for a particular node.
With Chef’s flexible reporting handlers and knife plugins, it’s a breeze. Kallistec pointed me towards Nom Nom Nom, a report-handler and redis-backed-sinatra server to collect information about a Chef-client runs. It looks pretty nice, but Redis and Sinatra seem extraneous when Chef is already running CouchDB and a web app.
The report handler gets any resources that were updated in that Chef-client run (i.e., Chef executed commands to converge these resource), and persists some information about those resources under the node[:log] attribute. By default it stores info about the last 10 updated resources.
Here’s an example client to use the new report handler. Check out the Opscode wiki page for more info.
Now that our nodes are persisting the data, we can use Chef 0.10.0′s handle Knife plguins to create a simple command line tool to query a node’s update log.
And, we’re logging node updates, for easy debugging. The command line output is nicely formatted with Highline.
I’m also playing around with using a similar report handler to Jabber the Ops chatroom to notify about executed actions, but I’d like a little more control over what resources report changes, and haven’t found a good way yet.