Category Archives: sysadmin

capistrano considered annoying

So, for the last 6+ months, I’ve had one major elephant on my plate. The eating has been slow, and honestly not very delicious. But something happened yesterday. I realized I had finished.

The proverbial lights came on and I understood Capistrano. I didn’t really like it. But it finally and suddenly made sense after having been a constant irritation for so long.

Capistrano is a deployment system for rails applications. On the tin, it says it can “easily be used to deploy any language” but they’re just kidding themselves. It’s a tool written in ruby meant to deal with the hassles of ruby website deployment. To a non-ruby developer (or more accurately, a non-rails developer), it is so much incomprehensible smoke and mirrors.

I’ve been a programmer for 25 years. I’ve been using Linux for about 20 years, professionally for about 15. I’ve been tinkering with ruby off and on for 5. I’ve recently started teaching my daughter ruby as her first programming language. I should really have been able to figure out this stupid tool much more quickly than this.

I think the bulk of my mental block stems from the way in which I was introduced to Capistrano. Last year, I started at a new job where the head of IT had been using cap to deploy all of our rails apps in ways I did not agree with 100%. There wasn’t anything wrong with what he was doing and it makes plenty of sense. It’s just not the way I’d ever done it myself.

He told me about how cap worked, and explained that we were transitioning between cap 2 and cap 3 – which in the grand tradition of all things ruby is not a backwards compatible change. But before he left the company a few months later, he never explained how to actually bootstrap capistrano on a new system.

One of my first assignments was to make a production environment for an application that one of our rails devs was writing. Staging already worked fine, so I just had to reproduce things with more resources, etc… or so I thought.

There was too much noise.

If I had been a rails developer for a living, I would have been better prepared to sort through the noise. But alas, I was not. I’ve used plenty of ruby over the years for small projects here and there – and for one very large project quite a while ago. But never had a need to use rails itself for anything.

The problem with any web framework is that it is, by necessity, a whole bunch of code sitting in your project written by people who aren’t on your current team. It’s like inheriting a hopefully well documented legacy project along with some outdated contact information from one of the original authors.

Over the years, I have deployed a lot of PHP and Java web pages/applications. I’ve done it by hand and with the assistance of deployment tools. It all makes sense to me.

With PHP, you deploy the source code and assets to a directory accessible by your php interpreter and tell the web server how to reach it. PHP and the web server have config files of their own. The site typically also has its own config files which tend to be clearly labeled as such.

With Java, you deploy a single file that contains all of your compressed assets where your servlet engine of choice (let’s say Tomcat) can see it. The application’s configuration is usually baked into the bundle that you deploy. Tomcat has its own config files separate from the application’s source tree. You run Tomcat, and it serves the website. You then put a real web server in front, similar to how you would with a PHP site.

Ruby… kind of straddles the line between these two, and it hands an additional wrench to the monkey. Rails applications are deployed as source code like PHP. They run a dedicated application server like Java. However, unlike with Java, that application server IS the rails app. You invoke it directly, and it runs itself. This means that app server and application configuration files are kind of mixed together – and it’s hard to tell where one aspect of things ends and another begins.

Ruby also typically makes the general assumption that the application itself will install its own dependencies. Whereas with PHP and Java, you must often install various libraries separately to support their execution – ruby apps ship with dependency lists that you can easily invoke to install everything automatically. So there’s that extra deployment information bundled in with everything else.

So, to recap. To deploy a rails app, you install ruby on a box. You then install one extra utility for ruby that doesn’t ship with it. Then you upload your source code to the server and run that tool. This will go out to the internet and fetch and install all of your dependencies (ala pecl or cpan). Then you can launch the application itself. Then you tell the web server where to find the ruby app just like with anything else.

This is all sane enough, it’s just a different ordering from the other languages. Capistrano gives our monkey a third wrench to hold (in its tail, I guess?). Instead of deploying the application to a self-contained directory, cap starts making symlinks around. It versions subsequent deploys of your app (ostensibly to make rollbacks easier).

There are a lot of symptoms of version paranoia here (yay for abandoning backwards compatibility). When cap runs its bundler install, those libraries aren’t installed system wide, they go into an unprivileged application specific location. This would allow you to run different versions of the same library across multiple applications but makes for a major headache trying to find things if you don’t launch the app just so.

Hang on, one of my programmers is having issues with a Capistrano deploy not making one of the symlinks it was supposed to…

30 minutes later…


The problem was because I originally set things up while referencing information from conflicting versions of various tools. Two config file entries later, he’s back and working and I have more grey hairs because of Capistrano.

Seriously. That just happened while I was trying to rant about versioning and documentation. Things like this would not fly anywhere else.

The only real problem with Capistrano is that it is a Rails tool written by Rails developers for Rails developers. They make a lot of cultural assumptions and subsequent shortcuts. It is not very accessible to those of different cultural backgrounds.

Because cap v2 and v3 are different and the internet is forever and because people are lazy, it’s easy to read incorrect documentation that doesn’t remember to identify which versions of tools it applies to.

Because ruby applications ship with so many different moving pieces all jumbled together in one big fat directory, it is impossible for the uninitiated to identify where config files for one of the myriad components end and where the configs for another begin.

Because I inherited an existing project, the SNR was practically zero.

And don’t even get me started on how Capistrano demands passwordless sudo…

I’m sick and fed up with it all, and it’s one of the best tools that I’ve ever had the misfortune of using.

Now I get to go back and add God to the mix before collecting my thoughts and attempting to write a useful tutorial for general Linux admins to get up and running with Capistrano.


For -years- I have hoped and searched and wished and moaned for lack of a halfway decent terminal emulator on Windows. Specifically, one that:

  1. Lets me use my standard unix toolset.
  2. Gives me a command line interface to the host machine WITHOUT requiring me to do something ridiculous like ssh’ing to localhost or firing up an x11 server…
  3. Realizes that sometimes the display is wider than 80 characters…
  4. Provides customizable color codes (#006 on black really stinks).
  5. Doesn’t set TERM=something-nobody-supports.

One wouldn’t think this was too much to ask. But none of the major open source projects of which I am aware provide this. I even tried my hand at writing one myself but got distracted before it was any good.

So, for years, I have used Cygwin xterms and rxvt as a mildly tolerable alternative to, well, nothing.

Today, a coworker and I discovered a 3-year-old blog post promoting Console, a GPL licensed CMD.exe replacement that matches all of my base criteria plus my big dream feature of tabs. TABS!

Console2, Where have you been all my life?!

The project is ancient – but I was using linux desktops for work back in its early days so that probably accounts for my missing it back then.

In the grand tradition of old Sourceforge projects, there is no installer. You just decompress it somewhere and run the exe directly.

When I launched it the first time, I was unsurprised by the 80×25 courier 10 cmd.exe shell it launched by default. I opened the settings menu and was very very pleased with what I found on the first screen. A few minutes later, I had it pointing at my cygwin install:

And a few minutes later:

Continue reading console2

virtualbox rules

Nothing much to say here, but with absolutely minimal pain and suffering, I have 64-bit linux virtual machines running on top of my 32-bit windows XP install. This pleases me.

The recipe:

  1. Compatible CPU with VT-x/AMD-V enabled in the BIOS
  2. Innotek/Oracle/Sun VirtualBox (a current version) with hardware virtualization enabled
  3. Profit!

The one downside to this? 64-bit VM’s running on 32-bit host OS can’t see multiple cpu’s. Boo. Hoo. I’ll just run more VM’s!

64-bit centos installer 64-bit ubuntu livecd

dead simple traditional style rotation

In response to my two-step rotation post earlier this week, I figure I may as well share the logic I use for a more traditional logfile rotation scheme.

I think this is as simple as I can possibly make it:

A sample series of executions might look like this:
ammon@wernstrom:/path/to/log/file$ touch abc.log
ammon@wernstrom:/path/to/log/file$ php rotate.php
– rotating /path/to/log/file/abc.log -> 0
– creating /path/to/log/file/abc.log
ammon@wernstrom:/path/to/log/file$ php rotate.php
– rotating /path/to/log/file/abc.log.0 -> 1
– rotating /path/to/log/file/abc.log -> 0
– creating /path/to/log/file/abc.log
ammon@wernstrom:/path/to/log/file$ php rotate.php
– rotating /path/to/log/file/abc.log.1 -> 2
– rotating /path/to/log/file/abc.log.0 -> 1
– rotating /path/to/log/file/abc.log -> 0
– creating /path/to/log/file/abc.log
ammon@wernstrom:/path/to/log/file$ php rotate.php
– deleting /path/to/log/file/abc.log.2
– rotating /path/to/log/file/abc.log.1 -> 2
– rotating /path/to/log/file/abc.log.0 -> 1
– rotating /path/to/log/file/abc.log -> 0
– creating /path/to/log/file/abc.log

This doesn’t have any failsafes, doesn’t compress anything, depends on an external call to ‘ls’, and it actually deletes old files in stead of overwriting them… but it is the shortest, simplest method I’ve come up with to get the job done.

If I feel like making this a full-fledged series, I might actually post a more thorough implementation later 😉

simple two-step logfile rotation

This is the result of 10 minutes of pounding on the keyboard after yet another disappointing experience with trying to get logrotate to do something vaguely more flexible.

This simple script scans all normal files in a log directory, and if they are older than a certain cutoff, moves them into a holding directory for old logs. Future passes will check files in the old directory for another age setting and will delete them. That’s all there is to it.

Configure your cutoffs, directories of interest, and optionally plug in a better logging mechanism and you’re set. (Oh, and change the #! if necessary, of course).

In the grand tradition of my publishing little building-block shell scripts of interest, here goes another one. This is a simple cron job that I run daily on a number of hosts to generate storage usage growth. (This is in addition to Cacti and Nagios which poll some of this data already but for different reasons and with different granularity).

The FILES variable should be populated with a whitespace separated list of files, directories, and block devices to track.

The DB_ABCD variables should be populated with appropriate credentials to talk to a mysql server.

The actual script looks something like this:

I am putting my data into a table called “storage_usage” in a database called “metrics”:

Obviously, this could be tweaked in any different number of ways, based on your needs. One tweak you might want to consider if you’re running it in a daily cron is to remove the echo so you don’t get an email report of every run. Also, if you might want to record more than one snapshot per file per host per day – in the which case you probably need to change the type of the timestamp column to a datetime. Or there might be cases where you want to change the replace to an insert or… whatever 😉