Image size is definitely a case of “less is more.” Many images on the web are larger than they need to be, and there has been a recent surge in the movement to strip out excess bits while maintaining a lossless image. Smush.it™ is a great example. Unfortunately, even though it runs on open source image utilities, there is no API and Yahoo! seems to have no plan to release an open source version.

Have no fear! It isn’t doing anything complicated, so why not just do it ourselves. Plus, the authors basically laid out what they do in the excellent Even Faster Web Sites.

Image Squeeze

Image Squeeze is a pluggable library for optimizing images. The only library it requires is ImageMagick (specifically it uses identify to figure out what type of image it’s looking at). The default processors included with the library require pngcrush, jpegtran, gifsicle (for animated gifs), and ImageMagick’s convert. Here’s a quick example:

• set up an ImageSqueeze with the default processors
  

  squeezer = ImageSqueeze.new(:default_processors => true)

• in-place squeeze of our png
  

  squeezer.squeeze!(‘my_logo.png’)

If you don’t want to install one of those for some reason, don’t worry, image_squeeze just won’t include the processor if the utility isn’t available.

Adding your own processors

Let’s say PngCrush isn’t good enough for you, and you want to add an OptiPNG processor as well. Easy enough, we can just plug that in.

So you just need to implement two methods, once which returns an image type (constants defined in ImageSqueeze) and a squeeze method (takes an input filename and leaves a new image in the output filename). Note: The default processors are all lossless quality, if you implement your own it’s up to you to verify that you aren’t losing quality.

Now let’s load it in with our defaults,

squeezer = ImageSqueeze.new(:default_processors => true, :processors => [ImageSqueeze::OptiPNGProcessor])

Or if you don’t like the defaults, just use your own processors

squeezer = ImageSqueeze.new(:processors => [ImageSqueeze::OptiPNGProcessor])

That was pretty useful, and OptiPNG is great, so I just added it to the defaults. Progress!

Now just hook this into wherever you handle uploaded images and you’re set. I’d definitely recommend doing this out-of-process as far as the server request, or you’ll kill your request times, but I’ll leave that up to you.

One thing I’d really like to do is make a pre-commit hook that process any images you are checking in so you don’t have to worry about it. But that’s for another time…

Woohoo less bytes!

While working on switching Kongregate from Mongrel to Unicorn I was googling around for a monit recipe for watching Unicorn child processes and unfortunately couldn’t find anything. The real issue is that monit will only watch either just the parent process or a the entire process family (parent including all of its children). What I really needed was the ability to watch each child process individually, and kill them when their memory usage grows out of control.

I was able to find another process monitoring tool called bluepill which looks like it was built almost explicitly for this reason. While monit can be a pain in the ass, it’s stable and I don’t have any other reason to switch. Ideally monit will eventually allow monitoring child process, until then I found a decent workaround.

The problem

Since the ideal case of monit watching child processes doesn’t exist, why not have monit just watch each child process individually. Well the odd thing here is that monit wants to start and stop each process, but the Unicorn master starts each worker, so we’ll have monit just worry about killing the children.

init.d script

For my init.d script, I use the script that is packaged with the Unicorn source with a simple change, I add a kill_worker method.

Unicorn master monit config

This is the simple part, but we want monit to watch the master worker, it shouldn’t ever grow about our memory limit, but we do want monit to make sure it’s running.

Unicorn workers monit config

Here’s where the magic happens…I guess. So the weird part about monit watching Unicorns is that monit wants to start each process it watches. We’ll just trick monit into doing a no-op for the start and let it watch the pid files to figure out when the master starts the workers.

Unicorn post_fork

The final piece of the puzzle is making each child write out its own pid file, which we do with a simple after_fork hook. I use port for the pid file, for slightly easier debugging, but there is no real reason for it.

In the end the final solution is kind of cheesy, since monit just keeps doing nothing until the master starts the workers, but once everything gets started up you are golden.

Finally, here’s the monit commands to start everything up.

monit start all -g unicorn
monit start all -g unicorn_workers

Chances are at some point you’ve wanted to get the values of a single column from ActiveRecord. You might have done something like this:

user_ids = User.find(:all).map(&:id)

That’s great and all, but hopefully somebody points out that you can pass ActiveRecord#find a :select key.

user_ids = User.find(:all, :select => 'id').map(&:id)

Now we’re getting somewhere. On a “wide” table (ie. one with a lot of columns, large columns, or binary types stored separately) this alone can be a huge improvement, as just sending all that data across the wire can really waste time. While ActiveRecord is generally good enough performance wise, sometimes getting a little closer to the metal is appropriate, and in comes find_ids.

user_ids = User.find_ids

Benchmarks

Here’s some benchmarks against a fairly “wide” table with 100 records, 10,000 times for each style:

                              user     system      total        real
find(:all).map           26.460000   3.130000  29.590000 ( 72.656614)
find(:all, :select).map  12.770000   0.560000  13.330000 ( 16.232474)
find_ids                  7.870000   0.340000   8.210000 ( 10.465719)

Examples

But that’s not all, it also natively supports other columns, associations, and named scopes.

emails = User.find_column_values('email')
User.wristbands.find_ids
User.scoped(:order => 'created_at desc').find_ids

Installation

script/plugin install git://github.com/stopdropandrew/find_ids.git

Issues

It currently only works with MySQL, but submissions for other adapters are welcome.

I’m still in the process of learning Git, and while it’s been fairly rough in the beginning, I feel like I’m starting to round the curve of really being productive. While we are still primarily using Subversion at Kongregate, I’ve been trying to use Git whenever possible.

For a side project I’ve been working on, I wanted to have my authoritative Git repo in our colo, behind our VPN, to both speed up deploys and to know that we don’t need to rely on a third party service. But, at the same time, I wanted to be able to use Github’s awesome UI.

hooks/post-receive

There’s not a lot to it, but just set up Github as a remote for your server side repo, then drop this into your hooks/post-receive and be sure to make it executable:

#!/usr/bin/env ruby
STDIN.read.split("\n").each do |line|
   oldrev, newrev, refname = line.split(' ')
   
   if refname.match(/^refs\/heads\/(.*)/)
     branch = $1
   
     `git push origin #{branch}`
   else
     puts "#{refname} was weird, not sure what to do."
   end
end

Nothing too special, but I was pretty happy to get it all working without too much effort.