Monday, September 19, 2011

From Zero to Multi-Mechanize

I'm working on a new project and the preference is for Python for test code. The current piece of the project is creating some load tests. Now, there's no good reason to write my own load test framework from scratch, so I did some research and started evaluating Multi-Mechanize.

Getting it set up on my Snow Leopard system wasn't completely trivial, so here's what I had to do:

  1. Upgrade to Tkinter 8.5.10. This is needed to get 64-bit Python to run properly with matplotlib.
    1. See for background information.
    2. Download ActiveTcl 8.5.10
    3. This is a standard .dmg file, so double click and follow the bouncing prompts to install. Take all the defaults.
  2. Upgrade Python to version 2.7. This is needed to run the latest version of matplotlib.
    1. Grab Mac OS X 64-bit/32-bit x86-64/i386 Installer (2.7.2) for Mac OS X 10.6 and 10.7 from
    2. This is a standard .dmg file, so double click and follow the instructions in the installer. Again, take the defaults.
  3. Install numpy. This is a prerequisite for matplotlib.
    1. Go to the installer site. Do not click the link at the top! It's tempting, but that's for OS X 10.3 and won't work. (And by tempting, I mean I did this.) Scroll down and grab this one instead:
    2. Look! Another .dmg file. You got it. Double click, bouncing prompts.
  4. Install matplotlib. This one you'll have to actually compile on your own.
    1. Do not download the .dmg file; that's also for OS X 10.3 and won't work. Instead, grab the source from here:  matplotlib-1.0.1.tar.gz
    2. Unpack the source and put it in a useful (temporary) location.
    3. Open matplotlib-1.0.1/ in your favorite text editor.
    4. Line 832 looks like this:  (Tkinter.__version__.split()[-2], Tkinter.TkVersion, Tkinter.TclVersion))
    5. Remove the [-2], so that it looks like this: (Tkinter.__version__.split(), Tkinter.TkVersion, Tkinter.TclVersion))
    6. In matplotlib-1.0.1 copy setup.cfg.template to setup.cfg
    7. In setup.cfg, uncomment line 10 and change it to read: basedirlist = /usr/X11
    8. Install! python2.7 install
  5. Install mechanize. Needed for, surprise, multi-mechanize.
    1. Download it: mechanize-0.2.5.tar.gz
    2. Unpack the source and put it in a useful (temporary) location.
    3. Install: python install
  6. Install multi-mechanize. Last step, promise.
    1. Download it from the project site.
    2. Unzip it and put it in a safe location. This one isn't an installer; it just runs, so watch where you put it.
Overall the process is tweaky but not difficult. Just make sure you get the right packaging for everything (32-bit versus 64-bit, versions of OS X), and it works just fine. Now off to play with a new (to me) load testing tool!


  1. digging up an old post here...

    wow. that doesn't sound like an easy install. How did you make out with the tool once you got working?

    some comments:

    - Mac is the hardest system to install the dependencies on. Windows is pretty trivial, and any modern Linux distro is dead-simple.

    - I am currently working on making the installation process more standard and automatic. (pypi, pip,, etc). It should be dramatically easier to setup in the next release.

    - I am launching a new website soon and moving development to GitHub. (excpect more news soon) will redirect there. follow updates on the discussion group:

    -Corey Goldberg

  2. Corey, I should thank you properly for writing MultiMechanize in the first place! It was certainly easier than writing it myself.

    To be fair, the real killer wasn't the tool itself, but dealing with 64-bit versus 32-bit python and dependencies. I had similar experiences setting up numpy and scipy on a 64-bit Windows machine.

    As for actual use, it did what I needed it to do. I did have some issues with memory utilization getting awfully high, mostly because my workflow involved repeatedly bytencoding and uploading a file (thanks, platform I was using!). Python's garbage collection leaves much to be desired in situations like these, so memory goes through the roof pretty quickly at any volume.

    A bit of tweaking around doing things in subprocesses versus threads and I got through it at volume.

    And like I said, it's far better than having had to write it myself! Best of luck moving forward with it; can't wait to see how it grows.

  3. Hi Catherine, did you have to install with the same process on Mac OS 10.7?

  4. Rodrigo, I haven't had to try it, since I simply upgraded to Lion. I would guess the process would be somewhat similar, with the added bonus of being careful about your gcc versions if you're also using the newer Xcode.

  5. I've been running MultiMechanize for past little while testing some of our components, but it looks based on the similarity of the graphs that we're hitting some network bandwidth limits running the tests form single machine.

    Is there plan to support multiple nodes? How do you aggregate all the date into one graph?