The Problem
The original problem presented to me was that Firefox developers would like to find regressions. My mentor wrote a tool (mozregression) that downloaded nightly builds and would bisect to find a regression range. However it would be nice for developers to be able to pinpoint an exact changeset and to have the regression finder check merge commits (branch aware). So I wrote mozcommitbuilder by wrapping mercurial’s internal bisect mechanism. The first iteration of the tool would download a clean copy of trunk from mozilla-central, configure the build environment, build, and run Firefox. It was able to bisect interactively, asking whether each built changeset was good or bad.

The ultimate goal was automation — fully automatic regression-finding that would take off and run on its own.

Regression Finding

For those who are unfamiliar with regression finding, basically what it entails is knowing that some feature is broken that used to be working. That means that somewhere along the line, someone / something broke the feature and the goal is to find when exactly things broke. The way we do this is by going back and doing a binary search: find the middle changeset, test it, and decide whether the break happened forward or backwards in time. It gets tricky when there are branches because then time isn’t just linear: both branches need to be checked when it comes to a merged changeset, because the parent branch could have caused the breakage.

Luckily, mercurial’s built-in bisect is very robust. The commitbuilder tool wraps mercurial and uses its internal bisection mechanism — it is branch aware in that it prints out the changeset for a missed branch following a bisection: so finding a regression on a merge commit may require more than one pass with the tool, but it gets the job done. However, to make matters easier, just four days ago hg 1.9 came out, which has a fully branch-aware bisect feature (see hg bisect –extend) and mozcommitbuilder leverages that on systems with the latest mercurial. With one pass, the commitbuilder tool is able to find regressions by building, running, and interactively prompting.

Fully-automated Regression Hunting
As of the most recent iteration, it is also able to automate regression-finding using custom conditional scripts. Instead of telling the regression finder that a given changeset is good or bad manually, the tool can be given a script to use as its truth condition. The way this works is that a user can define a conditional script that looks something like this:

mycondition.py
def interesting(args, directory):
  if(something == True):
    return "good"
  else:
    return "bad"


And the user can call commitbuilder with something like this:
mozcommitbuilder -g 70168 -b 72300 -c mycondition.py arg1 arg2


The result is that the builder will download a local copy of Mozilla-central’s repository, it will switch to the relevant revision, build Firefox (and detect how many cores are on the system for maximum processor utilization), run Firefox, run the condition (can be anything, like a mochitest), and based on its output, it will bisect/build/test until the regression is found.

There are some interesting use cases with this tool: for example, it is currently being used in conjunction with some fuzzer scripts which he uses as the condition (automated bug finding + using the tool to find the exact regression / cause of the bug). Another use case currently in development is a condition script that runs automated tests multiple times in a row while hunting, in order to find nondeterministic regressions.

This sounds useful!
sudo pip install mozcommitbuilder --upgrade
Feel free to open an issue or pull request on github. I’m generally quick about responding to those.