Python unittest and Hudson

Hudson uses the error code of the last command in your script to determine whether your build has failed (http://stackoverflow.com/questions/4334591/setup-a-test-build-job-in-hudson-that-detects-when-make-fails-to-compile). How does this affect things if you're running Python/Django unit tests, or more importantly, if you run Selenium tests against multiple browsers? The Python unittest returns the error code based on a wasSuccessful() command, which in turn is determined by whether any failures/errors occurred;

unittest.py:
 sys.exit(not result.wasSuccessful())

...where wasSuccessful() corresponds to this line:

   def wasSuccessful(self):
        "Tells whether or not this result was a success"
        return len(self.failures) == len(self.errors) == 0

Now suppose you build your own test runner, most notably if you're running Selenium to launch multiple browsers:

 """
_environments = [
    # Firefox runs faster than IE7, so check first even though our customers use the latter browser more.
    {'platform' : "WINDOWS",
     'browserName' : "firefox",
     'version' :  "3.6"
    },
    {'platform' : "WINDOWS",
     'browserName' : "iexplore",
     'version' :  "7"
    },
]
    def run_suite(self, suite, **kwargs):
        r = None
        for env in _environments:
            print "Running %s" % env

            # We need Django nose to run XML coverage outputs on Hudson.  For multiple browser support, we need
            # to change the --xunit-file parameter.
            if hasattr(settings, 'NOSE_ARGS'):
                nose_utils.swap_nose_args(suite, env)

            selenium_cfg.current_env = env
            test = super(SeleniumTestSuiteRunner, self).run_suite(suite, **kwargs)
            if not r:
                r = test
            else:
                r.failures.append(test.failures)
                r.errors.append(test.errors)

If we try to use the code above as the test-runner, Hudson always will trigger a build failure. Why? Well it turns out whate need to make sure is that we use extend() instead of append(). If we use append(), we appending multiple lists within a list (i.e.):

>>> a = []
>>> a.append([])
>>> a
[[]]
>>> a.append([])
>>> a
[[], []]

The fix is shown below.

52  52              if not r:
53  53                  r = test
54  54              else:
55                   r.failures.append(test.failures)
56                   r.errors.append(test.errors)
55                  r.failures.extend(test.failures)
56                  r.errors.extend(test.errors)
57  57