Tag Archives: py.test

py.test subprocess fixture callables for testing Twisted apps

TL;DR: fork and os._exit(0) the fixture callable

Getting py.test to test Twisted apps is supported to some extent, albeit somewhat briefly documented; also there’s a py.test plugin to help with testing Twisted apps.

Tests usually require fixtures to be set up. Let’s assume your tests require running something in a separate process, for example a server such as MySQL. What to do? Ok, you can use subprocess.Popen, or Twisted spawnProcess to spin up the database. Note that you should probably not use multiprocessing: It uses its own loop for which there is no support in Twisted.

But what if it’s Python code you want to run? Yes, you can put it into a module and run the module using the above methods. However, if you want to use a Python callable defined in your test module you’re out of luck: neither subprocess.Popen and spawnProcess: nor can run a Python callable in a subprocess.

In that case, you need os.fork. Simply run the callable in the child, and depending on your use case, either wait for it to complete in parent, or kill it at the end of the test. However there’s one gotcha, at least when using py.test: since you’re forking a running test, py.test will now report two tests running and completing. The solution is to exit the child abnormally; simple sys.exit() will raise an exception, but doing os._exit(0) does not.

Here’s example code that spins up a simple test HTTP server for one request, and checks that content fetched by HTTP client matches that served by the HTTP server:

import os
from httpserver import BaseHTTPServer
import pytest
import treq
from turq import TurqHandler

def serve_request(host, port, rulecode):
   TurqHandler.rules = parse_rules(rulecode)
   server = BaseHTTPServer.HTTPServer((host,port), TurqHandler)
   server.handle_request() # nothing more needed fro this one test

@pytest.inlineCallbacks
def test_something():
   pid = os.fork()

   # set up fixture in child
   if pid == 0:
      serve_request("127.0.0.1", 8080, "path('*').text('Hello')")
      os._exit(0)

   # proceed in parent (test), wait a bit first for the server fixture to come up
   time.sleep(0.5) 

   # make a request
   r = treq.get("http://127.0.0.1:8080")
 
   # kill server in child if we cannot connect
   try:
      response = yield r
   except Exception as exc:
      os.kill(pid, signal.SIGKILL)
      raise

   responsetext = yield treq.content(response)
   assert responsetext == "Hello"

I don’t know whether the same technique works with nose and/or Twisted Trial – let me know if you find out!

Advertisements

Comparison of py.test and nose for Python testing

I happened upon this useful comparison of py.test and nose at the testing-in-python mailing list, by Kenny (theotherwhitemeat at gmail). He spent some time evaluating testing tools for Python with a focus on py.test and nose . This article is a reformat of his mailing list post. I assume no credit for the content. The list of references [1] … [13] is at the end of article.

py.test

  • parallelizable: threading + SMP support [3] [4]
  • better documentation: [1] [3]
  • can generate script that does all py.test functions, obviating the need to distribute py.test [1][10]
  • integrate tests into your distribution (py.test –genscript=runtests.py), to create a standalone version of py.test [10]
  • can run nose, unittest, doctest style tests [1] [2]
  • test detection via globs, configurable [3]
  • test failure output more discernible than nose [3] [9]
  • easier, more flexible assertions than nose [8]
  • setup speed is sub-second different from nose, also test speeds can be managed via distribution (threads + SMP via xdist) [9] [11]
  • provides test isolation, if needed [9]
  • dependency injection via funcargs [10] [12] [13]
  • coverage plugin [11]

nose

  • documentation concerns, this may be outdated [3]
  • parallelization issues [3] [8]
  • slightly faster than py.test [4] [11]
  • test detection via regex (setup in cmdline or config file) [3]
  • can run unittest, doctest style tests [1] [2]
  • cannot run py.test style tests [1]

Conclusions

  • test formats are so similar, that nose or py.test can be used without much consequence until you’re writing more exotic tests (you could swap with little consequence)
  • nose is sub-second faster than py.test in execution time; this is typically unimportant
  • community seems to slightly favor py.test

References

  1. http://mail.scipy.org/pipermail/astropy/2011-July/001673.html
  2. http://pytest.org/latest/nose.html
  3. http://fedoraproject.org/wiki/User:Tflink/AutoQA_nose_pytest_comparison
  4. http://www.libcrack.so/2012/01/09/a-brief-analysis-of-python-testing-software/
  5. http://pythontesting.net/framework/nose/nose-introduction/
  6. http://wiki.python.org/moin/PythonTestingToolsTaxonomy
  7. http://docs.python-guide.org/en/latest/writing/tests.html#tools
  8. http://stackoverflow.com/questions/191673/preferred-python-unit-testing-framework
  9. http://thread.gmane.org/gmane.comp.python.testing.general/3748
  10. http://article.gmane.org/gmane.comp.python.testing.general/3752
  11. http://article.gmane.org/gmane.comp.python.testing.general/3765
  12. http://pytest.org/latest/funcargs.html
  13. http://holgerkrekel.net/2009/05/13/parametrizing-python-tests-generalized/