py.test subprocess fixture callables for testing Twisted apps

TL;DR: fork and os._exit(0) the fixture callable

Getting py.test to test Twisted apps is supported to some extent, albeit somewhat briefly documented; also there’s a py.test plugin to help with testing Twisted apps.

Tests usually require fixtures to be set up. Let’s assume your tests require running something in a separate process, for example a server such as MySQL. What to do? Ok, you can use subprocess.Popen, or Twisted spawnProcess to spin up the database. Note that you should probably not use multiprocessing: It uses its own loop for which there is no support in Twisted.

But what if it’s Python code you want to run? Yes, you can put it into a module and run the module using the above methods. However, if you want to use a Python callable defined in your test module you’re out of luck: neither subprocess.Popen and spawnProcess: nor can run a Python callable in a subprocess.

In that case, you need os.fork. Simply run the callable in the child, and depending on your use case, either wait for it to complete in parent, or kill it at the end of the test. However there’s one gotcha, at least when using py.test: since you’re forking a running test, py.test will now report two tests running and completing. The solution is to exit the child abnormally; simple sys.exit() will raise an exception, but doing os._exit(0) does not.

Here’s example code that spins up a simple test HTTP server for one request, and checks that content fetched by HTTP client matches that served by the HTTP server:

import os
from httpserver import BaseHTTPServer
import pytest
import treq
from turq import TurqHandler

def serve_request(host, port, rulecode):
   TurqHandler.rules = parse_rules(rulecode)
   server = BaseHTTPServer.HTTPServer((host,port), TurqHandler)
   server.handle_request() # nothing more needed fro this one test

@pytest.inlineCallbacks
def test_something():
   pid = os.fork()

   # set up fixture in child
   if pid == 0:
      serve_request("127.0.0.1", 8080, "path('*').text('Hello')")
      os._exit(0)

   # proceed in parent (test), wait a bit first for the server fixture to come up
   time.sleep(0.5) 

   # make a request
   r = treq.get("http://127.0.0.1:8080")
 
   # kill server in child if we cannot connect
   try:
      response = yield r
   except Exception as exc:
      os.kill(pid, signal.SIGKILL)
      raise

   responsetext = yield treq.content(response)
   assert responsetext == "Hello"

I don’t know whether the same technique works with nose and/or Twisted Trial – let me know if you find out!

Advertisements

Comparison of py.test and nose for Python testing

I happened upon this useful comparison of py.test and nose at the testing-in-python mailing list, by Kenny (theotherwhitemeat at gmail). He spent some time evaluating testing tools for Python with a focus on py.test and nose . This article is a reformat of his mailing list post. I assume no credit for the content. The list of references [1] … [13] is at the end of article.

py.test

  • parallelizable: threading + SMP support [3] [4]
  • better documentation: [1] [3]
  • can generate script that does all py.test functions, obviating the need to distribute py.test [1][10]
  • integrate tests into your distribution (py.test –genscript=runtests.py), to create a standalone version of py.test [10]
  • can run nose, unittest, doctest style tests [1] [2]
  • test detection via globs, configurable [3]
  • test failure output more discernible than nose [3] [9]
  • easier, more flexible assertions than nose [8]
  • setup speed is sub-second different from nose, also test speeds can be managed via distribution (threads + SMP via xdist) [9] [11]
  • provides test isolation, if needed [9]
  • dependency injection via funcargs [10] [12] [13]
  • coverage plugin [11]

nose

  • documentation concerns, this may be outdated [3]
  • parallelization issues [3] [8]
  • slightly faster than py.test [4] [11]
  • test detection via regex (setup in cmdline or config file) [3]
  • can run unittest, doctest style tests [1] [2]
  • cannot run py.test style tests [1]

Conclusions

  • test formats are so similar, that nose or py.test can be used without much consequence until you’re writing more exotic tests (you could swap with little consequence)
  • nose is sub-second faster than py.test in execution time; this is typically unimportant
  • community seems to slightly favor py.test

References

  1. http://mail.scipy.org/pipermail/astropy/2011-July/001673.html
  2. http://pytest.org/latest/nose.html
  3. http://fedoraproject.org/wiki/User:Tflink/AutoQA_nose_pytest_comparison
  4. http://www.libcrack.so/2012/01/09/a-brief-analysis-of-python-testing-software/
  5. http://pythontesting.net/framework/nose/nose-introduction/
  6. http://wiki.python.org/moin/PythonTestingToolsTaxonomy
  7. http://docs.python-guide.org/en/latest/writing/tests.html#tools
  8. http://stackoverflow.com/questions/191673/preferred-python-unit-testing-framework
  9. http://thread.gmane.org/gmane.comp.python.testing.general/3748
  10. http://article.gmane.org/gmane.comp.python.testing.general/3752
  11. http://article.gmane.org/gmane.comp.python.testing.general/3765
  12. http://pytest.org/latest/funcargs.html
  13. http://holgerkrekel.net/2009/05/13/parametrizing-python-tests-generalized/

Scripting ssh and git securely with Python

Some programs such as SSH ask for password and key passphrase via TTY (rather than STDIN), which is not supported by Python’s subprocess.Popen() call – the otherwise recommended, convenient way or running programs in a separate process.

This issue also arises when using SSH and PKI keys with git, to secure communications with a git server without an username and password.

This example illustrates use of pty.fork() from Python stdlib to support communications with a child process via a tty (a pty, actually).

There’s also some more explanations on how pty.fork() works, from Stack Overflow. Furthermore, Paul Mikesell has written an extensive article on scripting SSH with Python.

Uploadify session workaround

Due to design of Flash that Uploadify uses, there is a known problem whereby the Flash component of Uploadify does not pass back the cookies. So things like sessions do not work as expected out-of-the-box.

Luckily, there’s a workaround: uploadify has a scriptData option that you can populate from the session cookie  (perhaps using a jQuery cookie plugin for convenience). The option value is then passed to the backend app, which can use it to switch to the proper session. For example as follows:

First, a small function to fetch the session id:

var sid = function () {
   return {'sessionid':$.cookie('sessionid')}
};

Then, as part of the Uploadify initialization, set scriptData to pass the return value:

$(document).ready(function() {
   $('#file_upload').uploadify({
     'uploader'  : '/uploadify/uploadify.swf',
     'script'    : '/upload_files',
     'scriptData': sid(), // <--- our function
});

This is just an incomplete example; for the rest of the options you want to initialize Uploadify with, see Uploadify docs.
As for backend processing, scriptData comes in as regular POST variables. I was building a simple Django app when encountering this problem, so here’s what I did to fetch the session id and switch to another Django session using it.

# get session id that was passed in via scriptData
sid = request.POST['sessionid']

# Switch to another session by triggering the
# SessionStore session loading machinery, using
# the proper session id that we passed in from
# Uploadify.
session = request.session
session._set_session_key(sid)
session._get_session()

That’s all.