Python 3 Testing: An Intro to unittest
The unittest module is actually a testing framework that was originally inspired by JUnit. It currently supports test automation, the sharing of setup and shutdown code, aggregating tests into collections and the independence of tests from the reporting framework.
The unittest frameworks supports the following concepts:
- Test Fixture – A fixture is what is used to setup a test so it can be run and also tears down when the test is finished. For example, you might need to create a temporary database before the test can be run and destroy it once the test finishes.
- Test Case – The test case is your actual test. It will typically check (or assert) that a specific response comes from a specific set of inputs. The unittest frameworks provides a base class called **TestCase** that you can use to create new test cases.
- Test Suite – The test suite is a collection of test cases, test suites or both.
- Test Runner – A runner is what controls or orchestrates the running of the tests or suites. It will also provide the outcome to the user (i.e. did they pass or fail). A runner can use a graphical user interface or be a simple text interface.
A Simple Example
I always find a code example or two to be the quickest way to learn how something new works. So let’s create a little module that we will call mymath.py. Then put the following code into it:
def add(a, b): return a + b def subtract(a, b): return a - b def multiply(a, b): return a * b def divide(numerator, denominator): return float(numerator) / denominator
This module defines four mathematical functions: add, subtract, multiply and divide. They do not do any error checking and they actually don’t do exactly what you might expect. For example, if you were to call the add function with two strings, it would happily concatenate them together and return them. But for the purposes of illustration, this module will do for creating a test case. So let’s actually write a test case for the add function! We will call this script test_mymath.py and save it in the same folder that contains mymath.py.
import mymath import unittest class TestAdd(unittest.TestCase): """ Test the add function from the mymath library """ def test_add_integers(self): """ Test that the addition of two integers returns the correct total """ result = mymath.add(1, 2) self.assertEqual(result, 3) def test_add_floats(self): """ Test that the addition of two floats returns the correct result """ result = mymath.add(10.5, 2) self.assertEqual(result, 12.5) def test_add_strings(self): """ Test the addition of two strings returns the two string as one concatenated string """ result = mymath.add('abc', 'def') self.assertEqual(result, 'abcdef') if __name__ == '__main__': unittest.main()
Let’s take a moment and go over how this code works. First we import our mymath module and Python’s unittest module. Then we subclass TestCase and add three tests, which translates into three methods. The first function tests the addition of two integers; the second function tests the addition of two floating point numbers; and the last function concatenates two strings together. Finally we call unittest’s main method at the end.
You will note that each method begins with the letters “test”. This is actually important! It tells the test runner which methods are tests that it should run. Each test should have at least one assert which will verify that the result is as we expected. The unittest module supports many different types of asserts. You can test for exceptions, for Boolean conditions, and for many other conditions.
Let’s try running out test. Open up a terminal and navigate to the folder that contains your mymath module and your test module:
python test_mymath.py
This will execute our test and we should get the following output:
... ---------------------------------------------------------------------- Ran 3 tests in 0.001s OK
You will note that there are three periods. Each period represents a test that has passed. Then it tells us that it ran 3 tests, the time it took and the result: OK. That tells us that all the tests passed successfully.
You can make the output a bit more verbose by passing in the -v flag:
python test_mymath.py -v
This will cause the following output to be printed to stdout:
test_add_floats (__main__.TestAdd) ... ok test_add_integers (__main__.TestAdd) ... ok test_add_strings (__main__.TestAdd) ... ok ---------------------------------------------------------------------- Ran 3 tests in 0.000s OK
As you can see, this shows us exactly what tests were run and the results of each test. This also leads us into our next section where we will learn about some of the commands we can use with unittest on the command line.
Command-Line Interface
The unittest module comes with a few other commands that you might find useful. To find out what they are, you can run the unittest module directly and pass it the -h as shown below:
python -m unittest -h
This will cause the following output to be printed to stdout. Note that I have cut out a portion of the output that covered Test Discovery command line options for brevity:
usage: python -m unittest [-h] [-v] [-q] [--locals] [-f] [-c] [-b] [tests [tests ...]] positional arguments: tests a list of any number of test modules, classes and test methods. optional arguments: -h, --help show this help message and exit -v, --verbose Verbose output -q, --quiet Quiet output --locals Show local variables in tracebacks -f, --failfast Stop on first fail or error -c, --catch Catch ctrl-C and display results so far -b, --buffer Buffer stdout and stderr during tests Examples: python -m unittest test_module - run tests from test_module python -m unittest module.TestClass - run tests from module.TestClass python -m unittest module.Class.test_method - run specified test method
Now we have some ideas of how we might call our test code if it didn’t have the call to unittest.main() at the bottom. In fact, go ahead and re-save that code with a different name, such as test_mymath2.py with the last two lines removed. Then run the following command:
python -m unittest test_mymath2.py
This should result in the same output we got previously:
... ---------------------------------------------------------------------- Ran 3 tests in 0.000s OK
The cool thing about using the unittest module on the command line is that we can use it to call specific functions in our test. Here’s an example:
python -m unittest test_mymath2.TestAdd.test_add_integers
This command will run only run test, so the output from this command should look like this:
. ---------------------------------------------------------------------- Ran 1 test in 0.000s OK
Alternatively, if you had multiple test cases in this test module, then you could call just one test case at a time, like this:
python -m unittest test_mymath2.TestAdd
All this does is call our TestAdd subclass and runs all the test methods in it. So the result should be the same as if we ran it in the first example:
... ---------------------------------------------------------------------- Ran 3 tests in 0.000s OK
The point of this exercise is that if you were to have additional test cases in this test module, then this method gives you a method to run individual test cases instead of all of them.
Creating a More Complex Test
Most code is a lot more complex than our mymath.py example. So let’s create a piece of code that depends on a database being in existence. We will create a simple script that can create the database with some initial data if it doesn’t exist along with a few functions that will allow us to query it, delete and update rows. We will name this script simple_db.py. This is a fairly long example, so bear with me:
import sqlite3 def create_database(): conn = sqlite3.connect("mydatabase.db") cursor = conn.cursor() # create a table cursor.execute("""CREATE TABLE albums (title text, artist text, release_date text, publisher text, media_type text) """) # insert some data cursor.execute("INSERT INTO albums VALUES " "('Glow', 'Andy Hunter', '7/24/2012'," "'Xplore Records', 'MP3')") # save data to database conn.commit() # insert multiple records using the more secure "?" method albums = [('Exodus', 'Andy Hunter', '7/9/2002', 'Sparrow Records', 'CD'), ('Until We Have Faces', 'Red', '2/1/2011', 'Essential Records', 'CD'), ('The End is Where We Begin', 'Thousand Foot Krutch', '4/17/2012', 'TFKmusic', 'CD'), ('The Good Life', 'Trip Lee', '4/10/2012', 'Reach Records', 'CD')] cursor.executemany("INSERT INTO albums VALUES (?,?,?,?,?)", albums) conn.commit() def delete_artist(artist): """ Delete an artist from the database """ conn = sqlite3.connect("mydatabase.db") cursor = conn.cursor() sql = """ DELETE FROM albums WHERE artist = ? """ cursor.execute(sql, [(artist)]) conn.commit() cursor.close() conn.close() def update_artist(artist, new_name): """ Update the artist name """ conn = sqlite3.connect("mydatabase.db") cursor = conn.cursor() sql = """ UPDATE albums SET artist = ? WHERE artist = ? """ cursor.execute(sql, (new_name, artist)) conn.commit() cursor.close() conn.close() def select_all_albums(artist): """ Query the database for all the albums by a particular artist """ conn = sqlite3.connect("mydatabase.db") cursor = conn.cursor() sql = "SELECT * FROM albums WHERE artist=?" cursor.execute(sql, [(artist)]) result = cursor.fetchall() cursor.close() conn.close() return result if __name__ == '__main__': import os if not os.path.exists("mydatabase.db"): create_database() delete_artist('Andy Hunter') update_artist('Red', 'Redder') print(select_all_albums('Thousand Foot Krutch'))
You can play around with this code a bit to see how it works. Once you’re comfortable with it, then we can move on to testing it.
Now some might argue that creating a database and destroying it for each test is pretty big overhead. And they might have a good point. However, to test certain functionality, you sometimes need to do this sort of thing. Besides, you don’t usually need to create the entire production database just for sanity checks.
Anyway, this is once again for illustrative purposes. The unittest module allows us to override setUp and tearDown methods for these types of things. So we will create a setUp method that will create the database and a tearDown method that will delete it when the test is over. Note that the setup and tear down will occur for each test. This prevents the tests from altering the database in such a way that a subsequent test will fail.
Let’s take a look at the first part of the test case class:
import os import simple_db import sqlite3 import unittest class TestMusicDatabase(unittest.TestCase): """ Test the music database """ def setUp(self): """ Setup a temporary database """ conn = sqlite3.connect("mydatabase.db") cursor = conn.cursor() # create a table cursor.execute("""CREATE TABLE albums (title text, artist text, release_date text, publisher text, media_type text) """) # insert some data cursor.execute("INSERT INTO albums VALUES " "('Glow', 'Andy Hunter', '7/24/2012'," "'Xplore Records', 'MP3')") # save data to database conn.commit() # insert multiple records using the more secure "?" method albums = [('Exodus', 'Andy Hunter', '7/9/2002', 'Sparrow Records', 'CD'), ('Until We Have Faces', 'Red', '2/1/2011', 'Essential Records', 'CD'), ('The End is Where We Begin', 'Thousand Foot Krutch', '4/17/2012', 'TFKmusic', 'CD'), ('The Good Life', 'Trip Lee', '4/10/2012', 'Reach Records', 'CD')] cursor.executemany("INSERT INTO albums VALUES (?,?,?,?,?)", albums) conn.commit() def tearDown(self): """ Delete the database """ os.remove("mydatabase.db")
The setUp method will create our database and then populate it with some data. The tearDown method will delete our database file. If you were using something like MySQL or Microsoft SQL Server for your database, then you would probably just drop the table, but with sqlite, we can just delete the whole thing.
Now let’s add a couple of actual tests to our code. You can just add these to the end of the test class above:
def test_updating_artist(self): """ Tests that we can successfully update an artist's name """ simple_db.update_artist('Red', 'Redder') actual = simple_db.select_all_albums('Redder') expected = [('Until We Have Faces', 'Redder', '2/1/2011', 'Essential Records', 'CD')] self.assertListEqual(expected, actual) def test_artist_does_not_exist(self): """ Test that an artist does not exist """ result = simple_db.select_all_albums('Redder') self.assertFalse(result)
The first test will update the name of one of the artists to the string Redder. Then we do a query to make sure that the new artist name exists. The next test will also check to see if the artist known as “Redder” exists. This time it shouldn’t as the database was deleted and recreated between tests. Let’s try running it to see what happens:
python -m unittest test_db.py
The command above should result in the output below, although your runtime will probably differ:
.. ---------------------------------------------------------------------- Ran 2 tests in 0.032s OK
Pretty cool, eh? Now we can move on to learn about test suites!
Creating Test Suites
As was mentioned at the beginning, a test suite is just a collection of test cases, test suites or both. Most of the time, when you call unittest.main(), it will do the right thing and gather all the module’s test cases for you before executing them. But sometimes you will want to be the one in control. In that circumstance, you can use the TestSuite class. Here’s an example of how you might use it:
import unittest from test_mymath import TestAdd def my_suite(): suite = unittest.TestSuite() result = unittest.TestResult() suite.addTest(unittest.makeSuite(TestAdd)) runner = unittest.TextTestRunner() print(runner.run(suite)) my_suite()
Creating your own suite is a slightly convoluted process. First you need to create an instance of TestSuite and an instance of TestResult. The TestResult class just holds the results of the tests. Next we call addTest on our suite object. This is where it gets a bit weird. If you just pass in TestAdd, then it has to be an instance of TestAdd and TestAdd must also implement a runTest method. Since we didn’t do that, we use unittest’s makeSuite function to turn our TestCase class into a suite.
The last step is to actually run the suite, which means that we need a runner if we want nice output. Thus, we create an instance of TextTestRunner and have it run our suite. If you do that and you print out what it returns, you should get something like this printed out to your screen:
... ---------------------------------------------------------------------- Ran 3 tests in 0.001s OK <unittest.runner.TextTestResult run=3 errors=0 failures=0>
An alternative is to just call suite.run(result) and print out its result. However all that will give you is a TestResult object that looks very similar to that last line of output above. If you want the more usual output, then you will want to use a runner.
How to Skip Tests
The unittest module supports skipping tests as of Python 3.1. There are a few use cases for skipping tests:
- You might want to skip a test if the version of a library doesn’t support what you want to test
- The test is dependent on the operating system it is running under
- Or you have some other criteria for skipping a test
Let’s change our test case so it has a couple of tests that will be skipped:
import mymath import sys import unittest class TestAdd(unittest.TestCase): """ Test the add function from the mymath module """ def test_add_integers(self): """ Test that the addition of two integers returns the correct total """ result = mymath.add(1, 2) self.assertEqual(result, 3) def test_add_floats(self): """ Test that the addition of two floats returns the correct result """ result = mymath.add(10.5, 2) self.assertEqual(result, 12.5) @unittest.skip('Skip this test') def test_add_strings(self): """ Test the addition of two strings returns the two string as one concatenated string """ result = mymath.add('abc', 'def') self.assertEqual(result, 'abcdef') @unittest.skipUnless(sys.platform.startswith("win"), "requires Windows") def test_adding_on_windows(self): result = mymath.add(1, 2) self.assertEqual(result, 3)
Here we demonstrate two different methods of skipping a test: skip and skipUnless. You will notice that we are decorating the functions that need to be skipped. The skip decorator can be used to skip any test for any reason. The skipUnless decorator will skip a test unless the condition returns True. So if you run this test on Mac or Linux, it will get skipped. There is also a skipIf decorator that will skip a test if the condition is True.
You can run this script with the verbose flag to see why it’s skipping tests:
python -m unittest test_mymath.py -v
This command will result in the following output:
test_add_floats (test_mymath4.TestAdd) ... ok test_add_integers (test_mymath4.TestAdd) ... ok test_add_strings (test_mymath4.TestAdd) ... skipped 'Skip this test' test_adding_on_windows (test_mymath4.TestAdd) ... skipped 'requires Windows' ---------------------------------------------------------------------- Ran 4 tests in 0.000s OK (skipped=2)
his output tells us that we attempted to run four tests, but skipped two of them.
There is also an expectedFailure decorator that you can add to a test that you know will fail. I’ll leave that one for you to try out on your own.
Integrating with doctest
The unittest module can be used with Python’s doctest module as well. If you have created a lot of modules that have doctests in them, you will usually want to be able to run them systematically. This is where unittest comes in. The unittest module supports Test Discovery starting in Python 3.2. Test Discovery basically allows unittest to look at a the contents of a directory and determine from the file name which ones might contain tests. It then loads the test by importing them.
Let’s create a new empty folder and inside of it, we will create a file called my_docs.py. It will need to have the following code:
def add(a, b): """ Return the addition of the arguments: a + b >>> add(1, 2) 3 >>> add(-1, 10) 9 >>> add('a', 'b') 'ab' >>> add(1, '2') Traceback (most recent call last): File "test.py", line 17, in <module> add(1, '2') File "test.py", line 14, in add return a + b TypeError: unsupported operand type(s) for +: 'int' and 'str' """ return a + b def subtract(a, b): """ Returns the result of subtracting b from a >>> subtract(2, 1) 1 >>> subtract(10, 10) 0 >>> subtract(7, 10) -3 """ return a - b
Now we need to to create another module in the same location as this one that will turn our doctests into unittests. Let’s call this file test_doctests.py. Put the following code inside of it:
import doctest import my_docs import unittest def load_tests(loader, tests, ignore): tests.addTests(doctest.DocTestSuite(my_docs)) return tests
The function name is actually required here for Test Discovery to work, according to the documentation for the doctest module. What we’re doing here is adding a suite to the tests object in much the same way as we did earlier. In this case, we are using doctest’s DocTestSuite class. You can give this class a setUp and tearDown method as parameters should your tests need them. To run this code, you will need to execute the following command in your new folder:
python -m unittest discover
On my machine, I received the following output:
.. ---------------------------------------------------------------------- Ran 2 tests in 0.002s OK
You will note that when you use unittest to run doctest, each docstring is considered a single test. If you were to run the docstrings with doctest directly, then you will notice that doctest will say there are more tests. Other than that, it works pretty much as expected.
Wrapping Up
We covered a lot in this article. You learned the basics of using the unittest module. Then we moved on and learned how to use unittest from the command line. We also found out how to set up and tear down tests. You discovered how to create test suites too. Finally we learned how to turn a series of doctests into unittests. Be sure to spend some time reading the documentation on both of these fine libraries as there is a lot of additional functionality that was not covered here.
Related Reading
