【转载】Python Testing

Python provides powerful tools and other third-party testing frameworks that should be part of any development. The most straightforward way to get started with testing is with the built-in unittest module.

from unittest import TestCase, main

class MyTest(TestCase):
    def setUp(self):
        # setup variables for testing
        self.x = range(10)
    def test_sum(self):
        # function performs test
        assert sum(self.x) == 45

if __name__ == '__main__':
    main() # this runs the tests

Running this code should produce the following output:

----------------------------------------------------------------------
Ran 1 test in 0.001s

OK

This indicates that the test context (usually called the test fixture) was created and the test function was run with a successful outcome. In case, something goes wrong, you should see something like the following

======================================================================
FAIL: test_sum (__main__.MyTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "testing_codes_001.py", line 9, in test_sum
    assert sum(self.x) == 455
AssertionError
----------------------------------------------------------------------
Ran 1 test in 0.001s
FAILED (failures=1)

This gives detail regarding the nature of the failure and where it occurred in the test. When you start developing your own code as a separate module, you can include your code in the test file using import and then have the individual test functions use them. That way, you can be sure to keep the testing codes separate from your main code development which minimizes the chance that you will make the same mistake in both codes. It is not necessary to write a separate test case for every function in your code, but you should write tests that exercise common code pathways. This means that many functions can be called for a single test, but there should not be so many functions aggregated into a single test that it becomes hard to debug or potentially obscures errors between functions.

Although unit testing dramatically improves code quality, it is important to realize that the testing is only as meaningful as the individual tests and the coding discipline when developing and using tests. In other words, tests that do not evolve with the code or do not represent common run-time situations are not ultimately helpful, and can provide a false sense of code quality.

The unittest module provides many other ways to assert the results of tests.

Testing with Nose

Nose is a third-party Python module that provides additional functionality and is a superset of the tools provided in unittest. notably, nose tests do not have to use inheritance to work. Nose provides a powerful configuration framework that can provide very low-level granular control over the individual tests. Most important, Nose integrates with configurable code coverage tools that will report the percentage of a codebase that was exercised by a given suite of tests. A common way to use Nose is the following

Terminal> nosetests -c noseconfig

The nosetests program then reads the configuration settings in the noseconfig file and executes and reports on all of the tests (or a subset of the tests) provided. Sometimes it is not feasible to always run the deep and comprehensive tests that you have developed but rather to run a small subset as part of a continuous integration plan wherein tests are run continuously as part of active development. Such smaller tests are sometimes called smoke tests because they are sensitive enough upon failure to raise suspicions about deeper problems in the code that should be investigated with more comprehensive testing, but are still fast enough to run as part of continuous development. Even though Nose has been in maintenance for the last few years it is still used in many Python codes.

Testing with Pytest

As is the case with open source software, some modules accrue more support and attention than others and become dominant in the marketplace of ideas. This is particularly true of pytest which is now the dominant third-party Python testing framework. Pytest provides a superset of the functionality provided by unittest and nosetest so migrating to Pytest should be straightforward. However, Pytest provides many powerful new abstractions and much more in-depth detail for the performance of tests, which makes them more useful overall in terms of debugging and maintenance. Consider the following output code:

def test_tuples_successful():
    # no problems here!
    assert (1,2,3) == (1,2,3)

def test_tuples_unsuccessful():
    # notice the positions of the tuple elements
    assert (1,2,3) == (1,3,2)

We can run this with pytest as in the following:

Terminal> pytest testing_codes_002.py

with the following output:

======================================= FAILURES ========================================
_______________________________ test_tuples_unsuccessful ________________________________

    def test_tuples_unsuccessful():
>       assert (1,2,3) == (1,3,2)
E       assert (1, 2, 3) == (1, 3, 2)
E         At index 1 diff: 2 != 3
E         Use -v to get the full diff

testing_codes_002.py:5: AssertionError
========================== 1 failed, 1 passed in 0.08 seconds ===========================

The great part about this output as compared to unittest is the level of detail. Notice that the output tells you exactly where the failure occurred in the code, namely At index 1. Passing the -v flag to pytest can produce a file diff that contains the formatted difference between what was expected and was supplied for the given test. Under the hood, pytest hijacks Python’s assertion machinery to make reports more specfic.

For scientific applications, it is usually important to test for approximate equality of floating-point numbers. This can be done with pytest.approx as in the following:

a = 1.34
assert pytest.approx(1.33,0.1) == a

This says that equality should match to a relative decision of 10 percent. There are options for absolute position also.

Pytest Features

Pytest supports categorizing (i.e., marking) certain tests. This makes it possible to choose when certain intensive tests are run or skipped. For example, if want to mark a test as a smoke test (a test that is quick but can eliminate bigger problems that require deeper introspection) the @pytest.mark.smoke on the test functions in the named file means that it will be invoked using the following command line

Terminal> pytest -m smoke test_function.py

You can have multiple so-marked tests and call them using some basic Python logic.

Terminal> pytest -m 'smoke and db' test_function.py

where other functions have been marked with the decorator pytest.mark.db. You can make up your own names for marks. Tests that are marked with pytest.mark.skip are skipped otherwise. You can also supply an argument to this decorator with a keyword argument reason for documentation purposes. There is also a pytest.mark.skipif decorator that takes a position argument that is any valid Python code for the conditional along with the usual reason keyword argument. Tests that are expected to fail can likewise be marked with the pytest.mark.xfail decorator.

Outside of using markers to categorize and run tests, Pytest provides commandline arguments for drilling down to specific functions in directories that you want to run.

Terminal> pytest <full-path-to-filename>
Terminal> pytest <full-path-to-filename>::individual_test_function

The last line uses the :: syntax to specify the individual test function you want. This works for test classes also. For example, if you want to exercise a specific method test of a class, you can specify that with the following command line using more :: characters.

Terminal> pytest filename::TestClass::test_method

Furthermore, Pytest provides a -k flag that will filter on all functions that match the input string. These input strings can be logically combined using basic Python logical expressions, as discussed above.

Pytest uses fixtures via pytest.fixture for setup and teardown. These can be shared and scoped.

@pytest.fixture()
def setup_data():
   return 10

def test_data(setup_data):
   assert setup_data() == 10

Note that the test uses the output of the text fixture in its assert. Pytest uses the conftest.py function to organize and share fixtures. You do not have to import from this file because it is automatically read by Pytest. Given test function can use multiple test fixtures specified in the function signature. Each can have corresponding scope,

@pytest.fixture(scope='function')
def function_scope():
    'run once per test function'
    pass


@pytest.fixture(scope='module')
def module_scope():
   'run once per module regardless of how many test functions or methods are used'
   pass

@pytest.fixture(scope='session')
def session_scope():
   'run once per session. All test methods and functions share one setup and teardown'
    pass


@pytest.fixture(scope='class')
def class_scope():
   'run once per test class regardless of how many test methods are in the class'
    pass

def test_function(function_scope, session_scope, module_scope)
   pass

Because this can easily get very complicated, Pytest provides the --setup-show command line argument to debug how this works. If it gets too cumbersome to provide the fixtures as part of the test function signature, then you can use the pytest.mark.usefixtures decorator whos argument is the fixture name (e.g., pytest.mark.usefixtures(module_scope)). There is also a very useful --pdb flag which will drop into the usual Python debugger if a test fails. Naturally, all of this complexity needs management and configuration and there are other Pytest features that make this easy to control with nested scopes for test fixtures. Interestingly and relevant for data applications, Pytest also has a pytest.parameterize decorator 1 that can from cartesian products over the inputs, which avoids the tedium of creating separate test cases for different combinations of inputs, especially with a long function signature.

Using Hypothesis for Testing

The hypothesis package helps with more complex testing by automatically generating test cases using your defined requirements. The key concept is to carefully define properties of the functions you want to test as well as the data they ingest. Consider the following code,

from hypothesis import given
from hypothesis.strategies import integers

def adder(x,y):
   return x+y

@given(integers(),integers())
def test_adder(x,y):
    # addition should commute
    assert adder(x,y)==adder(y,x)

So that running the test_adder() function makes the text() strategy generate many different test inputs that are used in the assert to indicate whether or not the test has passed. Hypothesis generates many challenging input samples so that running the test_adder function takes substantially longer than a regular single-input test. The number of so-generated inputs can be configured with the max_size keyword argument to integers(). The most important thing about hypothesis is that because it automatically generates specific values for your test cases, it allows you to think more abstractly about what attributes your functions should have. In the above example, the test is checking for the attribute of commutative property of addition. In in addition to the usual built-in Python types, hypothesis can generate Pandas elements for testing Dataframes and Series objects. The following block creates a test case for a Pandas Series object.

from hypothesis.extra.pandas import series
series(dtype=int).example()

Here is an example for a Pandas Dataframe

from hypothesis.extra.pandas import data_frames,columns

def summer(df):
   return df[['A','B']].sum()

@given(data_frames(columns=columns(['A','B'],dtype=float)))
def test_summer(df):
   assert summer(df).sum()>=0

test_summer()

This test will fail because there is nothing in the logic of the code to ensure that the output is non-empty.

Well-structured code with its own assert need special handling using assume, as shown below,

from hypothesis import assume
import hypothesis.strategies as st
from hypothesis.strategies import integers, floats

def foo(x):
   assert x>0
   return 2*x

@given(integers()|floats())
def test_foo(x):
   assume(x>0)
   assert foo(x)>0

Data structures like lists and directories are supported. For example, if the function input is a list then st.lists(st.integers(),min_size=3) can generate test input lists of integers of length three or more. Likewise, st.fixed_dictionaries can generate test dictionaries with a specified strategy for each key.

Strategies can be individually managed using map and filter methods on the strategies. For example, st.integers().map(str) will convert degenerated integers into strings and st.integers().filter(lambda i:i>=0) will exclude negative integers, even though this can be inefficient if many random integers are generated they do not satisfy the predicate. The st.just() strategy will only generate a specific value and the st.sampled_from() will choose an element from a given sequence.

Strategies can be combined using one_of() which is similar to generating a union. The nothing() strategy plays the role of None. This means that st.one_of(st.integers(), st.nothing()) == st.integers()

Custom strategies can be implemented with builds() which generates values and then feeds them into a target function that produces the test function input arguments. If the target function is type-annotated, then builds() will use that to infer the corresponding strategies for the target’s inputs. The st.register_type_strategy() function can also infer the correct strategies from type-annotations. Likewise the st.from_regex(), st.from_dtype(), and st.from_field(). Further advanced methods for creating recursive strategies are implemented via st.recursive() , which is useful for testing nested data structures like JSON objects. The st.composite() function allows even more customized strategies that can combine other strategies beyond what st.filter() and st.map() can provide. There are also user-contributed third-party extensions that provide other strategies. Hypothesis provides a ghost-writing module that can help generate test logic, which can be helpful if you are modeling your functions against ones that hypothesis knows about (e.g., sorted).

More abstractly, Hypothesis can check certain properties of functions, such as idempotence, which is the property that repeated function calls do not change the output of the first function application. For example, the built-in sorted function is idempotent because once a sequence is sorted the applying sorted again does not change the input.

Hypothesis supports many other advanced use cases and is under constant development. Nonetheless, even at this point in its development, it provides a powerful framework for abstracting and automating complex tests that can thoroughly enforce code behaviors.

https://ece143.unpingco.com/testing_codes.html