1.. _tools: 2 3******************************************** 4Testing and Ensuring Type Annotation Quality 5******************************************** 6 7Testing Annotation Accuracy 8=========================== 9 10When creating a package with type annotations, authors may want to validate 11that the annotations they publish meet their expectations. 12This is especially important for library authors, for whom the published 13annotations are part of the public interface to their package. 14 15There are several approaches to this problem, and this document will show 16a few of them. 17 18.. note:: 19 20 For simplicity, we will assume that type-checking is done with ``mypy``. 21 Many of these strategies can be applied to other type-checkers as well. 22 23Testing Using ``mypy --warn-unused-ignores`` 24-------------------------------------------- 25 26Clever use of ``--warn-unused-ignores`` can be used to check that certain 27expressions are or are not well-typed. 28 29The idea is to write normal python files which contain valid expressions along 30with invalid expressions annotated with ``type: ignore`` comments. When 31``mypy --warn-unused-ignores`` is run on these files, it should pass. 32A directory of test files, ``typing_tests/``, can be maintained. 33 34This strategy does not offer strong guarantees about the types under test, but 35it requires no additional tooling. 36 37If the following file is under test 38 39.. code-block:: python 40 41 # foo.py 42 def bar(x: int) -> str: 43 return str(x) 44 45Then the following file tests ``foo.py``: 46 47.. code-block:: python 48 49 bar(42) 50 bar("42") # type: ignore [arg-type] 51 bar(y=42) # type: ignore [call-arg] 52 r1: str = bar(42) 53 r2: int = bar(42) # type: ignore [assignment] 54 55Checking ``reveal_type`` output from ``mypy.api.run`` 56----------------------------------------------------- 57 58``mypy`` provides a subpackage named ``api`` for invoking ``mypy`` from a 59python process. In combination with ``reveal_type``, this can be used to write 60a function which gets the ``reveal_type`` output from an expression. Once 61that's obtained, tests can assert strings and regular expression matches 62against it. 63 64This approach requires writing a set of helpers to provide a good testing 65experience, and it runs mypy once per test case (which can be slow). 66However, it builds only on ``mypy`` and the test framework of your choice. 67 68The following example could be integrated into a testsuite written in 69any framework: 70 71.. code-block:: python 72 73 import re 74 from mypy import api 75 76 def get_reveal_type_output(filename): 77 result = api.run([filename]) 78 stdout = result[0] 79 match = re.search(r'note: Revealed type is "([^"]+)"', stdout) 80 assert match is not None 81 return match.group(1) 82 83 84For example, we can use the above to provide a ``run_reveal_type`` pytest 85fixture which generates a temporary file and uses it as the input to 86``get_reveal_type_output``: 87 88.. code-block:: python 89 90 import os 91 import pytest 92 93 @pytest.fixture 94 def _in_tmp_path(tmp_path): 95 cur = os.getcwd() 96 try: 97 os.chdir(tmp_path) 98 yield 99 finally: 100 os.chdir(cur) 101 102 @pytest.fixture 103 def run_reveal_type(tmp_path, _in_tmp_path): 104 content_path = tmp_path / "reveal_type_test.py" 105 106 def func(code_snippet, *, preamble = ""): 107 content_path.write_text(preamble + f"reveal_type({code_snippet})") 108 return get_reveal_type_output("reveal_type_test.py") 109 110 return func 111 112 113For more details, see `the documentation on mypy.api 114<https://mypy.readthedocs.io/en/stable/extending_mypy.html#integrating-mypy-into-another-python-application>`_. 115 116pytest-mypy-plugins 117------------------- 118 119`pytest-mypy-plugins <https://github.com/typeddjango/pytest-mypy-plugins>`_ is 120a plugin for ``pytest`` which defines typing test cases as YAML data. 121The test cases are run through ``mypy`` and the output of ``reveal_type`` can 122be asserted. 123 124This project supports complex typing arrangements like ``pytest`` parametrized 125tests and per-test ``mypy`` configuration. It requires that you are using 126``pytest`` to run your tests, and runs ``mypy`` in a subprocess per test case. 127 128This is an example of a parametrized test with ``pytest-mypy-plugins``: 129 130.. code-block:: yaml 131 132 - case: with_params 133 parametrized: 134 - val: 1 135 rt: builtins.int 136 - val: 1.0 137 rt: builtins.float 138 main: | 139 reveal_type({[ val }}) # N: Revealed type is '{{ rt }}' 140 141Improving Type Completeness 142=========================== 143 144One of the goals of many libraries is to ensure that they are "fully type 145annotated", meaning that they provide complete and accurate type annotations 146for all functions, classes, and objects. Having full annotations is referred to 147as "type completeness" or "type coverage". 148 149Here are some tips for increasing the type completeness score for your 150library: 151 152- Make type completeness an output of your testing process. Several type 153 checkers have options for generating useful output, warnings, or even 154 reports. 155- If your package includes tests or sample code, consider removing them 156 from the distribution. If there is good reason to include them, 157 consider placing them in a directory that begins with an underscore 158 so they are not considered part of your library’s interface. 159- If your package includes submodules that are meant to be 160 implementation details, rename those files to begin with an 161 underscore. 162- If a symbol is not intended to be part of the library’s interface and 163 is considered an implementation detail, rename it such that it begins 164 with an underscore. It will then be considered private and excluded 165 from the type completeness check. 166- If your package exposes types from other libraries, work with the 167 maintainers of these other libraries to achieve type completeness. 168 169.. warning:: 170 171 The ways in which different type checkers evaluate and help you achieve 172 better type coverage may differ. Some of the above recommendations may or 173 may not be helpful to you, depending on which type checking tools you use. 174 175``mypy`` disallow options 176------------------------- 177 178``mypy`` offers several options which can detect untyped code. 179More details can be found in `the mypy documentation on these options 180<https://mypy.readthedocs.io/en/latest/command_line.html#untyped-definitions-and-calls>`_. 181 182Some basic usages which make ``mypy`` error on untyped data are:: 183 184 mypy --disallow-untyped-defs 185 mypy --disallow-incomplete-defs 186 187``pyright`` type verification 188----------------------------- 189 190pyright has a special command line flag, ``--verifytypes``, for verifying 191type completeness. You can learn more about it from 192`the pyright documentation on verifying type completeness 193<https://github.com/microsoft/pyright/blob/main/docs/typed-libraries.md#verifying-type-completeness>`_. 194 195``mypy`` reports 196---------------- 197 198``mypy`` offers several options options for generating reports on its analysis. 199See `the mypy documentation on report generation 200<https://mypy.readthedocs.io/en/stable/command_line.html#report-generation>`_ for details. 201