1:mod:`functools` --- Higher-order functions and operations on callable objects 2============================================================================== 3 4.. module:: functools 5 :synopsis: Higher-order functions and operations on callable objects. 6 7.. moduleauthor:: Peter Harris <scav@blueyonder.co.uk> 8.. moduleauthor:: Raymond Hettinger <python@rcn.com> 9.. moduleauthor:: Nick Coghlan <ncoghlan@gmail.com> 10.. moduleauthor:: Łukasz Langa <lukasz@langa.pl> 11.. sectionauthor:: Peter Harris <scav@blueyonder.co.uk> 12 13**Source code:** :source:`Lib/functools.py` 14 15-------------- 16 17The :mod:`functools` module is for higher-order functions: functions that act on 18or return other functions. In general, any callable object can be treated as a 19function for the purposes of this module. 20 21The :mod:`functools` module defines the following functions: 22 23.. function:: cmp_to_key(func) 24 25 Transform an old-style comparison function to a :term:`key function`. Used 26 with tools that accept key functions (such as :func:`sorted`, :func:`min`, 27 :func:`max`, :func:`heapq.nlargest`, :func:`heapq.nsmallest`, 28 :func:`itertools.groupby`). This function is primarily used as a transition 29 tool for programs being converted from Python 2 which supported the use of 30 comparison functions. 31 32 A comparison function is any callable that accept two arguments, compares them, 33 and returns a negative number for less-than, zero for equality, or a positive 34 number for greater-than. A key function is a callable that accepts one 35 argument and returns another value to be used as the sort key. 36 37 Example:: 38 39 sorted(iterable, key=cmp_to_key(locale.strcoll)) # locale-aware sort order 40 41 For sorting examples and a brief sorting tutorial, see :ref:`sortinghowto`. 42 43 .. versionadded:: 3.2 44 45 46.. decorator:: lru_cache(maxsize=128, typed=False) 47 48 Decorator to wrap a function with a memoizing callable that saves up to the 49 *maxsize* most recent calls. It can save time when an expensive or I/O bound 50 function is periodically called with the same arguments. 51 52 Since a dictionary is used to cache results, the positional and keyword 53 arguments to the function must be hashable. 54 55 Distinct argument patterns may be considered to be distinct calls with 56 separate cache entries. For example, `f(a=1, b=2)` and `f(b=2, a=1)` 57 differ in their keyword argument order and may have two separate cache 58 entries. 59 60 If *maxsize* is set to ``None``, the LRU feature is disabled and the cache can 61 grow without bound. The LRU feature performs best when *maxsize* is a 62 power-of-two. 63 64 If *typed* is set to true, function arguments of different types will be 65 cached separately. For example, ``f(3)`` and ``f(3.0)`` will be treated 66 as distinct calls with distinct results. 67 68 To help measure the effectiveness of the cache and tune the *maxsize* 69 parameter, the wrapped function is instrumented with a :func:`cache_info` 70 function that returns a :term:`named tuple` showing *hits*, *misses*, 71 *maxsize* and *currsize*. In a multi-threaded environment, the hits 72 and misses are approximate. 73 74 The decorator also provides a :func:`cache_clear` function for clearing or 75 invalidating the cache. 76 77 The original underlying function is accessible through the 78 :attr:`__wrapped__` attribute. This is useful for introspection, for 79 bypassing the cache, or for rewrapping the function with a different cache. 80 81 An `LRU (least recently used) cache 82 <https://en.wikipedia.org/wiki/Cache_algorithms#Examples>`_ works 83 best when the most recent calls are the best predictors of upcoming calls (for 84 example, the most popular articles on a news server tend to change each day). 85 The cache's size limit assures that the cache does not grow without bound on 86 long-running processes such as web servers. 87 88 In general, the LRU cache should only be used when you want to reuse 89 previously computed values. Accordingly, it doesn't make sense to cache 90 functions with side-effects, functions that need to create distinct mutable 91 objects on each call, or impure functions such as time() or random(). 92 93 Example of an LRU cache for static web content:: 94 95 @lru_cache(maxsize=32) 96 def get_pep(num): 97 'Retrieve text of a Python Enhancement Proposal' 98 resource = 'http://www.python.org/dev/peps/pep-%04d/' % num 99 try: 100 with urllib.request.urlopen(resource) as s: 101 return s.read() 102 except urllib.error.HTTPError: 103 return 'Not Found' 104 105 >>> for n in 8, 290, 308, 320, 8, 218, 320, 279, 289, 320, 9991: 106 ... pep = get_pep(n) 107 ... print(n, len(pep)) 108 109 >>> get_pep.cache_info() 110 CacheInfo(hits=3, misses=8, maxsize=32, currsize=8) 111 112 Example of efficiently computing 113 `Fibonacci numbers <https://en.wikipedia.org/wiki/Fibonacci_number>`_ 114 using a cache to implement a 115 `dynamic programming <https://en.wikipedia.org/wiki/Dynamic_programming>`_ 116 technique:: 117 118 @lru_cache(maxsize=None) 119 def fib(n): 120 if n < 2: 121 return n 122 return fib(n-1) + fib(n-2) 123 124 >>> [fib(n) for n in range(16)] 125 [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610] 126 127 >>> fib.cache_info() 128 CacheInfo(hits=28, misses=16, maxsize=None, currsize=16) 129 130 .. versionadded:: 3.2 131 132 .. versionchanged:: 3.3 133 Added the *typed* option. 134 135.. decorator:: total_ordering 136 137 Given a class defining one or more rich comparison ordering methods, this 138 class decorator supplies the rest. This simplifies the effort involved 139 in specifying all of the possible rich comparison operations: 140 141 The class must define one of :meth:`__lt__`, :meth:`__le__`, 142 :meth:`__gt__`, or :meth:`__ge__`. 143 In addition, the class should supply an :meth:`__eq__` method. 144 145 For example:: 146 147 @total_ordering 148 class Student: 149 def _is_valid_operand(self, other): 150 return (hasattr(other, "lastname") and 151 hasattr(other, "firstname")) 152 def __eq__(self, other): 153 if not self._is_valid_operand(other): 154 return NotImplemented 155 return ((self.lastname.lower(), self.firstname.lower()) == 156 (other.lastname.lower(), other.firstname.lower())) 157 def __lt__(self, other): 158 if not self._is_valid_operand(other): 159 return NotImplemented 160 return ((self.lastname.lower(), self.firstname.lower()) < 161 (other.lastname.lower(), other.firstname.lower())) 162 163 .. note:: 164 165 While this decorator makes it easy to create well behaved totally 166 ordered types, it *does* come at the cost of slower execution and 167 more complex stack traces for the derived comparison methods. If 168 performance benchmarking indicates this is a bottleneck for a given 169 application, implementing all six rich comparison methods instead is 170 likely to provide an easy speed boost. 171 172 .. versionadded:: 3.2 173 174 .. versionchanged:: 3.4 175 Returning NotImplemented from the underlying comparison function for 176 unrecognised types is now supported. 177 178.. function:: partial(func, *args, **keywords) 179 180 Return a new :ref:`partial object<partial-objects>` which when called 181 will behave like *func* called with the positional arguments *args* 182 and keyword arguments *keywords*. If more arguments are supplied to the 183 call, they are appended to *args*. If additional keyword arguments are 184 supplied, they extend and override *keywords*. 185 Roughly equivalent to:: 186 187 def partial(func, *args, **keywords): 188 def newfunc(*fargs, **fkeywords): 189 newkeywords = keywords.copy() 190 newkeywords.update(fkeywords) 191 return func(*args, *fargs, **newkeywords) 192 newfunc.func = func 193 newfunc.args = args 194 newfunc.keywords = keywords 195 return newfunc 196 197 The :func:`partial` is used for partial function application which "freezes" 198 some portion of a function's arguments and/or keywords resulting in a new object 199 with a simplified signature. For example, :func:`partial` can be used to create 200 a callable that behaves like the :func:`int` function where the *base* argument 201 defaults to two: 202 203 >>> from functools import partial 204 >>> basetwo = partial(int, base=2) 205 >>> basetwo.__doc__ = 'Convert base 2 string to an int.' 206 >>> basetwo('10010') 207 18 208 209 210.. class:: partialmethod(func, *args, **keywords) 211 212 Return a new :class:`partialmethod` descriptor which behaves 213 like :class:`partial` except that it is designed to be used as a method 214 definition rather than being directly callable. 215 216 *func* must be a :term:`descriptor` or a callable (objects which are both, 217 like normal functions, are handled as descriptors). 218 219 When *func* is a descriptor (such as a normal Python function, 220 :func:`classmethod`, :func:`staticmethod`, :func:`abstractmethod` or 221 another instance of :class:`partialmethod`), calls to ``__get__`` are 222 delegated to the underlying descriptor, and an appropriate 223 :ref:`partial object<partial-objects>` returned as the result. 224 225 When *func* is a non-descriptor callable, an appropriate bound method is 226 created dynamically. This behaves like a normal Python function when 227 used as a method: the *self* argument will be inserted as the first 228 positional argument, even before the *args* and *keywords* supplied to 229 the :class:`partialmethod` constructor. 230 231 Example:: 232 233 >>> class Cell(object): 234 ... def __init__(self): 235 ... self._alive = False 236 ... @property 237 ... def alive(self): 238 ... return self._alive 239 ... def set_state(self, state): 240 ... self._alive = bool(state) 241 ... set_alive = partialmethod(set_state, True) 242 ... set_dead = partialmethod(set_state, False) 243 ... 244 >>> c = Cell() 245 >>> c.alive 246 False 247 >>> c.set_alive() 248 >>> c.alive 249 True 250 251 .. versionadded:: 3.4 252 253 254.. function:: reduce(function, iterable[, initializer]) 255 256 Apply *function* of two arguments cumulatively to the items of *sequence*, from 257 left to right, so as to reduce the sequence to a single value. For example, 258 ``reduce(lambda x, y: x+y, [1, 2, 3, 4, 5])`` calculates ``((((1+2)+3)+4)+5)``. 259 The left argument, *x*, is the accumulated value and the right argument, *y*, is 260 the update value from the *sequence*. If the optional *initializer* is present, 261 it is placed before the items of the sequence in the calculation, and serves as 262 a default when the sequence is empty. If *initializer* is not given and 263 *sequence* contains only one item, the first item is returned. 264 265 Roughly equivalent to:: 266 267 def reduce(function, iterable, initializer=None): 268 it = iter(iterable) 269 if initializer is None: 270 value = next(it) 271 else: 272 value = initializer 273 for element in it: 274 value = function(value, element) 275 return value 276 277 278.. decorator:: singledispatch 279 280 Transform a function into a :term:`single-dispatch <single 281 dispatch>` :term:`generic function`. 282 283 To define a generic function, decorate it with the ``@singledispatch`` 284 decorator. Note that the dispatch happens on the type of the first argument, 285 create your function accordingly:: 286 287 >>> from functools import singledispatch 288 >>> @singledispatch 289 ... def fun(arg, verbose=False): 290 ... if verbose: 291 ... print("Let me just say,", end=" ") 292 ... print(arg) 293 294 To add overloaded implementations to the function, use the :func:`register` 295 attribute of the generic function. It is a decorator. For functions 296 annotated with types, the decorator will infer the type of the first 297 argument automatically:: 298 299 >>> @fun.register 300 ... def _(arg: int, verbose=False): 301 ... if verbose: 302 ... print("Strength in numbers, eh?", end=" ") 303 ... print(arg) 304 ... 305 >>> @fun.register 306 ... def _(arg: list, verbose=False): 307 ... if verbose: 308 ... print("Enumerate this:") 309 ... for i, elem in enumerate(arg): 310 ... print(i, elem) 311 312 For code which doesn't use type annotations, the appropriate type 313 argument can be passed explicitly to the decorator itself:: 314 315 >>> @fun.register(complex) 316 ... def _(arg, verbose=False): 317 ... if verbose: 318 ... print("Better than complicated.", end=" ") 319 ... print(arg.real, arg.imag) 320 ... 321 322 323 To enable registering lambdas and pre-existing functions, the 324 :func:`register` attribute can be used in a functional form:: 325 326 >>> def nothing(arg, verbose=False): 327 ... print("Nothing.") 328 ... 329 >>> fun.register(type(None), nothing) 330 331 The :func:`register` attribute returns the undecorated function which 332 enables decorator stacking, pickling, as well as creating unit tests for 333 each variant independently:: 334 335 >>> @fun.register(float) 336 ... @fun.register(Decimal) 337 ... def fun_num(arg, verbose=False): 338 ... if verbose: 339 ... print("Half of your number:", end=" ") 340 ... print(arg / 2) 341 ... 342 >>> fun_num is fun 343 False 344 345 When called, the generic function dispatches on the type of the first 346 argument:: 347 348 >>> fun("Hello, world.") 349 Hello, world. 350 >>> fun("test.", verbose=True) 351 Let me just say, test. 352 >>> fun(42, verbose=True) 353 Strength in numbers, eh? 42 354 >>> fun(['spam', 'spam', 'eggs', 'spam'], verbose=True) 355 Enumerate this: 356 0 spam 357 1 spam 358 2 eggs 359 3 spam 360 >>> fun(None) 361 Nothing. 362 >>> fun(1.23) 363 0.615 364 365 Where there is no registered implementation for a specific type, its 366 method resolution order is used to find a more generic implementation. 367 The original function decorated with ``@singledispatch`` is registered 368 for the base ``object`` type, which means it is used if no better 369 implementation is found. 370 371 To check which implementation will the generic function choose for 372 a given type, use the ``dispatch()`` attribute:: 373 374 >>> fun.dispatch(float) 375 <function fun_num at 0x1035a2840> 376 >>> fun.dispatch(dict) # note: default implementation 377 <function fun at 0x103fe0000> 378 379 To access all registered implementations, use the read-only ``registry`` 380 attribute:: 381 382 >>> fun.registry.keys() 383 dict_keys([<class 'NoneType'>, <class 'int'>, <class 'object'>, 384 <class 'decimal.Decimal'>, <class 'list'>, 385 <class 'float'>]) 386 >>> fun.registry[float] 387 <function fun_num at 0x1035a2840> 388 >>> fun.registry[object] 389 <function fun at 0x103fe0000> 390 391 .. versionadded:: 3.4 392 393 .. versionchanged:: 3.7 394 The :func:`register` attribute supports using type annotations. 395 396 397.. function:: update_wrapper(wrapper, wrapped, assigned=WRAPPER_ASSIGNMENTS, updated=WRAPPER_UPDATES) 398 399 Update a *wrapper* function to look like the *wrapped* function. The optional 400 arguments are tuples to specify which attributes of the original function are 401 assigned directly to the matching attributes on the wrapper function and which 402 attributes of the wrapper function are updated with the corresponding attributes 403 from the original function. The default values for these arguments are the 404 module level constants ``WRAPPER_ASSIGNMENTS`` (which assigns to the wrapper 405 function's ``__module__``, ``__name__``, ``__qualname__``, ``__annotations__`` 406 and ``__doc__``, the documentation string) and ``WRAPPER_UPDATES`` (which 407 updates the wrapper function's ``__dict__``, i.e. the instance dictionary). 408 409 To allow access to the original function for introspection and other purposes 410 (e.g. bypassing a caching decorator such as :func:`lru_cache`), this function 411 automatically adds a ``__wrapped__`` attribute to the wrapper that refers to 412 the function being wrapped. 413 414 The main intended use for this function is in :term:`decorator` functions which 415 wrap the decorated function and return the wrapper. If the wrapper function is 416 not updated, the metadata of the returned function will reflect the wrapper 417 definition rather than the original function definition, which is typically less 418 than helpful. 419 420 :func:`update_wrapper` may be used with callables other than functions. Any 421 attributes named in *assigned* or *updated* that are missing from the object 422 being wrapped are ignored (i.e. this function will not attempt to set them 423 on the wrapper function). :exc:`AttributeError` is still raised if the 424 wrapper function itself is missing any attributes named in *updated*. 425 426 .. versionadded:: 3.2 427 Automatic addition of the ``__wrapped__`` attribute. 428 429 .. versionadded:: 3.2 430 Copying of the ``__annotations__`` attribute by default. 431 432 .. versionchanged:: 3.2 433 Missing attributes no longer trigger an :exc:`AttributeError`. 434 435 .. versionchanged:: 3.4 436 The ``__wrapped__`` attribute now always refers to the wrapped 437 function, even if that function defined a ``__wrapped__`` attribute. 438 (see :issue:`17482`) 439 440 441.. decorator:: wraps(wrapped, assigned=WRAPPER_ASSIGNMENTS, updated=WRAPPER_UPDATES) 442 443 This is a convenience function for invoking :func:`update_wrapper` as a 444 function decorator when defining a wrapper function. It is equivalent to 445 ``partial(update_wrapper, wrapped=wrapped, assigned=assigned, updated=updated)``. 446 For example:: 447 448 >>> from functools import wraps 449 >>> def my_decorator(f): 450 ... @wraps(f) 451 ... def wrapper(*args, **kwds): 452 ... print('Calling decorated function') 453 ... return f(*args, **kwds) 454 ... return wrapper 455 ... 456 >>> @my_decorator 457 ... def example(): 458 ... """Docstring""" 459 ... print('Called example function') 460 ... 461 >>> example() 462 Calling decorated function 463 Called example function 464 >>> example.__name__ 465 'example' 466 >>> example.__doc__ 467 'Docstring' 468 469 Without the use of this decorator factory, the name of the example function 470 would have been ``'wrapper'``, and the docstring of the original :func:`example` 471 would have been lost. 472 473 474.. _partial-objects: 475 476:class:`partial` Objects 477------------------------ 478 479:class:`partial` objects are callable objects created by :func:`partial`. They 480have three read-only attributes: 481 482 483.. attribute:: partial.func 484 485 A callable object or function. Calls to the :class:`partial` object will be 486 forwarded to :attr:`func` with new arguments and keywords. 487 488 489.. attribute:: partial.args 490 491 The leftmost positional arguments that will be prepended to the positional 492 arguments provided to a :class:`partial` object call. 493 494 495.. attribute:: partial.keywords 496 497 The keyword arguments that will be supplied when the :class:`partial` object is 498 called. 499 500:class:`partial` objects are like :class:`function` objects in that they are 501callable, weak referencable, and can have attributes. There are some important 502differences. For instance, the :attr:`~definition.__name__` and :attr:`__doc__` attributes 503are not created automatically. Also, :class:`partial` objects defined in 504classes behave like static methods and do not transform into bound methods 505during instance attribute look-up. 506