Lines Matching +full:is +full:- +full:generator +full:- +full:function
11 :term:`iterator`\s and :term:`generator`\s and relevant library modules such as
20 skip to the next section on :ref:`functional-howto-iterators`.
30 perform the computation efficiently. SQL is the declarative language you're
35 * **Object-oriented** programs manipulate collections of objects. Objects have
37 some way. Smalltalk and Java are object-oriented languages. C++ and Python
38 are languages that support object-oriented programming, but don't force the
39 use of object-oriented features.
43 internal state that affects the output produced for a given input. Well-known
50 multi-paradigm languages that support several different approaches.
51 Lisp, C++, and Python are multi-paradigm; you can write programs or
52 libraries that are largely procedural, object-oriented, or functional
55 object-oriented while the processing logic is procedural or
58 In a functional program, input flows through a set of functions. Each function
61 that aren't visible in the function's return value. Functions that have no side
63 not using data structures that get updated as a program runs; every function's
69 example is a call to the :func:`print` or :func:`time.sleep` function, neither
75 functional-appearing interface but will use non-functional features internally.
76 For example, the implementation of a function will still use assignments to
79 Functional programming can be considered the opposite of object-oriented
85 that take and return instances representing objects in your application (e-mail
99 ------------------
101 A theoretical benefit is that it's easier to construct a mathematical proof that
102 a functional program is correct.
105 mathematically prove programs correct. This is different from testing a program
106 on numerous inputs and concluding that its output is usually correct, or reading
107 a program's source code and concluding that the code looks right; the goal is
111 The technique used to prove programs correct is to write down **invariants**,
114 **before** the line is executed, the slightly different invariants X' and Y' are
115 true **after** the line is executed. This continues until you reach the end of
124 Unfortunately, proving programs correct is largely impractical and not relevant
135 ----------
137 A more practical benefit of functional programming is that it forces you to
139 result. It's easier to specify and write a small function that does one thing
140 than a large function that performs a complicated transformation. Small
145 -----------------------------
147 Testing and debugging a functional-style program is easier.
149 Debugging is simplified because functions are generally small and clearly
150 specified. When a program doesn't work, each function is an interface point
152 inputs and outputs to quickly isolate the function that's responsible for a bug.
154 Testing is easier because each function is a potential subject for a unit test.
161 -------------
163 As you work on a functional-style program, you'll write a number of functions
166 variety of programs. For example, a function that takes a directory path and
167 returns all the XML files in the directory, or a function that takes a filename
175 .. _functional-howto-iterators:
181 foundation for writing functional-style programs: iterators.
183 An iterator is an object representing a stream of data; this object returns the
191 The built-in :func:`iter` function takes an arbitrary object and tries to return
194 built-in data types support iteration, the most common being lists and
195 dictionaries. An object is called :term:`iterable` if you can get an iterator
238 N elements, you can unpack them into an N-tuple:
246 Built-in functions such as :func:`max` and :func:`min` can take a single
248 and ``"not in"`` operators also support iterators: ``X in iterator`` is true if
249 X is found in the stream returned by the iterator. You'll run into obvious
250 problems if the iterator is infinite; :func:`max`, :func:`min`
264 ---------------------------------
290 Note that starting with Python 3.7, dictionary iteration order is guaranteed
330 Generator expressions and list comprehensions
339 List comprehensions and generator expressions (short form: "listcomps" and
346 >>> # Generator expression -- returns iterator
349 >>> # List comprehension -- returns list
357 With a list comprehension, you get back a Python list; ``stripped_list`` is a
358 list containing the resulting lines, not an iterator. Generator expressions
362 large amount of data. Generator expressions are preferable in these situations.
364 Generator expressions are surrounded by parentheses ("()") and list
365 comprehensions are surrounded by square brackets ("[]"). Generator expressions
383 is only evaluated and added to the result when ``condition`` is true.
385 Generator expressions always have to be written inside parentheses, but the
386 parentheses signalling a function call also count. If you want to create an
387 iterator that will be immediately passed to a function you can write::
394 ``sequence2`` is looped over from the beginning. ``sequence3`` is then looped
397 To put it another way, a list comprehension or generator expression is
417 list is 9 elements long:
426 To avoid introducing an ambiguity into Python's grammar, if ``expression`` is
428 comprehension below is a syntax error, while the second one is correct::
443 You're doubtless familiar with how regular function calls work in Python or C.
444 When you call a function, it gets a private namespace where its local variables
445 are created. When the function reaches a ``return`` statement, the local
446 variables are destroyed and the value is returned to the caller. A later call
447 to the same function creates a new private namespace and a fresh set of local
449 function? What if you could later resume the function where it left off? This
450 is what generators provide; they can be thought of as resumable functions.
452 Here's the simplest example of a generator function:
458 Any function containing a :keyword:`yield` keyword is a generator function;
459 this is detected by Python's :term:`bytecode` compiler which compiles the
460 function specially as a result.
462 When you call a generator function, it doesn't return a single value; instead it
463 returns a generator object that supports the iterator protocol. On executing
464 the ``yield`` expression, the generator outputs the value of ``i``, similar to a
466 statement is that on reaching a ``yield`` the generator's state of execution is
468 generator's :meth:`~generator.__next__` method, the function will resume
471 Here's a sample usage of the ``generate_ints()`` generator:
475 <generator object generate_ints at ...>
491 Inside a generator function, ``return value`` causes ``StopIteration(value)``
492 to be raised from the :meth:`~generator.__next__` method. Once this happens, or
493 the bottom of the function is reached, the procession of values ends and the
494 generator cannot yield any further values.
497 and storing all the local variables of the generator as instance variables. For
501 However, for a moderately complicated generator, writing a corresponding class
506 a number of more interesting examples. Here's one generator that implements an
507 in-order traversal of a tree using generators recursively. ::
509 # A recursive generator that generates Tree leaves in in-order.
520 Two other examples in ``test_generators.py`` produce solutions for the N-Queens
527 Passing values into a generator
528 -------------------------------
530 In Python 2.4 and earlier, generators only produced output. Once a generator's
532 information into the function when its execution is resumed. You could hack
533 together this ability by making the generator look at a global variable or by
537 In Python 2.5 there's a simple way to pass values into a generator.
548 (:pep:`342` explains the exact rules, which are that a ``yield``-expression must
549 always be parenthesized except when it occurs at the top-level expression on the
550 right-hand side of an assignment. This means you can write ``val = yield i``
554 Values are sent into a generator by calling its :meth:`send(value)
555 <generator.send>` method. This method resumes the generator's code and the
557 :meth:`~generator.__next__` method is called, the ``yield`` returns ``None``.
569 if val is not None:
593 :meth:`~generator.send` method will be the only method used to resume your
594 generator function.
596 In addition to :meth:`~generator.send`, there are two other methods on
599 * :meth:`throw(value) <generator.throw>` is used to
600 raise an exception inside the generator; the exception is raised by the
601 ``yield`` expression where the generator's execution is paused.
603 * :meth:`~generator.close` raises a :exc:`GeneratorExit` exception inside the
604 generator to terminate the iteration. On receiving this exception, the
605 generator's code must either raise :exc:`GeneratorExit` or
606 :exc:`StopIteration`; catching the exception and doing anything else is
607 illegal and will trigger a :exc:`RuntimeError`. :meth:`~generator.close`
608 will also be called by Python's garbage collector when the generator is
609 garbage-collected.
614 The cumulative effect of these changes is to turn generators from one-way
619 function, and a ``return`` statement), but coroutines can be entered, exited,
623 Built-in functions
626 Let's look in more detail at built-in functions often used with iterators.
628 Two of Python's built-in functions, :func:`map` and :func:`filter` duplicate the
629 features of generator expressions:
645 sequence elements that meet a certain condition, and is similarly duplicated by
646 list comprehensions. A **predicate** is a function that returns the truth
664 iterable returning 2-tuples containing the count (from *start*) and
673 :func:`enumerate` is often used when looping through a list and recording the
699 The :func:`any(iter) <any>` and :func:`all(iter) <all>` built-ins look at the
701 in the iterable is a true value, and :func:`all` returns ``True`` if all of the
724 It doesn't construct an in-memory list and exhaust all the input iterators
726 requested. (The technical term for this behaviour is `lazy evaluation
729 This iterator is intended to be used with iterables that are all of the same
751 * Functions for treating an iterator's elements as function arguments.
753 * A function for grouping an iterator's output.
756 ----------------------
777 element *n* times, or returns the element endlessly if *n* is not provided. ::
795 get *stop-start* elements, and if you supply a value for *step*, elements
809 If you don't supply a value for *n*, the default is 2. Replicating iterators
811 significant memory if the iterator is large and one of the new iterators is
817 where iterA ->
820 and iterB ->
825 -----------------------------
845 ------------------
850 :func:`itertools.filterfalse(predicate, iter) <itertools.filterfalse>` is the
882 element of *selectors* is true, stopping whenever either one is exhausted::
889 ----------------------
892 returns an iterator giving all possible *r*-tuple combinations of the
907 *iterable* returned them. For example, the number 1 is always before
908 2, 3, 4, or 5 in the examples above. A similar function,
925 If you don't supply a value for *r* the length of the iterable is used,
939 function relaxes a different constraint: elements can be repeated
940 within a single tuple. Conceptually an element is selected for the
941 first position of each tuple and then is replaced before the second
942 element is selected. ::
953 -----------------
955 The last function I'll discuss, :func:`itertools.groupby(iter, key_func=None)
956 <itertools.groupby>`, is the most complicated. ``key_func(elem)`` is a function
958 don't supply a key function, the key is simply each element itself.
962 2-tuples containing a key value and an iterator for the elements with that key.
976 ('AL', iterator-1),
977 ('AK', iterator-2),
978 ('AZ', iterator-3), ...
981 iterator-1 =>
983 iterator-2 =>
985 iterator-3 =>
990 the underlying iterable, so you have to consume the results of iterator-1 before
991 requesting iterator-2 and its corresponding key.
997 The :mod:`functools` module contains some higher-order functions.
998 A **higher-order function** takes one or more functions as input and returns a
999 new function. The most useful tool in this module is the
1000 :func:`functools.partial` function.
1004 Consider a Python function ``f(a, b, c)``; you may wish to create a new function
1006 one of ``f()``'s parameters. This is called "partial function application".
1009 ``(function, arg1, arg2, ..., kwarg1=value1, kwarg2=value2)``. The resulting
1010 object is callable, so you can just call it to invoke ``function`` with the
1011 filled-in arguments.
1027 therefore, can't be applied to infinite iterables. *func* must be a function
1032 and continues until the iterable is exhausted. If the iterable returns no
1033 values at all, a :exc:`TypeError` exception is raised. If the initial value is
1034 supplied, it's used as a starting point and ``func(initial_value, A)`` is the
1050 elements of the iterable. This case is so common that there's a special
1051 built-in called :func:`sum` to compute it:
1073 A related function is :func:`itertools.accumulate(iterable, func=operator.add)
1086 -------------------
1090 in functional-style code because they save you from writing trivial functions
1107 When writing functional-style programs, you'll often need little functions that
1110 If there's a Python built-in or a module function that's suitable, you don't
1111 need to define a new function at all::
1116 If the function you need doesn't exist, you need to write it. One way to write
1117 small functions is to use the :keyword:`lambda` expression. ``lambda`` takes a
1119 an anonymous function that returns the value of the expression::
1125 An alternative is to just use the ``def`` statement and define a function in the
1134 Which alternative is preferable? That's a style question; my usual course is to
1137 One reason for my preference is that ``lambda`` is quite limited in the
1163 Or the :func:`sum` built-in and a generator expression::
1172 1. Write a lambda function.
1180 about whether this lambda-free style is better.
1208 -------
1215 design approaches described in these chapters are applicable to functional-style
1228 Python-specific
1229 ---------------
1232 :title-reference:`Text Processing in Python` discusses functional programming
1233 for text processing, in the section titled "Utilizing Higher-Order Functions in
1236 Mertz also wrote a 3-part series of articles on functional programming
1238 `part 1 <https://developer.ibm.com/articles/l-prog/>`__,
1239 `part 2 <https://developer.ibm.com/tutorials/l-prog2/>`__, and
1240 `part 3 <https://developer.ibm.com/tutorials/l-prog3/>`__,
1244 --------------------
1252 :pep:`289`: "Generator Expressions"
1254 :pep:`342`: "Coroutines via Enhanced Generators" describes the new generator
1259 Handy little function for printing part of an iterator -- used
1265 for elem in slice[:-1]:
1268 print(elem[-1])