• Home
  • Line#
  • Scopes#
  • Navigate#
  • Raw
  • Download
1.. highlightlang:: c
2
3**********************
4Argument Clinic How-To
5**********************
6
7:author: Larry Hastings
8
9
10.. topic:: Abstract
11
12  Argument Clinic is a preprocessor for CPython C files.
13  Its purpose is to automate all the boilerplate involved
14  with writing argument parsing code for "builtins".
15  This document shows you how to convert your first C
16  function to work with Argument Clinic, and then introduces
17  some advanced topics on Argument Clinic usage.
18
19  Currently Argument Clinic is considered internal-only
20  for CPython.  Its use is not supported for files outside
21  CPython, and no guarantees are made regarding backwards
22  compatibility for future versions.  In other words: if you
23  maintain an external C extension for CPython, you're welcome
24  to experiment with Argument Clinic in your own code.  But the
25  version of Argument Clinic that ships with the next version
26  of CPython *could* be totally incompatible and break all your code.
27
28The Goals Of Argument Clinic
29============================
30
31Argument Clinic's primary goal
32is to take over responsibility for all argument parsing code
33inside CPython.  This means that, when you convert a function
34to work with Argument Clinic, that function should no longer
35do any of its own argument parsing—the code generated by
36Argument Clinic should be a "black box" to you, where CPython
37calls in at the top, and your code gets called at the bottom,
38with ``PyObject *args`` (and maybe ``PyObject *kwargs``)
39magically converted into the C variables and types you need.
40
41In order for Argument Clinic to accomplish its primary goal,
42it must be easy to use.  Currently, working with CPython's
43argument parsing library is a chore, requiring maintaining
44redundant information in a surprising number of places.
45When you use Argument Clinic, you don't have to repeat yourself.
46
47Obviously, no one would want to use Argument Clinic unless
48it's solving their problem—and without creating new problems of
49its own.
50So it's paramount that Argument Clinic generate correct code.
51It'd be nice if the code was faster, too, but at the very least
52it should not introduce a major speed regression.  (Eventually Argument
53Clinic *should* make a major speedup possible—we could
54rewrite its code generator to produce tailor-made argument
55parsing code, rather than calling the general-purpose CPython
56argument parsing library.  That would make for the fastest
57argument parsing possible!)
58
59Additionally, Argument Clinic must be flexible enough to
60work with any approach to argument parsing.  Python has
61some functions with some very strange parsing behaviors;
62Argument Clinic's goal is to support all of them.
63
64Finally, the original motivation for Argument Clinic was
65to provide introspection "signatures" for CPython builtins.
66It used to be, the introspection query functions would throw
67an exception if you passed in a builtin.  With Argument
68Clinic, that's a thing of the past!
69
70One idea you should keep in mind, as you work with
71Argument Clinic: the more information you give it, the
72better job it'll be able to do.
73Argument Clinic is admittedly relatively simple right
74now.  But as it evolves it will get more sophisticated,
75and it should be able to do many interesting and smart
76things with all the information you give it.
77
78
79Basic Concepts And Usage
80========================
81
82Argument Clinic ships with CPython; you'll find it in ``Tools/clinic/clinic.py``.
83If you run that script, specifying a C file as an argument:
84
85.. code-block:: shell-session
86
87    $ python3 Tools/clinic/clinic.py foo.c
88
89Argument Clinic will scan over the file looking for lines that
90look exactly like this:
91
92.. code-block:: none
93
94    /*[clinic input]
95
96When it finds one, it reads everything up to a line that looks
97exactly like this:
98
99.. code-block:: none
100
101    [clinic start generated code]*/
102
103Everything in between these two lines is input for Argument Clinic.
104All of these lines, including the beginning and ending comment
105lines, are collectively called an Argument Clinic "block".
106
107When Argument Clinic parses one of these blocks, it
108generates output.  This output is rewritten into the C file
109immediately after the block, followed by a comment containing a checksum.
110The Argument Clinic block now looks like this:
111
112.. code-block:: none
113
114    /*[clinic input]
115    ... clinic input goes here ...
116    [clinic start generated code]*/
117    ... clinic output goes here ...
118    /*[clinic end generated code: checksum=...]*/
119
120If you run Argument Clinic on the same file a second time, Argument Clinic
121will discard the old output and write out the new output with a fresh checksum
122line.  However, if the input hasn't changed, the output won't change either.
123
124You should never modify the output portion of an Argument Clinic block.  Instead,
125change the input until it produces the output you want.  (That's the purpose of the
126checksum—to detect if someone changed the output, as these edits would be lost
127the next time Argument Clinic writes out fresh output.)
128
129For the sake of clarity, here's the terminology we'll use with Argument Clinic:
130
131* The first line of the comment (``/*[clinic input]``) is the *start line*.
132* The last line of the initial comment (``[clinic start generated code]*/``) is the *end line*.
133* The last line (``/*[clinic end generated code: checksum=...]*/``) is the *checksum line*.
134* In between the start line and the end line is the *input*.
135* In between the end line and the checksum line is the *output*.
136* All the text collectively, from the start line to the checksum line inclusively,
137  is the *block*.  (A block that hasn't been successfully processed by Argument
138  Clinic yet doesn't have output or a checksum line, but it's still considered
139  a block.)
140
141
142Converting Your First Function
143==============================
144
145The best way to get a sense of how Argument Clinic works is to
146convert a function to work with it.  Here, then, are the bare
147minimum steps you'd need to follow to convert a function to
148work with Argument Clinic.  Note that for code you plan to
149check in to CPython, you really should take the conversion farther,
150using some of the advanced concepts you'll see later on in
151the document (like "return converters" and "self converters").
152But we'll keep it simple for this walkthrough so you can learn.
153
154Let's dive in!
155
1560. Make sure you're working with a freshly updated checkout
157   of the CPython trunk.
158
1591. Find a Python builtin that calls either :c:func:`PyArg_ParseTuple`
160   or :c:func:`PyArg_ParseTupleAndKeywords`, and hasn't been converted
161   to work with Argument Clinic yet.
162   For my example I'm using ``_pickle.Pickler.dump()``.
163
1642. If the call to the ``PyArg_Parse`` function uses any of the
165   following format units:
166
167   .. code-block:: none
168
169       O&
170       O!
171       es
172       es#
173       et
174       et#
175
176   or if it has multiple calls to :c:func:`PyArg_ParseTuple`,
177   you should choose a different function.  Argument Clinic *does*
178   support all of these scenarios.  But these are advanced
179   topics—let's do something simpler for your first function.
180
181   Also, if the function has multiple calls to :c:func:`PyArg_ParseTuple`
182   or :c:func:`PyArg_ParseTupleAndKeywords` where it supports different
183   types for the same argument, or if the function uses something besides
184   PyArg_Parse functions to parse its arguments, it probably
185   isn't suitable for conversion to Argument Clinic.  Argument Clinic
186   doesn't support generic functions or polymorphic parameters.
187
1883. Add the following boilerplate above the function, creating our block::
189
190    /*[clinic input]
191    [clinic start generated code]*/
192
1934. Cut the docstring and paste it in between the ``[clinic]`` lines,
194   removing all the junk that makes it a properly quoted C string.
195   When you're done you should have just the text, based at the left
196   margin, with no line wider than 80 characters.
197   (Argument Clinic will preserve indents inside the docstring.)
198
199   If the old docstring had a first line that looked like a function
200   signature, throw that line away.  (The docstring doesn't need it
201   anymore—when you use ``help()`` on your builtin in the future,
202   the first line will be built automatically based on the function's
203   signature.)
204
205   Sample::
206
207    /*[clinic input]
208    Write a pickled representation of obj to the open file.
209    [clinic start generated code]*/
210
2115. If your docstring doesn't have a "summary" line, Argument Clinic will
212   complain.  So let's make sure it has one.  The "summary" line should
213   be a paragraph consisting of a single 80-column line
214   at the beginning of the docstring.
215
216   (Our example docstring consists solely of a summary line, so the sample
217   code doesn't have to change for this step.)
218
2196. Above the docstring, enter the name of the function, followed
220   by a blank line.  This should be the Python name of the function,
221   and should be the full dotted path
222   to the function—it should start with the name of the module,
223   include any sub-modules, and if the function is a method on
224   a class it should include the class name too.
225
226   Sample::
227
228    /*[clinic input]
229    _pickle.Pickler.dump
230
231    Write a pickled representation of obj to the open file.
232    [clinic start generated code]*/
233
2347. If this is the first time that module or class has been used with Argument
235   Clinic in this C file,
236   you must declare the module and/or class.  Proper Argument Clinic hygiene
237   prefers declaring these in a separate block somewhere near the
238   top of the C file, in the same way that include files and statics go at
239   the top.  (In our sample code we'll just show the two blocks next to
240   each other.)
241
242   The name of the class and module should be the same as the one
243   seen by Python.  Check the name defined in the :c:type:`PyModuleDef`
244   or :c:type:`PyTypeObject` as appropriate.
245
246   When you declare a class, you must also specify two aspects of its type
247   in C: the type declaration you'd use for a pointer to an instance of
248   this class, and a pointer to the :c:type:`PyTypeObject` for this class.
249
250   Sample::
251
252       /*[clinic input]
253       module _pickle
254       class _pickle.Pickler "PicklerObject *" "&Pickler_Type"
255       [clinic start generated code]*/
256
257       /*[clinic input]
258       _pickle.Pickler.dump
259
260       Write a pickled representation of obj to the open file.
261       [clinic start generated code]*/
262
263
264
265
2668. Declare each of the parameters to the function.  Each parameter
267   should get its own line.  All the parameter lines should be
268   indented from the function name and the docstring.
269
270   The general form of these parameter lines is as follows:
271
272   .. code-block:: none
273
274       name_of_parameter: converter
275
276   If the parameter has a default value, add that after the
277   converter:
278
279   .. code-block:: none
280
281       name_of_parameter: converter = default_value
282
283   Argument Clinic's support for "default values" is quite sophisticated;
284   please see :ref:`the section below on default values <default_values>`
285   for more information.
286
287   Add a blank line below the parameters.
288
289   What's a "converter"?  It establishes both the type
290   of the variable used in C, and the method to convert the Python
291   value into a C value at runtime.
292   For now you're going to use what's called a "legacy converter"—a
293   convenience syntax intended to make porting old code into Argument
294   Clinic easier.
295
296   For each parameter, copy the "format unit" for that
297   parameter from the ``PyArg_Parse()`` format argument and
298   specify *that* as its converter, as a quoted
299   string.  ("format unit" is the formal name for the one-to-three
300   character substring of the ``format`` parameter that tells
301   the argument parsing function what the type of the variable
302   is and how to convert it.  For more on format units please
303   see :ref:`arg-parsing`.)
304
305   For multicharacter format units like ``z#``, use the
306   entire two-or-three character string.
307
308   Sample::
309
310        /*[clinic input]
311        module _pickle
312        class _pickle.Pickler "PicklerObject *" "&Pickler_Type"
313        [clinic start generated code]*/
314
315        /*[clinic input]
316        _pickle.Pickler.dump
317
318           obj: 'O'
319
320       Write a pickled representation of obj to the open file.
321       [clinic start generated code]*/
322
3239. If your function has ``|`` in the format string, meaning some
324   parameters have default values, you can ignore it.  Argument
325   Clinic infers which parameters are optional based on whether
326   or not they have default values.
327
328   If your function has ``$`` in the format string, meaning it
329   takes keyword-only arguments, specify ``*`` on a line by
330   itself before the first keyword-only argument, indented the
331   same as the parameter lines.
332
333   (``_pickle.Pickler.dump`` has neither, so our sample is unchanged.)
334
335
33610. If the existing C function calls :c:func:`PyArg_ParseTuple`
337    (as opposed to :c:func:`PyArg_ParseTupleAndKeywords`), then all its
338    arguments are positional-only.
339
340    To mark all parameters as positional-only in Argument Clinic,
341    add a ``/`` on a line by itself after the last parameter,
342    indented the same as the parameter lines.
343
344    Currently this is all-or-nothing; either all parameters are
345    positional-only, or none of them are.  (In the future Argument
346    Clinic may relax this restriction.)
347
348    Sample::
349
350        /*[clinic input]
351        module _pickle
352        class _pickle.Pickler "PicklerObject *" "&Pickler_Type"
353        [clinic start generated code]*/
354
355        /*[clinic input]
356        _pickle.Pickler.dump
357
358            obj: 'O'
359            /
360
361        Write a pickled representation of obj to the open file.
362        [clinic start generated code]*/
363
36411. It's helpful to write a per-parameter docstring for each parameter.
365    But per-parameter docstrings are optional; you can skip this step
366    if you prefer.
367
368    Here's how to add a per-parameter docstring.  The first line
369    of the per-parameter docstring must be indented further than the
370    parameter definition.  The left margin of this first line establishes
371    the left margin for the whole per-parameter docstring; all the text
372    you write will be outdented by this amount.  You can write as much
373    text as you like, across multiple lines if you wish.
374
375    Sample::
376
377        /*[clinic input]
378        module _pickle
379        class _pickle.Pickler "PicklerObject *" "&Pickler_Type"
380        [clinic start generated code]*/
381
382        /*[clinic input]
383        _pickle.Pickler.dump
384
385            obj: 'O'
386                The object to be pickled.
387            /
388
389        Write a pickled representation of obj to the open file.
390        [clinic start generated code]*/
391
39212. Save and close the file, then run ``Tools/clinic/clinic.py`` on
393    it.  With luck everything worked---your block now has output, and
394    a ``.c.h`` file has been generated! Reopen the file in your
395    text editor to see::
396
397       /*[clinic input]
398       _pickle.Pickler.dump
399
400           obj: 'O'
401               The object to be pickled.
402           /
403
404       Write a pickled representation of obj to the open file.
405       [clinic start generated code]*/
406
407       static PyObject *
408       _pickle_Pickler_dump(PicklerObject *self, PyObject *obj)
409       /*[clinic end generated code: output=87ecad1261e02ac7 input=552eb1c0f52260d9]*/
410
411    Obviously, if Argument Clinic didn't produce any output, it's because
412    it found an error in your input.  Keep fixing your errors and retrying
413    until Argument Clinic processes your file without complaint.
414
415    For readability, most of the glue code has been generated to a ``.c.h``
416    file.  You'll need to include that in your original ``.c`` file,
417    typically right after the clinic module block::
418
419       #include "clinic/_pickle.c.h"
420
42113. Double-check that the argument-parsing code Argument Clinic generated
422    looks basically the same as the existing code.
423
424    First, ensure both places use the same argument-parsing function.
425    The existing code must call either
426    :c:func:`PyArg_ParseTuple` or :c:func:`PyArg_ParseTupleAndKeywords`;
427    ensure that the code generated by Argument Clinic calls the
428    *exact* same function.
429
430    Second, the format string passed in to :c:func:`PyArg_ParseTuple` or
431    :c:func:`PyArg_ParseTupleAndKeywords` should be *exactly* the same
432    as the hand-written one in the existing function, up to the colon
433    or semi-colon.
434
435    (Argument Clinic always generates its format strings
436    with a ``:`` followed by the name of the function.  If the
437    existing code's format string ends with ``;``, to provide
438    usage help, this change is harmless—don't worry about it.)
439
440    Third, for parameters whose format units require two arguments
441    (like a length variable, or an encoding string, or a pointer
442    to a conversion function), ensure that the second argument is
443    *exactly* the same between the two invocations.
444
445    Fourth, inside the output portion of the block you'll find a preprocessor
446    macro defining the appropriate static :c:type:`PyMethodDef` structure for
447    this builtin::
448
449        #define __PICKLE_PICKLER_DUMP_METHODDEF    \
450        {"dump", (PyCFunction)__pickle_Pickler_dump, METH_O, __pickle_Pickler_dump__doc__},
451
452    This static structure should be *exactly* the same as the existing static
453    :c:type:`PyMethodDef` structure for this builtin.
454
455    If any of these items differ in *any way*,
456    adjust your Argument Clinic function specification and rerun
457    ``Tools/clinic/clinic.py`` until they *are* the same.
458
459
46014. Notice that the last line of its output is the declaration
461    of your "impl" function.  This is where the builtin's implementation goes.
462    Delete the existing prototype of the function you're modifying, but leave
463    the opening curly brace.  Now delete its argument parsing code and the
464    declarations of all the variables it dumps the arguments into.
465    Notice how the Python arguments are now arguments to this impl function;
466    if the implementation used different names for these variables, fix it.
467
468    Let's reiterate, just because it's kind of weird.  Your code should now
469    look like this::
470
471        static return_type
472        your_function_impl(...)
473        /*[clinic end generated code: checksum=...]*/
474        {
475        ...
476
477    Argument Clinic generated the checksum line and the function prototype just
478    above it.  You should write the opening (and closing) curly braces for the
479    function, and the implementation inside.
480
481    Sample::
482
483        /*[clinic input]
484        module _pickle
485        class _pickle.Pickler "PicklerObject *" "&Pickler_Type"
486        [clinic start generated code]*/
487        /*[clinic end generated code: checksum=da39a3ee5e6b4b0d3255bfef95601890afd80709]*/
488
489        /*[clinic input]
490        _pickle.Pickler.dump
491
492            obj: 'O'
493                The object to be pickled.
494            /
495
496        Write a pickled representation of obj to the open file.
497        [clinic start generated code]*/
498
499        PyDoc_STRVAR(__pickle_Pickler_dump__doc__,
500        "Write a pickled representation of obj to the open file.\n"
501        "\n"
502        ...
503        static PyObject *
504        _pickle_Pickler_dump_impl(PicklerObject *self, PyObject *obj)
505        /*[clinic end generated code: checksum=3bd30745bf206a48f8b576a1da3d90f55a0a4187]*/
506        {
507            /* Check whether the Pickler was initialized correctly (issue3664).
508               Developers often forget to call __init__() in their subclasses, which
509               would trigger a segfault without this check. */
510            if (self->write == NULL) {
511                PyErr_Format(PicklingError,
512                             "Pickler.__init__() was not called by %s.__init__()",
513                             Py_TYPE(self)->tp_name);
514                return NULL;
515            }
516
517            if (_Pickler_ClearBuffer(self) < 0)
518                return NULL;
519
520            ...
521
52215. Remember the macro with the :c:type:`PyMethodDef` structure for this
523    function?  Find the existing :c:type:`PyMethodDef` structure for this
524    function and replace it with a reference to the macro.  (If the builtin
525    is at module scope, this will probably be very near the end of the file;
526    if the builtin is a class method, this will probably be below but relatively
527    near to the implementation.)
528
529    Note that the body of the macro contains a trailing comma.  So when you
530    replace the existing static :c:type:`PyMethodDef` structure with the macro,
531    *don't* add a comma to the end.
532
533    Sample::
534
535        static struct PyMethodDef Pickler_methods[] = {
536            __PICKLE_PICKLER_DUMP_METHODDEF
537            __PICKLE_PICKLER_CLEAR_MEMO_METHODDEF
538            {NULL, NULL}                /* sentinel */
539        };
540
541
54216. Compile, then run the relevant portions of the regression-test suite.
543    This change should not introduce any new compile-time warnings or errors,
544    and there should be no externally-visible change to Python's behavior.
545
546    Well, except for one difference: ``inspect.signature()`` run on your function
547    should now provide a valid signature!
548
549    Congratulations, you've ported your first function to work with Argument Clinic!
550
551Advanced Topics
552===============
553
554Now that you've had some experience working with Argument Clinic, it's time
555for some advanced topics.
556
557
558Symbolic default values
559-----------------------
560
561The default value you provide for a parameter can't be any arbitrary
562expression.  Currently the following are explicitly supported:
563
564* Numeric constants (integer and float)
565* String constants
566* ``True``, ``False``, and ``None``
567* Simple symbolic constants like ``sys.maxsize``, which must
568  start with the name of the module
569
570In case you're curious, this is implemented in  ``from_builtin()``
571in ``Lib/inspect.py``.
572
573(In the future, this may need to get even more elaborate,
574to allow full expressions like ``CONSTANT - 1``.)
575
576
577Renaming the C functions and variables generated by Argument Clinic
578-------------------------------------------------------------------
579
580Argument Clinic automatically names the functions it generates for you.
581Occasionally this may cause a problem, if the generated name collides with
582the name of an existing C function.  There's an easy solution: override the names
583used for the C functions.  Just add the keyword ``"as"``
584to your function declaration line, followed by the function name you wish to use.
585Argument Clinic will use that function name for the base (generated) function,
586then add ``"_impl"`` to the end and use that for the name of the impl function.
587
588For example, if we wanted to rename the C function names generated for
589``pickle.Pickler.dump``, it'd look like this::
590
591    /*[clinic input]
592    pickle.Pickler.dump as pickler_dumper
593
594    ...
595
596The base function would now be named ``pickler_dumper()``,
597and the impl function would now be named ``pickler_dumper_impl()``.
598
599
600Similarly, you may have a problem where you want to give a parameter
601a specific Python name, but that name may be inconvenient in C.  Argument
602Clinic allows you to give a parameter different names in Python and in C,
603using the same ``"as"`` syntax::
604
605    /*[clinic input]
606    pickle.Pickler.dump
607
608        obj: object
609        file as file_obj: object
610        protocol: object = NULL
611        *
612        fix_imports: bool = True
613
614Here, the name used in Python (in the signature and the ``keywords``
615array) would be ``file``, but the C variable would be named ``file_obj``.
616
617You can use this to rename the ``self`` parameter too!
618
619
620Converting functions using PyArg_UnpackTuple
621--------------------------------------------
622
623To convert a function parsing its arguments with :c:func:`PyArg_UnpackTuple`,
624simply write out all the arguments, specifying each as an ``object``.  You
625may specify the ``type`` argument to cast the type as appropriate.  All
626arguments should be marked positional-only (add a ``/`` on a line by itself
627after the last argument).
628
629Currently the generated code will use :c:func:`PyArg_ParseTuple`, but this
630will change soon.
631
632Optional Groups
633---------------
634
635Some legacy functions have a tricky approach to parsing their arguments:
636they count the number of positional arguments, then use a ``switch`` statement
637to call one of several different :c:func:`PyArg_ParseTuple` calls depending on
638how many positional arguments there are.  (These functions cannot accept
639keyword-only arguments.)  This approach was used to simulate optional
640arguments back before :c:func:`PyArg_ParseTupleAndKeywords` was created.
641
642While functions using this approach can often be converted to
643use :c:func:`PyArg_ParseTupleAndKeywords`, optional arguments, and default values,
644it's not always possible.  Some of these legacy functions have
645behaviors :c:func:`PyArg_ParseTupleAndKeywords` doesn't directly support.
646The most obvious example is the builtin function ``range()``, which has
647an optional argument on the *left* side of its required argument!
648Another example is ``curses.window.addch()``, which has a group of two
649arguments that must always be specified together.  (The arguments are
650called ``x`` and ``y``; if you call the function passing in ``x``,
651you must also pass in ``y``—and if you don't pass in ``x`` you may not
652pass in ``y`` either.)
653
654In any case, the goal of Argument Clinic is to support argument parsing
655for all existing CPython builtins without changing their semantics.
656Therefore Argument Clinic supports
657this alternate approach to parsing, using what are called *optional groups*.
658Optional groups are groups of arguments that must all be passed in together.
659They can be to the left or the right of the required arguments.  They
660can *only* be used with positional-only parameters.
661
662.. note:: Optional groups are *only* intended for use when converting
663          functions that make multiple calls to :c:func:`PyArg_ParseTuple`!
664          Functions that use *any* other approach for parsing arguments
665          should *almost never* be converted to Argument Clinic using
666          optional groups.  Functions using optional groups currently
667          cannot have accurate signatures in Python, because Python just
668          doesn't understand the concept.  Please avoid using optional
669          groups wherever possible.
670
671To specify an optional group, add a ``[`` on a line by itself before
672the parameters you wish to group together, and a ``]`` on a line by itself
673after these parameters.  As an example, here's how ``curses.window.addch``
674uses optional groups to make the first two parameters and the last
675parameter optional::
676
677    /*[clinic input]
678
679    curses.window.addch
680
681        [
682        x: int
683          X-coordinate.
684        y: int
685          Y-coordinate.
686        ]
687
688        ch: object
689          Character to add.
690
691        [
692        attr: long
693          Attributes for the character.
694        ]
695        /
696
697    ...
698
699
700Notes:
701
702* For every optional group, one additional parameter will be passed into the
703  impl function representing the group.  The parameter will be an int named
704  ``group_{direction}_{number}``,
705  where ``{direction}`` is either ``right`` or ``left`` depending on whether the group
706  is before or after the required parameters, and ``{number}`` is a monotonically
707  increasing number (starting at 1) indicating how far away the group is from
708  the required parameters.  When the impl is called, this parameter will be set
709  to zero if this group was unused, and set to non-zero if this group was used.
710  (By used or unused, I mean whether or not the parameters received arguments
711  in this invocation.)
712
713* If there are no required arguments, the optional groups will behave
714  as if they're to the right of the required arguments.
715
716* In the case of ambiguity, the argument parsing code
717  favors parameters on the left (before the required parameters).
718
719* Optional groups can only contain positional-only parameters.
720
721* Optional groups are *only* intended for legacy code.  Please do not
722  use optional groups for new code.
723
724
725Using real Argument Clinic converters, instead of "legacy converters"
726---------------------------------------------------------------------
727
728To save time, and to minimize how much you need to learn
729to achieve your first port to Argument Clinic, the walkthrough above tells
730you to use "legacy converters".  "Legacy converters" are a convenience,
731designed explicitly to make porting existing code to Argument Clinic
732easier.  And to be clear, their use is acceptable when porting code for
733Python 3.4.
734
735However, in the long term we probably want all our blocks to
736use Argument Clinic's real syntax for converters.  Why?  A couple
737reasons:
738
739* The proper converters are far easier to read and clearer in their intent.
740* There are some format units that are unsupported as "legacy converters",
741  because they require arguments, and the legacy converter syntax doesn't
742  support specifying arguments.
743* In the future we may have a new argument parsing library that isn't
744  restricted to what :c:func:`PyArg_ParseTuple` supports; this flexibility
745  won't be available to parameters using legacy converters.
746
747Therefore, if you don't mind a little extra effort, please use the normal
748converters instead of legacy converters.
749
750In a nutshell, the syntax for Argument Clinic (non-legacy) converters
751looks like a Python function call.  However, if there are no explicit
752arguments to the function (all functions take their default values),
753you may omit the parentheses.  Thus ``bool`` and ``bool()`` are exactly
754the same converters.
755
756All arguments to Argument Clinic converters are keyword-only.
757All Argument Clinic converters accept the following arguments:
758
759  ``c_default``
760    The default value for this parameter when defined in C.
761    Specifically, this will be the initializer for the variable declared
762    in the "parse function".  See :ref:`the section on default values <default_values>`
763    for how to use this.
764    Specified as a string.
765
766  ``annotation``
767    The annotation value for this parameter.  Not currently supported,
768    because PEP 8 mandates that the Python library may not use
769    annotations.
770
771In addition, some converters accept additional arguments.  Here is a list
772of these arguments, along with their meanings:
773
774  ``accept``
775    A set of Python types (and possibly pseudo-types);
776    this restricts the allowable Python argument to values of these types.
777    (This is not a general-purpose facility; as a rule it only supports
778    specific lists of types as shown in the legacy converter table.)
779
780    To accept ``None``, add ``NoneType`` to this set.
781
782  ``bitwise``
783    Only supported for unsigned integers.  The native integer value of this
784    Python argument will be written to the parameter without any range checking,
785    even for negative values.
786
787  ``converter``
788    Only supported by the ``object`` converter.  Specifies the name of a
789    :ref:`C "converter function" <o_ampersand>`
790    to use to convert this object to a native type.
791
792  ``encoding``
793    Only supported for strings.  Specifies the encoding to use when converting
794    this string from a Python str (Unicode) value into a C ``char *`` value.
795
796
797  ``subclass_of``
798    Only supported for the ``object`` converter.  Requires that the Python
799    value be a subclass of a Python type, as expressed in C.
800
801  ``type``
802    Only supported for the ``object`` and ``self`` converters.  Specifies
803    the C type that will be used to declare the variable.  Default value is
804    ``"PyObject *"``.
805
806  ``zeroes``
807    Only supported for strings.  If true, embedded NUL bytes (``'\\0'``) are
808    permitted inside the value.  The length of the string will be passed in
809    to the impl function, just after the string parameter, as a parameter named
810    ``<parameter_name>_length``.
811
812Please note, not every possible combination of arguments will work.
813Usually these arguments are implemented by specific ``PyArg_ParseTuple``
814*format units*, with specific behavior.  For example, currently you cannot
815call ``unsigned_short`` without also specifying ``bitwise=True``.
816Although it's perfectly reasonable to think this would work, these semantics don't
817map to any existing format unit.  So Argument Clinic doesn't support it.  (Or, at
818least, not yet.)
819
820Below is a table showing the mapping of legacy converters into real
821Argument Clinic converters.  On the left is the legacy converter,
822on the right is the text you'd replace it with.
823
824=========   =================================================================================
825``'B'``     ``unsigned_char(bitwise=True)``
826``'b'``     ``unsigned_char``
827``'c'``     ``char``
828``'C'``     ``int(accept={str})``
829``'d'``     ``double``
830``'D'``     ``Py_complex``
831``'es'``    ``str(encoding='name_of_encoding')``
832``'es#'``   ``str(encoding='name_of_encoding', zeroes=True)``
833``'et'``    ``str(encoding='name_of_encoding', accept={bytes, bytearray, str})``
834``'et#'``   ``str(encoding='name_of_encoding', accept={bytes, bytearray, str}, zeroes=True)``
835``'f'``     ``float``
836``'h'``     ``short``
837``'H'``     ``unsigned_short(bitwise=True)``
838``'i'``     ``int``
839``'I'``     ``unsigned_int(bitwise=True)``
840``'k'``     ``unsigned_long(bitwise=True)``
841``'K'``     ``unsigned_long_long(bitwise=True)``
842``'l'``     ``long``
843``'L'``     ``long long``
844``'n'``     ``Py_ssize_t``
845``'O'``     ``object``
846``'O!'``    ``object(subclass_of='&PySomething_Type')``
847``'O&'``    ``object(converter='name_of_c_function')``
848``'p'``     ``bool``
849``'S'``     ``PyBytesObject``
850``'s'``     ``str``
851``'s#'``    ``str(zeroes=True)``
852``'s*'``    ``Py_buffer(accept={buffer, str})``
853``'U'``     ``unicode``
854``'u'``     ``Py_UNICODE``
855``'u#'``    ``Py_UNICODE(zeroes=True)``
856``'w*'``    ``Py_buffer(accept={rwbuffer})``
857``'Y'``     ``PyByteArrayObject``
858``'y'``     ``str(accept={bytes})``
859``'y#'``    ``str(accept={robuffer}, zeroes=True)``
860``'y*'``    ``Py_buffer``
861``'Z'``     ``Py_UNICODE(accept={str, NoneType})``
862``'Z#'``    ``Py_UNICODE(accept={str, NoneType}, zeroes=True)``
863``'z'``     ``str(accept={str, NoneType})``
864``'z#'``    ``str(accept={str, NoneType}, zeroes=True)``
865``'z*'``    ``Py_buffer(accept={buffer, str, NoneType})``
866=========   =================================================================================
867
868As an example, here's our sample ``pickle.Pickler.dump`` using the proper
869converter::
870
871    /*[clinic input]
872    pickle.Pickler.dump
873
874        obj: object
875            The object to be pickled.
876        /
877
878    Write a pickled representation of obj to the open file.
879    [clinic start generated code]*/
880
881Argument Clinic will show you all the converters it has
882available.  For each converter it'll show you all the parameters
883it accepts, along with the default value for each parameter.
884Just run ``Tools/clinic/clinic.py --converters`` to see the full list.
885
886Py_buffer
887---------
888
889When using the ``Py_buffer`` converter
890(or the ``'s*'``, ``'w*'``, ``'*y'``, or ``'z*'`` legacy converters),
891you *must* not call :c:func:`PyBuffer_Release` on the provided buffer.
892Argument Clinic generates code that does it for you (in the parsing function).
893
894
895
896Advanced converters
897-------------------
898
899Remember those format units you skipped for your first
900time because they were advanced?  Here's how to handle those too.
901
902The trick is, all those format units take arguments—either
903conversion functions, or types, or strings specifying an encoding.
904(But "legacy converters" don't support arguments.  That's why we
905skipped them for your first function.)  The argument you specified
906to the format unit is now an argument to the converter; this
907argument is either ``converter`` (for ``O&``), ``subclass_of`` (for ``O!``),
908or ``encoding`` (for all the format units that start with ``e``).
909
910When using ``subclass_of``, you may also want to use the other
911custom argument for ``object()``: ``type``, which lets you set the type
912actually used for the parameter.  For example, if you want to ensure
913that the object is a subclass of ``PyUnicode_Type``, you probably want
914to use the converter ``object(type='PyUnicodeObject *', subclass_of='&PyUnicode_Type')``.
915
916One possible problem with using Argument Clinic: it takes away some possible
917flexibility for the format units starting with ``e``.  When writing a
918``PyArg_Parse`` call by hand, you could theoretically decide at runtime what
919encoding string to pass in to :c:func:`PyArg_ParseTuple`.   But now this string must
920be hard-coded at Argument-Clinic-preprocessing-time.  This limitation is deliberate;
921it made supporting this format unit much easier, and may allow for future optimizations.
922This restriction doesn't seem unreasonable; CPython itself always passes in static
923hard-coded encoding strings for parameters whose format units start with ``e``.
924
925
926.. _default_values:
927
928Parameter default values
929------------------------
930
931Default values for parameters can be any of a number of values.
932At their simplest, they can be string, int, or float literals:
933
934.. code-block:: none
935
936    foo: str = "abc"
937    bar: int = 123
938    bat: float = 45.6
939
940They can also use any of Python's built-in constants:
941
942.. code-block:: none
943
944    yep:  bool = True
945    nope: bool = False
946    nada: object = None
947
948There's also special support for a default value of ``NULL``, and
949for simple expressions, documented in the following sections.
950
951
952The ``NULL`` default value
953--------------------------
954
955For string and object parameters, you can set them to ``None`` to indicate
956that there's no default.  However, that means the C variable will be
957initialized to ``Py_None``.  For convenience's sakes, there's a special
958value called ``NULL`` for just this reason: from Python's perspective it
959behaves like a default value of ``None``, but the C variable is initialized
960with ``NULL``.
961
962Expressions specified as default values
963---------------------------------------
964
965The default value for a parameter can be more than just a literal value.
966It can be an entire expression, using math operators and looking up attributes
967on objects.  However, this support isn't exactly simple, because of some
968non-obvious semantics.
969
970Consider the following example:
971
972.. code-block:: none
973
974    foo: Py_ssize_t = sys.maxsize - 1
975
976``sys.maxsize`` can have different values on different platforms.  Therefore
977Argument Clinic can't simply evaluate that expression locally and hard-code it
978in C.  So it stores the default in such a way that it will get evaluated at
979runtime, when the user asks for the function's signature.
980
981What namespace is available when the expression is evaluated?  It's evaluated
982in the context of the module the builtin came from.  So, if your module has an
983attribute called "``max_widgets``", you may simply use it:
984
985.. code-block:: none
986
987    foo: Py_ssize_t = max_widgets
988
989If the symbol isn't found in the current module, it fails over to looking in
990``sys.modules``.  That's how it can find ``sys.maxsize`` for example.  (Since you
991don't know in advance what modules the user will load into their interpreter,
992it's best to restrict yourself to modules that are preloaded by Python itself.)
993
994Evaluating default values only at runtime means Argument Clinic can't compute
995the correct equivalent C default value.  So you need to tell it explicitly.
996When you use an expression, you must also specify the equivalent expression
997in C, using the ``c_default`` parameter to the converter:
998
999.. code-block:: none
1000
1001    foo: Py_ssize_t(c_default="PY_SSIZE_T_MAX - 1") = sys.maxsize - 1
1002
1003Another complication: Argument Clinic can't know in advance whether or not the
1004expression you supply is valid.  It parses it to make sure it looks legal, but
1005it can't *actually* know.  You must be very careful when using expressions to
1006specify values that are guaranteed to be valid at runtime!
1007
1008Finally, because expressions must be representable as static C values, there
1009are many restrictions on legal expressions.  Here's a list of Python features
1010you're not permitted to use:
1011
1012* Function calls.
1013* Inline if statements (``3 if foo else 5``).
1014* Automatic sequence unpacking (``*[1, 2, 3]``).
1015* List/set/dict comprehensions and generator expressions.
1016* Tuple/list/set/dict literals.
1017
1018
1019
1020Using a return converter
1021------------------------
1022
1023By default the impl function Argument Clinic generates for you returns ``PyObject *``.
1024But your C function often computes some C type, then converts it into the ``PyObject *``
1025at the last moment.  Argument Clinic handles converting your inputs from Python types
1026into native C types—why not have it convert your return value from a native C type
1027into a Python type too?
1028
1029That's what a "return converter" does.  It changes your impl function to return
1030some C type, then adds code to the generated (non-impl) function to handle converting
1031that value into the appropriate ``PyObject *``.
1032
1033The syntax for return converters is similar to that of parameter converters.
1034You specify the return converter like it was a return annotation on the
1035function itself.  Return converters behave much the same as parameter converters;
1036they take arguments, the arguments are all keyword-only, and if you're not changing
1037any of the default arguments you can omit the parentheses.
1038
1039(If you use both ``"as"`` *and* a return converter for your function,
1040the ``"as"`` should come before the return converter.)
1041
1042There's one additional complication when using return converters: how do you
1043indicate an error has occurred?  Normally, a function returns a valid (non-``NULL``)
1044pointer for success, and ``NULL`` for failure.  But if you use an integer return converter,
1045all integers are valid.  How can Argument Clinic detect an error?  Its solution: each return
1046converter implicitly looks for a special value that indicates an error.  If you return
1047that value, and an error has been set (``PyErr_Occurred()`` returns a true
1048value), then the generated code will propagate the error.  Otherwise it will
1049encode the value you return like normal.
1050
1051Currently Argument Clinic supports only a few return converters:
1052
1053.. code-block:: none
1054
1055    bool
1056    int
1057    unsigned int
1058    long
1059    unsigned int
1060    size_t
1061    Py_ssize_t
1062    float
1063    double
1064    DecodeFSDefault
1065
1066None of these take parameters.  For the first three, return -1 to indicate
1067error.  For ``DecodeFSDefault``, the return type is ``const char *``; return a NULL
1068pointer to indicate an error.
1069
1070(There's also an experimental ``NoneType`` converter, which lets you
1071return ``Py_None`` on success or ``NULL`` on failure, without having
1072to increment the reference count on ``Py_None``.  I'm not sure it adds
1073enough clarity to be worth using.)
1074
1075To see all the return converters Argument Clinic supports, along with
1076their parameters (if any),
1077just run ``Tools/clinic/clinic.py --converters`` for the full list.
1078
1079
1080Cloning existing functions
1081--------------------------
1082
1083If you have a number of functions that look similar, you may be able to
1084use Clinic's "clone" feature.  When you clone an existing function,
1085you reuse:
1086
1087* its parameters, including
1088
1089  * their names,
1090
1091  * their converters, with all parameters,
1092
1093  * their default values,
1094
1095  * their per-parameter docstrings,
1096
1097  * their *kind* (whether they're positional only,
1098    positional or keyword, or keyword only), and
1099
1100* its return converter.
1101
1102The only thing not copied from the original function is its docstring;
1103the syntax allows you to specify a new docstring.
1104
1105Here's the syntax for cloning a function::
1106
1107    /*[clinic input]
1108    module.class.new_function [as c_basename] = module.class.existing_function
1109
1110    Docstring for new_function goes here.
1111    [clinic start generated code]*/
1112
1113(The functions can be in different modules or classes.  I wrote
1114``module.class`` in the sample just to illustrate that you must
1115use the full path to *both* functions.)
1116
1117Sorry, there's no syntax for partially-cloning a function, or cloning a function
1118then modifying it.  Cloning is an all-or nothing proposition.
1119
1120Also, the function you are cloning from must have been previously defined
1121in the current file.
1122
1123Calling Python code
1124-------------------
1125
1126The rest of the advanced topics require you to write Python code
1127which lives inside your C file and modifies Argument Clinic's
1128runtime state.  This is simple: you simply define a Python block.
1129
1130A Python block uses different delimiter lines than an Argument
1131Clinic function block.  It looks like this::
1132
1133    /*[python input]
1134    # python code goes here
1135    [python start generated code]*/
1136
1137All the code inside the Python block is executed at the
1138time it's parsed.  All text written to stdout inside the block
1139is redirected into the "output" after the block.
1140
1141As an example, here's a Python block that adds a static integer
1142variable to the C code::
1143
1144    /*[python input]
1145    print('static int __ignored_unused_variable__ = 0;')
1146    [python start generated code]*/
1147    static int __ignored_unused_variable__ = 0;
1148    /*[python checksum:...]*/
1149
1150
1151Using a "self converter"
1152------------------------
1153
1154Argument Clinic automatically adds a "self" parameter for you
1155using a default converter.  It automatically sets the ``type``
1156of this parameter to the "pointer to an instance" you specified
1157when you declared the type.  However, you can override
1158Argument Clinic's converter and specify one yourself.
1159Just add your own ``self`` parameter as the first parameter in a
1160block, and ensure that its converter is an instance of
1161``self_converter`` or a subclass thereof.
1162
1163What's the point?  This lets you override the type of ``self``,
1164or give it a different default name.
1165
1166How do you specify the custom type you want to cast ``self`` to?
1167If you only have one or two functions with the same type for ``self``,
1168you can directly use Argument Clinic's existing ``self`` converter,
1169passing in the type you want to use as the ``type`` parameter::
1170
1171    /*[clinic input]
1172
1173    _pickle.Pickler.dump
1174
1175      self: self(type="PicklerObject *")
1176      obj: object
1177      /
1178
1179    Write a pickled representation of the given object to the open file.
1180    [clinic start generated code]*/
1181
1182On the other hand, if you have a lot of functions that will use the same
1183type for ``self``, it's best to create your own converter, subclassing
1184``self_converter`` but overwriting the ``type`` member::
1185
1186    /*[python input]
1187    class PicklerObject_converter(self_converter):
1188        type = "PicklerObject *"
1189    [python start generated code]*/
1190
1191    /*[clinic input]
1192
1193    _pickle.Pickler.dump
1194
1195      self: PicklerObject
1196      obj: object
1197      /
1198
1199    Write a pickled representation of the given object to the open file.
1200    [clinic start generated code]*/
1201
1202
1203
1204Writing a custom converter
1205--------------------------
1206
1207As we hinted at in the previous section... you can write your own converters!
1208A converter is simply a Python class that inherits from ``CConverter``.
1209The main purpose of a custom converter is if you have a parameter using
1210the ``O&`` format unit—parsing this parameter means calling
1211a :c:func:`PyArg_ParseTuple` "converter function".
1212
1213Your converter class should be named ``*something*_converter``.
1214If the name follows this convention, then your converter class
1215will be automatically registered with Argument Clinic; its name
1216will be the name of your class with the ``_converter`` suffix
1217stripped off.  (This is accomplished with a metaclass.)
1218
1219You shouldn't subclass ``CConverter.__init__``.  Instead, you should
1220write a ``converter_init()`` function.  ``converter_init()``
1221always accepts a ``self`` parameter; after that, all additional
1222parameters *must* be keyword-only.  Any arguments passed in to
1223the converter in Argument Clinic will be passed along to your
1224``converter_init()``.
1225
1226There are some additional members of ``CConverter`` you may wish
1227to specify in your subclass.  Here's the current list:
1228
1229``type``
1230    The C type to use for this variable.
1231    ``type`` should be a Python string specifying the type, e.g. ``int``.
1232    If this is a pointer type, the type string should end with ``' *'``.
1233
1234``default``
1235    The Python default value for this parameter, as a Python value.
1236    Or the magic value ``unspecified`` if there is no default.
1237
1238``py_default``
1239    ``default`` as it should appear in Python code,
1240    as a string.
1241    Or ``None`` if there is no default.
1242
1243``c_default``
1244    ``default`` as it should appear in C code,
1245    as a string.
1246    Or ``None`` if there is no default.
1247
1248``c_ignored_default``
1249    The default value used to initialize the C variable when
1250    there is no default, but not specifying a default may
1251    result in an "uninitialized variable" warning.  This can
1252    easily happen when using option groups—although
1253    properly-written code will never actually use this value,
1254    the variable does get passed in to the impl, and the
1255    C compiler will complain about the "use" of the
1256    uninitialized value.  This value should always be a
1257    non-empty string.
1258
1259``converter``
1260    The name of the C converter function, as a string.
1261
1262``impl_by_reference``
1263    A boolean value.  If true,
1264    Argument Clinic will add a ``&`` in front of the name of
1265    the variable when passing it into the impl function.
1266
1267``parse_by_reference``
1268    A boolean value.  If true,
1269    Argument Clinic will add a ``&`` in front of the name of
1270    the variable when passing it into :c:func:`PyArg_ParseTuple`.
1271
1272
1273Here's the simplest example of a custom converter, from ``Modules/zlibmodule.c``::
1274
1275    /*[python input]
1276
1277    class ssize_t_converter(CConverter):
1278        type = 'Py_ssize_t'
1279        converter = 'ssize_t_converter'
1280
1281    [python start generated code]*/
1282    /*[python end generated code: output=da39a3ee5e6b4b0d input=35521e4e733823c7]*/
1283
1284This block adds a converter to Argument Clinic named ``ssize_t``.  Parameters
1285declared as ``ssize_t`` will be declared as type ``Py_ssize_t``, and will
1286be parsed by the ``'O&'`` format unit, which will call the
1287``ssize_t_converter`` converter function.  ``ssize_t`` variables
1288automatically support default values.
1289
1290More sophisticated custom converters can insert custom C code to
1291handle initialization and cleanup.
1292You can see more examples of custom converters in the CPython
1293source tree; grep the C files for the string ``CConverter``.
1294
1295Writing a custom return converter
1296---------------------------------
1297
1298Writing a custom return converter is much like writing
1299a custom converter.  Except it's somewhat simpler, because return
1300converters are themselves much simpler.
1301
1302Return converters must subclass ``CReturnConverter``.
1303There are no examples yet of custom return converters,
1304because they are not widely used yet.  If you wish to
1305write your own return converter, please read ``Tools/clinic/clinic.py``,
1306specifically the implementation of ``CReturnConverter`` and
1307all its subclasses.
1308
1309METH_O and METH_NOARGS
1310----------------------------------------------
1311
1312To convert a function using ``METH_O``, make sure the function's
1313single argument is using the ``object`` converter, and mark the
1314arguments as positional-only::
1315
1316    /*[clinic input]
1317    meth_o_sample
1318
1319         argument: object
1320         /
1321    [clinic start generated code]*/
1322
1323
1324To convert a function using ``METH_NOARGS``, just don't specify
1325any arguments.
1326
1327You can still use a self converter, a return converter, and specify
1328a ``type`` argument to the object converter for ``METH_O``.
1329
1330tp_new and tp_init functions
1331----------------------------------------------
1332
1333You can convert ``tp_new`` and ``tp_init`` functions.  Just name
1334them ``__new__`` or ``__init__`` as appropriate.  Notes:
1335
1336* The function name generated for ``__new__`` doesn't end in ``__new__``
1337  like it would by default.  It's just the name of the class, converted
1338  into a valid C identifier.
1339
1340* No ``PyMethodDef`` ``#define`` is generated for these functions.
1341
1342* ``__init__`` functions return ``int``, not ``PyObject *``.
1343
1344* Use the docstring as the class docstring.
1345
1346* Although ``__new__`` and ``__init__`` functions must always
1347  accept both the ``args`` and ``kwargs`` objects, when converting
1348  you may specify any signature for these functions that you like.
1349  (If your function doesn't support keywords, the parsing function
1350  generated will throw an exception if it receives any.)
1351
1352Changing and redirecting Clinic's output
1353----------------------------------------
1354
1355It can be inconvenient to have Clinic's output interspersed with
1356your conventional hand-edited C code.  Luckily, Clinic is configurable:
1357you can buffer up its output for printing later (or earlier!), or write
1358its output to a separate file.  You can also add a prefix or suffix to
1359every line of Clinic's generated output.
1360
1361While changing Clinic's output in this manner can be a boon to readability,
1362it may result in Clinic code using types before they are defined, or
1363your code attempting to use Clinic-generated code before it is defined.
1364These problems can be easily solved by rearranging the declarations in your file,
1365or moving where Clinic's generated code goes.  (This is why the default behavior
1366of Clinic is to output everything into the current block; while many people
1367consider this hampers readability, it will never require rearranging your
1368code to fix definition-before-use problems.)
1369
1370Let's start with defining some terminology:
1371
1372*field*
1373  A field, in this context, is a subsection of Clinic's output.
1374  For example, the ``#define`` for the ``PyMethodDef`` structure
1375  is a field, called ``methoddef_define``.  Clinic has seven
1376  different fields it can output per function definition:
1377
1378  .. code-block:: none
1379
1380      docstring_prototype
1381      docstring_definition
1382      methoddef_define
1383      impl_prototype
1384      parser_prototype
1385      parser_definition
1386      impl_definition
1387
1388  All the names are of the form ``"<a>_<b>"``,
1389  where ``"<a>"`` is the semantic object represented (the parsing function,
1390  the impl function, the docstring, or the methoddef structure) and ``"<b>"``
1391  represents what kind of statement the field is.  Field names that end in
1392  ``"_prototype"``
1393  represent forward declarations of that thing, without the actual body/data
1394  of the thing; field names that end in ``"_definition"`` represent the actual
1395  definition of the thing, with the body/data of the thing.  (``"methoddef"``
1396  is special, it's the only one that ends with ``"_define"``, representing that
1397  it's a preprocessor #define.)
1398
1399*destination*
1400  A destination is a place Clinic can write output to.  There are
1401  five built-in destinations:
1402
1403  ``block``
1404    The default destination: printed in the output section of
1405    the current Clinic block.
1406
1407  ``buffer``
1408    A text buffer where you can save text for later.  Text sent
1409    here is appended to the end of any existing text.  It's an
1410    error to have any text left in the buffer when Clinic finishes
1411    processing a file.
1412
1413  ``file``
1414    A separate "clinic file" that will be created automatically by Clinic.
1415    The filename chosen for the file is ``{basename}.clinic{extension}``,
1416    where ``basename`` and ``extension`` were assigned the output
1417    from ``os.path.splitext()`` run on the current file.  (Example:
1418    the ``file`` destination for ``_pickle.c`` would be written to
1419    ``_pickle.clinic.c``.)
1420
1421    **Important: When using a** ``file`` **destination, you**
1422    *must check in* **the generated file!**
1423
1424  ``two-pass``
1425    A buffer like ``buffer``.  However, a two-pass buffer can only
1426    be dumped once, and it prints out all text sent to it during
1427    all processing, even from Clinic blocks *after* the dumping point.
1428
1429  ``suppress``
1430    The text is suppressed—thrown away.
1431
1432
1433Clinic defines five new directives that let you reconfigure its output.
1434
1435The first new directive is ``dump``:
1436
1437.. code-block:: none
1438
1439   dump <destination>
1440
1441This dumps the current contents of the named destination into the output of
1442the current block, and empties it.  This only works with ``buffer`` and
1443``two-pass`` destinations.
1444
1445The second new directive is ``output``.  The most basic form of ``output``
1446is like this:
1447
1448.. code-block:: none
1449
1450    output <field> <destination>
1451
1452This tells Clinic to output *field* to *destination*.  ``output`` also
1453supports a special meta-destination, called ``everything``, which tells
1454Clinic to output *all* fields to that *destination*.
1455
1456``output`` has a number of other functions:
1457
1458.. code-block:: none
1459
1460    output push
1461    output pop
1462    output preset <preset>
1463
1464
1465``output push`` and ``output pop`` allow you to push and pop
1466configurations on an internal configuration stack, so that you
1467can temporarily modify the output configuration, then easily restore
1468the previous configuration.  Simply push before your change to save
1469the current configuration, then pop when you wish to restore the
1470previous configuration.
1471
1472``output preset`` sets Clinic's output to one of several built-in
1473preset configurations, as follows:
1474
1475  ``block``
1476    Clinic's original starting configuration.  Writes everything
1477    immediately after the input block.
1478
1479    Suppress the ``parser_prototype``
1480    and ``docstring_prototype``, write everything else to ``block``.
1481
1482  ``file``
1483    Designed to write everything to the "clinic file" that it can.
1484    You then ``#include`` this file near the top of your file.
1485    You may need to rearrange your file to make this work, though
1486    usually this just means creating forward declarations for various
1487    ``typedef`` and ``PyTypeObject`` definitions.
1488
1489    Suppress the ``parser_prototype``
1490    and ``docstring_prototype``, write the ``impl_definition`` to
1491    ``block``, and write everything else to ``file``.
1492
1493    The default filename is ``"{dirname}/clinic/{basename}.h"``.
1494
1495  ``buffer``
1496    Save up most of the output from Clinic, to be written into
1497    your file near the end.  For Python files implementing modules
1498    or builtin types, it's recommended that you dump the buffer
1499    just above the static structures for your module or
1500    builtin type; these are normally very near the end.  Using
1501    ``buffer`` may require even more editing than ``file``, if
1502    your file has static ``PyMethodDef`` arrays defined in the
1503    middle of the file.
1504
1505    Suppress the ``parser_prototype``, ``impl_prototype``,
1506    and ``docstring_prototype``, write the ``impl_definition`` to
1507    ``block``, and write everything else to ``file``.
1508
1509  ``two-pass``
1510    Similar to the ``buffer`` preset, but writes forward declarations to
1511    the ``two-pass`` buffer, and definitions to the ``buffer``.
1512    This is similar to the ``buffer`` preset, but may require
1513    less editing than ``buffer``.  Dump the ``two-pass`` buffer
1514    near the top of your file, and dump the ``buffer`` near
1515    the end just like you would when using the ``buffer`` preset.
1516
1517    Suppresses the ``impl_prototype``, write the ``impl_definition``
1518    to ``block``, write ``docstring_prototype``, ``methoddef_define``,
1519    and ``parser_prototype`` to ``two-pass``, write everything else
1520    to ``buffer``.
1521
1522  ``partial-buffer``
1523    Similar to the ``buffer`` preset, but writes more things to ``block``,
1524    only writing the really big chunks of generated code to ``buffer``.
1525    This avoids the definition-before-use problem of ``buffer`` completely,
1526    at the small cost of having slightly more stuff in the block's output.
1527    Dump the ``buffer`` near the end, just like you would when using
1528    the ``buffer`` preset.
1529
1530    Suppresses the ``impl_prototype``, write the ``docstring_definition``
1531    and ``parser_definition`` to ``buffer``, write everything else to ``block``.
1532
1533The third new directive is ``destination``:
1534
1535.. code-block:: none
1536
1537    destination <name> <command> [...]
1538
1539This performs an operation on the destination named ``name``.
1540
1541There are two defined subcommands: ``new`` and ``clear``.
1542
1543The ``new`` subcommand works like this:
1544
1545.. code-block:: none
1546
1547    destination <name> new <type>
1548
1549This creates a new destination with name ``<name>`` and type ``<type>``.
1550
1551There are five destination types:
1552
1553    ``suppress``
1554        Throws the text away.
1555
1556    ``block``
1557        Writes the text to the current block.  This is what Clinic
1558        originally did.
1559
1560    ``buffer``
1561        A simple text buffer, like the "buffer" builtin destination above.
1562
1563    ``file``
1564        A text file.  The file destination takes an extra argument,
1565        a template to use for building the filename, like so:
1566
1567            destination <name> new <type> <file_template>
1568
1569        The template can use three strings internally that will be replaced
1570        by bits of the filename:
1571
1572            {path}
1573                The full path to the file, including directory and full filename.
1574            {dirname}
1575                The name of the directory the file is in.
1576            {basename}
1577                Just the name of the file, not including the directory.
1578            {basename_root}
1579                Basename with the extension clipped off
1580                (everything up to but not including the last '.').
1581            {basename_extension}
1582                The last '.' and everything after it.  If the basename
1583                does not contain a period, this will be the empty string.
1584
1585        If there are no periods in the filename, {basename} and {filename}
1586        are the same, and {extension} is empty.  "{basename}{extension}"
1587        is always exactly the same as "{filename}"."
1588
1589    ``two-pass``
1590        A two-pass buffer, like the "two-pass" builtin destination above.
1591
1592
1593The ``clear`` subcommand works like this:
1594
1595.. code-block:: none
1596
1597    destination <name> clear
1598
1599It removes all the accumulated text up to this point in the destination.
1600(I don't know what you'd need this for, but I thought maybe it'd be
1601useful while someone's experimenting.)
1602
1603The fourth new directive is ``set``:
1604
1605.. code-block:: none
1606
1607    set line_prefix "string"
1608    set line_suffix "string"
1609
1610``set`` lets you set two internal variables in Clinic.
1611``line_prefix`` is a string that will be prepended to every line of Clinic's output;
1612``line_suffix`` is a string that will be appended to every line of Clinic's output.
1613
1614Both of these support two format strings:
1615
1616  ``{block comment start}``
1617    Turns into the string ``/*``, the start-comment text sequence for C files.
1618
1619  ``{block comment end}``
1620    Turns into the string ``*/``, the end-comment text sequence for C files.
1621
1622The final new directive is one you shouldn't need to use directly,
1623called ``preserve``:
1624
1625.. code-block:: none
1626
1627    preserve
1628
1629This tells Clinic that the current contents of the output should be kept, unmodified.
1630This is used internally by Clinic when dumping output into ``file`` files; wrapping
1631it in a Clinic block lets Clinic use its existing checksum functionality to ensure
1632the file was not modified by hand before it gets overwritten.
1633
1634
1635The #ifdef trick
1636----------------------------------------------
1637
1638If you're converting a function that isn't available on all platforms,
1639there's a trick you can use to make life a little easier.  The existing
1640code probably looks like this::
1641
1642    #ifdef HAVE_FUNCTIONNAME
1643    static module_functionname(...)
1644    {
1645    ...
1646    }
1647    #endif /* HAVE_FUNCTIONNAME */
1648
1649And then in the ``PyMethodDef`` structure at the bottom the existing code
1650will have:
1651
1652.. code-block:: none
1653
1654    #ifdef HAVE_FUNCTIONNAME
1655    {'functionname', ... },
1656    #endif /* HAVE_FUNCTIONNAME */
1657
1658In this scenario, you should enclose the body of your impl function inside the ``#ifdef``,
1659like so::
1660
1661    #ifdef HAVE_FUNCTIONNAME
1662    /*[clinic input]
1663    module.functionname
1664    ...
1665    [clinic start generated code]*/
1666    static module_functionname(...)
1667    {
1668    ...
1669    }
1670    #endif /* HAVE_FUNCTIONNAME */
1671
1672Then, remove those three lines from the ``PyMethodDef`` structure,
1673replacing them with the macro Argument Clinic generated:
1674
1675.. code-block:: none
1676
1677    MODULE_FUNCTIONNAME_METHODDEF
1678
1679(You can find the real name for this macro inside the generated code.
1680Or you can calculate it yourself: it's the name of your function as defined
1681on the first line of your block, but with periods changed to underscores,
1682uppercased, and ``"_METHODDEF"`` added to the end.)
1683
1684Perhaps you're wondering: what if ``HAVE_FUNCTIONNAME`` isn't defined?
1685The ``MODULE_FUNCTIONNAME_METHODDEF`` macro won't be defined either!
1686
1687Here's where Argument Clinic gets very clever.  It actually detects that the
1688Argument Clinic block might be deactivated by the ``#ifdef``.  When that
1689happens, it generates a little extra code that looks like this::
1690
1691    #ifndef MODULE_FUNCTIONNAME_METHODDEF
1692        #define MODULE_FUNCTIONNAME_METHODDEF
1693    #endif /* !defined(MODULE_FUNCTIONNAME_METHODDEF) */
1694
1695That means the macro always works.  If the function is defined, this turns
1696into the correct structure, including the trailing comma.  If the function is
1697undefined, this turns into nothing.
1698
1699However, this causes one ticklish problem: where should Argument Clinic put this
1700extra code when using the "block" output preset?  It can't go in the output block,
1701because that could be deactivated by the ``#ifdef``.  (That's the whole point!)
1702
1703In this situation, Argument Clinic writes the extra code to the "buffer" destination.
1704This may mean that you get a complaint from Argument Clinic:
1705
1706.. code-block:: none
1707
1708    Warning in file "Modules/posixmodule.c" on line 12357:
1709    Destination buffer 'buffer' not empty at end of file, emptying.
1710
1711When this happens, just open your file, find the ``dump buffer`` block that
1712Argument Clinic added to your file (it'll be at the very bottom), then
1713move it above the ``PyMethodDef`` structure where that macro is used.
1714
1715
1716
1717Using Argument Clinic in Python files
1718-------------------------------------
1719
1720It's actually possible to use Argument Clinic to preprocess Python files.
1721There's no point to using Argument Clinic blocks, of course, as the output
1722wouldn't make any sense to the Python interpreter.  But using Argument Clinic
1723to run Python blocks lets you use Python as a Python preprocessor!
1724
1725Since Python comments are different from C comments, Argument Clinic
1726blocks embedded in Python files look slightly different.  They look like this:
1727
1728.. code-block:: python3
1729
1730    #/*[python input]
1731    #print("def foo(): pass")
1732    #[python start generated code]*/
1733    def foo(): pass
1734    #/*[python checksum:...]*/
1735