1.. highlight:: c 2 3********************** 4Argument Clinic How-To 5********************** 6 7:author: Larry Hastings 8 9 10.. topic:: Abstract 11 12 Argument Clinic is a preprocessor for CPython C files. 13 Its purpose is to automate all the boilerplate involved 14 with writing argument parsing code for "builtins". 15 This document shows you how to convert your first C 16 function to work with Argument Clinic, and then introduces 17 some advanced topics on Argument Clinic usage. 18 19 Currently Argument Clinic is considered internal-only 20 for CPython. Its use is not supported for files outside 21 CPython, and no guarantees are made regarding backwards 22 compatibility for future versions. In other words: if you 23 maintain an external C extension for CPython, you're welcome 24 to experiment with Argument Clinic in your own code. But the 25 version of Argument Clinic that ships with the next version 26 of CPython *could* be totally incompatible and break all your code. 27 28The Goals Of Argument Clinic 29============================ 30 31Argument Clinic's primary goal 32is to take over responsibility for all argument parsing code 33inside CPython. This means that, when you convert a function 34to work with Argument Clinic, that function should no longer 35do any of its own argument parsing—the code generated by 36Argument Clinic should be a "black box" to you, where CPython 37calls in at the top, and your code gets called at the bottom, 38with ``PyObject *args`` (and maybe ``PyObject *kwargs``) 39magically converted into the C variables and types you need. 40 41In order for Argument Clinic to accomplish its primary goal, 42it must be easy to use. Currently, working with CPython's 43argument parsing library is a chore, requiring maintaining 44redundant information in a surprising number of places. 45When you use Argument Clinic, you don't have to repeat yourself. 46 47Obviously, no one would want to use Argument Clinic unless 48it's solving their problem—and without creating new problems of 49its own. 50So it's paramount that Argument Clinic generate correct code. 51It'd be nice if the code was faster, too, but at the very least 52it should not introduce a major speed regression. (Eventually Argument 53Clinic *should* make a major speedup possible—we could 54rewrite its code generator to produce tailor-made argument 55parsing code, rather than calling the general-purpose CPython 56argument parsing library. That would make for the fastest 57argument parsing possible!) 58 59Additionally, Argument Clinic must be flexible enough to 60work with any approach to argument parsing. Python has 61some functions with some very strange parsing behaviors; 62Argument Clinic's goal is to support all of them. 63 64Finally, the original motivation for Argument Clinic was 65to provide introspection "signatures" for CPython builtins. 66It used to be, the introspection query functions would throw 67an exception if you passed in a builtin. With Argument 68Clinic, that's a thing of the past! 69 70One idea you should keep in mind, as you work with 71Argument Clinic: the more information you give it, the 72better job it'll be able to do. 73Argument Clinic is admittedly relatively simple right 74now. But as it evolves it will get more sophisticated, 75and it should be able to do many interesting and smart 76things with all the information you give it. 77 78 79Basic Concepts And Usage 80======================== 81 82Argument Clinic ships with CPython; you'll find it in ``Tools/clinic/clinic.py``. 83If you run that script, specifying a C file as an argument: 84 85.. code-block:: shell-session 86 87 $ python3 Tools/clinic/clinic.py foo.c 88 89Argument Clinic will scan over the file looking for lines that 90look exactly like this: 91 92.. code-block:: none 93 94 /*[clinic input] 95 96When it finds one, it reads everything up to a line that looks 97exactly like this: 98 99.. code-block:: none 100 101 [clinic start generated code]*/ 102 103Everything in between these two lines is input for Argument Clinic. 104All of these lines, including the beginning and ending comment 105lines, are collectively called an Argument Clinic "block". 106 107When Argument Clinic parses one of these blocks, it 108generates output. This output is rewritten into the C file 109immediately after the block, followed by a comment containing a checksum. 110The Argument Clinic block now looks like this: 111 112.. code-block:: none 113 114 /*[clinic input] 115 ... clinic input goes here ... 116 [clinic start generated code]*/ 117 ... clinic output goes here ... 118 /*[clinic end generated code: checksum=...]*/ 119 120If you run Argument Clinic on the same file a second time, Argument Clinic 121will discard the old output and write out the new output with a fresh checksum 122line. However, if the input hasn't changed, the output won't change either. 123 124You should never modify the output portion of an Argument Clinic block. Instead, 125change the input until it produces the output you want. (That's the purpose of the 126checksum—to detect if someone changed the output, as these edits would be lost 127the next time Argument Clinic writes out fresh output.) 128 129For the sake of clarity, here's the terminology we'll use with Argument Clinic: 130 131* The first line of the comment (``/*[clinic input]``) is the *start line*. 132* The last line of the initial comment (``[clinic start generated code]*/``) is the *end line*. 133* The last line (``/*[clinic end generated code: checksum=...]*/``) is the *checksum line*. 134* In between the start line and the end line is the *input*. 135* In between the end line and the checksum line is the *output*. 136* All the text collectively, from the start line to the checksum line inclusively, 137 is the *block*. (A block that hasn't been successfully processed by Argument 138 Clinic yet doesn't have output or a checksum line, but it's still considered 139 a block.) 140 141 142Converting Your First Function 143============================== 144 145The best way to get a sense of how Argument Clinic works is to 146convert a function to work with it. Here, then, are the bare 147minimum steps you'd need to follow to convert a function to 148work with Argument Clinic. Note that for code you plan to 149check in to CPython, you really should take the conversion farther, 150using some of the advanced concepts you'll see later on in 151the document (like "return converters" and "self converters"). 152But we'll keep it simple for this walkthrough so you can learn. 153 154Let's dive in! 155 1560. Make sure you're working with a freshly updated checkout 157 of the CPython trunk. 158 1591. Find a Python builtin that calls either :c:func:`PyArg_ParseTuple` 160 or :c:func:`PyArg_ParseTupleAndKeywords`, and hasn't been converted 161 to work with Argument Clinic yet. 162 For my example I'm using ``_pickle.Pickler.dump()``. 163 1642. If the call to the ``PyArg_Parse`` function uses any of the 165 following format units: 166 167 .. code-block:: none 168 169 O& 170 O! 171 es 172 es# 173 et 174 et# 175 176 or if it has multiple calls to :c:func:`PyArg_ParseTuple`, 177 you should choose a different function. Argument Clinic *does* 178 support all of these scenarios. But these are advanced 179 topics—let's do something simpler for your first function. 180 181 Also, if the function has multiple calls to :c:func:`PyArg_ParseTuple` 182 or :c:func:`PyArg_ParseTupleAndKeywords` where it supports different 183 types for the same argument, or if the function uses something besides 184 PyArg_Parse functions to parse its arguments, it probably 185 isn't suitable for conversion to Argument Clinic. Argument Clinic 186 doesn't support generic functions or polymorphic parameters. 187 1883. Add the following boilerplate above the function, creating our block:: 189 190 /*[clinic input] 191 [clinic start generated code]*/ 192 1934. Cut the docstring and paste it in between the ``[clinic]`` lines, 194 removing all the junk that makes it a properly quoted C string. 195 When you're done you should have just the text, based at the left 196 margin, with no line wider than 80 characters. 197 (Argument Clinic will preserve indents inside the docstring.) 198 199 If the old docstring had a first line that looked like a function 200 signature, throw that line away. (The docstring doesn't need it 201 anymore—when you use ``help()`` on your builtin in the future, 202 the first line will be built automatically based on the function's 203 signature.) 204 205 Sample:: 206 207 /*[clinic input] 208 Write a pickled representation of obj to the open file. 209 [clinic start generated code]*/ 210 2115. If your docstring doesn't have a "summary" line, Argument Clinic will 212 complain. So let's make sure it has one. The "summary" line should 213 be a paragraph consisting of a single 80-column line 214 at the beginning of the docstring. 215 216 (Our example docstring consists solely of a summary line, so the sample 217 code doesn't have to change for this step.) 218 2196. Above the docstring, enter the name of the function, followed 220 by a blank line. This should be the Python name of the function, 221 and should be the full dotted path 222 to the function—it should start with the name of the module, 223 include any sub-modules, and if the function is a method on 224 a class it should include the class name too. 225 226 Sample:: 227 228 /*[clinic input] 229 _pickle.Pickler.dump 230 231 Write a pickled representation of obj to the open file. 232 [clinic start generated code]*/ 233 2347. If this is the first time that module or class has been used with Argument 235 Clinic in this C file, 236 you must declare the module and/or class. Proper Argument Clinic hygiene 237 prefers declaring these in a separate block somewhere near the 238 top of the C file, in the same way that include files and statics go at 239 the top. (In our sample code we'll just show the two blocks next to 240 each other.) 241 242 The name of the class and module should be the same as the one 243 seen by Python. Check the name defined in the :c:type:`PyModuleDef` 244 or :c:type:`PyTypeObject` as appropriate. 245 246 When you declare a class, you must also specify two aspects of its type 247 in C: the type declaration you'd use for a pointer to an instance of 248 this class, and a pointer to the :c:type:`PyTypeObject` for this class. 249 250 Sample:: 251 252 /*[clinic input] 253 module _pickle 254 class _pickle.Pickler "PicklerObject *" "&Pickler_Type" 255 [clinic start generated code]*/ 256 257 /*[clinic input] 258 _pickle.Pickler.dump 259 260 Write a pickled representation of obj to the open file. 261 [clinic start generated code]*/ 262 263 264 265 2668. Declare each of the parameters to the function. Each parameter 267 should get its own line. All the parameter lines should be 268 indented from the function name and the docstring. 269 270 The general form of these parameter lines is as follows: 271 272 .. code-block:: none 273 274 name_of_parameter: converter 275 276 If the parameter has a default value, add that after the 277 converter: 278 279 .. code-block:: none 280 281 name_of_parameter: converter = default_value 282 283 Argument Clinic's support for "default values" is quite sophisticated; 284 please see :ref:`the section below on default values <default_values>` 285 for more information. 286 287 Add a blank line below the parameters. 288 289 What's a "converter"? It establishes both the type 290 of the variable used in C, and the method to convert the Python 291 value into a C value at runtime. 292 For now you're going to use what's called a "legacy converter"—a 293 convenience syntax intended to make porting old code into Argument 294 Clinic easier. 295 296 For each parameter, copy the "format unit" for that 297 parameter from the ``PyArg_Parse()`` format argument and 298 specify *that* as its converter, as a quoted 299 string. ("format unit" is the formal name for the one-to-three 300 character substring of the ``format`` parameter that tells 301 the argument parsing function what the type of the variable 302 is and how to convert it. For more on format units please 303 see :ref:`arg-parsing`.) 304 305 For multicharacter format units like ``z#``, use the 306 entire two-or-three character string. 307 308 Sample:: 309 310 /*[clinic input] 311 module _pickle 312 class _pickle.Pickler "PicklerObject *" "&Pickler_Type" 313 [clinic start generated code]*/ 314 315 /*[clinic input] 316 _pickle.Pickler.dump 317 318 obj: 'O' 319 320 Write a pickled representation of obj to the open file. 321 [clinic start generated code]*/ 322 3239. If your function has ``|`` in the format string, meaning some 324 parameters have default values, you can ignore it. Argument 325 Clinic infers which parameters are optional based on whether 326 or not they have default values. 327 328 If your function has ``$`` in the format string, meaning it 329 takes keyword-only arguments, specify ``*`` on a line by 330 itself before the first keyword-only argument, indented the 331 same as the parameter lines. 332 333 (``_pickle.Pickler.dump`` has neither, so our sample is unchanged.) 334 335 33610. If the existing C function calls :c:func:`PyArg_ParseTuple` 337 (as opposed to :c:func:`PyArg_ParseTupleAndKeywords`), then all its 338 arguments are positional-only. 339 340 To mark all parameters as positional-only in Argument Clinic, 341 add a ``/`` on a line by itself after the last parameter, 342 indented the same as the parameter lines. 343 344 Currently this is all-or-nothing; either all parameters are 345 positional-only, or none of them are. (In the future Argument 346 Clinic may relax this restriction.) 347 348 Sample:: 349 350 /*[clinic input] 351 module _pickle 352 class _pickle.Pickler "PicklerObject *" "&Pickler_Type" 353 [clinic start generated code]*/ 354 355 /*[clinic input] 356 _pickle.Pickler.dump 357 358 obj: 'O' 359 / 360 361 Write a pickled representation of obj to the open file. 362 [clinic start generated code]*/ 363 36411. It's helpful to write a per-parameter docstring for each parameter. 365 But per-parameter docstrings are optional; you can skip this step 366 if you prefer. 367 368 Here's how to add a per-parameter docstring. The first line 369 of the per-parameter docstring must be indented further than the 370 parameter definition. The left margin of this first line establishes 371 the left margin for the whole per-parameter docstring; all the text 372 you write will be outdented by this amount. You can write as much 373 text as you like, across multiple lines if you wish. 374 375 Sample:: 376 377 /*[clinic input] 378 module _pickle 379 class _pickle.Pickler "PicklerObject *" "&Pickler_Type" 380 [clinic start generated code]*/ 381 382 /*[clinic input] 383 _pickle.Pickler.dump 384 385 obj: 'O' 386 The object to be pickled. 387 / 388 389 Write a pickled representation of obj to the open file. 390 [clinic start generated code]*/ 391 39212. Save and close the file, then run ``Tools/clinic/clinic.py`` on 393 it. With luck everything worked---your block now has output, and 394 a ``.c.h`` file has been generated! Reopen the file in your 395 text editor to see:: 396 397 /*[clinic input] 398 _pickle.Pickler.dump 399 400 obj: 'O' 401 The object to be pickled. 402 / 403 404 Write a pickled representation of obj to the open file. 405 [clinic start generated code]*/ 406 407 static PyObject * 408 _pickle_Pickler_dump(PicklerObject *self, PyObject *obj) 409 /*[clinic end generated code: output=87ecad1261e02ac7 input=552eb1c0f52260d9]*/ 410 411 Obviously, if Argument Clinic didn't produce any output, it's because 412 it found an error in your input. Keep fixing your errors and retrying 413 until Argument Clinic processes your file without complaint. 414 415 For readability, most of the glue code has been generated to a ``.c.h`` 416 file. You'll need to include that in your original ``.c`` file, 417 typically right after the clinic module block:: 418 419 #include "clinic/_pickle.c.h" 420 42113. Double-check that the argument-parsing code Argument Clinic generated 422 looks basically the same as the existing code. 423 424 First, ensure both places use the same argument-parsing function. 425 The existing code must call either 426 :c:func:`PyArg_ParseTuple` or :c:func:`PyArg_ParseTupleAndKeywords`; 427 ensure that the code generated by Argument Clinic calls the 428 *exact* same function. 429 430 Second, the format string passed in to :c:func:`PyArg_ParseTuple` or 431 :c:func:`PyArg_ParseTupleAndKeywords` should be *exactly* the same 432 as the hand-written one in the existing function, up to the colon 433 or semi-colon. 434 435 (Argument Clinic always generates its format strings 436 with a ``:`` followed by the name of the function. If the 437 existing code's format string ends with ``;``, to provide 438 usage help, this change is harmless—don't worry about it.) 439 440 Third, for parameters whose format units require two arguments 441 (like a length variable, or an encoding string, or a pointer 442 to a conversion function), ensure that the second argument is 443 *exactly* the same between the two invocations. 444 445 Fourth, inside the output portion of the block you'll find a preprocessor 446 macro defining the appropriate static :c:type:`PyMethodDef` structure for 447 this builtin:: 448 449 #define __PICKLE_PICKLER_DUMP_METHODDEF \ 450 {"dump", (PyCFunction)__pickle_Pickler_dump, METH_O, __pickle_Pickler_dump__doc__}, 451 452 This static structure should be *exactly* the same as the existing static 453 :c:type:`PyMethodDef` structure for this builtin. 454 455 If any of these items differ in *any way*, 456 adjust your Argument Clinic function specification and rerun 457 ``Tools/clinic/clinic.py`` until they *are* the same. 458 459 46014. Notice that the last line of its output is the declaration 461 of your "impl" function. This is where the builtin's implementation goes. 462 Delete the existing prototype of the function you're modifying, but leave 463 the opening curly brace. Now delete its argument parsing code and the 464 declarations of all the variables it dumps the arguments into. 465 Notice how the Python arguments are now arguments to this impl function; 466 if the implementation used different names for these variables, fix it. 467 468 Let's reiterate, just because it's kind of weird. Your code should now 469 look like this:: 470 471 static return_type 472 your_function_impl(...) 473 /*[clinic end generated code: checksum=...]*/ 474 { 475 ... 476 477 Argument Clinic generated the checksum line and the function prototype just 478 above it. You should write the opening (and closing) curly braces for the 479 function, and the implementation inside. 480 481 Sample:: 482 483 /*[clinic input] 484 module _pickle 485 class _pickle.Pickler "PicklerObject *" "&Pickler_Type" 486 [clinic start generated code]*/ 487 /*[clinic end generated code: checksum=da39a3ee5e6b4b0d3255bfef95601890afd80709]*/ 488 489 /*[clinic input] 490 _pickle.Pickler.dump 491 492 obj: 'O' 493 The object to be pickled. 494 / 495 496 Write a pickled representation of obj to the open file. 497 [clinic start generated code]*/ 498 499 PyDoc_STRVAR(__pickle_Pickler_dump__doc__, 500 "Write a pickled representation of obj to the open file.\n" 501 "\n" 502 ... 503 static PyObject * 504 _pickle_Pickler_dump_impl(PicklerObject *self, PyObject *obj) 505 /*[clinic end generated code: checksum=3bd30745bf206a48f8b576a1da3d90f55a0a4187]*/ 506 { 507 /* Check whether the Pickler was initialized correctly (issue3664). 508 Developers often forget to call __init__() in their subclasses, which 509 would trigger a segfault without this check. */ 510 if (self->write == NULL) { 511 PyErr_Format(PicklingError, 512 "Pickler.__init__() was not called by %s.__init__()", 513 Py_TYPE(self)->tp_name); 514 return NULL; 515 } 516 517 if (_Pickler_ClearBuffer(self) < 0) 518 return NULL; 519 520 ... 521 52215. Remember the macro with the :c:type:`PyMethodDef` structure for this 523 function? Find the existing :c:type:`PyMethodDef` structure for this 524 function and replace it with a reference to the macro. (If the builtin 525 is at module scope, this will probably be very near the end of the file; 526 if the builtin is a class method, this will probably be below but relatively 527 near to the implementation.) 528 529 Note that the body of the macro contains a trailing comma. So when you 530 replace the existing static :c:type:`PyMethodDef` structure with the macro, 531 *don't* add a comma to the end. 532 533 Sample:: 534 535 static struct PyMethodDef Pickler_methods[] = { 536 __PICKLE_PICKLER_DUMP_METHODDEF 537 __PICKLE_PICKLER_CLEAR_MEMO_METHODDEF 538 {NULL, NULL} /* sentinel */ 539 }; 540 541 54216. Compile, then run the relevant portions of the regression-test suite. 543 This change should not introduce any new compile-time warnings or errors, 544 and there should be no externally-visible change to Python's behavior. 545 546 Well, except for one difference: ``inspect.signature()`` run on your function 547 should now provide a valid signature! 548 549 Congratulations, you've ported your first function to work with Argument Clinic! 550 551Advanced Topics 552=============== 553 554Now that you've had some experience working with Argument Clinic, it's time 555for some advanced topics. 556 557 558Symbolic default values 559----------------------- 560 561The default value you provide for a parameter can't be any arbitrary 562expression. Currently the following are explicitly supported: 563 564* Numeric constants (integer and float) 565* String constants 566* ``True``, ``False``, and ``None`` 567* Simple symbolic constants like ``sys.maxsize``, which must 568 start with the name of the module 569 570In case you're curious, this is implemented in ``from_builtin()`` 571in ``Lib/inspect.py``. 572 573(In the future, this may need to get even more elaborate, 574to allow full expressions like ``CONSTANT - 1``.) 575 576 577Renaming the C functions and variables generated by Argument Clinic 578------------------------------------------------------------------- 579 580Argument Clinic automatically names the functions it generates for you. 581Occasionally this may cause a problem, if the generated name collides with 582the name of an existing C function. There's an easy solution: override the names 583used for the C functions. Just add the keyword ``"as"`` 584to your function declaration line, followed by the function name you wish to use. 585Argument Clinic will use that function name for the base (generated) function, 586then add ``"_impl"`` to the end and use that for the name of the impl function. 587 588For example, if we wanted to rename the C function names generated for 589``pickle.Pickler.dump``, it'd look like this:: 590 591 /*[clinic input] 592 pickle.Pickler.dump as pickler_dumper 593 594 ... 595 596The base function would now be named ``pickler_dumper()``, 597and the impl function would now be named ``pickler_dumper_impl()``. 598 599 600Similarly, you may have a problem where you want to give a parameter 601a specific Python name, but that name may be inconvenient in C. Argument 602Clinic allows you to give a parameter different names in Python and in C, 603using the same ``"as"`` syntax:: 604 605 /*[clinic input] 606 pickle.Pickler.dump 607 608 obj: object 609 file as file_obj: object 610 protocol: object = NULL 611 * 612 fix_imports: bool = True 613 614Here, the name used in Python (in the signature and the ``keywords`` 615array) would be ``file``, but the C variable would be named ``file_obj``. 616 617You can use this to rename the ``self`` parameter too! 618 619 620Converting functions using PyArg_UnpackTuple 621-------------------------------------------- 622 623To convert a function parsing its arguments with :c:func:`PyArg_UnpackTuple`, 624simply write out all the arguments, specifying each as an ``object``. You 625may specify the ``type`` argument to cast the type as appropriate. All 626arguments should be marked positional-only (add a ``/`` on a line by itself 627after the last argument). 628 629Currently the generated code will use :c:func:`PyArg_ParseTuple`, but this 630will change soon. 631 632Optional Groups 633--------------- 634 635Some legacy functions have a tricky approach to parsing their arguments: 636they count the number of positional arguments, then use a ``switch`` statement 637to call one of several different :c:func:`PyArg_ParseTuple` calls depending on 638how many positional arguments there are. (These functions cannot accept 639keyword-only arguments.) This approach was used to simulate optional 640arguments back before :c:func:`PyArg_ParseTupleAndKeywords` was created. 641 642While functions using this approach can often be converted to 643use :c:func:`PyArg_ParseTupleAndKeywords`, optional arguments, and default values, 644it's not always possible. Some of these legacy functions have 645behaviors :c:func:`PyArg_ParseTupleAndKeywords` doesn't directly support. 646The most obvious example is the builtin function ``range()``, which has 647an optional argument on the *left* side of its required argument! 648Another example is ``curses.window.addch()``, which has a group of two 649arguments that must always be specified together. (The arguments are 650called ``x`` and ``y``; if you call the function passing in ``x``, 651you must also pass in ``y``—and if you don't pass in ``x`` you may not 652pass in ``y`` either.) 653 654In any case, the goal of Argument Clinic is to support argument parsing 655for all existing CPython builtins without changing their semantics. 656Therefore Argument Clinic supports 657this alternate approach to parsing, using what are called *optional groups*. 658Optional groups are groups of arguments that must all be passed in together. 659They can be to the left or the right of the required arguments. They 660can *only* be used with positional-only parameters. 661 662.. note:: Optional groups are *only* intended for use when converting 663 functions that make multiple calls to :c:func:`PyArg_ParseTuple`! 664 Functions that use *any* other approach for parsing arguments 665 should *almost never* be converted to Argument Clinic using 666 optional groups. Functions using optional groups currently 667 cannot have accurate signatures in Python, because Python just 668 doesn't understand the concept. Please avoid using optional 669 groups wherever possible. 670 671To specify an optional group, add a ``[`` on a line by itself before 672the parameters you wish to group together, and a ``]`` on a line by itself 673after these parameters. As an example, here's how ``curses.window.addch`` 674uses optional groups to make the first two parameters and the last 675parameter optional:: 676 677 /*[clinic input] 678 679 curses.window.addch 680 681 [ 682 x: int 683 X-coordinate. 684 y: int 685 Y-coordinate. 686 ] 687 688 ch: object 689 Character to add. 690 691 [ 692 attr: long 693 Attributes for the character. 694 ] 695 / 696 697 ... 698 699 700Notes: 701 702* For every optional group, one additional parameter will be passed into the 703 impl function representing the group. The parameter will be an int named 704 ``group_{direction}_{number}``, 705 where ``{direction}`` is either ``right`` or ``left`` depending on whether the group 706 is before or after the required parameters, and ``{number}`` is a monotonically 707 increasing number (starting at 1) indicating how far away the group is from 708 the required parameters. When the impl is called, this parameter will be set 709 to zero if this group was unused, and set to non-zero if this group was used. 710 (By used or unused, I mean whether or not the parameters received arguments 711 in this invocation.) 712 713* If there are no required arguments, the optional groups will behave 714 as if they're to the right of the required arguments. 715 716* In the case of ambiguity, the argument parsing code 717 favors parameters on the left (before the required parameters). 718 719* Optional groups can only contain positional-only parameters. 720 721* Optional groups are *only* intended for legacy code. Please do not 722 use optional groups for new code. 723 724 725Using real Argument Clinic converters, instead of "legacy converters" 726--------------------------------------------------------------------- 727 728To save time, and to minimize how much you need to learn 729to achieve your first port to Argument Clinic, the walkthrough above tells 730you to use "legacy converters". "Legacy converters" are a convenience, 731designed explicitly to make porting existing code to Argument Clinic 732easier. And to be clear, their use is acceptable when porting code for 733Python 3.4. 734 735However, in the long term we probably want all our blocks to 736use Argument Clinic's real syntax for converters. Why? A couple 737reasons: 738 739* The proper converters are far easier to read and clearer in their intent. 740* There are some format units that are unsupported as "legacy converters", 741 because they require arguments, and the legacy converter syntax doesn't 742 support specifying arguments. 743* In the future we may have a new argument parsing library that isn't 744 restricted to what :c:func:`PyArg_ParseTuple` supports; this flexibility 745 won't be available to parameters using legacy converters. 746 747Therefore, if you don't mind a little extra effort, please use the normal 748converters instead of legacy converters. 749 750In a nutshell, the syntax for Argument Clinic (non-legacy) converters 751looks like a Python function call. However, if there are no explicit 752arguments to the function (all functions take their default values), 753you may omit the parentheses. Thus ``bool`` and ``bool()`` are exactly 754the same converters. 755 756All arguments to Argument Clinic converters are keyword-only. 757All Argument Clinic converters accept the following arguments: 758 759 ``c_default`` 760 The default value for this parameter when defined in C. 761 Specifically, this will be the initializer for the variable declared 762 in the "parse function". See :ref:`the section on default values <default_values>` 763 for how to use this. 764 Specified as a string. 765 766 ``annotation`` 767 The annotation value for this parameter. Not currently supported, 768 because :pep:`8` mandates that the Python library may not use 769 annotations. 770 771In addition, some converters accept additional arguments. Here is a list 772of these arguments, along with their meanings: 773 774 ``accept`` 775 A set of Python types (and possibly pseudo-types); 776 this restricts the allowable Python argument to values of these types. 777 (This is not a general-purpose facility; as a rule it only supports 778 specific lists of types as shown in the legacy converter table.) 779 780 To accept ``None``, add ``NoneType`` to this set. 781 782 ``bitwise`` 783 Only supported for unsigned integers. The native integer value of this 784 Python argument will be written to the parameter without any range checking, 785 even for negative values. 786 787 ``converter`` 788 Only supported by the ``object`` converter. Specifies the name of a 789 :ref:`C "converter function" <o_ampersand>` 790 to use to convert this object to a native type. 791 792 ``encoding`` 793 Only supported for strings. Specifies the encoding to use when converting 794 this string from a Python str (Unicode) value into a C ``char *`` value. 795 796 797 ``subclass_of`` 798 Only supported for the ``object`` converter. Requires that the Python 799 value be a subclass of a Python type, as expressed in C. 800 801 ``type`` 802 Only supported for the ``object`` and ``self`` converters. Specifies 803 the C type that will be used to declare the variable. Default value is 804 ``"PyObject *"``. 805 806 ``zeroes`` 807 Only supported for strings. If true, embedded NUL bytes (``'\\0'``) are 808 permitted inside the value. The length of the string will be passed in 809 to the impl function, just after the string parameter, as a parameter named 810 ``<parameter_name>_length``. 811 812Please note, not every possible combination of arguments will work. 813Usually these arguments are implemented by specific ``PyArg_ParseTuple`` 814*format units*, with specific behavior. For example, currently you cannot 815call ``unsigned_short`` without also specifying ``bitwise=True``. 816Although it's perfectly reasonable to think this would work, these semantics don't 817map to any existing format unit. So Argument Clinic doesn't support it. (Or, at 818least, not yet.) 819 820Below is a table showing the mapping of legacy converters into real 821Argument Clinic converters. On the left is the legacy converter, 822on the right is the text you'd replace it with. 823 824========= ================================================================================= 825``'B'`` ``unsigned_char(bitwise=True)`` 826``'b'`` ``unsigned_char`` 827``'c'`` ``char`` 828``'C'`` ``int(accept={str})`` 829``'d'`` ``double`` 830``'D'`` ``Py_complex`` 831``'es'`` ``str(encoding='name_of_encoding')`` 832``'es#'`` ``str(encoding='name_of_encoding', zeroes=True)`` 833``'et'`` ``str(encoding='name_of_encoding', accept={bytes, bytearray, str})`` 834``'et#'`` ``str(encoding='name_of_encoding', accept={bytes, bytearray, str}, zeroes=True)`` 835``'f'`` ``float`` 836``'h'`` ``short`` 837``'H'`` ``unsigned_short(bitwise=True)`` 838``'i'`` ``int`` 839``'I'`` ``unsigned_int(bitwise=True)`` 840``'k'`` ``unsigned_long(bitwise=True)`` 841``'K'`` ``unsigned_long_long(bitwise=True)`` 842``'l'`` ``long`` 843``'L'`` ``long long`` 844``'n'`` ``Py_ssize_t`` 845``'O'`` ``object`` 846``'O!'`` ``object(subclass_of='&PySomething_Type')`` 847``'O&'`` ``object(converter='name_of_c_function')`` 848``'p'`` ``bool`` 849``'S'`` ``PyBytesObject`` 850``'s'`` ``str`` 851``'s#'`` ``str(zeroes=True)`` 852``'s*'`` ``Py_buffer(accept={buffer, str})`` 853``'U'`` ``unicode`` 854``'u'`` ``Py_UNICODE`` 855``'u#'`` ``Py_UNICODE(zeroes=True)`` 856``'w*'`` ``Py_buffer(accept={rwbuffer})`` 857``'Y'`` ``PyByteArrayObject`` 858``'y'`` ``str(accept={bytes})`` 859``'y#'`` ``str(accept={robuffer}, zeroes=True)`` 860``'y*'`` ``Py_buffer`` 861``'Z'`` ``Py_UNICODE(accept={str, NoneType})`` 862``'Z#'`` ``Py_UNICODE(accept={str, NoneType}, zeroes=True)`` 863``'z'`` ``str(accept={str, NoneType})`` 864``'z#'`` ``str(accept={str, NoneType}, zeroes=True)`` 865``'z*'`` ``Py_buffer(accept={buffer, str, NoneType})`` 866========= ================================================================================= 867 868As an example, here's our sample ``pickle.Pickler.dump`` using the proper 869converter:: 870 871 /*[clinic input] 872 pickle.Pickler.dump 873 874 obj: object 875 The object to be pickled. 876 / 877 878 Write a pickled representation of obj to the open file. 879 [clinic start generated code]*/ 880 881One advantage of real converters is that they're more flexible than legacy 882converters. For example, the ``unsigned_int`` converter (and all the 883``unsigned_`` converters) can be specified without ``bitwise=True``. Their 884default behavior performs range checking on the value, and they won't accept 885negative numbers. You just can't do that with a legacy converter! 886 887Argument Clinic will show you all the converters it has 888available. For each converter it'll show you all the parameters 889it accepts, along with the default value for each parameter. 890Just run ``Tools/clinic/clinic.py --converters`` to see the full list. 891 892Py_buffer 893--------- 894 895When using the ``Py_buffer`` converter 896(or the ``'s*'``, ``'w*'``, ``'*y'``, or ``'z*'`` legacy converters), 897you *must* not call :c:func:`PyBuffer_Release` on the provided buffer. 898Argument Clinic generates code that does it for you (in the parsing function). 899 900 901 902Advanced converters 903------------------- 904 905Remember those format units you skipped for your first 906time because they were advanced? Here's how to handle those too. 907 908The trick is, all those format units take arguments—either 909conversion functions, or types, or strings specifying an encoding. 910(But "legacy converters" don't support arguments. That's why we 911skipped them for your first function.) The argument you specified 912to the format unit is now an argument to the converter; this 913argument is either ``converter`` (for ``O&``), ``subclass_of`` (for ``O!``), 914or ``encoding`` (for all the format units that start with ``e``). 915 916When using ``subclass_of``, you may also want to use the other 917custom argument for ``object()``: ``type``, which lets you set the type 918actually used for the parameter. For example, if you want to ensure 919that the object is a subclass of ``PyUnicode_Type``, you probably want 920to use the converter ``object(type='PyUnicodeObject *', subclass_of='&PyUnicode_Type')``. 921 922One possible problem with using Argument Clinic: it takes away some possible 923flexibility for the format units starting with ``e``. When writing a 924``PyArg_Parse`` call by hand, you could theoretically decide at runtime what 925encoding string to pass in to :c:func:`PyArg_ParseTuple`. But now this string must 926be hard-coded at Argument-Clinic-preprocessing-time. This limitation is deliberate; 927it made supporting this format unit much easier, and may allow for future optimizations. 928This restriction doesn't seem unreasonable; CPython itself always passes in static 929hard-coded encoding strings for parameters whose format units start with ``e``. 930 931 932.. _default_values: 933 934Parameter default values 935------------------------ 936 937Default values for parameters can be any of a number of values. 938At their simplest, they can be string, int, or float literals: 939 940.. code-block:: none 941 942 foo: str = "abc" 943 bar: int = 123 944 bat: float = 45.6 945 946They can also use any of Python's built-in constants: 947 948.. code-block:: none 949 950 yep: bool = True 951 nope: bool = False 952 nada: object = None 953 954There's also special support for a default value of ``NULL``, and 955for simple expressions, documented in the following sections. 956 957 958The ``NULL`` default value 959-------------------------- 960 961For string and object parameters, you can set them to ``None`` to indicate 962that there's no default. However, that means the C variable will be 963initialized to ``Py_None``. For convenience's sakes, there's a special 964value called ``NULL`` for just this reason: from Python's perspective it 965behaves like a default value of ``None``, but the C variable is initialized 966with ``NULL``. 967 968Expressions specified as default values 969--------------------------------------- 970 971The default value for a parameter can be more than just a literal value. 972It can be an entire expression, using math operators and looking up attributes 973on objects. However, this support isn't exactly simple, because of some 974non-obvious semantics. 975 976Consider the following example: 977 978.. code-block:: none 979 980 foo: Py_ssize_t = sys.maxsize - 1 981 982``sys.maxsize`` can have different values on different platforms. Therefore 983Argument Clinic can't simply evaluate that expression locally and hard-code it 984in C. So it stores the default in such a way that it will get evaluated at 985runtime, when the user asks for the function's signature. 986 987What namespace is available when the expression is evaluated? It's evaluated 988in the context of the module the builtin came from. So, if your module has an 989attribute called "``max_widgets``", you may simply use it: 990 991.. code-block:: none 992 993 foo: Py_ssize_t = max_widgets 994 995If the symbol isn't found in the current module, it fails over to looking in 996``sys.modules``. That's how it can find ``sys.maxsize`` for example. (Since you 997don't know in advance what modules the user will load into their interpreter, 998it's best to restrict yourself to modules that are preloaded by Python itself.) 999 1000Evaluating default values only at runtime means Argument Clinic can't compute 1001the correct equivalent C default value. So you need to tell it explicitly. 1002When you use an expression, you must also specify the equivalent expression 1003in C, using the ``c_default`` parameter to the converter: 1004 1005.. code-block:: none 1006 1007 foo: Py_ssize_t(c_default="PY_SSIZE_T_MAX - 1") = sys.maxsize - 1 1008 1009Another complication: Argument Clinic can't know in advance whether or not the 1010expression you supply is valid. It parses it to make sure it looks legal, but 1011it can't *actually* know. You must be very careful when using expressions to 1012specify values that are guaranteed to be valid at runtime! 1013 1014Finally, because expressions must be representable as static C values, there 1015are many restrictions on legal expressions. Here's a list of Python features 1016you're not permitted to use: 1017 1018* Function calls. 1019* Inline if statements (``3 if foo else 5``). 1020* Automatic sequence unpacking (``*[1, 2, 3]``). 1021* List/set/dict comprehensions and generator expressions. 1022* Tuple/list/set/dict literals. 1023 1024 1025 1026Using a return converter 1027------------------------ 1028 1029By default the impl function Argument Clinic generates for you returns ``PyObject *``. 1030But your C function often computes some C type, then converts it into the ``PyObject *`` 1031at the last moment. Argument Clinic handles converting your inputs from Python types 1032into native C types—why not have it convert your return value from a native C type 1033into a Python type too? 1034 1035That's what a "return converter" does. It changes your impl function to return 1036some C type, then adds code to the generated (non-impl) function to handle converting 1037that value into the appropriate ``PyObject *``. 1038 1039The syntax for return converters is similar to that of parameter converters. 1040You specify the return converter like it was a return annotation on the 1041function itself. Return converters behave much the same as parameter converters; 1042they take arguments, the arguments are all keyword-only, and if you're not changing 1043any of the default arguments you can omit the parentheses. 1044 1045(If you use both ``"as"`` *and* a return converter for your function, 1046the ``"as"`` should come before the return converter.) 1047 1048There's one additional complication when using return converters: how do you 1049indicate an error has occurred? Normally, a function returns a valid (non-``NULL``) 1050pointer for success, and ``NULL`` for failure. But if you use an integer return converter, 1051all integers are valid. How can Argument Clinic detect an error? Its solution: each return 1052converter implicitly looks for a special value that indicates an error. If you return 1053that value, and an error has been set (``PyErr_Occurred()`` returns a true 1054value), then the generated code will propagate the error. Otherwise it will 1055encode the value you return like normal. 1056 1057Currently Argument Clinic supports only a few return converters: 1058 1059.. code-block:: none 1060 1061 bool 1062 int 1063 unsigned int 1064 long 1065 unsigned int 1066 size_t 1067 Py_ssize_t 1068 float 1069 double 1070 DecodeFSDefault 1071 1072None of these take parameters. For the first three, return -1 to indicate 1073error. For ``DecodeFSDefault``, the return type is ``const char *``; return a ``NULL`` 1074pointer to indicate an error. 1075 1076(There's also an experimental ``NoneType`` converter, which lets you 1077return ``Py_None`` on success or ``NULL`` on failure, without having 1078to increment the reference count on ``Py_None``. I'm not sure it adds 1079enough clarity to be worth using.) 1080 1081To see all the return converters Argument Clinic supports, along with 1082their parameters (if any), 1083just run ``Tools/clinic/clinic.py --converters`` for the full list. 1084 1085 1086Cloning existing functions 1087-------------------------- 1088 1089If you have a number of functions that look similar, you may be able to 1090use Clinic's "clone" feature. When you clone an existing function, 1091you reuse: 1092 1093* its parameters, including 1094 1095 * their names, 1096 1097 * their converters, with all parameters, 1098 1099 * their default values, 1100 1101 * their per-parameter docstrings, 1102 1103 * their *kind* (whether they're positional only, 1104 positional or keyword, or keyword only), and 1105 1106* its return converter. 1107 1108The only thing not copied from the original function is its docstring; 1109the syntax allows you to specify a new docstring. 1110 1111Here's the syntax for cloning a function:: 1112 1113 /*[clinic input] 1114 module.class.new_function [as c_basename] = module.class.existing_function 1115 1116 Docstring for new_function goes here. 1117 [clinic start generated code]*/ 1118 1119(The functions can be in different modules or classes. I wrote 1120``module.class`` in the sample just to illustrate that you must 1121use the full path to *both* functions.) 1122 1123Sorry, there's no syntax for partially-cloning a function, or cloning a function 1124then modifying it. Cloning is an all-or nothing proposition. 1125 1126Also, the function you are cloning from must have been previously defined 1127in the current file. 1128 1129Calling Python code 1130------------------- 1131 1132The rest of the advanced topics require you to write Python code 1133which lives inside your C file and modifies Argument Clinic's 1134runtime state. This is simple: you simply define a Python block. 1135 1136A Python block uses different delimiter lines than an Argument 1137Clinic function block. It looks like this:: 1138 1139 /*[python input] 1140 # python code goes here 1141 [python start generated code]*/ 1142 1143All the code inside the Python block is executed at the 1144time it's parsed. All text written to stdout inside the block 1145is redirected into the "output" after the block. 1146 1147As an example, here's a Python block that adds a static integer 1148variable to the C code:: 1149 1150 /*[python input] 1151 print('static int __ignored_unused_variable__ = 0;') 1152 [python start generated code]*/ 1153 static int __ignored_unused_variable__ = 0; 1154 /*[python checksum:...]*/ 1155 1156 1157Using a "self converter" 1158------------------------ 1159 1160Argument Clinic automatically adds a "self" parameter for you 1161using a default converter. It automatically sets the ``type`` 1162of this parameter to the "pointer to an instance" you specified 1163when you declared the type. However, you can override 1164Argument Clinic's converter and specify one yourself. 1165Just add your own ``self`` parameter as the first parameter in a 1166block, and ensure that its converter is an instance of 1167``self_converter`` or a subclass thereof. 1168 1169What's the point? This lets you override the type of ``self``, 1170or give it a different default name. 1171 1172How do you specify the custom type you want to cast ``self`` to? 1173If you only have one or two functions with the same type for ``self``, 1174you can directly use Argument Clinic's existing ``self`` converter, 1175passing in the type you want to use as the ``type`` parameter:: 1176 1177 /*[clinic input] 1178 1179 _pickle.Pickler.dump 1180 1181 self: self(type="PicklerObject *") 1182 obj: object 1183 / 1184 1185 Write a pickled representation of the given object to the open file. 1186 [clinic start generated code]*/ 1187 1188On the other hand, if you have a lot of functions that will use the same 1189type for ``self``, it's best to create your own converter, subclassing 1190``self_converter`` but overwriting the ``type`` member:: 1191 1192 /*[python input] 1193 class PicklerObject_converter(self_converter): 1194 type = "PicklerObject *" 1195 [python start generated code]*/ 1196 1197 /*[clinic input] 1198 1199 _pickle.Pickler.dump 1200 1201 self: PicklerObject 1202 obj: object 1203 / 1204 1205 Write a pickled representation of the given object to the open file. 1206 [clinic start generated code]*/ 1207 1208 1209 1210Writing a custom converter 1211-------------------------- 1212 1213As we hinted at in the previous section... you can write your own converters! 1214A converter is simply a Python class that inherits from ``CConverter``. 1215The main purpose of a custom converter is if you have a parameter using 1216the ``O&`` format unit—parsing this parameter means calling 1217a :c:func:`PyArg_ParseTuple` "converter function". 1218 1219Your converter class should be named ``*something*_converter``. 1220If the name follows this convention, then your converter class 1221will be automatically registered with Argument Clinic; its name 1222will be the name of your class with the ``_converter`` suffix 1223stripped off. (This is accomplished with a metaclass.) 1224 1225You shouldn't subclass ``CConverter.__init__``. Instead, you should 1226write a ``converter_init()`` function. ``converter_init()`` 1227always accepts a ``self`` parameter; after that, all additional 1228parameters *must* be keyword-only. Any arguments passed in to 1229the converter in Argument Clinic will be passed along to your 1230``converter_init()``. 1231 1232There are some additional members of ``CConverter`` you may wish 1233to specify in your subclass. Here's the current list: 1234 1235``type`` 1236 The C type to use for this variable. 1237 ``type`` should be a Python string specifying the type, e.g. ``int``. 1238 If this is a pointer type, the type string should end with ``' *'``. 1239 1240``default`` 1241 The Python default value for this parameter, as a Python value. 1242 Or the magic value ``unspecified`` if there is no default. 1243 1244``py_default`` 1245 ``default`` as it should appear in Python code, 1246 as a string. 1247 Or ``None`` if there is no default. 1248 1249``c_default`` 1250 ``default`` as it should appear in C code, 1251 as a string. 1252 Or ``None`` if there is no default. 1253 1254``c_ignored_default`` 1255 The default value used to initialize the C variable when 1256 there is no default, but not specifying a default may 1257 result in an "uninitialized variable" warning. This can 1258 easily happen when using option groups—although 1259 properly-written code will never actually use this value, 1260 the variable does get passed in to the impl, and the 1261 C compiler will complain about the "use" of the 1262 uninitialized value. This value should always be a 1263 non-empty string. 1264 1265``converter`` 1266 The name of the C converter function, as a string. 1267 1268``impl_by_reference`` 1269 A boolean value. If true, 1270 Argument Clinic will add a ``&`` in front of the name of 1271 the variable when passing it into the impl function. 1272 1273``parse_by_reference`` 1274 A boolean value. If true, 1275 Argument Clinic will add a ``&`` in front of the name of 1276 the variable when passing it into :c:func:`PyArg_ParseTuple`. 1277 1278 1279Here's the simplest example of a custom converter, from ``Modules/zlibmodule.c``:: 1280 1281 /*[python input] 1282 1283 class ssize_t_converter(CConverter): 1284 type = 'Py_ssize_t' 1285 converter = 'ssize_t_converter' 1286 1287 [python start generated code]*/ 1288 /*[python end generated code: output=da39a3ee5e6b4b0d input=35521e4e733823c7]*/ 1289 1290This block adds a converter to Argument Clinic named ``ssize_t``. Parameters 1291declared as ``ssize_t`` will be declared as type ``Py_ssize_t``, and will 1292be parsed by the ``'O&'`` format unit, which will call the 1293``ssize_t_converter`` converter function. ``ssize_t`` variables 1294automatically support default values. 1295 1296More sophisticated custom converters can insert custom C code to 1297handle initialization and cleanup. 1298You can see more examples of custom converters in the CPython 1299source tree; grep the C files for the string ``CConverter``. 1300 1301Writing a custom return converter 1302--------------------------------- 1303 1304Writing a custom return converter is much like writing 1305a custom converter. Except it's somewhat simpler, because return 1306converters are themselves much simpler. 1307 1308Return converters must subclass ``CReturnConverter``. 1309There are no examples yet of custom return converters, 1310because they are not widely used yet. If you wish to 1311write your own return converter, please read ``Tools/clinic/clinic.py``, 1312specifically the implementation of ``CReturnConverter`` and 1313all its subclasses. 1314 1315METH_O and METH_NOARGS 1316---------------------------------------------- 1317 1318To convert a function using ``METH_O``, make sure the function's 1319single argument is using the ``object`` converter, and mark the 1320arguments as positional-only:: 1321 1322 /*[clinic input] 1323 meth_o_sample 1324 1325 argument: object 1326 / 1327 [clinic start generated code]*/ 1328 1329 1330To convert a function using ``METH_NOARGS``, just don't specify 1331any arguments. 1332 1333You can still use a self converter, a return converter, and specify 1334a ``type`` argument to the object converter for ``METH_O``. 1335 1336tp_new and tp_init functions 1337---------------------------------------------- 1338 1339You can convert ``tp_new`` and ``tp_init`` functions. Just name 1340them ``__new__`` or ``__init__`` as appropriate. Notes: 1341 1342* The function name generated for ``__new__`` doesn't end in ``__new__`` 1343 like it would by default. It's just the name of the class, converted 1344 into a valid C identifier. 1345 1346* No ``PyMethodDef`` ``#define`` is generated for these functions. 1347 1348* ``__init__`` functions return ``int``, not ``PyObject *``. 1349 1350* Use the docstring as the class docstring. 1351 1352* Although ``__new__`` and ``__init__`` functions must always 1353 accept both the ``args`` and ``kwargs`` objects, when converting 1354 you may specify any signature for these functions that you like. 1355 (If your function doesn't support keywords, the parsing function 1356 generated will throw an exception if it receives any.) 1357 1358Changing and redirecting Clinic's output 1359---------------------------------------- 1360 1361It can be inconvenient to have Clinic's output interspersed with 1362your conventional hand-edited C code. Luckily, Clinic is configurable: 1363you can buffer up its output for printing later (or earlier!), or write 1364its output to a separate file. You can also add a prefix or suffix to 1365every line of Clinic's generated output. 1366 1367While changing Clinic's output in this manner can be a boon to readability, 1368it may result in Clinic code using types before they are defined, or 1369your code attempting to use Clinic-generated code before it is defined. 1370These problems can be easily solved by rearranging the declarations in your file, 1371or moving where Clinic's generated code goes. (This is why the default behavior 1372of Clinic is to output everything into the current block; while many people 1373consider this hampers readability, it will never require rearranging your 1374code to fix definition-before-use problems.) 1375 1376Let's start with defining some terminology: 1377 1378*field* 1379 A field, in this context, is a subsection of Clinic's output. 1380 For example, the ``#define`` for the ``PyMethodDef`` structure 1381 is a field, called ``methoddef_define``. Clinic has seven 1382 different fields it can output per function definition: 1383 1384 .. code-block:: none 1385 1386 docstring_prototype 1387 docstring_definition 1388 methoddef_define 1389 impl_prototype 1390 parser_prototype 1391 parser_definition 1392 impl_definition 1393 1394 All the names are of the form ``"<a>_<b>"``, 1395 where ``"<a>"`` is the semantic object represented (the parsing function, 1396 the impl function, the docstring, or the methoddef structure) and ``"<b>"`` 1397 represents what kind of statement the field is. Field names that end in 1398 ``"_prototype"`` 1399 represent forward declarations of that thing, without the actual body/data 1400 of the thing; field names that end in ``"_definition"`` represent the actual 1401 definition of the thing, with the body/data of the thing. (``"methoddef"`` 1402 is special, it's the only one that ends with ``"_define"``, representing that 1403 it's a preprocessor #define.) 1404 1405*destination* 1406 A destination is a place Clinic can write output to. There are 1407 five built-in destinations: 1408 1409 ``block`` 1410 The default destination: printed in the output section of 1411 the current Clinic block. 1412 1413 ``buffer`` 1414 A text buffer where you can save text for later. Text sent 1415 here is appended to the end of any existing text. It's an 1416 error to have any text left in the buffer when Clinic finishes 1417 processing a file. 1418 1419 ``file`` 1420 A separate "clinic file" that will be created automatically by Clinic. 1421 The filename chosen for the file is ``{basename}.clinic{extension}``, 1422 where ``basename`` and ``extension`` were assigned the output 1423 from ``os.path.splitext()`` run on the current file. (Example: 1424 the ``file`` destination for ``_pickle.c`` would be written to 1425 ``_pickle.clinic.c``.) 1426 1427 **Important: When using a** ``file`` **destination, you** 1428 *must check in* **the generated file!** 1429 1430 ``two-pass`` 1431 A buffer like ``buffer``. However, a two-pass buffer can only 1432 be dumped once, and it prints out all text sent to it during 1433 all processing, even from Clinic blocks *after* the dumping point. 1434 1435 ``suppress`` 1436 The text is suppressed—thrown away. 1437 1438 1439Clinic defines five new directives that let you reconfigure its output. 1440 1441The first new directive is ``dump``: 1442 1443.. code-block:: none 1444 1445 dump <destination> 1446 1447This dumps the current contents of the named destination into the output of 1448the current block, and empties it. This only works with ``buffer`` and 1449``two-pass`` destinations. 1450 1451The second new directive is ``output``. The most basic form of ``output`` 1452is like this: 1453 1454.. code-block:: none 1455 1456 output <field> <destination> 1457 1458This tells Clinic to output *field* to *destination*. ``output`` also 1459supports a special meta-destination, called ``everything``, which tells 1460Clinic to output *all* fields to that *destination*. 1461 1462``output`` has a number of other functions: 1463 1464.. code-block:: none 1465 1466 output push 1467 output pop 1468 output preset <preset> 1469 1470 1471``output push`` and ``output pop`` allow you to push and pop 1472configurations on an internal configuration stack, so that you 1473can temporarily modify the output configuration, then easily restore 1474the previous configuration. Simply push before your change to save 1475the current configuration, then pop when you wish to restore the 1476previous configuration. 1477 1478``output preset`` sets Clinic's output to one of several built-in 1479preset configurations, as follows: 1480 1481 ``block`` 1482 Clinic's original starting configuration. Writes everything 1483 immediately after the input block. 1484 1485 Suppress the ``parser_prototype`` 1486 and ``docstring_prototype``, write everything else to ``block``. 1487 1488 ``file`` 1489 Designed to write everything to the "clinic file" that it can. 1490 You then ``#include`` this file near the top of your file. 1491 You may need to rearrange your file to make this work, though 1492 usually this just means creating forward declarations for various 1493 ``typedef`` and ``PyTypeObject`` definitions. 1494 1495 Suppress the ``parser_prototype`` 1496 and ``docstring_prototype``, write the ``impl_definition`` to 1497 ``block``, and write everything else to ``file``. 1498 1499 The default filename is ``"{dirname}/clinic/{basename}.h"``. 1500 1501 ``buffer`` 1502 Save up most of the output from Clinic, to be written into 1503 your file near the end. For Python files implementing modules 1504 or builtin types, it's recommended that you dump the buffer 1505 just above the static structures for your module or 1506 builtin type; these are normally very near the end. Using 1507 ``buffer`` may require even more editing than ``file``, if 1508 your file has static ``PyMethodDef`` arrays defined in the 1509 middle of the file. 1510 1511 Suppress the ``parser_prototype``, ``impl_prototype``, 1512 and ``docstring_prototype``, write the ``impl_definition`` to 1513 ``block``, and write everything else to ``file``. 1514 1515 ``two-pass`` 1516 Similar to the ``buffer`` preset, but writes forward declarations to 1517 the ``two-pass`` buffer, and definitions to the ``buffer``. 1518 This is similar to the ``buffer`` preset, but may require 1519 less editing than ``buffer``. Dump the ``two-pass`` buffer 1520 near the top of your file, and dump the ``buffer`` near 1521 the end just like you would when using the ``buffer`` preset. 1522 1523 Suppresses the ``impl_prototype``, write the ``impl_definition`` 1524 to ``block``, write ``docstring_prototype``, ``methoddef_define``, 1525 and ``parser_prototype`` to ``two-pass``, write everything else 1526 to ``buffer``. 1527 1528 ``partial-buffer`` 1529 Similar to the ``buffer`` preset, but writes more things to ``block``, 1530 only writing the really big chunks of generated code to ``buffer``. 1531 This avoids the definition-before-use problem of ``buffer`` completely, 1532 at the small cost of having slightly more stuff in the block's output. 1533 Dump the ``buffer`` near the end, just like you would when using 1534 the ``buffer`` preset. 1535 1536 Suppresses the ``impl_prototype``, write the ``docstring_definition`` 1537 and ``parser_definition`` to ``buffer``, write everything else to ``block``. 1538 1539The third new directive is ``destination``: 1540 1541.. code-block:: none 1542 1543 destination <name> <command> [...] 1544 1545This performs an operation on the destination named ``name``. 1546 1547There are two defined subcommands: ``new`` and ``clear``. 1548 1549The ``new`` subcommand works like this: 1550 1551.. code-block:: none 1552 1553 destination <name> new <type> 1554 1555This creates a new destination with name ``<name>`` and type ``<type>``. 1556 1557There are five destination types: 1558 1559 ``suppress`` 1560 Throws the text away. 1561 1562 ``block`` 1563 Writes the text to the current block. This is what Clinic 1564 originally did. 1565 1566 ``buffer`` 1567 A simple text buffer, like the "buffer" builtin destination above. 1568 1569 ``file`` 1570 A text file. The file destination takes an extra argument, 1571 a template to use for building the filename, like so: 1572 1573 destination <name> new <type> <file_template> 1574 1575 The template can use three strings internally that will be replaced 1576 by bits of the filename: 1577 1578 {path} 1579 The full path to the file, including directory and full filename. 1580 {dirname} 1581 The name of the directory the file is in. 1582 {basename} 1583 Just the name of the file, not including the directory. 1584 {basename_root} 1585 Basename with the extension clipped off 1586 (everything up to but not including the last '.'). 1587 {basename_extension} 1588 The last '.' and everything after it. If the basename 1589 does not contain a period, this will be the empty string. 1590 1591 If there are no periods in the filename, {basename} and {filename} 1592 are the same, and {extension} is empty. "{basename}{extension}" 1593 is always exactly the same as "{filename}"." 1594 1595 ``two-pass`` 1596 A two-pass buffer, like the "two-pass" builtin destination above. 1597 1598 1599The ``clear`` subcommand works like this: 1600 1601.. code-block:: none 1602 1603 destination <name> clear 1604 1605It removes all the accumulated text up to this point in the destination. 1606(I don't know what you'd need this for, but I thought maybe it'd be 1607useful while someone's experimenting.) 1608 1609The fourth new directive is ``set``: 1610 1611.. code-block:: none 1612 1613 set line_prefix "string" 1614 set line_suffix "string" 1615 1616``set`` lets you set two internal variables in Clinic. 1617``line_prefix`` is a string that will be prepended to every line of Clinic's output; 1618``line_suffix`` is a string that will be appended to every line of Clinic's output. 1619 1620Both of these support two format strings: 1621 1622 ``{block comment start}`` 1623 Turns into the string ``/*``, the start-comment text sequence for C files. 1624 1625 ``{block comment end}`` 1626 Turns into the string ``*/``, the end-comment text sequence for C files. 1627 1628The final new directive is one you shouldn't need to use directly, 1629called ``preserve``: 1630 1631.. code-block:: none 1632 1633 preserve 1634 1635This tells Clinic that the current contents of the output should be kept, unmodified. 1636This is used internally by Clinic when dumping output into ``file`` files; wrapping 1637it in a Clinic block lets Clinic use its existing checksum functionality to ensure 1638the file was not modified by hand before it gets overwritten. 1639 1640 1641The #ifdef trick 1642---------------------------------------------- 1643 1644If you're converting a function that isn't available on all platforms, 1645there's a trick you can use to make life a little easier. The existing 1646code probably looks like this:: 1647 1648 #ifdef HAVE_FUNCTIONNAME 1649 static module_functionname(...) 1650 { 1651 ... 1652 } 1653 #endif /* HAVE_FUNCTIONNAME */ 1654 1655And then in the ``PyMethodDef`` structure at the bottom the existing code 1656will have: 1657 1658.. code-block:: none 1659 1660 #ifdef HAVE_FUNCTIONNAME 1661 {'functionname', ... }, 1662 #endif /* HAVE_FUNCTIONNAME */ 1663 1664In this scenario, you should enclose the body of your impl function inside the ``#ifdef``, 1665like so:: 1666 1667 #ifdef HAVE_FUNCTIONNAME 1668 /*[clinic input] 1669 module.functionname 1670 ... 1671 [clinic start generated code]*/ 1672 static module_functionname(...) 1673 { 1674 ... 1675 } 1676 #endif /* HAVE_FUNCTIONNAME */ 1677 1678Then, remove those three lines from the ``PyMethodDef`` structure, 1679replacing them with the macro Argument Clinic generated: 1680 1681.. code-block:: none 1682 1683 MODULE_FUNCTIONNAME_METHODDEF 1684 1685(You can find the real name for this macro inside the generated code. 1686Or you can calculate it yourself: it's the name of your function as defined 1687on the first line of your block, but with periods changed to underscores, 1688uppercased, and ``"_METHODDEF"`` added to the end.) 1689 1690Perhaps you're wondering: what if ``HAVE_FUNCTIONNAME`` isn't defined? 1691The ``MODULE_FUNCTIONNAME_METHODDEF`` macro won't be defined either! 1692 1693Here's where Argument Clinic gets very clever. It actually detects that the 1694Argument Clinic block might be deactivated by the ``#ifdef``. When that 1695happens, it generates a little extra code that looks like this:: 1696 1697 #ifndef MODULE_FUNCTIONNAME_METHODDEF 1698 #define MODULE_FUNCTIONNAME_METHODDEF 1699 #endif /* !defined(MODULE_FUNCTIONNAME_METHODDEF) */ 1700 1701That means the macro always works. If the function is defined, this turns 1702into the correct structure, including the trailing comma. If the function is 1703undefined, this turns into nothing. 1704 1705However, this causes one ticklish problem: where should Argument Clinic put this 1706extra code when using the "block" output preset? It can't go in the output block, 1707because that could be deactivated by the ``#ifdef``. (That's the whole point!) 1708 1709In this situation, Argument Clinic writes the extra code to the "buffer" destination. 1710This may mean that you get a complaint from Argument Clinic: 1711 1712.. code-block:: none 1713 1714 Warning in file "Modules/posixmodule.c" on line 12357: 1715 Destination buffer 'buffer' not empty at end of file, emptying. 1716 1717When this happens, just open your file, find the ``dump buffer`` block that 1718Argument Clinic added to your file (it'll be at the very bottom), then 1719move it above the ``PyMethodDef`` structure where that macro is used. 1720 1721 1722 1723Using Argument Clinic in Python files 1724------------------------------------- 1725 1726It's actually possible to use Argument Clinic to preprocess Python files. 1727There's no point to using Argument Clinic blocks, of course, as the output 1728wouldn't make any sense to the Python interpreter. But using Argument Clinic 1729to run Python blocks lets you use Python as a Python preprocessor! 1730 1731Since Python comments are different from C comments, Argument Clinic 1732blocks embedded in Python files look slightly different. They look like this: 1733 1734.. code-block:: python3 1735 1736 #/*[python input] 1737 #print("def foo(): pass") 1738 #[python start generated code]*/ 1739 def foo(): pass 1740 #/*[python checksum:...]*/ 1741