1.. highlight:: c 2 3 4.. _initialization: 5 6***************************************** 7Initialization, Finalization, and Threads 8***************************************** 9 10See :ref:`Python Initialization Configuration <init-config>` for details 11on how to configure the interpreter prior to initialization. 12 13.. _pre-init-safe: 14 15Before Python Initialization 16============================ 17 18In an application embedding Python, the :c:func:`Py_Initialize` function must 19be called before using any other Python/C API functions; with the exception of 20a few functions and the :ref:`global configuration variables 21<global-conf-vars>`. 22 23The following functions can be safely called before Python is initialized: 24 25* Functions that initialize the interpreter: 26 27 * :c:func:`Py_Initialize` 28 * :c:func:`Py_InitializeEx` 29 * :c:func:`Py_InitializeFromConfig` 30 * :c:func:`Py_BytesMain` 31 * :c:func:`Py_Main` 32 * the runtime pre-initialization functions covered in :ref:`init-config` 33 34* Configuration functions: 35 36 * :c:func:`PyImport_AppendInittab` 37 * :c:func:`PyImport_ExtendInittab` 38 * :c:func:`!PyInitFrozenExtensions` 39 * :c:func:`PyMem_SetAllocator` 40 * :c:func:`PyMem_SetupDebugHooks` 41 * :c:func:`PyObject_SetArenaAllocator` 42 * :c:func:`Py_SetProgramName` 43 * :c:func:`Py_SetPythonHome` 44 * :c:func:`PySys_ResetWarnOptions` 45 * the configuration functions covered in :ref:`init-config` 46 47* Informative functions: 48 49 * :c:func:`Py_IsInitialized` 50 * :c:func:`PyMem_GetAllocator` 51 * :c:func:`PyObject_GetArenaAllocator` 52 * :c:func:`Py_GetBuildInfo` 53 * :c:func:`Py_GetCompiler` 54 * :c:func:`Py_GetCopyright` 55 * :c:func:`Py_GetPlatform` 56 * :c:func:`Py_GetVersion` 57 * :c:func:`Py_IsInitialized` 58 59* Utilities: 60 61 * :c:func:`Py_DecodeLocale` 62 * the status reporting and utility functions covered in :ref:`init-config` 63 64* Memory allocators: 65 66 * :c:func:`PyMem_RawMalloc` 67 * :c:func:`PyMem_RawRealloc` 68 * :c:func:`PyMem_RawCalloc` 69 * :c:func:`PyMem_RawFree` 70 71* Synchronization: 72 73 * :c:func:`PyMutex_Lock` 74 * :c:func:`PyMutex_Unlock` 75 76.. note:: 77 78 Despite their apparent similarity to some of the functions listed above, 79 the following functions **should not be called** before the interpreter has 80 been initialized: :c:func:`Py_EncodeLocale`, :c:func:`Py_GetPath`, 81 :c:func:`Py_GetPrefix`, :c:func:`Py_GetExecPrefix`, 82 :c:func:`Py_GetProgramFullPath`, :c:func:`Py_GetPythonHome`, 83 :c:func:`Py_GetProgramName`, :c:func:`PyEval_InitThreads`, and 84 :c:func:`Py_RunMain`. 85 86 87.. _global-conf-vars: 88 89Global configuration variables 90============================== 91 92Python has variables for the global configuration to control different features 93and options. By default, these flags are controlled by :ref:`command line 94options <using-on-interface-options>`. 95 96When a flag is set by an option, the value of the flag is the number of times 97that the option was set. For example, ``-b`` sets :c:data:`Py_BytesWarningFlag` 98to 1 and ``-bb`` sets :c:data:`Py_BytesWarningFlag` to 2. 99 100.. c:var:: int Py_BytesWarningFlag 101 102 This API is kept for backward compatibility: setting 103 :c:member:`PyConfig.bytes_warning` should be used instead, see :ref:`Python 104 Initialization Configuration <init-config>`. 105 106 Issue a warning when comparing :class:`bytes` or :class:`bytearray` with 107 :class:`str` or :class:`bytes` with :class:`int`. Issue an error if greater 108 or equal to ``2``. 109 110 Set by the :option:`-b` option. 111 112 .. deprecated-removed:: 3.12 3.14 113 114.. c:var:: int Py_DebugFlag 115 116 This API is kept for backward compatibility: setting 117 :c:member:`PyConfig.parser_debug` should be used instead, see :ref:`Python 118 Initialization Configuration <init-config>`. 119 120 Turn on parser debugging output (for expert only, depending on compilation 121 options). 122 123 Set by the :option:`-d` option and the :envvar:`PYTHONDEBUG` environment 124 variable. 125 126 .. deprecated-removed:: 3.12 3.14 127 128.. c:var:: int Py_DontWriteBytecodeFlag 129 130 This API is kept for backward compatibility: setting 131 :c:member:`PyConfig.write_bytecode` should be used instead, see :ref:`Python 132 Initialization Configuration <init-config>`. 133 134 If set to non-zero, Python won't try to write ``.pyc`` files on the 135 import of source modules. 136 137 Set by the :option:`-B` option and the :envvar:`PYTHONDONTWRITEBYTECODE` 138 environment variable. 139 140 .. deprecated-removed:: 3.12 3.14 141 142.. c:var:: int Py_FrozenFlag 143 144 This API is kept for backward compatibility: setting 145 :c:member:`PyConfig.pathconfig_warnings` should be used instead, see 146 :ref:`Python Initialization Configuration <init-config>`. 147 148 Suppress error messages when calculating the module search path in 149 :c:func:`Py_GetPath`. 150 151 Private flag used by ``_freeze_module`` and ``frozenmain`` programs. 152 153 .. deprecated-removed:: 3.12 3.14 154 155.. c:var:: int Py_HashRandomizationFlag 156 157 This API is kept for backward compatibility: setting 158 :c:member:`PyConfig.hash_seed` and :c:member:`PyConfig.use_hash_seed` should 159 be used instead, see :ref:`Python Initialization Configuration 160 <init-config>`. 161 162 Set to ``1`` if the :envvar:`PYTHONHASHSEED` environment variable is set to 163 a non-empty string. 164 165 If the flag is non-zero, read the :envvar:`PYTHONHASHSEED` environment 166 variable to initialize the secret hash seed. 167 168 .. deprecated-removed:: 3.12 3.14 169 170.. c:var:: int Py_IgnoreEnvironmentFlag 171 172 This API is kept for backward compatibility: setting 173 :c:member:`PyConfig.use_environment` should be used instead, see 174 :ref:`Python Initialization Configuration <init-config>`. 175 176 Ignore all :envvar:`!PYTHON*` environment variables, e.g. 177 :envvar:`PYTHONPATH` and :envvar:`PYTHONHOME`, that might be set. 178 179 Set by the :option:`-E` and :option:`-I` options. 180 181 .. deprecated-removed:: 3.12 3.14 182 183.. c:var:: int Py_InspectFlag 184 185 This API is kept for backward compatibility: setting 186 :c:member:`PyConfig.inspect` should be used instead, see 187 :ref:`Python Initialization Configuration <init-config>`. 188 189 When a script is passed as first argument or the :option:`-c` option is used, 190 enter interactive mode after executing the script or the command, even when 191 :data:`sys.stdin` does not appear to be a terminal. 192 193 Set by the :option:`-i` option and the :envvar:`PYTHONINSPECT` environment 194 variable. 195 196 .. deprecated-removed:: 3.12 3.14 197 198.. c:var:: int Py_InteractiveFlag 199 200 This API is kept for backward compatibility: setting 201 :c:member:`PyConfig.interactive` should be used instead, see 202 :ref:`Python Initialization Configuration <init-config>`. 203 204 Set by the :option:`-i` option. 205 206 .. deprecated:: 3.12 207 208.. c:var:: int Py_IsolatedFlag 209 210 This API is kept for backward compatibility: setting 211 :c:member:`PyConfig.isolated` should be used instead, see 212 :ref:`Python Initialization Configuration <init-config>`. 213 214 Run Python in isolated mode. In isolated mode :data:`sys.path` contains 215 neither the script's directory nor the user's site-packages directory. 216 217 Set by the :option:`-I` option. 218 219 .. versionadded:: 3.4 220 221 .. deprecated-removed:: 3.12 3.14 222 223.. c:var:: int Py_LegacyWindowsFSEncodingFlag 224 225 This API is kept for backward compatibility: setting 226 :c:member:`PyPreConfig.legacy_windows_fs_encoding` should be used instead, see 227 :ref:`Python Initialization Configuration <init-config>`. 228 229 If the flag is non-zero, use the ``mbcs`` encoding with ``replace`` error 230 handler, instead of the UTF-8 encoding with ``surrogatepass`` error handler, 231 for the :term:`filesystem encoding and error handler`. 232 233 Set to ``1`` if the :envvar:`PYTHONLEGACYWINDOWSFSENCODING` environment 234 variable is set to a non-empty string. 235 236 See :pep:`529` for more details. 237 238 .. availability:: Windows. 239 240 .. deprecated-removed:: 3.12 3.14 241 242.. c:var:: int Py_LegacyWindowsStdioFlag 243 244 This API is kept for backward compatibility: setting 245 :c:member:`PyConfig.legacy_windows_stdio` should be used instead, see 246 :ref:`Python Initialization Configuration <init-config>`. 247 248 If the flag is non-zero, use :class:`io.FileIO` instead of 249 :class:`!io._WindowsConsoleIO` for :mod:`sys` standard streams. 250 251 Set to ``1`` if the :envvar:`PYTHONLEGACYWINDOWSSTDIO` environment 252 variable is set to a non-empty string. 253 254 See :pep:`528` for more details. 255 256 .. availability:: Windows. 257 258 .. deprecated-removed:: 3.12 3.14 259 260.. c:var:: int Py_NoSiteFlag 261 262 This API is kept for backward compatibility: setting 263 :c:member:`PyConfig.site_import` should be used instead, see 264 :ref:`Python Initialization Configuration <init-config>`. 265 266 Disable the import of the module :mod:`site` and the site-dependent 267 manipulations of :data:`sys.path` that it entails. Also disable these 268 manipulations if :mod:`site` is explicitly imported later (call 269 :func:`site.main` if you want them to be triggered). 270 271 Set by the :option:`-S` option. 272 273 .. deprecated-removed:: 3.12 3.14 274 275.. c:var:: int Py_NoUserSiteDirectory 276 277 This API is kept for backward compatibility: setting 278 :c:member:`PyConfig.user_site_directory` should be used instead, see 279 :ref:`Python Initialization Configuration <init-config>`. 280 281 Don't add the :data:`user site-packages directory <site.USER_SITE>` to 282 :data:`sys.path`. 283 284 Set by the :option:`-s` and :option:`-I` options, and the 285 :envvar:`PYTHONNOUSERSITE` environment variable. 286 287 .. deprecated-removed:: 3.12 3.14 288 289.. c:var:: int Py_OptimizeFlag 290 291 This API is kept for backward compatibility: setting 292 :c:member:`PyConfig.optimization_level` should be used instead, see 293 :ref:`Python Initialization Configuration <init-config>`. 294 295 Set by the :option:`-O` option and the :envvar:`PYTHONOPTIMIZE` environment 296 variable. 297 298 .. deprecated-removed:: 3.12 3.14 299 300.. c:var:: int Py_QuietFlag 301 302 This API is kept for backward compatibility: setting 303 :c:member:`PyConfig.quiet` should be used instead, see :ref:`Python 304 Initialization Configuration <init-config>`. 305 306 Don't display the copyright and version messages even in interactive mode. 307 308 Set by the :option:`-q` option. 309 310 .. versionadded:: 3.2 311 312 .. deprecated-removed:: 3.12 3.14 313 314.. c:var:: int Py_UnbufferedStdioFlag 315 316 This API is kept for backward compatibility: setting 317 :c:member:`PyConfig.buffered_stdio` should be used instead, see :ref:`Python 318 Initialization Configuration <init-config>`. 319 320 Force the stdout and stderr streams to be unbuffered. 321 322 Set by the :option:`-u` option and the :envvar:`PYTHONUNBUFFERED` 323 environment variable. 324 325 .. deprecated-removed:: 3.12 3.14 326 327.. c:var:: int Py_VerboseFlag 328 329 This API is kept for backward compatibility: setting 330 :c:member:`PyConfig.verbose` should be used instead, see :ref:`Python 331 Initialization Configuration <init-config>`. 332 333 Print a message each time a module is initialized, showing the place 334 (filename or built-in module) from which it is loaded. If greater or equal 335 to ``2``, print a message for each file that is checked for when 336 searching for a module. Also provides information on module cleanup at exit. 337 338 Set by the :option:`-v` option and the :envvar:`PYTHONVERBOSE` environment 339 variable. 340 341 .. deprecated-removed:: 3.12 3.14 342 343 344Initializing and finalizing the interpreter 345=========================================== 346 347 348.. c:function:: void Py_Initialize() 349 350 .. index:: 351 single: PyEval_InitThreads() 352 single: modules (in module sys) 353 single: path (in module sys) 354 pair: module; builtins 355 pair: module; __main__ 356 pair: module; sys 357 triple: module; search; path 358 single: Py_FinalizeEx (C function) 359 360 Initialize the Python interpreter. In an application embedding Python, 361 this should be called before using any other Python/C API functions; see 362 :ref:`Before Python Initialization <pre-init-safe>` for the few exceptions. 363 364 This initializes the table of loaded modules (``sys.modules``), and creates 365 the fundamental modules :mod:`builtins`, :mod:`__main__` and :mod:`sys`. 366 It also initializes the module search path (``sys.path``). It does not set 367 ``sys.argv``; use the :ref:`Python Initialization Configuration <init-config>` 368 API for that. This is a no-op when called for a second time (without calling 369 :c:func:`Py_FinalizeEx` first). There is no return value; it is a fatal 370 error if the initialization fails. 371 372 Use :c:func:`Py_InitializeFromConfig` to customize the 373 :ref:`Python Initialization Configuration <init-config>`. 374 375 .. note:: 376 On Windows, changes the console mode from ``O_TEXT`` to ``O_BINARY``, 377 which will also affect non-Python uses of the console using the C Runtime. 378 379 380.. c:function:: void Py_InitializeEx(int initsigs) 381 382 This function works like :c:func:`Py_Initialize` if *initsigs* is ``1``. If 383 *initsigs* is ``0``, it skips initialization registration of signal handlers, 384 which may be useful when CPython is embedded as part of a larger application. 385 386 Use :c:func:`Py_InitializeFromConfig` to customize the 387 :ref:`Python Initialization Configuration <init-config>`. 388 389 390.. c:function:: PyStatus Py_InitializeFromConfig(const PyConfig *config) 391 392 Initialize Python from *config* configuration, as described in 393 :ref:`init-from-config`. 394 395 See the :ref:`init-config` section for details on pre-initializing the 396 interpreter, populating the runtime configuration structure, and querying 397 the returned status structure. 398 399 400.. c:function:: int Py_IsInitialized() 401 402 Return true (nonzero) when the Python interpreter has been initialized, false 403 (zero) if not. After :c:func:`Py_FinalizeEx` is called, this returns false until 404 :c:func:`Py_Initialize` is called again. 405 406 407.. c:function:: int Py_IsFinalizing() 408 409 Return true (non-zero) if the main Python interpreter is 410 :term:`shutting down <interpreter shutdown>`. Return false (zero) otherwise. 411 412 .. versionadded:: 3.13 413 414 415.. c:function:: int Py_FinalizeEx() 416 417 Undo all initializations made by :c:func:`Py_Initialize` and subsequent use of 418 Python/C API functions, and destroy all sub-interpreters (see 419 :c:func:`Py_NewInterpreter` below) that were created and not yet destroyed since 420 the last call to :c:func:`Py_Initialize`. Ideally, this frees all memory 421 allocated by the Python interpreter. This is a no-op when called for a second 422 time (without calling :c:func:`Py_Initialize` again first). 423 424 Since this is the reverse of :c:func:`Py_Initialize`, it should be called 425 in the same thread with the same interpreter active. That means 426 the main thread and the main interpreter. 427 This should never be called while :c:func:`Py_RunMain` is running. 428 429 Normally the return value is ``0``. 430 If there were errors during finalization (flushing buffered data), 431 ``-1`` is returned. 432 433 This function is provided for a number of reasons. An embedding application 434 might want to restart Python without having to restart the application itself. 435 An application that has loaded the Python interpreter from a dynamically 436 loadable library (or DLL) might want to free all memory allocated by Python 437 before unloading the DLL. During a hunt for memory leaks in an application a 438 developer might want to free all memory allocated by Python before exiting from 439 the application. 440 441 **Bugs and caveats:** The destruction of modules and objects in modules is done 442 in random order; this may cause destructors (:meth:`~object.__del__` methods) to fail 443 when they depend on other objects (even functions) or modules. Dynamically 444 loaded extension modules loaded by Python are not unloaded. Small amounts of 445 memory allocated by the Python interpreter may not be freed (if you find a leak, 446 please report it). Memory tied up in circular references between objects is not 447 freed. Some memory allocated by extension modules may not be freed. Some 448 extensions may not work properly if their initialization routine is called more 449 than once; this can happen if an application calls :c:func:`Py_Initialize` and 450 :c:func:`Py_FinalizeEx` more than once. 451 452 .. audit-event:: cpython._PySys_ClearAuditHooks "" c.Py_FinalizeEx 453 454 .. versionadded:: 3.6 455 456 457.. c:function:: void Py_Finalize() 458 459 This is a backwards-compatible version of :c:func:`Py_FinalizeEx` that 460 disregards the return value. 461 462 463.. c:function:: int Py_BytesMain(int argc, char **argv) 464 465 Similar to :c:func:`Py_Main` but *argv* is an array of bytes strings, 466 allowing the calling application to delegate the text decoding step to 467 the CPython runtime. 468 469 .. versionadded:: 3.8 470 471 472.. c:function:: int Py_Main(int argc, wchar_t **argv) 473 474 The main program for the standard interpreter, encapsulating a full 475 initialization/finalization cycle, as well as additional 476 behaviour to implement reading configurations settings from the environment 477 and command line, and then executing ``__main__`` in accordance with 478 :ref:`using-on-cmdline`. 479 480 This is made available for programs which wish to support the full CPython 481 command line interface, rather than just embedding a Python runtime in a 482 larger application. 483 484 The *argc* and *argv* parameters are similar to those which are passed to a 485 C program's :c:func:`main` function, except that the *argv* entries are first 486 converted to ``wchar_t`` using :c:func:`Py_DecodeLocale`. It is also 487 important to note that the argument list entries may be modified to point to 488 strings other than those passed in (however, the contents of the strings 489 pointed to by the argument list are not modified). 490 491 The return value will be ``0`` if the interpreter exits normally (i.e., 492 without an exception), ``1`` if the interpreter exits due to an exception, 493 or ``2`` if the argument list does not represent a valid Python command 494 line. 495 496 Note that if an otherwise unhandled :exc:`SystemExit` is raised, this 497 function will not return ``1``, but exit the process, as long as 498 ``Py_InspectFlag`` is not set. If ``Py_InspectFlag`` is set, execution will 499 drop into the interactive Python prompt, at which point a second otherwise 500 unhandled :exc:`SystemExit` will still exit the process, while any other 501 means of exiting will set the return value as described above. 502 503 In terms of the CPython runtime configuration APIs documented in the 504 :ref:`runtime configuration <init-config>` section (and without accounting 505 for error handling), ``Py_Main`` is approximately equivalent to:: 506 507 PyConfig config; 508 PyConfig_InitPythonConfig(&config); 509 PyConfig_SetArgv(&config, argc, argv); 510 Py_InitializeFromConfig(&config); 511 PyConfig_Clear(&config); 512 513 Py_RunMain(); 514 515 In normal usage, an embedding application will call this function 516 *instead* of calling :c:func:`Py_Initialize`, :c:func:`Py_InitializeEx` or 517 :c:func:`Py_InitializeFromConfig` directly, and all settings will be applied 518 as described elsewhere in this documentation. If this function is instead 519 called *after* a preceding runtime initialization API call, then exactly 520 which environmental and command line configuration settings will be updated 521 is version dependent (as it depends on which settings correctly support 522 being modified after they have already been set once when the runtime was 523 first initialized). 524 525 526.. c:function:: int Py_RunMain(void) 527 528 Executes the main module in a fully configured CPython runtime. 529 530 Executes the command (:c:member:`PyConfig.run_command`), the script 531 (:c:member:`PyConfig.run_filename`) or the module 532 (:c:member:`PyConfig.run_module`) specified on the command line or in the 533 configuration. If none of these values are set, runs the interactive Python 534 prompt (REPL) using the ``__main__`` module's global namespace. 535 536 If :c:member:`PyConfig.inspect` is not set (the default), the return value 537 will be ``0`` if the interpreter exits normally (that is, without raising 538 an exception), or ``1`` if the interpreter exits due to an exception. If an 539 otherwise unhandled :exc:`SystemExit` is raised, the function will immediately 540 exit the process instead of returning ``1``. 541 542 If :c:member:`PyConfig.inspect` is set (such as when the :option:`-i` option 543 is used), rather than returning when the interpreter exits, execution will 544 instead resume in an interactive Python prompt (REPL) using the ``__main__`` 545 module's global namespace. If the interpreter exited with an exception, it 546 is immediately raised in the REPL session. The function return value is 547 then determined by the way the *REPL session* terminates: returning ``0`` 548 if the session terminates without raising an unhandled exception, exiting 549 immediately for an unhandled :exc:`SystemExit`, and returning ``1`` for 550 any other unhandled exception. 551 552 This function always finalizes the Python interpreter regardless of whether 553 it returns a value or immediately exits the process due to an unhandled 554 :exc:`SystemExit` exception. 555 556 See :ref:`Python Configuration <init-python-config>` for an example of a 557 customized Python that always runs in isolated mode using 558 :c:func:`Py_RunMain`. 559 560 561Process-wide parameters 562======================= 563 564 565.. c:function:: void Py_SetProgramName(const wchar_t *name) 566 567 .. index:: 568 single: Py_Initialize() 569 single: main() 570 single: Py_GetPath() 571 572 This API is kept for backward compatibility: setting 573 :c:member:`PyConfig.program_name` should be used instead, see :ref:`Python 574 Initialization Configuration <init-config>`. 575 576 This function should be called before :c:func:`Py_Initialize` is called for 577 the first time, if it is called at all. It tells the interpreter the value 578 of the ``argv[0]`` argument to the :c:func:`main` function of the program 579 (converted to wide characters). 580 This is used by :c:func:`Py_GetPath` and some other functions below to find 581 the Python run-time libraries relative to the interpreter executable. The 582 default value is ``'python'``. The argument should point to a 583 zero-terminated wide character string in static storage whose contents will not 584 change for the duration of the program's execution. No code in the Python 585 interpreter will change the contents of this storage. 586 587 Use :c:func:`Py_DecodeLocale` to decode a bytes string to get a 588 :c:expr:`wchar_*` string. 589 590 .. deprecated:: 3.11 591 592 593.. c:function:: wchar_t* Py_GetProgramName() 594 595 Return the program name set with :c:member:`PyConfig.program_name`, or the default. 596 The returned string points into static storage; the caller should not modify its 597 value. 598 599 This function should not be called before :c:func:`Py_Initialize`, otherwise 600 it returns ``NULL``. 601 602 .. versionchanged:: 3.10 603 It now returns ``NULL`` if called before :c:func:`Py_Initialize`. 604 605 .. deprecated-removed:: 3.13 3.15 606 Get :data:`sys.executable` instead. 607 608 609.. c:function:: wchar_t* Py_GetPrefix() 610 611 Return the *prefix* for installed platform-independent files. This is derived 612 through a number of complicated rules from the program name set with 613 :c:member:`PyConfig.program_name` and some environment variables; for example, if the 614 program name is ``'/usr/local/bin/python'``, the prefix is ``'/usr/local'``. The 615 returned string points into static storage; the caller should not modify its 616 value. This corresponds to the :makevar:`prefix` variable in the top-level 617 :file:`Makefile` and the :option:`--prefix` argument to the :program:`configure` 618 script at build time. The value is available to Python code as ``sys.base_prefix``. 619 It is only useful on Unix. See also the next function. 620 621 This function should not be called before :c:func:`Py_Initialize`, otherwise 622 it returns ``NULL``. 623 624 .. versionchanged:: 3.10 625 It now returns ``NULL`` if called before :c:func:`Py_Initialize`. 626 627 .. deprecated-removed:: 3.13 3.15 628 Get :data:`sys.base_prefix` instead, or :data:`sys.prefix` if 629 :ref:`virtual environments <venv-def>` need to be handled. 630 631 632.. c:function:: wchar_t* Py_GetExecPrefix() 633 634 Return the *exec-prefix* for installed platform-*dependent* files. This is 635 derived through a number of complicated rules from the program name set with 636 :c:member:`PyConfig.program_name` and some environment variables; for example, if the 637 program name is ``'/usr/local/bin/python'``, the exec-prefix is 638 ``'/usr/local'``. The returned string points into static storage; the caller 639 should not modify its value. This corresponds to the :makevar:`exec_prefix` 640 variable in the top-level :file:`Makefile` and the ``--exec-prefix`` 641 argument to the :program:`configure` script at build time. The value is 642 available to Python code as ``sys.base_exec_prefix``. It is only useful on 643 Unix. 644 645 Background: The exec-prefix differs from the prefix when platform dependent 646 files (such as executables and shared libraries) are installed in a different 647 directory tree. In a typical installation, platform dependent files may be 648 installed in the :file:`/usr/local/plat` subtree while platform independent may 649 be installed in :file:`/usr/local`. 650 651 Generally speaking, a platform is a combination of hardware and software 652 families, e.g. Sparc machines running the Solaris 2.x operating system are 653 considered the same platform, but Intel machines running Solaris 2.x are another 654 platform, and Intel machines running Linux are yet another platform. Different 655 major revisions of the same operating system generally also form different 656 platforms. Non-Unix operating systems are a different story; the installation 657 strategies on those systems are so different that the prefix and exec-prefix are 658 meaningless, and set to the empty string. Note that compiled Python bytecode 659 files are platform independent (but not independent from the Python version by 660 which they were compiled!). 661 662 System administrators will know how to configure the :program:`mount` or 663 :program:`automount` programs to share :file:`/usr/local` between platforms 664 while having :file:`/usr/local/plat` be a different filesystem for each 665 platform. 666 667 This function should not be called before :c:func:`Py_Initialize`, otherwise 668 it returns ``NULL``. 669 670 .. versionchanged:: 3.10 671 It now returns ``NULL`` if called before :c:func:`Py_Initialize`. 672 673 .. deprecated-removed:: 3.13 3.15 674 Get :data:`sys.base_exec_prefix` instead, or :data:`sys.exec_prefix` if 675 :ref:`virtual environments <venv-def>` need to be handled. 676 677 678.. c:function:: wchar_t* Py_GetProgramFullPath() 679 680 .. index:: 681 single: executable (in module sys) 682 683 Return the full program name of the Python executable; this is computed as a 684 side-effect of deriving the default module search path from the program name 685 (set by :c:member:`PyConfig.program_name`). The returned string points into 686 static storage; the caller should not modify its value. The value is available 687 to Python code as ``sys.executable``. 688 689 This function should not be called before :c:func:`Py_Initialize`, otherwise 690 it returns ``NULL``. 691 692 .. versionchanged:: 3.10 693 It now returns ``NULL`` if called before :c:func:`Py_Initialize`. 694 695 .. deprecated-removed:: 3.13 3.15 696 Get :data:`sys.executable` instead. 697 698 699.. c:function:: wchar_t* Py_GetPath() 700 701 .. index:: 702 triple: module; search; path 703 single: path (in module sys) 704 705 Return the default module search path; this is computed from the program name 706 (set by :c:member:`PyConfig.program_name`) and some environment variables. 707 The returned string consists of a series of directory names separated by a 708 platform dependent delimiter character. The delimiter character is ``':'`` 709 on Unix and macOS, ``';'`` on Windows. The returned string points into 710 static storage; the caller should not modify its value. The list 711 :data:`sys.path` is initialized with this value on interpreter startup; it 712 can be (and usually is) modified later to change the search path for loading 713 modules. 714 715 This function should not be called before :c:func:`Py_Initialize`, otherwise 716 it returns ``NULL``. 717 718 .. XXX should give the exact rules 719 720 .. versionchanged:: 3.10 721 It now returns ``NULL`` if called before :c:func:`Py_Initialize`. 722 723 .. deprecated-removed:: 3.13 3.15 724 Get :data:`sys.path` instead. 725 726 727.. c:function:: const char* Py_GetVersion() 728 729 Return the version of this Python interpreter. This is a string that looks 730 something like :: 731 732 "3.0a5+ (py3k:63103M, May 12 2008, 00:53:55) \n[GCC 4.2.3]" 733 734 .. index:: single: version (in module sys) 735 736 The first word (up to the first space character) is the current Python version; 737 the first characters are the major and minor version separated by a 738 period. The returned string points into static storage; the caller should not 739 modify its value. The value is available to Python code as :data:`sys.version`. 740 741 See also the :c:var:`Py_Version` constant. 742 743 744.. c:function:: const char* Py_GetPlatform() 745 746 .. index:: single: platform (in module sys) 747 748 Return the platform identifier for the current platform. On Unix, this is 749 formed from the "official" name of the operating system, converted to lower 750 case, followed by the major revision number; e.g., for Solaris 2.x, which is 751 also known as SunOS 5.x, the value is ``'sunos5'``. On macOS, it is 752 ``'darwin'``. On Windows, it is ``'win'``. The returned string points into 753 static storage; the caller should not modify its value. The value is available 754 to Python code as ``sys.platform``. 755 756 757.. c:function:: const char* Py_GetCopyright() 758 759 Return the official copyright string for the current Python version, for example 760 761 ``'Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam'`` 762 763 .. index:: single: copyright (in module sys) 764 765 The returned string points into static storage; the caller should not modify its 766 value. The value is available to Python code as ``sys.copyright``. 767 768 769.. c:function:: const char* Py_GetCompiler() 770 771 Return an indication of the compiler used to build the current Python version, 772 in square brackets, for example:: 773 774 "[GCC 2.7.2.2]" 775 776 .. index:: single: version (in module sys) 777 778 The returned string points into static storage; the caller should not modify its 779 value. The value is available to Python code as part of the variable 780 ``sys.version``. 781 782 783.. c:function:: const char* Py_GetBuildInfo() 784 785 Return information about the sequence number and build date and time of the 786 current Python interpreter instance, for example :: 787 788 "#67, Aug 1 1997, 22:34:28" 789 790 .. index:: single: version (in module sys) 791 792 The returned string points into static storage; the caller should not modify its 793 value. The value is available to Python code as part of the variable 794 ``sys.version``. 795 796 797.. c:function:: void PySys_SetArgvEx(int argc, wchar_t **argv, int updatepath) 798 799 .. index:: 800 single: main() 801 single: Py_FatalError() 802 single: argv (in module sys) 803 804 This API is kept for backward compatibility: setting 805 :c:member:`PyConfig.argv`, :c:member:`PyConfig.parse_argv` and 806 :c:member:`PyConfig.safe_path` should be used instead, see :ref:`Python 807 Initialization Configuration <init-config>`. 808 809 Set :data:`sys.argv` based on *argc* and *argv*. These parameters are 810 similar to those passed to the program's :c:func:`main` function with the 811 difference that the first entry should refer to the script file to be 812 executed rather than the executable hosting the Python interpreter. If there 813 isn't a script that will be run, the first entry in *argv* can be an empty 814 string. If this function fails to initialize :data:`sys.argv`, a fatal 815 condition is signalled using :c:func:`Py_FatalError`. 816 817 If *updatepath* is zero, this is all the function does. If *updatepath* 818 is non-zero, the function also modifies :data:`sys.path` according to the 819 following algorithm: 820 821 - If the name of an existing script is passed in ``argv[0]``, the absolute 822 path of the directory where the script is located is prepended to 823 :data:`sys.path`. 824 - Otherwise (that is, if *argc* is ``0`` or ``argv[0]`` doesn't point 825 to an existing file name), an empty string is prepended to 826 :data:`sys.path`, which is the same as prepending the current working 827 directory (``"."``). 828 829 Use :c:func:`Py_DecodeLocale` to decode a bytes string to get a 830 :c:expr:`wchar_*` string. 831 832 See also :c:member:`PyConfig.orig_argv` and :c:member:`PyConfig.argv` 833 members of the :ref:`Python Initialization Configuration <init-config>`. 834 835 .. note:: 836 It is recommended that applications embedding the Python interpreter 837 for purposes other than executing a single script pass ``0`` as *updatepath*, 838 and update :data:`sys.path` themselves if desired. 839 See :cve:`2008-5983`. 840 841 On versions before 3.1.3, you can achieve the same effect by manually 842 popping the first :data:`sys.path` element after having called 843 :c:func:`PySys_SetArgv`, for example using:: 844 845 PyRun_SimpleString("import sys; sys.path.pop(0)\n"); 846 847 .. versionadded:: 3.1.3 848 849 .. XXX impl. doesn't seem consistent in allowing ``0``/``NULL`` for the params; 850 check w/ Guido. 851 852 .. deprecated:: 3.11 853 854 855.. c:function:: void PySys_SetArgv(int argc, wchar_t **argv) 856 857 This API is kept for backward compatibility: setting 858 :c:member:`PyConfig.argv` and :c:member:`PyConfig.parse_argv` should be used 859 instead, see :ref:`Python Initialization Configuration <init-config>`. 860 861 This function works like :c:func:`PySys_SetArgvEx` with *updatepath* set 862 to ``1`` unless the :program:`python` interpreter was started with the 863 :option:`-I`. 864 865 Use :c:func:`Py_DecodeLocale` to decode a bytes string to get a 866 :c:expr:`wchar_*` string. 867 868 See also :c:member:`PyConfig.orig_argv` and :c:member:`PyConfig.argv` 869 members of the :ref:`Python Initialization Configuration <init-config>`. 870 871 .. versionchanged:: 3.4 The *updatepath* value depends on :option:`-I`. 872 873 .. deprecated:: 3.11 874 875 876.. c:function:: void Py_SetPythonHome(const wchar_t *home) 877 878 This API is kept for backward compatibility: setting 879 :c:member:`PyConfig.home` should be used instead, see :ref:`Python 880 Initialization Configuration <init-config>`. 881 882 Set the default "home" directory, that is, the location of the standard 883 Python libraries. See :envvar:`PYTHONHOME` for the meaning of the 884 argument string. 885 886 The argument should point to a zero-terminated character string in static 887 storage whose contents will not change for the duration of the program's 888 execution. No code in the Python interpreter will change the contents of 889 this storage. 890 891 Use :c:func:`Py_DecodeLocale` to decode a bytes string to get a 892 :c:expr:`wchar_*` string. 893 894 .. deprecated:: 3.11 895 896 897.. c:function:: wchar_t* Py_GetPythonHome() 898 899 Return the default "home", that is, the value set by 900 :c:member:`PyConfig.home`, or the value of the :envvar:`PYTHONHOME` 901 environment variable if it is set. 902 903 This function should not be called before :c:func:`Py_Initialize`, otherwise 904 it returns ``NULL``. 905 906 .. versionchanged:: 3.10 907 It now returns ``NULL`` if called before :c:func:`Py_Initialize`. 908 909 .. deprecated-removed:: 3.13 3.15 910 Get :c:member:`PyConfig.home` or :envvar:`PYTHONHOME` environment 911 variable instead. 912 913 914.. _threads: 915 916Thread State and the Global Interpreter Lock 917============================================ 918 919.. index:: 920 single: global interpreter lock 921 single: interpreter lock 922 single: lock, interpreter 923 924The Python interpreter is not fully thread-safe. In order to support 925multi-threaded Python programs, there's a global lock, called the :term:`global 926interpreter lock` or :term:`GIL`, that must be held by the current thread before 927it can safely access Python objects. Without the lock, even the simplest 928operations could cause problems in a multi-threaded program: for example, when 929two threads simultaneously increment the reference count of the same object, the 930reference count could end up being incremented only once instead of twice. 931 932.. index:: single: setswitchinterval (in module sys) 933 934Therefore, the rule exists that only the thread that has acquired the 935:term:`GIL` may operate on Python objects or call Python/C API functions. 936In order to emulate concurrency of execution, the interpreter regularly 937tries to switch threads (see :func:`sys.setswitchinterval`). The lock is also 938released around potentially blocking I/O operations like reading or writing 939a file, so that other Python threads can run in the meantime. 940 941.. index:: 942 single: PyThreadState (C type) 943 944The Python interpreter keeps some thread-specific bookkeeping information 945inside a data structure called :c:type:`PyThreadState`. There's also one 946global variable pointing to the current :c:type:`PyThreadState`: it can 947be retrieved using :c:func:`PyThreadState_Get`. 948 949Releasing the GIL from extension code 950------------------------------------- 951 952Most extension code manipulating the :term:`GIL` has the following simple 953structure:: 954 955 Save the thread state in a local variable. 956 Release the global interpreter lock. 957 ... Do some blocking I/O operation ... 958 Reacquire the global interpreter lock. 959 Restore the thread state from the local variable. 960 961This is so common that a pair of macros exists to simplify it:: 962 963 Py_BEGIN_ALLOW_THREADS 964 ... Do some blocking I/O operation ... 965 Py_END_ALLOW_THREADS 966 967.. index:: 968 single: Py_BEGIN_ALLOW_THREADS (C macro) 969 single: Py_END_ALLOW_THREADS (C macro) 970 971The :c:macro:`Py_BEGIN_ALLOW_THREADS` macro opens a new block and declares a 972hidden local variable; the :c:macro:`Py_END_ALLOW_THREADS` macro closes the 973block. 974 975The block above expands to the following code:: 976 977 PyThreadState *_save; 978 979 _save = PyEval_SaveThread(); 980 ... Do some blocking I/O operation ... 981 PyEval_RestoreThread(_save); 982 983.. index:: 984 single: PyEval_RestoreThread (C function) 985 single: PyEval_SaveThread (C function) 986 987Here is how these functions work: the global interpreter lock is used to protect the pointer to the 988current thread state. When releasing the lock and saving the thread state, 989the current thread state pointer must be retrieved before the lock is released 990(since another thread could immediately acquire the lock and store its own thread 991state in the global variable). Conversely, when acquiring the lock and restoring 992the thread state, the lock must be acquired before storing the thread state 993pointer. 994 995.. note:: 996 Calling system I/O functions is the most common use case for releasing 997 the GIL, but it can also be useful before calling long-running computations 998 which don't need access to Python objects, such as compression or 999 cryptographic functions operating over memory buffers. For example, the 1000 standard :mod:`zlib` and :mod:`hashlib` modules release the GIL when 1001 compressing or hashing data. 1002 1003 1004.. _gilstate: 1005 1006Non-Python created threads 1007-------------------------- 1008 1009When threads are created using the dedicated Python APIs (such as the 1010:mod:`threading` module), a thread state is automatically associated to them 1011and the code showed above is therefore correct. However, when threads are 1012created from C (for example by a third-party library with its own thread 1013management), they don't hold the GIL, nor is there a thread state structure 1014for them. 1015 1016If you need to call Python code from these threads (often this will be part 1017of a callback API provided by the aforementioned third-party library), 1018you must first register these threads with the interpreter by 1019creating a thread state data structure, then acquiring the GIL, and finally 1020storing their thread state pointer, before you can start using the Python/C 1021API. When you are done, you should reset the thread state pointer, release 1022the GIL, and finally free the thread state data structure. 1023 1024The :c:func:`PyGILState_Ensure` and :c:func:`PyGILState_Release` functions do 1025all of the above automatically. The typical idiom for calling into Python 1026from a C thread is:: 1027 1028 PyGILState_STATE gstate; 1029 gstate = PyGILState_Ensure(); 1030 1031 /* Perform Python actions here. */ 1032 result = CallSomeFunction(); 1033 /* evaluate result or handle exception */ 1034 1035 /* Release the thread. No Python API allowed beyond this point. */ 1036 PyGILState_Release(gstate); 1037 1038Note that the ``PyGILState_*`` functions assume there is only one global 1039interpreter (created automatically by :c:func:`Py_Initialize`). Python 1040supports the creation of additional interpreters (using 1041:c:func:`Py_NewInterpreter`), but mixing multiple interpreters and the 1042``PyGILState_*`` API is unsupported. 1043 1044 1045.. _fork-and-threads: 1046 1047Cautions about fork() 1048--------------------- 1049 1050Another important thing to note about threads is their behaviour in the face 1051of the C :c:func:`fork` call. On most systems with :c:func:`fork`, after a 1052process forks only the thread that issued the fork will exist. This has a 1053concrete impact both on how locks must be handled and on all stored state 1054in CPython's runtime. 1055 1056The fact that only the "current" thread remains 1057means any locks held by other threads will never be released. Python solves 1058this for :func:`os.fork` by acquiring the locks it uses internally before 1059the fork, and releasing them afterwards. In addition, it resets any 1060:ref:`lock-objects` in the child. When extending or embedding Python, there 1061is no way to inform Python of additional (non-Python) locks that need to be 1062acquired before or reset after a fork. OS facilities such as 1063:c:func:`!pthread_atfork` would need to be used to accomplish the same thing. 1064Additionally, when extending or embedding Python, calling :c:func:`fork` 1065directly rather than through :func:`os.fork` (and returning to or calling 1066into Python) may result in a deadlock by one of Python's internal locks 1067being held by a thread that is defunct after the fork. 1068:c:func:`PyOS_AfterFork_Child` tries to reset the necessary locks, but is not 1069always able to. 1070 1071The fact that all other threads go away also means that CPython's 1072runtime state there must be cleaned up properly, which :func:`os.fork` 1073does. This means finalizing all other :c:type:`PyThreadState` objects 1074belonging to the current interpreter and all other 1075:c:type:`PyInterpreterState` objects. Due to this and the special 1076nature of the :ref:`"main" interpreter <sub-interpreter-support>`, 1077:c:func:`fork` should only be called in that interpreter's "main" 1078thread, where the CPython global runtime was originally initialized. 1079The only exception is if :c:func:`exec` will be called immediately 1080after. 1081 1082 1083High-level API 1084-------------- 1085 1086These are the most commonly used types and functions when writing C extension 1087code, or when embedding the Python interpreter: 1088 1089.. c:type:: PyInterpreterState 1090 1091 This data structure represents the state shared by a number of cooperating 1092 threads. Threads belonging to the same interpreter share their module 1093 administration and a few other internal items. There are no public members in 1094 this structure. 1095 1096 Threads belonging to different interpreters initially share nothing, except 1097 process state like available memory, open file descriptors and such. The global 1098 interpreter lock is also shared by all threads, regardless of to which 1099 interpreter they belong. 1100 1101 1102.. c:type:: PyThreadState 1103 1104 This data structure represents the state of a single thread. The only public 1105 data member is: 1106 1107 .. c:member:: PyInterpreterState *interp 1108 1109 This thread's interpreter state. 1110 1111 1112.. c:function:: void PyEval_InitThreads() 1113 1114 .. index:: 1115 single: PyEval_AcquireThread() 1116 single: PyEval_ReleaseThread() 1117 single: PyEval_SaveThread() 1118 single: PyEval_RestoreThread() 1119 1120 Deprecated function which does nothing. 1121 1122 In Python 3.6 and older, this function created the GIL if it didn't exist. 1123 1124 .. versionchanged:: 3.9 1125 The function now does nothing. 1126 1127 .. versionchanged:: 3.7 1128 This function is now called by :c:func:`Py_Initialize()`, so you don't 1129 have to call it yourself anymore. 1130 1131 .. versionchanged:: 3.2 1132 This function cannot be called before :c:func:`Py_Initialize()` anymore. 1133 1134 .. deprecated:: 3.9 1135 1136 .. index:: pair: module; _thread 1137 1138 1139.. c:function:: PyThreadState* PyEval_SaveThread() 1140 1141 Release the global interpreter lock (if it has been created) and reset the 1142 thread state to ``NULL``, returning the previous thread state (which is not 1143 ``NULL``). If the lock has been created, the current thread must have 1144 acquired it. 1145 1146 1147.. c:function:: void PyEval_RestoreThread(PyThreadState *tstate) 1148 1149 Acquire the global interpreter lock (if it has been created) and set the 1150 thread state to *tstate*, which must not be ``NULL``. If the lock has been 1151 created, the current thread must not have acquired it, otherwise deadlock 1152 ensues. 1153 1154 .. note:: 1155 Calling this function from a thread when the runtime is finalizing 1156 will terminate the thread, even if the thread was not created by Python. 1157 You can use :c:func:`Py_IsFinalizing` or :func:`sys.is_finalizing` to 1158 check if the interpreter is in process of being finalized before calling 1159 this function to avoid unwanted termination. 1160 1161.. c:function:: PyThreadState* PyThreadState_Get() 1162 1163 Return the current thread state. The global interpreter lock must be held. 1164 When the current thread state is ``NULL``, this issues a fatal error (so that 1165 the caller needn't check for ``NULL``). 1166 1167 See also :c:func:`PyThreadState_GetUnchecked`. 1168 1169 1170.. c:function:: PyThreadState* PyThreadState_GetUnchecked() 1171 1172 Similar to :c:func:`PyThreadState_Get`, but don't kill the process with a 1173 fatal error if it is NULL. The caller is responsible to check if the result 1174 is NULL. 1175 1176 .. versionadded:: 3.13 1177 In Python 3.5 to 3.12, the function was private and known as 1178 ``_PyThreadState_UncheckedGet()``. 1179 1180 1181.. c:function:: PyThreadState* PyThreadState_Swap(PyThreadState *tstate) 1182 1183 Swap the current thread state with the thread state given by the argument 1184 *tstate*, which may be ``NULL``. The global interpreter lock must be held 1185 and is not released. 1186 1187 1188The following functions use thread-local storage, and are not compatible 1189with sub-interpreters: 1190 1191.. c:function:: PyGILState_STATE PyGILState_Ensure() 1192 1193 Ensure that the current thread is ready to call the Python C API regardless 1194 of the current state of Python, or of the global interpreter lock. This may 1195 be called as many times as desired by a thread as long as each call is 1196 matched with a call to :c:func:`PyGILState_Release`. In general, other 1197 thread-related APIs may be used between :c:func:`PyGILState_Ensure` and 1198 :c:func:`PyGILState_Release` calls as long as the thread state is restored to 1199 its previous state before the Release(). For example, normal usage of the 1200 :c:macro:`Py_BEGIN_ALLOW_THREADS` and :c:macro:`Py_END_ALLOW_THREADS` macros is 1201 acceptable. 1202 1203 The return value is an opaque "handle" to the thread state when 1204 :c:func:`PyGILState_Ensure` was called, and must be passed to 1205 :c:func:`PyGILState_Release` to ensure Python is left in the same state. Even 1206 though recursive calls are allowed, these handles *cannot* be shared - each 1207 unique call to :c:func:`PyGILState_Ensure` must save the handle for its call 1208 to :c:func:`PyGILState_Release`. 1209 1210 When the function returns, the current thread will hold the GIL and be able 1211 to call arbitrary Python code. Failure is a fatal error. 1212 1213 .. note:: 1214 Calling this function from a thread when the runtime is finalizing 1215 will terminate the thread, even if the thread was not created by Python. 1216 You can use :c:func:`Py_IsFinalizing` or :func:`sys.is_finalizing` to 1217 check if the interpreter is in process of being finalized before calling 1218 this function to avoid unwanted termination. 1219 1220.. c:function:: void PyGILState_Release(PyGILState_STATE) 1221 1222 Release any resources previously acquired. After this call, Python's state will 1223 be the same as it was prior to the corresponding :c:func:`PyGILState_Ensure` call 1224 (but generally this state will be unknown to the caller, hence the use of the 1225 GILState API). 1226 1227 Every call to :c:func:`PyGILState_Ensure` must be matched by a call to 1228 :c:func:`PyGILState_Release` on the same thread. 1229 1230 1231.. c:function:: PyThreadState* PyGILState_GetThisThreadState() 1232 1233 Get the current thread state for this thread. May return ``NULL`` if no 1234 GILState API has been used on the current thread. Note that the main thread 1235 always has such a thread-state, even if no auto-thread-state call has been 1236 made on the main thread. This is mainly a helper/diagnostic function. 1237 1238 1239.. c:function:: int PyGILState_Check() 1240 1241 Return ``1`` if the current thread is holding the GIL and ``0`` otherwise. 1242 This function can be called from any thread at any time. 1243 Only if it has had its Python thread state initialized and currently is 1244 holding the GIL will it return ``1``. 1245 This is mainly a helper/diagnostic function. It can be useful 1246 for example in callback contexts or memory allocation functions when 1247 knowing that the GIL is locked can allow the caller to perform sensitive 1248 actions or otherwise behave differently. 1249 1250 .. versionadded:: 3.4 1251 1252 1253The following macros are normally used without a trailing semicolon; look for 1254example usage in the Python source distribution. 1255 1256 1257.. c:macro:: Py_BEGIN_ALLOW_THREADS 1258 1259 This macro expands to ``{ PyThreadState *_save; _save = PyEval_SaveThread();``. 1260 Note that it contains an opening brace; it must be matched with a following 1261 :c:macro:`Py_END_ALLOW_THREADS` macro. See above for further discussion of this 1262 macro. 1263 1264 1265.. c:macro:: Py_END_ALLOW_THREADS 1266 1267 This macro expands to ``PyEval_RestoreThread(_save); }``. Note that it contains 1268 a closing brace; it must be matched with an earlier 1269 :c:macro:`Py_BEGIN_ALLOW_THREADS` macro. See above for further discussion of 1270 this macro. 1271 1272 1273.. c:macro:: Py_BLOCK_THREADS 1274 1275 This macro expands to ``PyEval_RestoreThread(_save);``: it is equivalent to 1276 :c:macro:`Py_END_ALLOW_THREADS` without the closing brace. 1277 1278 1279.. c:macro:: Py_UNBLOCK_THREADS 1280 1281 This macro expands to ``_save = PyEval_SaveThread();``: it is equivalent to 1282 :c:macro:`Py_BEGIN_ALLOW_THREADS` without the opening brace and variable 1283 declaration. 1284 1285 1286Low-level API 1287------------- 1288 1289All of the following functions must be called after :c:func:`Py_Initialize`. 1290 1291.. versionchanged:: 3.7 1292 :c:func:`Py_Initialize()` now initializes the :term:`GIL`. 1293 1294 1295.. c:function:: PyInterpreterState* PyInterpreterState_New() 1296 1297 Create a new interpreter state object. The global interpreter lock need not 1298 be held, but may be held if it is necessary to serialize calls to this 1299 function. 1300 1301 .. audit-event:: cpython.PyInterpreterState_New "" c.PyInterpreterState_New 1302 1303 1304.. c:function:: void PyInterpreterState_Clear(PyInterpreterState *interp) 1305 1306 Reset all information in an interpreter state object. The global interpreter 1307 lock must be held. 1308 1309 .. audit-event:: cpython.PyInterpreterState_Clear "" c.PyInterpreterState_Clear 1310 1311 1312.. c:function:: void PyInterpreterState_Delete(PyInterpreterState *interp) 1313 1314 Destroy an interpreter state object. The global interpreter lock need not be 1315 held. The interpreter state must have been reset with a previous call to 1316 :c:func:`PyInterpreterState_Clear`. 1317 1318 1319.. c:function:: PyThreadState* PyThreadState_New(PyInterpreterState *interp) 1320 1321 Create a new thread state object belonging to the given interpreter object. 1322 The global interpreter lock need not be held, but may be held if it is 1323 necessary to serialize calls to this function. 1324 1325 1326.. c:function:: void PyThreadState_Clear(PyThreadState *tstate) 1327 1328 Reset all information in a thread state object. The global interpreter lock 1329 must be held. 1330 1331 .. versionchanged:: 3.9 1332 This function now calls the :c:member:`PyThreadState.on_delete` callback. 1333 Previously, that happened in :c:func:`PyThreadState_Delete`. 1334 1335 .. versionchanged:: 3.13 1336 The :c:member:`PyThreadState.on_delete` callback was removed. 1337 1338 1339.. c:function:: void PyThreadState_Delete(PyThreadState *tstate) 1340 1341 Destroy a thread state object. The global interpreter lock need not be held. 1342 The thread state must have been reset with a previous call to 1343 :c:func:`PyThreadState_Clear`. 1344 1345 1346.. c:function:: void PyThreadState_DeleteCurrent(void) 1347 1348 Destroy the current thread state and release the global interpreter lock. 1349 Like :c:func:`PyThreadState_Delete`, the global interpreter lock must 1350 be held. The thread state must have been reset with a previous call 1351 to :c:func:`PyThreadState_Clear`. 1352 1353 1354.. c:function:: PyFrameObject* PyThreadState_GetFrame(PyThreadState *tstate) 1355 1356 Get the current frame of the Python thread state *tstate*. 1357 1358 Return a :term:`strong reference`. Return ``NULL`` if no frame is currently 1359 executing. 1360 1361 See also :c:func:`PyEval_GetFrame`. 1362 1363 *tstate* must not be ``NULL``. 1364 1365 .. versionadded:: 3.9 1366 1367 1368.. c:function:: uint64_t PyThreadState_GetID(PyThreadState *tstate) 1369 1370 Get the unique thread state identifier of the Python thread state *tstate*. 1371 1372 *tstate* must not be ``NULL``. 1373 1374 .. versionadded:: 3.9 1375 1376 1377.. c:function:: PyInterpreterState* PyThreadState_GetInterpreter(PyThreadState *tstate) 1378 1379 Get the interpreter of the Python thread state *tstate*. 1380 1381 *tstate* must not be ``NULL``. 1382 1383 .. versionadded:: 3.9 1384 1385 1386.. c:function:: void PyThreadState_EnterTracing(PyThreadState *tstate) 1387 1388 Suspend tracing and profiling in the Python thread state *tstate*. 1389 1390 Resume them using the :c:func:`PyThreadState_LeaveTracing` function. 1391 1392 .. versionadded:: 3.11 1393 1394 1395.. c:function:: void PyThreadState_LeaveTracing(PyThreadState *tstate) 1396 1397 Resume tracing and profiling in the Python thread state *tstate* suspended 1398 by the :c:func:`PyThreadState_EnterTracing` function. 1399 1400 See also :c:func:`PyEval_SetTrace` and :c:func:`PyEval_SetProfile` 1401 functions. 1402 1403 .. versionadded:: 3.11 1404 1405 1406.. c:function:: PyInterpreterState* PyInterpreterState_Get(void) 1407 1408 Get the current interpreter. 1409 1410 Issue a fatal error if there no current Python thread state or no current 1411 interpreter. It cannot return NULL. 1412 1413 The caller must hold the GIL. 1414 1415 .. versionadded:: 3.9 1416 1417 1418.. c:function:: int64_t PyInterpreterState_GetID(PyInterpreterState *interp) 1419 1420 Return the interpreter's unique ID. If there was any error in doing 1421 so then ``-1`` is returned and an error is set. 1422 1423 The caller must hold the GIL. 1424 1425 .. versionadded:: 3.7 1426 1427 1428.. c:function:: PyObject* PyInterpreterState_GetDict(PyInterpreterState *interp) 1429 1430 Return a dictionary in which interpreter-specific data may be stored. 1431 If this function returns ``NULL`` then no exception has been raised and 1432 the caller should assume no interpreter-specific dict is available. 1433 1434 This is not a replacement for :c:func:`PyModule_GetState()`, which 1435 extensions should use to store interpreter-specific state information. 1436 1437 .. versionadded:: 3.8 1438 1439.. c:type:: PyObject* (*_PyFrameEvalFunction)(PyThreadState *tstate, _PyInterpreterFrame *frame, int throwflag) 1440 1441 Type of a frame evaluation function. 1442 1443 The *throwflag* parameter is used by the ``throw()`` method of generators: 1444 if non-zero, handle the current exception. 1445 1446 .. versionchanged:: 3.9 1447 The function now takes a *tstate* parameter. 1448 1449 .. versionchanged:: 3.11 1450 The *frame* parameter changed from ``PyFrameObject*`` to ``_PyInterpreterFrame*``. 1451 1452.. c:function:: _PyFrameEvalFunction _PyInterpreterState_GetEvalFrameFunc(PyInterpreterState *interp) 1453 1454 Get the frame evaluation function. 1455 1456 See the :pep:`523` "Adding a frame evaluation API to CPython". 1457 1458 .. versionadded:: 3.9 1459 1460.. c:function:: void _PyInterpreterState_SetEvalFrameFunc(PyInterpreterState *interp, _PyFrameEvalFunction eval_frame) 1461 1462 Set the frame evaluation function. 1463 1464 See the :pep:`523` "Adding a frame evaluation API to CPython". 1465 1466 .. versionadded:: 3.9 1467 1468 1469.. c:function:: PyObject* PyThreadState_GetDict() 1470 1471 Return a dictionary in which extensions can store thread-specific state 1472 information. Each extension should use a unique key to use to store state in 1473 the dictionary. It is okay to call this function when no current thread state 1474 is available. If this function returns ``NULL``, no exception has been raised and 1475 the caller should assume no current thread state is available. 1476 1477 1478.. c:function:: int PyThreadState_SetAsyncExc(unsigned long id, PyObject *exc) 1479 1480 Asynchronously raise an exception in a thread. The *id* argument is the thread 1481 id of the target thread; *exc* is the exception object to be raised. This 1482 function does not steal any references to *exc*. To prevent naive misuse, you 1483 must write your own C extension to call this. Must be called with the GIL held. 1484 Returns the number of thread states modified; this is normally one, but will be 1485 zero if the thread id isn't found. If *exc* is ``NULL``, the pending 1486 exception (if any) for the thread is cleared. This raises no exceptions. 1487 1488 .. versionchanged:: 3.7 1489 The type of the *id* parameter changed from :c:expr:`long` to 1490 :c:expr:`unsigned long`. 1491 1492.. c:function:: void PyEval_AcquireThread(PyThreadState *tstate) 1493 1494 Acquire the global interpreter lock and set the current thread state to 1495 *tstate*, which must not be ``NULL``. The lock must have been created earlier. 1496 If this thread already has the lock, deadlock ensues. 1497 1498 .. note:: 1499 Calling this function from a thread when the runtime is finalizing 1500 will terminate the thread, even if the thread was not created by Python. 1501 You can use :c:func:`Py_IsFinalizing` or :func:`sys.is_finalizing` to 1502 check if the interpreter is in process of being finalized before calling 1503 this function to avoid unwanted termination. 1504 1505 .. versionchanged:: 3.8 1506 Updated to be consistent with :c:func:`PyEval_RestoreThread`, 1507 :c:func:`Py_END_ALLOW_THREADS`, and :c:func:`PyGILState_Ensure`, 1508 and terminate the current thread if called while the interpreter is finalizing. 1509 1510 :c:func:`PyEval_RestoreThread` is a higher-level function which is always 1511 available (even when threads have not been initialized). 1512 1513 1514.. c:function:: void PyEval_ReleaseThread(PyThreadState *tstate) 1515 1516 Reset the current thread state to ``NULL`` and release the global interpreter 1517 lock. The lock must have been created earlier and must be held by the current 1518 thread. The *tstate* argument, which must not be ``NULL``, is only used to check 1519 that it represents the current thread state --- if it isn't, a fatal error is 1520 reported. 1521 1522 :c:func:`PyEval_SaveThread` is a higher-level function which is always 1523 available (even when threads have not been initialized). 1524 1525 1526.. _sub-interpreter-support: 1527 1528Sub-interpreter support 1529======================= 1530 1531While in most uses, you will only embed a single Python interpreter, there 1532are cases where you need to create several independent interpreters in the 1533same process and perhaps even in the same thread. Sub-interpreters allow 1534you to do that. 1535 1536The "main" interpreter is the first one created when the runtime initializes. 1537It is usually the only Python interpreter in a process. Unlike sub-interpreters, 1538the main interpreter has unique process-global responsibilities like signal 1539handling. It is also responsible for execution during runtime initialization and 1540is usually the active interpreter during runtime finalization. The 1541:c:func:`PyInterpreterState_Main` function returns a pointer to its state. 1542 1543You can switch between sub-interpreters using the :c:func:`PyThreadState_Swap` 1544function. You can create and destroy them using the following functions: 1545 1546 1547.. c:type:: PyInterpreterConfig 1548 1549 Structure containing most parameters to configure a sub-interpreter. 1550 Its values are used only in :c:func:`Py_NewInterpreterFromConfig` and 1551 never modified by the runtime. 1552 1553 .. versionadded:: 3.12 1554 1555 Structure fields: 1556 1557 .. c:member:: int use_main_obmalloc 1558 1559 If this is ``0`` then the sub-interpreter will use its own 1560 "object" allocator state. 1561 Otherwise it will use (share) the main interpreter's. 1562 1563 If this is ``0`` then 1564 :c:member:`~PyInterpreterConfig.check_multi_interp_extensions` 1565 must be ``1`` (non-zero). 1566 If this is ``1`` then :c:member:`~PyInterpreterConfig.gil` 1567 must not be :c:macro:`PyInterpreterConfig_OWN_GIL`. 1568 1569 .. c:member:: int allow_fork 1570 1571 If this is ``0`` then the runtime will not support forking the 1572 process in any thread where the sub-interpreter is currently active. 1573 Otherwise fork is unrestricted. 1574 1575 Note that the :mod:`subprocess` module still works 1576 when fork is disallowed. 1577 1578 .. c:member:: int allow_exec 1579 1580 If this is ``0`` then the runtime will not support replacing the 1581 current process via exec (e.g. :func:`os.execv`) in any thread 1582 where the sub-interpreter is currently active. 1583 Otherwise exec is unrestricted. 1584 1585 Note that the :mod:`subprocess` module still works 1586 when exec is disallowed. 1587 1588 .. c:member:: int allow_threads 1589 1590 If this is ``0`` then the sub-interpreter's :mod:`threading` module 1591 won't create threads. 1592 Otherwise threads are allowed. 1593 1594 .. c:member:: int allow_daemon_threads 1595 1596 If this is ``0`` then the sub-interpreter's :mod:`threading` module 1597 won't create daemon threads. 1598 Otherwise daemon threads are allowed (as long as 1599 :c:member:`~PyInterpreterConfig.allow_threads` is non-zero). 1600 1601 .. c:member:: int check_multi_interp_extensions 1602 1603 If this is ``0`` then all extension modules may be imported, 1604 including legacy (single-phase init) modules, 1605 in any thread where the sub-interpreter is currently active. 1606 Otherwise only multi-phase init extension modules 1607 (see :pep:`489`) may be imported. 1608 (Also see :c:macro:`Py_mod_multiple_interpreters`.) 1609 1610 This must be ``1`` (non-zero) if 1611 :c:member:`~PyInterpreterConfig.use_main_obmalloc` is ``0``. 1612 1613 .. c:member:: int gil 1614 1615 This determines the operation of the GIL for the sub-interpreter. 1616 It may be one of the following: 1617 1618 .. c:namespace:: NULL 1619 1620 .. c:macro:: PyInterpreterConfig_DEFAULT_GIL 1621 1622 Use the default selection (:c:macro:`PyInterpreterConfig_SHARED_GIL`). 1623 1624 .. c:macro:: PyInterpreterConfig_SHARED_GIL 1625 1626 Use (share) the main interpreter's GIL. 1627 1628 .. c:macro:: PyInterpreterConfig_OWN_GIL 1629 1630 Use the sub-interpreter's own GIL. 1631 1632 If this is :c:macro:`PyInterpreterConfig_OWN_GIL` then 1633 :c:member:`PyInterpreterConfig.use_main_obmalloc` must be ``0``. 1634 1635 1636.. c:function:: PyStatus Py_NewInterpreterFromConfig(PyThreadState **tstate_p, const PyInterpreterConfig *config) 1637 1638 .. index:: 1639 pair: module; builtins 1640 pair: module; __main__ 1641 pair: module; sys 1642 single: stdout (in module sys) 1643 single: stderr (in module sys) 1644 single: stdin (in module sys) 1645 1646 Create a new sub-interpreter. This is an (almost) totally separate environment 1647 for the execution of Python code. In particular, the new interpreter has 1648 separate, independent versions of all imported modules, including the 1649 fundamental modules :mod:`builtins`, :mod:`__main__` and :mod:`sys`. The 1650 table of loaded modules (``sys.modules``) and the module search path 1651 (``sys.path``) are also separate. The new environment has no ``sys.argv`` 1652 variable. It has new standard I/O stream file objects ``sys.stdin``, 1653 ``sys.stdout`` and ``sys.stderr`` (however these refer to the same underlying 1654 file descriptors). 1655 1656 The given *config* controls the options with which the interpreter 1657 is initialized. 1658 1659 Upon success, *tstate_p* will be set to the first thread state 1660 created in the new 1661 sub-interpreter. This thread state is made in the current thread state. 1662 Note that no actual thread is created; see the discussion of thread states 1663 below. If creation of the new interpreter is unsuccessful, 1664 *tstate_p* is set to ``NULL``; 1665 no exception is set since the exception state is stored in the 1666 current thread state and there may not be a current thread state. 1667 1668 Like all other Python/C API functions, the global interpreter lock 1669 must be held before calling this function and is still held when it 1670 returns. Likewise a current thread state must be set on entry. On 1671 success, the returned thread state will be set as current. If the 1672 sub-interpreter is created with its own GIL then the GIL of the 1673 calling interpreter will be released. When the function returns, 1674 the new interpreter's GIL will be held by the current thread and 1675 the previously interpreter's GIL will remain released here. 1676 1677 .. versionadded:: 3.12 1678 1679 Sub-interpreters are most effective when isolated from each other, 1680 with certain functionality restricted:: 1681 1682 PyInterpreterConfig config = { 1683 .use_main_obmalloc = 0, 1684 .allow_fork = 0, 1685 .allow_exec = 0, 1686 .allow_threads = 1, 1687 .allow_daemon_threads = 0, 1688 .check_multi_interp_extensions = 1, 1689 .gil = PyInterpreterConfig_OWN_GIL, 1690 }; 1691 PyThreadState *tstate = NULL; 1692 PyStatus status = Py_NewInterpreterFromConfig(&tstate, &config); 1693 if (PyStatus_Exception(status)) { 1694 Py_ExitStatusException(status); 1695 } 1696 1697 Note that the config is used only briefly and does not get modified. 1698 During initialization the config's values are converted into various 1699 :c:type:`PyInterpreterState` values. A read-only copy of the config 1700 may be stored internally on the :c:type:`PyInterpreterState`. 1701 1702 .. index:: 1703 single: Py_FinalizeEx (C function) 1704 single: Py_Initialize (C function) 1705 1706 Extension modules are shared between (sub-)interpreters as follows: 1707 1708 * For modules using multi-phase initialization, 1709 e.g. :c:func:`PyModule_FromDefAndSpec`, a separate module object is 1710 created and initialized for each interpreter. 1711 Only C-level static and global variables are shared between these 1712 module objects. 1713 1714 * For modules using single-phase initialization, 1715 e.g. :c:func:`PyModule_Create`, the first time a particular extension 1716 is imported, it is initialized normally, and a (shallow) copy of its 1717 module's dictionary is squirreled away. 1718 When the same extension is imported by another (sub-)interpreter, a new 1719 module is initialized and filled with the contents of this copy; the 1720 extension's ``init`` function is not called. 1721 Objects in the module's dictionary thus end up shared across 1722 (sub-)interpreters, which might cause unwanted behavior (see 1723 `Bugs and caveats`_ below). 1724 1725 Note that this is different from what happens when an extension is 1726 imported after the interpreter has been completely re-initialized by 1727 calling :c:func:`Py_FinalizeEx` and :c:func:`Py_Initialize`; in that 1728 case, the extension's ``initmodule`` function *is* called again. 1729 As with multi-phase initialization, this means that only C-level static 1730 and global variables are shared between these modules. 1731 1732 .. index:: single: close (in module os) 1733 1734 1735.. c:function:: PyThreadState* Py_NewInterpreter(void) 1736 1737 .. index:: 1738 pair: module; builtins 1739 pair: module; __main__ 1740 pair: module; sys 1741 single: stdout (in module sys) 1742 single: stderr (in module sys) 1743 single: stdin (in module sys) 1744 1745 Create a new sub-interpreter. This is essentially just a wrapper 1746 around :c:func:`Py_NewInterpreterFromConfig` with a config that 1747 preserves the existing behavior. The result is an unisolated 1748 sub-interpreter that shares the main interpreter's GIL, allows 1749 fork/exec, allows daemon threads, and allows single-phase init 1750 modules. 1751 1752 1753.. c:function:: void Py_EndInterpreter(PyThreadState *tstate) 1754 1755 .. index:: single: Py_FinalizeEx (C function) 1756 1757 Destroy the (sub-)interpreter represented by the given thread state. 1758 The given thread state must be the current thread state. See the 1759 discussion of thread states below. When the call returns, 1760 the current thread state is ``NULL``. All thread states associated 1761 with this interpreter are destroyed. The global interpreter lock 1762 used by the target interpreter must be held before calling this 1763 function. No GIL is held when it returns. 1764 1765 :c:func:`Py_FinalizeEx` will destroy all sub-interpreters that 1766 haven't been explicitly destroyed at that point. 1767 1768 1769A Per-Interpreter GIL 1770--------------------- 1771 1772Using :c:func:`Py_NewInterpreterFromConfig` you can create 1773a sub-interpreter that is completely isolated from other interpreters, 1774including having its own GIL. The most important benefit of this 1775isolation is that such an interpreter can execute Python code without 1776being blocked by other interpreters or blocking any others. Thus a 1777single Python process can truly take advantage of multiple CPU cores 1778when running Python code. The isolation also encourages a different 1779approach to concurrency than that of just using threads. 1780(See :pep:`554`.) 1781 1782Using an isolated interpreter requires vigilance in preserving that 1783isolation. That especially means not sharing any objects or mutable 1784state without guarantees about thread-safety. Even objects that are 1785otherwise immutable (e.g. ``None``, ``(1, 5)``) can't normally be shared 1786because of the refcount. One simple but less-efficient approach around 1787this is to use a global lock around all use of some state (or object). 1788Alternately, effectively immutable objects (like integers or strings) 1789can be made safe in spite of their refcounts by making them :term:`immortal`. 1790In fact, this has been done for the builtin singletons, small integers, 1791and a number of other builtin objects. 1792 1793If you preserve isolation then you will have access to proper multi-core 1794computing without the complications that come with free-threading. 1795Failure to preserve isolation will expose you to the full consequences 1796of free-threading, including races and hard-to-debug crashes. 1797 1798Aside from that, one of the main challenges of using multiple isolated 1799interpreters is how to communicate between them safely (not break 1800isolation) and efficiently. The runtime and stdlib do not provide 1801any standard approach to this yet. A future stdlib module would help 1802mitigate the effort of preserving isolation and expose effective tools 1803for communicating (and sharing) data between interpreters. 1804 1805.. versionadded:: 3.12 1806 1807 1808Bugs and caveats 1809---------------- 1810 1811Because sub-interpreters (and the main interpreter) are part of the same 1812process, the insulation between them isn't perfect --- for example, using 1813low-level file operations like :func:`os.close` they can 1814(accidentally or maliciously) affect each other's open files. Because of the 1815way extensions are shared between (sub-)interpreters, some extensions may not 1816work properly; this is especially likely when using single-phase initialization 1817or (static) global variables. 1818It is possible to insert objects created in one sub-interpreter into 1819a namespace of another (sub-)interpreter; this should be avoided if possible. 1820 1821Special care should be taken to avoid sharing user-defined functions, 1822methods, instances or classes between sub-interpreters, since import 1823operations executed by such objects may affect the wrong (sub-)interpreter's 1824dictionary of loaded modules. It is equally important to avoid sharing 1825objects from which the above are reachable. 1826 1827Also note that combining this functionality with ``PyGILState_*`` APIs 1828is delicate, because these APIs assume a bijection between Python thread states 1829and OS-level threads, an assumption broken by the presence of sub-interpreters. 1830It is highly recommended that you don't switch sub-interpreters between a pair 1831of matching :c:func:`PyGILState_Ensure` and :c:func:`PyGILState_Release` calls. 1832Furthermore, extensions (such as :mod:`ctypes`) using these APIs to allow calling 1833of Python code from non-Python created threads will probably be broken when using 1834sub-interpreters. 1835 1836 1837Asynchronous Notifications 1838========================== 1839 1840A mechanism is provided to make asynchronous notifications to the main 1841interpreter thread. These notifications take the form of a function 1842pointer and a void pointer argument. 1843 1844 1845.. c:function:: int Py_AddPendingCall(int (*func)(void *), void *arg) 1846 1847 Schedule a function to be called from the main interpreter thread. On 1848 success, ``0`` is returned and *func* is queued for being called in the 1849 main thread. On failure, ``-1`` is returned without setting any exception. 1850 1851 When successfully queued, *func* will be *eventually* called from the 1852 main interpreter thread with the argument *arg*. It will be called 1853 asynchronously with respect to normally running Python code, but with 1854 both these conditions met: 1855 1856 * on a :term:`bytecode` boundary; 1857 * with the main thread holding the :term:`global interpreter lock` 1858 (*func* can therefore use the full C API). 1859 1860 *func* must return ``0`` on success, or ``-1`` on failure with an exception 1861 set. *func* won't be interrupted to perform another asynchronous 1862 notification recursively, but it can still be interrupted to switch 1863 threads if the global interpreter lock is released. 1864 1865 This function doesn't need a current thread state to run, and it doesn't 1866 need the global interpreter lock. 1867 1868 To call this function in a subinterpreter, the caller must hold the GIL. 1869 Otherwise, the function *func* can be scheduled to be called from the wrong 1870 interpreter. 1871 1872 .. warning:: 1873 This is a low-level function, only useful for very special cases. 1874 There is no guarantee that *func* will be called as quick as 1875 possible. If the main thread is busy executing a system call, 1876 *func* won't be called before the system call returns. This 1877 function is generally **not** suitable for calling Python code from 1878 arbitrary C threads. Instead, use the :ref:`PyGILState API<gilstate>`. 1879 1880 .. versionadded:: 3.1 1881 1882 .. versionchanged:: 3.9 1883 If this function is called in a subinterpreter, the function *func* is 1884 now scheduled to be called from the subinterpreter, rather than being 1885 called from the main interpreter. Each subinterpreter now has its own 1886 list of scheduled calls. 1887 1888.. _profiling: 1889 1890Profiling and Tracing 1891===================== 1892 1893.. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> 1894 1895 1896The Python interpreter provides some low-level support for attaching profiling 1897and execution tracing facilities. These are used for profiling, debugging, and 1898coverage analysis tools. 1899 1900This C interface allows the profiling or tracing code to avoid the overhead of 1901calling through Python-level callable objects, making a direct C function call 1902instead. The essential attributes of the facility have not changed; the 1903interface allows trace functions to be installed per-thread, and the basic 1904events reported to the trace function are the same as had been reported to the 1905Python-level trace functions in previous versions. 1906 1907 1908.. c:type:: int (*Py_tracefunc)(PyObject *obj, PyFrameObject *frame, int what, PyObject *arg) 1909 1910 The type of the trace function registered using :c:func:`PyEval_SetProfile` and 1911 :c:func:`PyEval_SetTrace`. The first parameter is the object passed to the 1912 registration function as *obj*, *frame* is the frame object to which the event 1913 pertains, *what* is one of the constants :c:data:`PyTrace_CALL`, 1914 :c:data:`PyTrace_EXCEPTION`, :c:data:`PyTrace_LINE`, :c:data:`PyTrace_RETURN`, 1915 :c:data:`PyTrace_C_CALL`, :c:data:`PyTrace_C_EXCEPTION`, :c:data:`PyTrace_C_RETURN`, 1916 or :c:data:`PyTrace_OPCODE`, and *arg* depends on the value of *what*: 1917 1918 +-------------------------------+----------------------------------------+ 1919 | Value of *what* | Meaning of *arg* | 1920 +===============================+========================================+ 1921 | :c:data:`PyTrace_CALL` | Always :c:data:`Py_None`. | 1922 +-------------------------------+----------------------------------------+ 1923 | :c:data:`PyTrace_EXCEPTION` | Exception information as returned by | 1924 | | :func:`sys.exc_info`. | 1925 +-------------------------------+----------------------------------------+ 1926 | :c:data:`PyTrace_LINE` | Always :c:data:`Py_None`. | 1927 +-------------------------------+----------------------------------------+ 1928 | :c:data:`PyTrace_RETURN` | Value being returned to the caller, | 1929 | | or ``NULL`` if caused by an exception. | 1930 +-------------------------------+----------------------------------------+ 1931 | :c:data:`PyTrace_C_CALL` | Function object being called. | 1932 +-------------------------------+----------------------------------------+ 1933 | :c:data:`PyTrace_C_EXCEPTION` | Function object being called. | 1934 +-------------------------------+----------------------------------------+ 1935 | :c:data:`PyTrace_C_RETURN` | Function object being called. | 1936 +-------------------------------+----------------------------------------+ 1937 | :c:data:`PyTrace_OPCODE` | Always :c:data:`Py_None`. | 1938 +-------------------------------+----------------------------------------+ 1939 1940.. c:var:: int PyTrace_CALL 1941 1942 The value of the *what* parameter to a :c:type:`Py_tracefunc` function when a new 1943 call to a function or method is being reported, or a new entry into a generator. 1944 Note that the creation of the iterator for a generator function is not reported 1945 as there is no control transfer to the Python bytecode in the corresponding 1946 frame. 1947 1948 1949.. c:var:: int PyTrace_EXCEPTION 1950 1951 The value of the *what* parameter to a :c:type:`Py_tracefunc` function when an 1952 exception has been raised. The callback function is called with this value for 1953 *what* when after any bytecode is processed after which the exception becomes 1954 set within the frame being executed. The effect of this is that as exception 1955 propagation causes the Python stack to unwind, the callback is called upon 1956 return to each frame as the exception propagates. Only trace functions receives 1957 these events; they are not needed by the profiler. 1958 1959 1960.. c:var:: int PyTrace_LINE 1961 1962 The value passed as the *what* parameter to a :c:type:`Py_tracefunc` function 1963 (but not a profiling function) when a line-number event is being reported. 1964 It may be disabled for a frame by setting :attr:`~frame.f_trace_lines` to 1965 *0* on that frame. 1966 1967 1968.. c:var:: int PyTrace_RETURN 1969 1970 The value for the *what* parameter to :c:type:`Py_tracefunc` functions when a 1971 call is about to return. 1972 1973 1974.. c:var:: int PyTrace_C_CALL 1975 1976 The value for the *what* parameter to :c:type:`Py_tracefunc` functions when a C 1977 function is about to be called. 1978 1979 1980.. c:var:: int PyTrace_C_EXCEPTION 1981 1982 The value for the *what* parameter to :c:type:`Py_tracefunc` functions when a C 1983 function has raised an exception. 1984 1985 1986.. c:var:: int PyTrace_C_RETURN 1987 1988 The value for the *what* parameter to :c:type:`Py_tracefunc` functions when a C 1989 function has returned. 1990 1991 1992.. c:var:: int PyTrace_OPCODE 1993 1994 The value for the *what* parameter to :c:type:`Py_tracefunc` functions (but not 1995 profiling functions) when a new opcode is about to be executed. This event is 1996 not emitted by default: it must be explicitly requested by setting 1997 :attr:`~frame.f_trace_opcodes` to *1* on the frame. 1998 1999 2000.. c:function:: void PyEval_SetProfile(Py_tracefunc func, PyObject *obj) 2001 2002 Set the profiler function to *func*. The *obj* parameter is passed to the 2003 function as its first parameter, and may be any Python object, or ``NULL``. If 2004 the profile function needs to maintain state, using a different value for *obj* 2005 for each thread provides a convenient and thread-safe place to store it. The 2006 profile function is called for all monitored events except :c:data:`PyTrace_LINE` 2007 :c:data:`PyTrace_OPCODE` and :c:data:`PyTrace_EXCEPTION`. 2008 2009 See also the :func:`sys.setprofile` function. 2010 2011 The caller must hold the :term:`GIL`. 2012 2013.. c:function:: void PyEval_SetProfileAllThreads(Py_tracefunc func, PyObject *obj) 2014 2015 Like :c:func:`PyEval_SetProfile` but sets the profile function in all running threads 2016 belonging to the current interpreter instead of the setting it only on the current thread. 2017 2018 The caller must hold the :term:`GIL`. 2019 2020 As :c:func:`PyEval_SetProfile`, this function ignores any exceptions raised while 2021 setting the profile functions in all threads. 2022 2023.. versionadded:: 3.12 2024 2025 2026.. c:function:: void PyEval_SetTrace(Py_tracefunc func, PyObject *obj) 2027 2028 Set the tracing function to *func*. This is similar to 2029 :c:func:`PyEval_SetProfile`, except the tracing function does receive line-number 2030 events and per-opcode events, but does not receive any event related to C function 2031 objects being called. Any trace function registered using :c:func:`PyEval_SetTrace` 2032 will not receive :c:data:`PyTrace_C_CALL`, :c:data:`PyTrace_C_EXCEPTION` or 2033 :c:data:`PyTrace_C_RETURN` as a value for the *what* parameter. 2034 2035 See also the :func:`sys.settrace` function. 2036 2037 The caller must hold the :term:`GIL`. 2038 2039.. c:function:: void PyEval_SetTraceAllThreads(Py_tracefunc func, PyObject *obj) 2040 2041 Like :c:func:`PyEval_SetTrace` but sets the tracing function in all running threads 2042 belonging to the current interpreter instead of the setting it only on the current thread. 2043 2044 The caller must hold the :term:`GIL`. 2045 2046 As :c:func:`PyEval_SetTrace`, this function ignores any exceptions raised while 2047 setting the trace functions in all threads. 2048 2049.. versionadded:: 3.12 2050 2051Reference tracing 2052================= 2053 2054.. versionadded:: 3.13 2055 2056.. c:type:: int (*PyRefTracer)(PyObject *, int event, void* data) 2057 2058 The type of the trace function registered using :c:func:`PyRefTracer_SetTracer`. 2059 The first parameter is a Python object that has been just created (when **event** 2060 is set to :c:data:`PyRefTracer_CREATE`) or about to be destroyed (when **event** 2061 is set to :c:data:`PyRefTracer_DESTROY`). The **data** argument is the opaque pointer 2062 that was provided when :c:func:`PyRefTracer_SetTracer` was called. 2063 2064.. versionadded:: 3.13 2065 2066.. c:var:: int PyRefTracer_CREATE 2067 2068 The value for the *event* parameter to :c:type:`PyRefTracer` functions when a Python 2069 object has been created. 2070 2071.. c:var:: int PyRefTracer_DESTROY 2072 2073 The value for the *event* parameter to :c:type:`PyRefTracer` functions when a Python 2074 object has been destroyed. 2075 2076.. c:function:: int PyRefTracer_SetTracer(PyRefTracer tracer, void *data) 2077 2078 Register a reference tracer function. The function will be called when a new 2079 Python has been created or when an object is going to be destroyed. If 2080 **data** is provided it must be an opaque pointer that will be provided when 2081 the tracer function is called. Return ``0`` on success. Set an exception and 2082 return ``-1`` on error. 2083 2084 Not that tracer functions **must not** create Python objects inside or 2085 otherwise the call will be re-entrant. The tracer also **must not** clear 2086 any existing exception or set an exception. The GIL will be held every time 2087 the tracer function is called. 2088 2089 The GIL must be held when calling this function. 2090 2091.. versionadded:: 3.13 2092 2093.. c:function:: PyRefTracer PyRefTracer_GetTracer(void** data) 2094 2095 Get the registered reference tracer function and the value of the opaque data 2096 pointer that was registered when :c:func:`PyRefTracer_SetTracer` was called. 2097 If no tracer was registered this function will return NULL and will set the 2098 **data** pointer to NULL. 2099 2100 The GIL must be held when calling this function. 2101 2102.. versionadded:: 3.13 2103 2104.. _advanced-debugging: 2105 2106Advanced Debugger Support 2107========================= 2108 2109.. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> 2110 2111 2112These functions are only intended to be used by advanced debugging tools. 2113 2114 2115.. c:function:: PyInterpreterState* PyInterpreterState_Head() 2116 2117 Return the interpreter state object at the head of the list of all such objects. 2118 2119 2120.. c:function:: PyInterpreterState* PyInterpreterState_Main() 2121 2122 Return the main interpreter state object. 2123 2124 2125.. c:function:: PyInterpreterState* PyInterpreterState_Next(PyInterpreterState *interp) 2126 2127 Return the next interpreter state object after *interp* from the list of all 2128 such objects. 2129 2130 2131.. c:function:: PyThreadState * PyInterpreterState_ThreadHead(PyInterpreterState *interp) 2132 2133 Return the pointer to the first :c:type:`PyThreadState` object in the list of 2134 threads associated with the interpreter *interp*. 2135 2136 2137.. c:function:: PyThreadState* PyThreadState_Next(PyThreadState *tstate) 2138 2139 Return the next thread state object after *tstate* from the list of all such 2140 objects belonging to the same :c:type:`PyInterpreterState` object. 2141 2142 2143.. _thread-local-storage: 2144 2145Thread Local Storage Support 2146============================ 2147 2148.. sectionauthor:: Masayuki Yamamoto <ma3yuki.8mamo10@gmail.com> 2149 2150The Python interpreter provides low-level support for thread-local storage 2151(TLS) which wraps the underlying native TLS implementation to support the 2152Python-level thread local storage API (:class:`threading.local`). The 2153CPython C level APIs are similar to those offered by pthreads and Windows: 2154use a thread key and functions to associate a :c:expr:`void*` value per 2155thread. 2156 2157The GIL does *not* need to be held when calling these functions; they supply 2158their own locking. 2159 2160Note that :file:`Python.h` does not include the declaration of the TLS APIs, 2161you need to include :file:`pythread.h` to use thread-local storage. 2162 2163.. note:: 2164 None of these API functions handle memory management on behalf of the 2165 :c:expr:`void*` values. You need to allocate and deallocate them yourself. 2166 If the :c:expr:`void*` values happen to be :c:expr:`PyObject*`, these 2167 functions don't do refcount operations on them either. 2168 2169.. _thread-specific-storage-api: 2170 2171Thread Specific Storage (TSS) API 2172--------------------------------- 2173 2174TSS API is introduced to supersede the use of the existing TLS API within the 2175CPython interpreter. This API uses a new type :c:type:`Py_tss_t` instead of 2176:c:expr:`int` to represent thread keys. 2177 2178.. versionadded:: 3.7 2179 2180.. seealso:: "A New C-API for Thread-Local Storage in CPython" (:pep:`539`) 2181 2182 2183.. c:type:: Py_tss_t 2184 2185 This data structure represents the state of a thread key, the definition of 2186 which may depend on the underlying TLS implementation, and it has an 2187 internal field representing the key's initialization state. There are no 2188 public members in this structure. 2189 2190 When :ref:`Py_LIMITED_API <stable>` is not defined, static allocation of 2191 this type by :c:macro:`Py_tss_NEEDS_INIT` is allowed. 2192 2193 2194.. c:macro:: Py_tss_NEEDS_INIT 2195 2196 This macro expands to the initializer for :c:type:`Py_tss_t` variables. 2197 Note that this macro won't be defined with :ref:`Py_LIMITED_API <stable>`. 2198 2199 2200Dynamic Allocation 2201~~~~~~~~~~~~~~~~~~ 2202 2203Dynamic allocation of the :c:type:`Py_tss_t`, required in extension modules 2204built with :ref:`Py_LIMITED_API <stable>`, where static allocation of this type 2205is not possible due to its implementation being opaque at build time. 2206 2207 2208.. c:function:: Py_tss_t* PyThread_tss_alloc() 2209 2210 Return a value which is the same state as a value initialized with 2211 :c:macro:`Py_tss_NEEDS_INIT`, or ``NULL`` in the case of dynamic allocation 2212 failure. 2213 2214 2215.. c:function:: void PyThread_tss_free(Py_tss_t *key) 2216 2217 Free the given *key* allocated by :c:func:`PyThread_tss_alloc`, after 2218 first calling :c:func:`PyThread_tss_delete` to ensure any associated 2219 thread locals have been unassigned. This is a no-op if the *key* 2220 argument is ``NULL``. 2221 2222 .. note:: 2223 A freed key becomes a dangling pointer. You should reset the key to 2224 ``NULL``. 2225 2226 2227Methods 2228~~~~~~~ 2229 2230The parameter *key* of these functions must not be ``NULL``. Moreover, the 2231behaviors of :c:func:`PyThread_tss_set` and :c:func:`PyThread_tss_get` are 2232undefined if the given :c:type:`Py_tss_t` has not been initialized by 2233:c:func:`PyThread_tss_create`. 2234 2235 2236.. c:function:: int PyThread_tss_is_created(Py_tss_t *key) 2237 2238 Return a non-zero value if the given :c:type:`Py_tss_t` has been initialized 2239 by :c:func:`PyThread_tss_create`. 2240 2241 2242.. c:function:: int PyThread_tss_create(Py_tss_t *key) 2243 2244 Return a zero value on successful initialization of a TSS key. The behavior 2245 is undefined if the value pointed to by the *key* argument is not 2246 initialized by :c:macro:`Py_tss_NEEDS_INIT`. This function can be called 2247 repeatedly on the same key -- calling it on an already initialized key is a 2248 no-op and immediately returns success. 2249 2250 2251.. c:function:: void PyThread_tss_delete(Py_tss_t *key) 2252 2253 Destroy a TSS key to forget the values associated with the key across all 2254 threads, and change the key's initialization state to uninitialized. A 2255 destroyed key is able to be initialized again by 2256 :c:func:`PyThread_tss_create`. This function can be called repeatedly on 2257 the same key -- calling it on an already destroyed key is a no-op. 2258 2259 2260.. c:function:: int PyThread_tss_set(Py_tss_t *key, void *value) 2261 2262 Return a zero value to indicate successfully associating a :c:expr:`void*` 2263 value with a TSS key in the current thread. Each thread has a distinct 2264 mapping of the key to a :c:expr:`void*` value. 2265 2266 2267.. c:function:: void* PyThread_tss_get(Py_tss_t *key) 2268 2269 Return the :c:expr:`void*` value associated with a TSS key in the current 2270 thread. This returns ``NULL`` if no value is associated with the key in the 2271 current thread. 2272 2273 2274.. _thread-local-storage-api: 2275 2276Thread Local Storage (TLS) API 2277------------------------------ 2278 2279.. deprecated:: 3.7 2280 This API is superseded by 2281 :ref:`Thread Specific Storage (TSS) API <thread-specific-storage-api>`. 2282 2283.. note:: 2284 This version of the API does not support platforms where the native TLS key 2285 is defined in a way that cannot be safely cast to ``int``. On such platforms, 2286 :c:func:`PyThread_create_key` will return immediately with a failure status, 2287 and the other TLS functions will all be no-ops on such platforms. 2288 2289Due to the compatibility problem noted above, this version of the API should not 2290be used in new code. 2291 2292.. c:function:: int PyThread_create_key() 2293.. c:function:: void PyThread_delete_key(int key) 2294.. c:function:: int PyThread_set_key_value(int key, void *value) 2295.. c:function:: void* PyThread_get_key_value(int key) 2296.. c:function:: void PyThread_delete_key_value(int key) 2297.. c:function:: void PyThread_ReInitTLS() 2298 2299Synchronization Primitives 2300========================== 2301 2302The C-API provides a basic mutual exclusion lock. 2303 2304.. c:type:: PyMutex 2305 2306 A mutual exclusion lock. The :c:type:`!PyMutex` should be initialized to 2307 zero to represent the unlocked state. For example:: 2308 2309 PyMutex mutex = {0}; 2310 2311 Instances of :c:type:`!PyMutex` should not be copied or moved. Both the 2312 contents and address of a :c:type:`!PyMutex` are meaningful, and it must 2313 remain at a fixed, writable location in memory. 2314 2315 .. note:: 2316 2317 A :c:type:`!PyMutex` currently occupies one byte, but the size should be 2318 considered unstable. The size may change in future Python releases 2319 without a deprecation period. 2320 2321 .. versionadded:: 3.13 2322 2323.. c:function:: void PyMutex_Lock(PyMutex *m) 2324 2325 Lock mutex *m*. If another thread has already locked it, the calling 2326 thread will block until the mutex is unlocked. While blocked, the thread 2327 will temporarily release the :term:`GIL` if it is held. 2328 2329 .. versionadded:: 3.13 2330 2331.. c:function:: void PyMutex_Unlock(PyMutex *m) 2332 2333 Unlock mutex *m*. The mutex must be locked --- otherwise, the function will 2334 issue a fatal error. 2335 2336 .. versionadded:: 3.13 2337 2338.. _python-critical-section-api: 2339 2340Python Critical Section API 2341--------------------------- 2342 2343The critical section API provides a deadlock avoidance layer on top of 2344per-object locks for :term:`free-threaded <free threading>` CPython. They are 2345intended to replace reliance on the :term:`global interpreter lock`, and are 2346no-ops in versions of Python with the global interpreter lock. 2347 2348Critical sections avoid deadlocks by implicitly suspending active critical 2349sections and releasing the locks during calls to :c:func:`PyEval_SaveThread`. 2350When :c:func:`PyEval_RestoreThread` is called, the most recent critical section 2351is resumed, and its locks reacquired. This means the critical section API 2352provides weaker guarantees than traditional locks -- they are useful because 2353their behavior is similar to the :term:`GIL`. 2354 2355The functions and structs used by the macros are exposed for cases 2356where C macros are not available. They should only be used as in the 2357given macro expansions. Note that the sizes and contents of the structures may 2358change in future Python versions. 2359 2360.. note:: 2361 2362 Operations that need to lock two objects at once must use 2363 :c:macro:`Py_BEGIN_CRITICAL_SECTION2`. You *cannot* use nested critical 2364 sections to lock more than one object at once, because the inner critical 2365 section may suspend the outer critical sections. This API does not provide 2366 a way to lock more than two objects at once. 2367 2368Example usage:: 2369 2370 static PyObject * 2371 set_field(MyObject *self, PyObject *value) 2372 { 2373 Py_BEGIN_CRITICAL_SECTION(self); 2374 Py_SETREF(self->field, Py_XNewRef(value)); 2375 Py_END_CRITICAL_SECTION(); 2376 Py_RETURN_NONE; 2377 } 2378 2379In the above example, :c:macro:`Py_SETREF` calls :c:macro:`Py_DECREF`, which 2380can call arbitrary code through an object's deallocation function. The critical 2381section API avoids potential deadlocks due to reentrancy and lock ordering 2382by allowing the runtime to temporarily suspend the critical section if the 2383code triggered by the finalizer blocks and calls :c:func:`PyEval_SaveThread`. 2384 2385.. c:macro:: Py_BEGIN_CRITICAL_SECTION(op) 2386 2387 Acquires the per-object lock for the object *op* and begins a 2388 critical section. 2389 2390 In the free-threaded build, this macro expands to:: 2391 2392 { 2393 PyCriticalSection _py_cs; 2394 PyCriticalSection_Begin(&_py_cs, (PyObject*)(op)) 2395 2396 In the default build, this macro expands to ``{``. 2397 2398 .. versionadded:: 3.13 2399 2400.. c:macro:: Py_END_CRITICAL_SECTION() 2401 2402 Ends the critical section and releases the per-object lock. 2403 2404 In the free-threaded build, this macro expands to:: 2405 2406 PyCriticalSection_End(&_py_cs); 2407 } 2408 2409 In the default build, this macro expands to ``}``. 2410 2411 .. versionadded:: 3.13 2412 2413.. c:macro:: Py_BEGIN_CRITICAL_SECTION2(a, b) 2414 2415 Acquires the per-objects locks for the objects *a* and *b* and begins a 2416 critical section. The locks are acquired in a consistent order (lowest 2417 address first) to avoid lock ordering deadlocks. 2418 2419 In the free-threaded build, this macro expands to:: 2420 2421 { 2422 PyCriticalSection2 _py_cs2; 2423 PyCriticalSection2_Begin(&_py_cs2, (PyObject*)(a), (PyObject*)(b)) 2424 2425 In the default build, this macro expands to ``{``. 2426 2427 .. versionadded:: 3.13 2428 2429.. c:macro:: Py_END_CRITICAL_SECTION2() 2430 2431 Ends the critical section and releases the per-object locks. 2432 2433 In the free-threaded build, this macro expands to:: 2434 2435 PyCriticalSection2_End(&_py_cs2); 2436 } 2437 2438 In the default build, this macro expands to ``}``. 2439 2440 .. versionadded:: 3.13 2441