1.. highlight:: c 2 3 4.. _initialization: 5 6***************************************** 7Initialization, Finalization, and Threads 8***************************************** 9 10See also :ref:`Python Initialization Configuration <init-config>`. 11 12.. _pre-init-safe: 13 14Before Python Initialization 15============================ 16 17In an application embedding Python, the :c:func:`Py_Initialize` function must 18be called before using any other Python/C API functions; with the exception of 19a few functions and the :ref:`global configuration variables 20<global-conf-vars>`. 21 22The following functions can be safely called before Python is initialized: 23 24* Configuration functions: 25 26 * :c:func:`PyImport_AppendInittab` 27 * :c:func:`PyImport_ExtendInittab` 28 * :c:func:`PyInitFrozenExtensions` 29 * :c:func:`PyMem_SetAllocator` 30 * :c:func:`PyMem_SetupDebugHooks` 31 * :c:func:`PyObject_SetArenaAllocator` 32 * :c:func:`Py_SetPath` 33 * :c:func:`Py_SetProgramName` 34 * :c:func:`Py_SetPythonHome` 35 * :c:func:`Py_SetStandardStreamEncoding` 36 * :c:func:`PySys_AddWarnOption` 37 * :c:func:`PySys_AddXOption` 38 * :c:func:`PySys_ResetWarnOptions` 39 40* Informative functions: 41 42 * :c:func:`Py_IsInitialized` 43 * :c:func:`PyMem_GetAllocator` 44 * :c:func:`PyObject_GetArenaAllocator` 45 * :c:func:`Py_GetBuildInfo` 46 * :c:func:`Py_GetCompiler` 47 * :c:func:`Py_GetCopyright` 48 * :c:func:`Py_GetPlatform` 49 * :c:func:`Py_GetVersion` 50 51* Utilities: 52 53 * :c:func:`Py_DecodeLocale` 54 55* Memory allocators: 56 57 * :c:func:`PyMem_RawMalloc` 58 * :c:func:`PyMem_RawRealloc` 59 * :c:func:`PyMem_RawCalloc` 60 * :c:func:`PyMem_RawFree` 61 62.. note:: 63 64 The following functions **should not be called** before 65 :c:func:`Py_Initialize`: :c:func:`Py_EncodeLocale`, :c:func:`Py_GetPath`, 66 :c:func:`Py_GetPrefix`, :c:func:`Py_GetExecPrefix`, 67 :c:func:`Py_GetProgramFullPath`, :c:func:`Py_GetPythonHome`, 68 :c:func:`Py_GetProgramName` and :c:func:`PyEval_InitThreads`. 69 70 71.. _global-conf-vars: 72 73Global configuration variables 74============================== 75 76Python has variables for the global configuration to control different features 77and options. By default, these flags are controlled by :ref:`command line 78options <using-on-interface-options>`. 79 80When a flag is set by an option, the value of the flag is the number of times 81that the option was set. For example, ``-b`` sets :c:data:`Py_BytesWarningFlag` 82to 1 and ``-bb`` sets :c:data:`Py_BytesWarningFlag` to 2. 83 84.. c:var:: Py_BytesWarningFlag 85 86 Issue a warning when comparing :class:`bytes` or :class:`bytearray` with 87 :class:`str` or :class:`bytes` with :class:`int`. Issue an error if greater 88 or equal to ``2``. 89 90 Set by the :option:`-b` option. 91 92.. c:var:: Py_DebugFlag 93 94 Turn on parser debugging output (for expert only, depending on compilation 95 options). 96 97 Set by the :option:`-d` option and the :envvar:`PYTHONDEBUG` environment 98 variable. 99 100.. c:var:: Py_DontWriteBytecodeFlag 101 102 If set to non-zero, Python won't try to write ``.pyc`` files on the 103 import of source modules. 104 105 Set by the :option:`-B` option and the :envvar:`PYTHONDONTWRITEBYTECODE` 106 environment variable. 107 108.. c:var:: Py_FrozenFlag 109 110 Suppress error messages when calculating the module search path in 111 :c:func:`Py_GetPath`. 112 113 Private flag used by ``_freeze_importlib`` and ``frozenmain`` programs. 114 115.. c:var:: Py_HashRandomizationFlag 116 117 Set to ``1`` if the :envvar:`PYTHONHASHSEED` environment variable is set to 118 a non-empty string. 119 120 If the flag is non-zero, read the :envvar:`PYTHONHASHSEED` environment 121 variable to initialize the secret hash seed. 122 123.. c:var:: Py_IgnoreEnvironmentFlag 124 125 Ignore all :envvar:`PYTHON*` environment variables, e.g. 126 :envvar:`PYTHONPATH` and :envvar:`PYTHONHOME`, that might be set. 127 128 Set by the :option:`-E` and :option:`-I` options. 129 130.. c:var:: Py_InspectFlag 131 132 When a script is passed as first argument or the :option:`-c` option is used, 133 enter interactive mode after executing the script or the command, even when 134 :data:`sys.stdin` does not appear to be a terminal. 135 136 Set by the :option:`-i` option and the :envvar:`PYTHONINSPECT` environment 137 variable. 138 139.. c:var:: Py_InteractiveFlag 140 141 Set by the :option:`-i` option. 142 143.. c:var:: Py_IsolatedFlag 144 145 Run Python in isolated mode. In isolated mode :data:`sys.path` contains 146 neither the script's directory nor the user's site-packages directory. 147 148 Set by the :option:`-I` option. 149 150 .. versionadded:: 3.4 151 152.. c:var:: Py_LegacyWindowsFSEncodingFlag 153 154 If the flag is non-zero, use the ``mbcs`` encoding instead of the UTF-8 155 encoding for the filesystem encoding. 156 157 Set to ``1`` if the :envvar:`PYTHONLEGACYWINDOWSFSENCODING` environment 158 variable is set to a non-empty string. 159 160 See :pep:`529` for more details. 161 162 .. availability:: Windows. 163 164.. c:var:: Py_LegacyWindowsStdioFlag 165 166 If the flag is non-zero, use :class:`io.FileIO` instead of 167 :class:`WindowsConsoleIO` for :mod:`sys` standard streams. 168 169 Set to ``1`` if the :envvar:`PYTHONLEGACYWINDOWSSTDIO` environment 170 variable is set to a non-empty string. 171 172 See :pep:`528` for more details. 173 174 .. availability:: Windows. 175 176.. c:var:: Py_NoSiteFlag 177 178 Disable the import of the module :mod:`site` and the site-dependent 179 manipulations of :data:`sys.path` that it entails. Also disable these 180 manipulations if :mod:`site` is explicitly imported later (call 181 :func:`site.main` if you want them to be triggered). 182 183 Set by the :option:`-S` option. 184 185.. c:var:: Py_NoUserSiteDirectory 186 187 Don't add the :data:`user site-packages directory <site.USER_SITE>` to 188 :data:`sys.path`. 189 190 Set by the :option:`-s` and :option:`-I` options, and the 191 :envvar:`PYTHONNOUSERSITE` environment variable. 192 193.. c:var:: Py_OptimizeFlag 194 195 Set by the :option:`-O` option and the :envvar:`PYTHONOPTIMIZE` environment 196 variable. 197 198.. c:var:: Py_QuietFlag 199 200 Don't display the copyright and version messages even in interactive mode. 201 202 Set by the :option:`-q` option. 203 204 .. versionadded:: 3.2 205 206.. c:var:: Py_UnbufferedStdioFlag 207 208 Force the stdout and stderr streams to be unbuffered. 209 210 Set by the :option:`-u` option and the :envvar:`PYTHONUNBUFFERED` 211 environment variable. 212 213.. c:var:: Py_VerboseFlag 214 215 Print a message each time a module is initialized, showing the place 216 (filename or built-in module) from which it is loaded. If greater or equal 217 to ``2``, print a message for each file that is checked for when 218 searching for a module. Also provides information on module cleanup at exit. 219 220 Set by the :option:`-v` option and the :envvar:`PYTHONVERBOSE` environment 221 variable. 222 223 224Initializing and finalizing the interpreter 225=========================================== 226 227 228.. c:function:: void Py_Initialize() 229 230 .. index:: 231 single: Py_SetProgramName() 232 single: PyEval_InitThreads() 233 single: modules (in module sys) 234 single: path (in module sys) 235 module: builtins 236 module: __main__ 237 module: sys 238 triple: module; search; path 239 single: PySys_SetArgv() 240 single: PySys_SetArgvEx() 241 single: Py_FinalizeEx() 242 243 Initialize the Python interpreter. In an application embedding Python, 244 this should be called before using any other Python/C API functions; see 245 :ref:`Before Python Initialization <pre-init-safe>` for the few exceptions. 246 247 This initializes 248 the table of loaded modules (``sys.modules``), and creates the fundamental 249 modules :mod:`builtins`, :mod:`__main__` and :mod:`sys`. It also initializes 250 the module search path (``sys.path``). It does not set ``sys.argv``; use 251 :c:func:`PySys_SetArgvEx` for that. This is a no-op when called for a second time 252 (without calling :c:func:`Py_FinalizeEx` first). There is no return value; it is a 253 fatal error if the initialization fails. 254 255 .. note:: 256 On Windows, changes the console mode from ``O_TEXT`` to ``O_BINARY``, which will 257 also affect non-Python uses of the console using the C Runtime. 258 259 260.. c:function:: void Py_InitializeEx(int initsigs) 261 262 This function works like :c:func:`Py_Initialize` if *initsigs* is ``1``. If 263 *initsigs* is ``0``, it skips initialization registration of signal handlers, which 264 might be useful when Python is embedded. 265 266 267.. c:function:: int Py_IsInitialized() 268 269 Return true (nonzero) when the Python interpreter has been initialized, false 270 (zero) if not. After :c:func:`Py_FinalizeEx` is called, this returns false until 271 :c:func:`Py_Initialize` is called again. 272 273 274.. c:function:: int Py_FinalizeEx() 275 276 Undo all initializations made by :c:func:`Py_Initialize` and subsequent use of 277 Python/C API functions, and destroy all sub-interpreters (see 278 :c:func:`Py_NewInterpreter` below) that were created and not yet destroyed since 279 the last call to :c:func:`Py_Initialize`. Ideally, this frees all memory 280 allocated by the Python interpreter. This is a no-op when called for a second 281 time (without calling :c:func:`Py_Initialize` again first). Normally the 282 return value is ``0``. If there were errors during finalization 283 (flushing buffered data), ``-1`` is returned. 284 285 This function is provided for a number of reasons. An embedding application 286 might want to restart Python without having to restart the application itself. 287 An application that has loaded the Python interpreter from a dynamically 288 loadable library (or DLL) might want to free all memory allocated by Python 289 before unloading the DLL. During a hunt for memory leaks in an application a 290 developer might want to free all memory allocated by Python before exiting from 291 the application. 292 293 **Bugs and caveats:** The destruction of modules and objects in modules is done 294 in random order; this may cause destructors (:meth:`__del__` methods) to fail 295 when they depend on other objects (even functions) or modules. Dynamically 296 loaded extension modules loaded by Python are not unloaded. Small amounts of 297 memory allocated by the Python interpreter may not be freed (if you find a leak, 298 please report it). Memory tied up in circular references between objects is not 299 freed. Some memory allocated by extension modules may not be freed. Some 300 extensions may not work properly if their initialization routine is called more 301 than once; this can happen if an application calls :c:func:`Py_Initialize` and 302 :c:func:`Py_FinalizeEx` more than once. 303 304 .. audit-event:: cpython._PySys_ClearAuditHooks "" c.Py_FinalizeEx 305 306 .. versionadded:: 3.6 307 308.. c:function:: void Py_Finalize() 309 310 This is a backwards-compatible version of :c:func:`Py_FinalizeEx` that 311 disregards the return value. 312 313 314Process-wide parameters 315======================= 316 317 318.. c:function:: int Py_SetStandardStreamEncoding(const char *encoding, const char *errors) 319 320 .. index:: 321 single: Py_Initialize() 322 single: main() 323 triple: stdin; stdout; sdterr 324 325 This function should be called before :c:func:`Py_Initialize`, if it is 326 called at all. It specifies which encoding and error handling to use 327 with standard IO, with the same meanings as in :func:`str.encode`. 328 329 It overrides :envvar:`PYTHONIOENCODING` values, and allows embedding code 330 to control IO encoding when the environment variable does not work. 331 332 *encoding* and/or *errors* may be ``NULL`` to use 333 :envvar:`PYTHONIOENCODING` and/or default values (depending on other 334 settings). 335 336 Note that :data:`sys.stderr` always uses the "backslashreplace" error 337 handler, regardless of this (or any other) setting. 338 339 If :c:func:`Py_FinalizeEx` is called, this function will need to be called 340 again in order to affect subsequent calls to :c:func:`Py_Initialize`. 341 342 Returns ``0`` if successful, a nonzero value on error (e.g. calling after the 343 interpreter has already been initialized). 344 345 .. versionadded:: 3.4 346 347 348.. c:function:: void Py_SetProgramName(const wchar_t *name) 349 350 .. index:: 351 single: Py_Initialize() 352 single: main() 353 single: Py_GetPath() 354 355 This function should be called before :c:func:`Py_Initialize` is called for 356 the first time, if it is called at all. It tells the interpreter the value 357 of the ``argv[0]`` argument to the :c:func:`main` function of the program 358 (converted to wide characters). 359 This is used by :c:func:`Py_GetPath` and some other functions below to find 360 the Python run-time libraries relative to the interpreter executable. The 361 default value is ``'python'``. The argument should point to a 362 zero-terminated wide character string in static storage whose contents will not 363 change for the duration of the program's execution. No code in the Python 364 interpreter will change the contents of this storage. 365 366 Use :c:func:`Py_DecodeLocale` to decode a bytes string to get a 367 :c:type:`wchar_*` string. 368 369 370.. c:function:: wchar* Py_GetProgramName() 371 372 .. index:: single: Py_SetProgramName() 373 374 Return the program name set with :c:func:`Py_SetProgramName`, or the default. 375 The returned string points into static storage; the caller should not modify its 376 value. 377 378 379.. c:function:: wchar_t* Py_GetPrefix() 380 381 Return the *prefix* for installed platform-independent files. This is derived 382 through a number of complicated rules from the program name set with 383 :c:func:`Py_SetProgramName` and some environment variables; for example, if the 384 program name is ``'/usr/local/bin/python'``, the prefix is ``'/usr/local'``. The 385 returned string points into static storage; the caller should not modify its 386 value. This corresponds to the :makevar:`prefix` variable in the top-level 387 :file:`Makefile` and the ``--prefix`` argument to the :program:`configure` 388 script at build time. The value is available to Python code as ``sys.prefix``. 389 It is only useful on Unix. See also the next function. 390 391 392.. c:function:: wchar_t* Py_GetExecPrefix() 393 394 Return the *exec-prefix* for installed platform-*dependent* files. This is 395 derived through a number of complicated rules from the program name set with 396 :c:func:`Py_SetProgramName` and some environment variables; for example, if the 397 program name is ``'/usr/local/bin/python'``, the exec-prefix is 398 ``'/usr/local'``. The returned string points into static storage; the caller 399 should not modify its value. This corresponds to the :makevar:`exec_prefix` 400 variable in the top-level :file:`Makefile` and the ``--exec-prefix`` 401 argument to the :program:`configure` script at build time. The value is 402 available to Python code as ``sys.exec_prefix``. It is only useful on Unix. 403 404 Background: The exec-prefix differs from the prefix when platform dependent 405 files (such as executables and shared libraries) are installed in a different 406 directory tree. In a typical installation, platform dependent files may be 407 installed in the :file:`/usr/local/plat` subtree while platform independent may 408 be installed in :file:`/usr/local`. 409 410 Generally speaking, a platform is a combination of hardware and software 411 families, e.g. Sparc machines running the Solaris 2.x operating system are 412 considered the same platform, but Intel machines running Solaris 2.x are another 413 platform, and Intel machines running Linux are yet another platform. Different 414 major revisions of the same operating system generally also form different 415 platforms. Non-Unix operating systems are a different story; the installation 416 strategies on those systems are so different that the prefix and exec-prefix are 417 meaningless, and set to the empty string. Note that compiled Python bytecode 418 files are platform independent (but not independent from the Python version by 419 which they were compiled!). 420 421 System administrators will know how to configure the :program:`mount` or 422 :program:`automount` programs to share :file:`/usr/local` between platforms 423 while having :file:`/usr/local/plat` be a different filesystem for each 424 platform. 425 426 427.. c:function:: wchar_t* Py_GetProgramFullPath() 428 429 .. index:: 430 single: Py_SetProgramName() 431 single: executable (in module sys) 432 433 Return the full program name of the Python executable; this is computed as a 434 side-effect of deriving the default module search path from the program name 435 (set by :c:func:`Py_SetProgramName` above). The returned string points into 436 static storage; the caller should not modify its value. The value is available 437 to Python code as ``sys.executable``. 438 439 440.. c:function:: wchar_t* Py_GetPath() 441 442 .. index:: 443 triple: module; search; path 444 single: path (in module sys) 445 single: Py_SetPath() 446 447 Return the default module search path; this is computed from the program name 448 (set by :c:func:`Py_SetProgramName` above) and some environment variables. 449 The returned string consists of a series of directory names separated by a 450 platform dependent delimiter character. The delimiter character is ``':'`` 451 on Unix and Mac OS X, ``';'`` on Windows. The returned string points into 452 static storage; the caller should not modify its value. The list 453 :data:`sys.path` is initialized with this value on interpreter startup; it 454 can be (and usually is) modified later to change the search path for loading 455 modules. 456 457 .. XXX should give the exact rules 458 459 460.. c:function:: void Py_SetPath(const wchar_t *) 461 462 .. index:: 463 triple: module; search; path 464 single: path (in module sys) 465 single: Py_GetPath() 466 467 Set the default module search path. If this function is called before 468 :c:func:`Py_Initialize`, then :c:func:`Py_GetPath` won't attempt to compute a 469 default search path but uses the one provided instead. This is useful if 470 Python is embedded by an application that has full knowledge of the location 471 of all modules. The path components should be separated by the platform 472 dependent delimiter character, which is ``':'`` on Unix and Mac OS X, ``';'`` 473 on Windows. 474 475 This also causes :data:`sys.executable` to be set to the program 476 full path (see :c:func:`Py_GetProgramFullPath`) and for :data:`sys.prefix` and 477 :data:`sys.exec_prefix` to be empty. It is up to the caller to modify these 478 if required after calling :c:func:`Py_Initialize`. 479 480 Use :c:func:`Py_DecodeLocale` to decode a bytes string to get a 481 :c:type:`wchar_*` string. 482 483 The path argument is copied internally, so the caller may free it after the 484 call completes. 485 486 .. versionchanged:: 3.8 487 The program full path is now used for :data:`sys.executable`, instead 488 of the program name. 489 490 491.. c:function:: const char* Py_GetVersion() 492 493 Return the version of this Python interpreter. This is a string that looks 494 something like :: 495 496 "3.0a5+ (py3k:63103M, May 12 2008, 00:53:55) \n[GCC 4.2.3]" 497 498 .. index:: single: version (in module sys) 499 500 The first word (up to the first space character) is the current Python version; 501 the first three characters are the major and minor version separated by a 502 period. The returned string points into static storage; the caller should not 503 modify its value. The value is available to Python code as :data:`sys.version`. 504 505 506.. c:function:: const char* Py_GetPlatform() 507 508 .. index:: single: platform (in module sys) 509 510 Return the platform identifier for the current platform. On Unix, this is 511 formed from the "official" name of the operating system, converted to lower 512 case, followed by the major revision number; e.g., for Solaris 2.x, which is 513 also known as SunOS 5.x, the value is ``'sunos5'``. On Mac OS X, it is 514 ``'darwin'``. On Windows, it is ``'win'``. The returned string points into 515 static storage; the caller should not modify its value. The value is available 516 to Python code as ``sys.platform``. 517 518 519.. c:function:: const char* Py_GetCopyright() 520 521 Return the official copyright string for the current Python version, for example 522 523 ``'Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam'`` 524 525 .. index:: single: copyright (in module sys) 526 527 The returned string points into static storage; the caller should not modify its 528 value. The value is available to Python code as ``sys.copyright``. 529 530 531.. c:function:: const char* Py_GetCompiler() 532 533 Return an indication of the compiler used to build the current Python version, 534 in square brackets, for example:: 535 536 "[GCC 2.7.2.2]" 537 538 .. index:: single: version (in module sys) 539 540 The returned string points into static storage; the caller should not modify its 541 value. The value is available to Python code as part of the variable 542 ``sys.version``. 543 544 545.. c:function:: const char* Py_GetBuildInfo() 546 547 Return information about the sequence number and build date and time of the 548 current Python interpreter instance, for example :: 549 550 "#67, Aug 1 1997, 22:34:28" 551 552 .. index:: single: version (in module sys) 553 554 The returned string points into static storage; the caller should not modify its 555 value. The value is available to Python code as part of the variable 556 ``sys.version``. 557 558 559.. c:function:: void PySys_SetArgvEx(int argc, wchar_t **argv, int updatepath) 560 561 .. index:: 562 single: main() 563 single: Py_FatalError() 564 single: argv (in module sys) 565 566 Set :data:`sys.argv` based on *argc* and *argv*. These parameters are 567 similar to those passed to the program's :c:func:`main` function with the 568 difference that the first entry should refer to the script file to be 569 executed rather than the executable hosting the Python interpreter. If there 570 isn't a script that will be run, the first entry in *argv* can be an empty 571 string. If this function fails to initialize :data:`sys.argv`, a fatal 572 condition is signalled using :c:func:`Py_FatalError`. 573 574 If *updatepath* is zero, this is all the function does. If *updatepath* 575 is non-zero, the function also modifies :data:`sys.path` according to the 576 following algorithm: 577 578 - If the name of an existing script is passed in ``argv[0]``, the absolute 579 path of the directory where the script is located is prepended to 580 :data:`sys.path`. 581 - Otherwise (that is, if *argc* is ``0`` or ``argv[0]`` doesn't point 582 to an existing file name), an empty string is prepended to 583 :data:`sys.path`, which is the same as prepending the current working 584 directory (``"."``). 585 586 Use :c:func:`Py_DecodeLocale` to decode a bytes string to get a 587 :c:type:`wchar_*` string. 588 589 .. note:: 590 It is recommended that applications embedding the Python interpreter 591 for purposes other than executing a single script pass ``0`` as *updatepath*, 592 and update :data:`sys.path` themselves if desired. 593 See `CVE-2008-5983 <https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2008-5983>`_. 594 595 On versions before 3.1.3, you can achieve the same effect by manually 596 popping the first :data:`sys.path` element after having called 597 :c:func:`PySys_SetArgv`, for example using:: 598 599 PyRun_SimpleString("import sys; sys.path.pop(0)\n"); 600 601 .. versionadded:: 3.1.3 602 603 .. XXX impl. doesn't seem consistent in allowing ``0``/``NULL`` for the params; 604 check w/ Guido. 605 606 607.. c:function:: void PySys_SetArgv(int argc, wchar_t **argv) 608 609 This function works like :c:func:`PySys_SetArgvEx` with *updatepath* set 610 to ``1`` unless the :program:`python` interpreter was started with the 611 :option:`-I`. 612 613 Use :c:func:`Py_DecodeLocale` to decode a bytes string to get a 614 :c:type:`wchar_*` string. 615 616 .. versionchanged:: 3.4 The *updatepath* value depends on :option:`-I`. 617 618 619.. c:function:: void Py_SetPythonHome(const wchar_t *home) 620 621 Set the default "home" directory, that is, the location of the standard 622 Python libraries. See :envvar:`PYTHONHOME` for the meaning of the 623 argument string. 624 625 The argument should point to a zero-terminated character string in static 626 storage whose contents will not change for the duration of the program's 627 execution. No code in the Python interpreter will change the contents of 628 this storage. 629 630 Use :c:func:`Py_DecodeLocale` to decode a bytes string to get a 631 :c:type:`wchar_*` string. 632 633 634.. c:function:: w_char* Py_GetPythonHome() 635 636 Return the default "home", that is, the value set by a previous call to 637 :c:func:`Py_SetPythonHome`, or the value of the :envvar:`PYTHONHOME` 638 environment variable if it is set. 639 640 641.. _threads: 642 643Thread State and the Global Interpreter Lock 644============================================ 645 646.. index:: 647 single: global interpreter lock 648 single: interpreter lock 649 single: lock, interpreter 650 651The Python interpreter is not fully thread-safe. In order to support 652multi-threaded Python programs, there's a global lock, called the :term:`global 653interpreter lock` or :term:`GIL`, that must be held by the current thread before 654it can safely access Python objects. Without the lock, even the simplest 655operations could cause problems in a multi-threaded program: for example, when 656two threads simultaneously increment the reference count of the same object, the 657reference count could end up being incremented only once instead of twice. 658 659.. index:: single: setswitchinterval() (in module sys) 660 661Therefore, the rule exists that only the thread that has acquired the 662:term:`GIL` may operate on Python objects or call Python/C API functions. 663In order to emulate concurrency of execution, the interpreter regularly 664tries to switch threads (see :func:`sys.setswitchinterval`). The lock is also 665released around potentially blocking I/O operations like reading or writing 666a file, so that other Python threads can run in the meantime. 667 668.. index:: 669 single: PyThreadState 670 single: PyThreadState 671 672The Python interpreter keeps some thread-specific bookkeeping information 673inside a data structure called :c:type:`PyThreadState`. There's also one 674global variable pointing to the current :c:type:`PyThreadState`: it can 675be retrieved using :c:func:`PyThreadState_Get`. 676 677Releasing the GIL from extension code 678------------------------------------- 679 680Most extension code manipulating the :term:`GIL` has the following simple 681structure:: 682 683 Save the thread state in a local variable. 684 Release the global interpreter lock. 685 ... Do some blocking I/O operation ... 686 Reacquire the global interpreter lock. 687 Restore the thread state from the local variable. 688 689This is so common that a pair of macros exists to simplify it:: 690 691 Py_BEGIN_ALLOW_THREADS 692 ... Do some blocking I/O operation ... 693 Py_END_ALLOW_THREADS 694 695.. index:: 696 single: Py_BEGIN_ALLOW_THREADS 697 single: Py_END_ALLOW_THREADS 698 699The :c:macro:`Py_BEGIN_ALLOW_THREADS` macro opens a new block and declares a 700hidden local variable; the :c:macro:`Py_END_ALLOW_THREADS` macro closes the 701block. 702 703The block above expands to the following code:: 704 705 PyThreadState *_save; 706 707 _save = PyEval_SaveThread(); 708 ... Do some blocking I/O operation ... 709 PyEval_RestoreThread(_save); 710 711.. index:: 712 single: PyEval_RestoreThread() 713 single: PyEval_SaveThread() 714 715Here is how these functions work: the global interpreter lock is used to protect the pointer to the 716current thread state. When releasing the lock and saving the thread state, 717the current thread state pointer must be retrieved before the lock is released 718(since another thread could immediately acquire the lock and store its own thread 719state in the global variable). Conversely, when acquiring the lock and restoring 720the thread state, the lock must be acquired before storing the thread state 721pointer. 722 723.. note:: 724 Calling system I/O functions is the most common use case for releasing 725 the GIL, but it can also be useful before calling long-running computations 726 which don't need access to Python objects, such as compression or 727 cryptographic functions operating over memory buffers. For example, the 728 standard :mod:`zlib` and :mod:`hashlib` modules release the GIL when 729 compressing or hashing data. 730 731 732.. _gilstate: 733 734Non-Python created threads 735-------------------------- 736 737When threads are created using the dedicated Python APIs (such as the 738:mod:`threading` module), a thread state is automatically associated to them 739and the code showed above is therefore correct. However, when threads are 740created from C (for example by a third-party library with its own thread 741management), they don't hold the GIL, nor is there a thread state structure 742for them. 743 744If you need to call Python code from these threads (often this will be part 745of a callback API provided by the aforementioned third-party library), 746you must first register these threads with the interpreter by 747creating a thread state data structure, then acquiring the GIL, and finally 748storing their thread state pointer, before you can start using the Python/C 749API. When you are done, you should reset the thread state pointer, release 750the GIL, and finally free the thread state data structure. 751 752The :c:func:`PyGILState_Ensure` and :c:func:`PyGILState_Release` functions do 753all of the above automatically. The typical idiom for calling into Python 754from a C thread is:: 755 756 PyGILState_STATE gstate; 757 gstate = PyGILState_Ensure(); 758 759 /* Perform Python actions here. */ 760 result = CallSomeFunction(); 761 /* evaluate result or handle exception */ 762 763 /* Release the thread. No Python API allowed beyond this point. */ 764 PyGILState_Release(gstate); 765 766Note that the :c:func:`PyGILState_\*` functions assume there is only one global 767interpreter (created automatically by :c:func:`Py_Initialize`). Python 768supports the creation of additional interpreters (using 769:c:func:`Py_NewInterpreter`), but mixing multiple interpreters and the 770:c:func:`PyGILState_\*` API is unsupported. 771 772 773.. _fork-and-threads: 774 775Cautions about fork() 776--------------------- 777 778Another important thing to note about threads is their behaviour in the face 779of the C :c:func:`fork` call. On most systems with :c:func:`fork`, after a 780process forks only the thread that issued the fork will exist. This has a 781concrete impact both on how locks must be handled and on all stored state 782in CPython's runtime. 783 784The fact that only the "current" thread remains 785means any locks held by other threads will never be released. Python solves 786this for :func:`os.fork` by acquiring the locks it uses internally before 787the fork, and releasing them afterwards. In addition, it resets any 788:ref:`lock-objects` in the child. When extending or embedding Python, there 789is no way to inform Python of additional (non-Python) locks that need to be 790acquired before or reset after a fork. OS facilities such as 791:c:func:`pthread_atfork` would need to be used to accomplish the same thing. 792Additionally, when extending or embedding Python, calling :c:func:`fork` 793directly rather than through :func:`os.fork` (and returning to or calling 794into Python) may result in a deadlock by one of Python's internal locks 795being held by a thread that is defunct after the fork. 796:c:func:`PyOS_AfterFork_Child` tries to reset the necessary locks, but is not 797always able to. 798 799The fact that all other threads go away also means that CPython's 800runtime state there must be cleaned up properly, which :func:`os.fork` 801does. This means finalizing all other :c:type:`PyThreadState` objects 802belonging to the current interpreter and all other 803:c:type:`PyInterpreterState` objects. Due to this and the special 804nature of the :ref:`"main" interpreter <sub-interpreter-support>`, 805:c:func:`fork` should only be called in that interpreter's "main" 806thread, where the CPython global runtime was originally initialized. 807The only exception is if :c:func:`exec` will be called immediately 808after. 809 810 811High-level API 812-------------- 813 814These are the most commonly used types and functions when writing C extension 815code, or when embedding the Python interpreter: 816 817.. c:type:: PyInterpreterState 818 819 This data structure represents the state shared by a number of cooperating 820 threads. Threads belonging to the same interpreter share their module 821 administration and a few other internal items. There are no public members in 822 this structure. 823 824 Threads belonging to different interpreters initially share nothing, except 825 process state like available memory, open file descriptors and such. The global 826 interpreter lock is also shared by all threads, regardless of to which 827 interpreter they belong. 828 829 830.. c:type:: PyThreadState 831 832 This data structure represents the state of a single thread. The only public 833 data member is :c:type:`PyInterpreterState \*`:attr:`interp`, which points to 834 this thread's interpreter state. 835 836 837.. c:function:: void PyEval_InitThreads() 838 839 .. index:: 840 single: PyEval_AcquireThread() 841 single: PyEval_ReleaseThread() 842 single: PyEval_SaveThread() 843 single: PyEval_RestoreThread() 844 845 Initialize and acquire the global interpreter lock. It should be called in the 846 main thread before creating a second thread or engaging in any other thread 847 operations such as ``PyEval_ReleaseThread(tstate)``. It is not needed before 848 calling :c:func:`PyEval_SaveThread` or :c:func:`PyEval_RestoreThread`. 849 850 This is a no-op when called for a second time. 851 852 .. versionchanged:: 3.7 853 This function is now called by :c:func:`Py_Initialize()`, so you don't 854 have to call it yourself anymore. 855 856 .. versionchanged:: 3.2 857 This function cannot be called before :c:func:`Py_Initialize()` anymore. 858 859 .. index:: module: _thread 860 861 862.. c:function:: int PyEval_ThreadsInitialized() 863 864 Returns a non-zero value if :c:func:`PyEval_InitThreads` has been called. This 865 function can be called without holding the GIL, and therefore can be used to 866 avoid calls to the locking API when running single-threaded. 867 868 .. versionchanged:: 3.7 869 The :term:`GIL` is now initialized by :c:func:`Py_Initialize()`. 870 871 872.. c:function:: PyThreadState* PyEval_SaveThread() 873 874 Release the global interpreter lock (if it has been created) and reset the 875 thread state to ``NULL``, returning the previous thread state (which is not 876 ``NULL``). If the lock has been created, the current thread must have 877 acquired it. 878 879 880.. c:function:: void PyEval_RestoreThread(PyThreadState *tstate) 881 882 Acquire the global interpreter lock (if it has been created) and set the 883 thread state to *tstate*, which must not be ``NULL``. If the lock has been 884 created, the current thread must not have acquired it, otherwise deadlock 885 ensues. 886 887 .. note:: 888 Calling this function from a thread when the runtime is finalizing 889 will terminate the thread, even if the thread was not created by Python. 890 You can use :c:func:`_Py_IsFinalizing` or :func:`sys.is_finalizing` to 891 check if the interpreter is in process of being finalized before calling 892 this function to avoid unwanted termination. 893 894.. c:function:: PyThreadState* PyThreadState_Get() 895 896 Return the current thread state. The global interpreter lock must be held. 897 When the current thread state is ``NULL``, this issues a fatal error (so that 898 the caller needn't check for ``NULL``). 899 900 901.. c:function:: PyThreadState* PyThreadState_Swap(PyThreadState *tstate) 902 903 Swap the current thread state with the thread state given by the argument 904 *tstate*, which may be ``NULL``. The global interpreter lock must be held 905 and is not released. 906 907 908The following functions use thread-local storage, and are not compatible 909with sub-interpreters: 910 911.. c:function:: PyGILState_STATE PyGILState_Ensure() 912 913 Ensure that the current thread is ready to call the Python C API regardless 914 of the current state of Python, or of the global interpreter lock. This may 915 be called as many times as desired by a thread as long as each call is 916 matched with a call to :c:func:`PyGILState_Release`. In general, other 917 thread-related APIs may be used between :c:func:`PyGILState_Ensure` and 918 :c:func:`PyGILState_Release` calls as long as the thread state is restored to 919 its previous state before the Release(). For example, normal usage of the 920 :c:macro:`Py_BEGIN_ALLOW_THREADS` and :c:macro:`Py_END_ALLOW_THREADS` macros is 921 acceptable. 922 923 The return value is an opaque "handle" to the thread state when 924 :c:func:`PyGILState_Ensure` was called, and must be passed to 925 :c:func:`PyGILState_Release` to ensure Python is left in the same state. Even 926 though recursive calls are allowed, these handles *cannot* be shared - each 927 unique call to :c:func:`PyGILState_Ensure` must save the handle for its call 928 to :c:func:`PyGILState_Release`. 929 930 When the function returns, the current thread will hold the GIL and be able 931 to call arbitrary Python code. Failure is a fatal error. 932 933 .. note:: 934 Calling this function from a thread when the runtime is finalizing 935 will terminate the thread, even if the thread was not created by Python. 936 You can use :c:func:`_Py_IsFinalizing` or :func:`sys.is_finalizing` to 937 check if the interpreter is in process of being finalized before calling 938 this function to avoid unwanted termination. 939 940.. c:function:: void PyGILState_Release(PyGILState_STATE) 941 942 Release any resources previously acquired. After this call, Python's state will 943 be the same as it was prior to the corresponding :c:func:`PyGILState_Ensure` call 944 (but generally this state will be unknown to the caller, hence the use of the 945 GILState API). 946 947 Every call to :c:func:`PyGILState_Ensure` must be matched by a call to 948 :c:func:`PyGILState_Release` on the same thread. 949 950 951.. c:function:: PyThreadState* PyGILState_GetThisThreadState() 952 953 Get the current thread state for this thread. May return ``NULL`` if no 954 GILState API has been used on the current thread. Note that the main thread 955 always has such a thread-state, even if no auto-thread-state call has been 956 made on the main thread. This is mainly a helper/diagnostic function. 957 958 959.. c:function:: int PyGILState_Check() 960 961 Return ``1`` if the current thread is holding the GIL and ``0`` otherwise. 962 This function can be called from any thread at any time. 963 Only if it has had its Python thread state initialized and currently is 964 holding the GIL will it return ``1``. 965 This is mainly a helper/diagnostic function. It can be useful 966 for example in callback contexts or memory allocation functions when 967 knowing that the GIL is locked can allow the caller to perform sensitive 968 actions or otherwise behave differently. 969 970 .. versionadded:: 3.4 971 972 973The following macros are normally used without a trailing semicolon; look for 974example usage in the Python source distribution. 975 976 977.. c:macro:: Py_BEGIN_ALLOW_THREADS 978 979 This macro expands to ``{ PyThreadState *_save; _save = PyEval_SaveThread();``. 980 Note that it contains an opening brace; it must be matched with a following 981 :c:macro:`Py_END_ALLOW_THREADS` macro. See above for further discussion of this 982 macro. 983 984 985.. c:macro:: Py_END_ALLOW_THREADS 986 987 This macro expands to ``PyEval_RestoreThread(_save); }``. Note that it contains 988 a closing brace; it must be matched with an earlier 989 :c:macro:`Py_BEGIN_ALLOW_THREADS` macro. See above for further discussion of 990 this macro. 991 992 993.. c:macro:: Py_BLOCK_THREADS 994 995 This macro expands to ``PyEval_RestoreThread(_save);``: it is equivalent to 996 :c:macro:`Py_END_ALLOW_THREADS` without the closing brace. 997 998 999.. c:macro:: Py_UNBLOCK_THREADS 1000 1001 This macro expands to ``_save = PyEval_SaveThread();``: it is equivalent to 1002 :c:macro:`Py_BEGIN_ALLOW_THREADS` without the opening brace and variable 1003 declaration. 1004 1005 1006Low-level API 1007------------- 1008 1009All of the following functions must be called after :c:func:`Py_Initialize`. 1010 1011.. versionchanged:: 3.7 1012 :c:func:`Py_Initialize()` now initializes the :term:`GIL`. 1013 1014 1015.. c:function:: PyInterpreterState* PyInterpreterState_New() 1016 1017 Create a new interpreter state object. The global interpreter lock need not 1018 be held, but may be held if it is necessary to serialize calls to this 1019 function. 1020 1021 .. audit-event:: cpython.PyInterpreterState_New "" c.PyInterpreterState_New 1022 1023 1024.. c:function:: void PyInterpreterState_Clear(PyInterpreterState *interp) 1025 1026 Reset all information in an interpreter state object. The global interpreter 1027 lock must be held. 1028 1029 .. audit-event:: cpython.PyInterpreterState_Clear "" c.PyInterpreterState_Clear 1030 1031 1032.. c:function:: void PyInterpreterState_Delete(PyInterpreterState *interp) 1033 1034 Destroy an interpreter state object. The global interpreter lock need not be 1035 held. The interpreter state must have been reset with a previous call to 1036 :c:func:`PyInterpreterState_Clear`. 1037 1038 1039.. c:function:: PyThreadState* PyThreadState_New(PyInterpreterState *interp) 1040 1041 Create a new thread state object belonging to the given interpreter object. 1042 The global interpreter lock need not be held, but may be held if it is 1043 necessary to serialize calls to this function. 1044 1045 1046.. c:function:: void PyThreadState_Clear(PyThreadState *tstate) 1047 1048 Reset all information in a thread state object. The global interpreter lock 1049 must be held. 1050 1051 1052.. c:function:: void PyThreadState_Delete(PyThreadState *tstate) 1053 1054 Destroy a thread state object. The global interpreter lock need not be held. 1055 The thread state must have been reset with a previous call to 1056 :c:func:`PyThreadState_Clear`. 1057 1058 1059.. c:function:: PY_INT64_T PyInterpreterState_GetID(PyInterpreterState *interp) 1060 1061 Return the interpreter's unique ID. If there was any error in doing 1062 so then ``-1`` is returned and an error is set. 1063 1064 .. versionadded:: 3.7 1065 1066 1067.. c:function:: PyObject* PyInterpreterState_GetDict(PyInterpreterState *interp) 1068 1069 Return a dictionary in which interpreter-specific data may be stored. 1070 If this function returns ``NULL`` then no exception has been raised and 1071 the caller should assume no interpreter-specific dict is available. 1072 1073 This is not a replacement for :c:func:`PyModule_GetState()`, which 1074 extensions should use to store interpreter-specific state information. 1075 1076 .. versionadded:: 3.8 1077 1078 1079.. c:function:: PyObject* PyThreadState_GetDict() 1080 1081 Return a dictionary in which extensions can store thread-specific state 1082 information. Each extension should use a unique key to use to store state in 1083 the dictionary. It is okay to call this function when no current thread state 1084 is available. If this function returns ``NULL``, no exception has been raised and 1085 the caller should assume no current thread state is available. 1086 1087 1088.. c:function:: int PyThreadState_SetAsyncExc(unsigned long id, PyObject *exc) 1089 1090 Asynchronously raise an exception in a thread. The *id* argument is the thread 1091 id of the target thread; *exc* is the exception object to be raised. This 1092 function does not steal any references to *exc*. To prevent naive misuse, you 1093 must write your own C extension to call this. Must be called with the GIL held. 1094 Returns the number of thread states modified; this is normally one, but will be 1095 zero if the thread id isn't found. If *exc* is :const:`NULL`, the pending 1096 exception (if any) for the thread is cleared. This raises no exceptions. 1097 1098 .. versionchanged:: 3.7 1099 The type of the *id* parameter changed from :c:type:`long` to 1100 :c:type:`unsigned long`. 1101 1102.. c:function:: void PyEval_AcquireThread(PyThreadState *tstate) 1103 1104 Acquire the global interpreter lock and set the current thread state to 1105 *tstate*, which should not be ``NULL``. The lock must have been created earlier. 1106 If this thread already has the lock, deadlock ensues. 1107 1108 .. note:: 1109 Calling this function from a thread when the runtime is finalizing 1110 will terminate the thread, even if the thread was not created by Python. 1111 You can use :c:func:`_Py_IsFinalizing` or :func:`sys.is_finalizing` to 1112 check if the interpreter is in process of being finalized before calling 1113 this function to avoid unwanted termination. 1114 1115 .. versionchanged:: 3.8 1116 Updated to be consistent with :c:func:`PyEval_RestoreThread`, 1117 :c:func:`Py_END_ALLOW_THREADS`, and :c:func:`PyGILState_Ensure`, 1118 and terminate the current thread if called while the interpreter is finalizing. 1119 1120 :c:func:`PyEval_RestoreThread` is a higher-level function which is always 1121 available (even when threads have not been initialized). 1122 1123 1124.. c:function:: void PyEval_ReleaseThread(PyThreadState *tstate) 1125 1126 Reset the current thread state to ``NULL`` and release the global interpreter 1127 lock. The lock must have been created earlier and must be held by the current 1128 thread. The *tstate* argument, which must not be ``NULL``, is only used to check 1129 that it represents the current thread state --- if it isn't, a fatal error is 1130 reported. 1131 1132 :c:func:`PyEval_SaveThread` is a higher-level function which is always 1133 available (even when threads have not been initialized). 1134 1135 1136.. c:function:: void PyEval_AcquireLock() 1137 1138 Acquire the global interpreter lock. The lock must have been created earlier. 1139 If this thread already has the lock, a deadlock ensues. 1140 1141 .. deprecated:: 3.2 1142 This function does not update the current thread state. Please use 1143 :c:func:`PyEval_RestoreThread` or :c:func:`PyEval_AcquireThread` 1144 instead. 1145 1146 .. note:: 1147 Calling this function from a thread when the runtime is finalizing 1148 will terminate the thread, even if the thread was not created by Python. 1149 You can use :c:func:`_Py_IsFinalizing` or :func:`sys.is_finalizing` to 1150 check if the interpreter is in process of being finalized before calling 1151 this function to avoid unwanted termination. 1152 1153 .. versionchanged:: 3.8 1154 Updated to be consistent with :c:func:`PyEval_RestoreThread`, 1155 :c:func:`Py_END_ALLOW_THREADS`, and :c:func:`PyGILState_Ensure`, 1156 and terminate the current thread if called while the interpreter is finalizing. 1157 1158 1159.. c:function:: void PyEval_ReleaseLock() 1160 1161 Release the global interpreter lock. The lock must have been created earlier. 1162 1163 .. deprecated:: 3.2 1164 This function does not update the current thread state. Please use 1165 :c:func:`PyEval_SaveThread` or :c:func:`PyEval_ReleaseThread` 1166 instead. 1167 1168 1169.. _sub-interpreter-support: 1170 1171Sub-interpreter support 1172======================= 1173 1174While in most uses, you will only embed a single Python interpreter, there 1175are cases where you need to create several independent interpreters in the 1176same process and perhaps even in the same thread. Sub-interpreters allow 1177you to do that. 1178 1179The "main" interpreter is the first one created when the runtime initializes. 1180It is usually the only Python interpreter in a process. Unlike sub-interpreters, 1181the main interpreter has unique process-global responsibilities like signal 1182handling. It is also responsible for execution during runtime initialization and 1183is usually the active interpreter during runtime finalization. The 1184:c:func:`PyInterpreterState_Main` function returns a pointer to its state. 1185 1186You can switch between sub-interpreters using the :c:func:`PyThreadState_Swap` 1187function. You can create and destroy them using the following functions: 1188 1189 1190.. c:function:: PyThreadState* Py_NewInterpreter() 1191 1192 .. index:: 1193 module: builtins 1194 module: __main__ 1195 module: sys 1196 single: stdout (in module sys) 1197 single: stderr (in module sys) 1198 single: stdin (in module sys) 1199 1200 Create a new sub-interpreter. This is an (almost) totally separate environment 1201 for the execution of Python code. In particular, the new interpreter has 1202 separate, independent versions of all imported modules, including the 1203 fundamental modules :mod:`builtins`, :mod:`__main__` and :mod:`sys`. The 1204 table of loaded modules (``sys.modules``) and the module search path 1205 (``sys.path``) are also separate. The new environment has no ``sys.argv`` 1206 variable. It has new standard I/O stream file objects ``sys.stdin``, 1207 ``sys.stdout`` and ``sys.stderr`` (however these refer to the same underlying 1208 file descriptors). 1209 1210 The return value points to the first thread state created in the new 1211 sub-interpreter. This thread state is made in the current thread state. 1212 Note that no actual thread is created; see the discussion of thread states 1213 below. If creation of the new interpreter is unsuccessful, ``NULL`` is 1214 returned; no exception is set since the exception state is stored in the 1215 current thread state and there may not be a current thread state. (Like all 1216 other Python/C API functions, the global interpreter lock must be held before 1217 calling this function and is still held when it returns; however, unlike most 1218 other Python/C API functions, there needn't be a current thread state on 1219 entry.) 1220 1221 .. index:: 1222 single: Py_FinalizeEx() 1223 single: Py_Initialize() 1224 1225 Extension modules are shared between (sub-)interpreters as follows: 1226 1227 * For modules using multi-phase initialization, 1228 e.g. :c:func:`PyModule_FromDefAndSpec`, a separate module object is 1229 created and initialized for each interpreter. 1230 Only C-level static and global variables are shared between these 1231 module objects. 1232 1233 * For modules using single-phase initialization, 1234 e.g. :c:func:`PyModule_Create`, the first time a particular extension 1235 is imported, it is initialized normally, and a (shallow) copy of its 1236 module's dictionary is squirreled away. 1237 When the same extension is imported by another (sub-)interpreter, a new 1238 module is initialized and filled with the contents of this copy; the 1239 extension's ``init`` function is not called. 1240 Objects in the module's dictionary thus end up shared across 1241 (sub-)interpreters, which might cause unwanted behavior (see 1242 `Bugs and caveats`_ below). 1243 1244 Note that this is different from what happens when an extension is 1245 imported after the interpreter has been completely re-initialized by 1246 calling :c:func:`Py_FinalizeEx` and :c:func:`Py_Initialize`; in that 1247 case, the extension's ``initmodule`` function *is* called again. 1248 As with multi-phase initialization, this means that only C-level static 1249 and global variables are shared between these modules. 1250 1251 .. index:: single: close() (in module os) 1252 1253 1254.. c:function:: void Py_EndInterpreter(PyThreadState *tstate) 1255 1256 .. index:: single: Py_FinalizeEx() 1257 1258 Destroy the (sub-)interpreter represented by the given thread state. The given 1259 thread state must be the current thread state. See the discussion of thread 1260 states below. When the call returns, the current thread state is ``NULL``. All 1261 thread states associated with this interpreter are destroyed. (The global 1262 interpreter lock must be held before calling this function and is still held 1263 when it returns.) :c:func:`Py_FinalizeEx` will destroy all sub-interpreters that 1264 haven't been explicitly destroyed at that point. 1265 1266 1267Bugs and caveats 1268---------------- 1269 1270Because sub-interpreters (and the main interpreter) are part of the same 1271process, the insulation between them isn't perfect --- for example, using 1272low-level file operations like :func:`os.close` they can 1273(accidentally or maliciously) affect each other's open files. Because of the 1274way extensions are shared between (sub-)interpreters, some extensions may not 1275work properly; this is especially likely when using single-phase initialization 1276or (static) global variables. 1277It is possible to insert objects created in one sub-interpreter into 1278a namespace of another (sub-)interpreter; this should be avoided if possible. 1279 1280Special care should be taken to avoid sharing user-defined functions, 1281methods, instances or classes between sub-interpreters, since import 1282operations executed by such objects may affect the wrong (sub-)interpreter's 1283dictionary of loaded modules. It is equally important to avoid sharing 1284objects from which the above are reachable. 1285 1286Also note that combining this functionality with :c:func:`PyGILState_\*` APIs 1287is delicate, because these APIs assume a bijection between Python thread states 1288and OS-level threads, an assumption broken by the presence of sub-interpreters. 1289It is highly recommended that you don't switch sub-interpreters between a pair 1290of matching :c:func:`PyGILState_Ensure` and :c:func:`PyGILState_Release` calls. 1291Furthermore, extensions (such as :mod:`ctypes`) using these APIs to allow calling 1292of Python code from non-Python created threads will probably be broken when using 1293sub-interpreters. 1294 1295 1296Asynchronous Notifications 1297========================== 1298 1299A mechanism is provided to make asynchronous notifications to the main 1300interpreter thread. These notifications take the form of a function 1301pointer and a void pointer argument. 1302 1303 1304.. c:function:: int Py_AddPendingCall(int (*func)(void *), void *arg) 1305 1306 .. index:: single: Py_AddPendingCall() 1307 1308 Schedule a function to be called from the main interpreter thread. On 1309 success, ``0`` is returned and *func* is queued for being called in the 1310 main thread. On failure, ``-1`` is returned without setting any exception. 1311 1312 When successfully queued, *func* will be *eventually* called from the 1313 main interpreter thread with the argument *arg*. It will be called 1314 asynchronously with respect to normally running Python code, but with 1315 both these conditions met: 1316 1317 * on a :term:`bytecode` boundary; 1318 * with the main thread holding the :term:`global interpreter lock` 1319 (*func* can therefore use the full C API). 1320 1321 *func* must return ``0`` on success, or ``-1`` on failure with an exception 1322 set. *func* won't be interrupted to perform another asynchronous 1323 notification recursively, but it can still be interrupted to switch 1324 threads if the global interpreter lock is released. 1325 1326 This function doesn't need a current thread state to run, and it doesn't 1327 need the global interpreter lock. 1328 1329 .. warning:: 1330 This is a low-level function, only useful for very special cases. 1331 There is no guarantee that *func* will be called as quick as 1332 possible. If the main thread is busy executing a system call, 1333 *func* won't be called before the system call returns. This 1334 function is generally **not** suitable for calling Python code from 1335 arbitrary C threads. Instead, use the :ref:`PyGILState API<gilstate>`. 1336 1337 .. versionadded:: 3.1 1338 1339.. _profiling: 1340 1341Profiling and Tracing 1342===================== 1343 1344.. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> 1345 1346 1347The Python interpreter provides some low-level support for attaching profiling 1348and execution tracing facilities. These are used for profiling, debugging, and 1349coverage analysis tools. 1350 1351This C interface allows the profiling or tracing code to avoid the overhead of 1352calling through Python-level callable objects, making a direct C function call 1353instead. The essential attributes of the facility have not changed; the 1354interface allows trace functions to be installed per-thread, and the basic 1355events reported to the trace function are the same as had been reported to the 1356Python-level trace functions in previous versions. 1357 1358 1359.. c:type:: int (*Py_tracefunc)(PyObject *obj, PyFrameObject *frame, int what, PyObject *arg) 1360 1361 The type of the trace function registered using :c:func:`PyEval_SetProfile` and 1362 :c:func:`PyEval_SetTrace`. The first parameter is the object passed to the 1363 registration function as *obj*, *frame* is the frame object to which the event 1364 pertains, *what* is one of the constants :const:`PyTrace_CALL`, 1365 :const:`PyTrace_EXCEPTION`, :const:`PyTrace_LINE`, :const:`PyTrace_RETURN`, 1366 :const:`PyTrace_C_CALL`, :const:`PyTrace_C_EXCEPTION`, :const:`PyTrace_C_RETURN`, 1367 or :const:`PyTrace_OPCODE`, and *arg* depends on the value of *what*: 1368 1369 +------------------------------+----------------------------------------+ 1370 | Value of *what* | Meaning of *arg* | 1371 +==============================+========================================+ 1372 | :const:`PyTrace_CALL` | Always :c:data:`Py_None`. | 1373 +------------------------------+----------------------------------------+ 1374 | :const:`PyTrace_EXCEPTION` | Exception information as returned by | 1375 | | :func:`sys.exc_info`. | 1376 +------------------------------+----------------------------------------+ 1377 | :const:`PyTrace_LINE` | Always :c:data:`Py_None`. | 1378 +------------------------------+----------------------------------------+ 1379 | :const:`PyTrace_RETURN` | Value being returned to the caller, | 1380 | | or ``NULL`` if caused by an exception. | 1381 +------------------------------+----------------------------------------+ 1382 | :const:`PyTrace_C_CALL` | Function object being called. | 1383 +------------------------------+----------------------------------------+ 1384 | :const:`PyTrace_C_EXCEPTION` | Function object being called. | 1385 +------------------------------+----------------------------------------+ 1386 | :const:`PyTrace_C_RETURN` | Function object being called. | 1387 +------------------------------+----------------------------------------+ 1388 | :const:`PyTrace_OPCODE` | Always :c:data:`Py_None`. | 1389 +------------------------------+----------------------------------------+ 1390 1391.. c:var:: int PyTrace_CALL 1392 1393 The value of the *what* parameter to a :c:type:`Py_tracefunc` function when a new 1394 call to a function or method is being reported, or a new entry into a generator. 1395 Note that the creation of the iterator for a generator function is not reported 1396 as there is no control transfer to the Python bytecode in the corresponding 1397 frame. 1398 1399 1400.. c:var:: int PyTrace_EXCEPTION 1401 1402 The value of the *what* parameter to a :c:type:`Py_tracefunc` function when an 1403 exception has been raised. The callback function is called with this value for 1404 *what* when after any bytecode is processed after which the exception becomes 1405 set within the frame being executed. The effect of this is that as exception 1406 propagation causes the Python stack to unwind, the callback is called upon 1407 return to each frame as the exception propagates. Only trace functions receives 1408 these events; they are not needed by the profiler. 1409 1410 1411.. c:var:: int PyTrace_LINE 1412 1413 The value passed as the *what* parameter to a :c:type:`Py_tracefunc` function 1414 (but not a profiling function) when a line-number event is being reported. 1415 It may be disabled for a frame by setting :attr:`f_trace_lines` to *0* on that frame. 1416 1417 1418.. c:var:: int PyTrace_RETURN 1419 1420 The value for the *what* parameter to :c:type:`Py_tracefunc` functions when a 1421 call is about to return. 1422 1423 1424.. c:var:: int PyTrace_C_CALL 1425 1426 The value for the *what* parameter to :c:type:`Py_tracefunc` functions when a C 1427 function is about to be called. 1428 1429 1430.. c:var:: int PyTrace_C_EXCEPTION 1431 1432 The value for the *what* parameter to :c:type:`Py_tracefunc` functions when a C 1433 function has raised an exception. 1434 1435 1436.. c:var:: int PyTrace_C_RETURN 1437 1438 The value for the *what* parameter to :c:type:`Py_tracefunc` functions when a C 1439 function has returned. 1440 1441 1442.. c:var:: int PyTrace_OPCODE 1443 1444 The value for the *what* parameter to :c:type:`Py_tracefunc` functions (but not 1445 profiling functions) when a new opcode is about to be executed. This event is 1446 not emitted by default: it must be explicitly requested by setting 1447 :attr:`f_trace_opcodes` to *1* on the frame. 1448 1449 1450.. c:function:: void PyEval_SetProfile(Py_tracefunc func, PyObject *obj) 1451 1452 Set the profiler function to *func*. The *obj* parameter is passed to the 1453 function as its first parameter, and may be any Python object, or ``NULL``. If 1454 the profile function needs to maintain state, using a different value for *obj* 1455 for each thread provides a convenient and thread-safe place to store it. The 1456 profile function is called for all monitored events except :const:`PyTrace_LINE` 1457 :const:`PyTrace_OPCODE` and :const:`PyTrace_EXCEPTION`. 1458 1459 1460.. c:function:: void PyEval_SetTrace(Py_tracefunc func, PyObject *obj) 1461 1462 Set the tracing function to *func*. This is similar to 1463 :c:func:`PyEval_SetProfile`, except the tracing function does receive line-number 1464 events and per-opcode events, but does not receive any event related to C function 1465 objects being called. Any trace function registered using :c:func:`PyEval_SetTrace` 1466 will not receive :const:`PyTrace_C_CALL`, :const:`PyTrace_C_EXCEPTION` or 1467 :const:`PyTrace_C_RETURN` as a value for the *what* parameter. 1468 1469.. _advanced-debugging: 1470 1471Advanced Debugger Support 1472========================= 1473 1474.. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> 1475 1476 1477These functions are only intended to be used by advanced debugging tools. 1478 1479 1480.. c:function:: PyInterpreterState* PyInterpreterState_Head() 1481 1482 Return the interpreter state object at the head of the list of all such objects. 1483 1484 1485.. c:function:: PyInterpreterState* PyInterpreterState_Main() 1486 1487 Return the main interpreter state object. 1488 1489 1490.. c:function:: PyInterpreterState* PyInterpreterState_Next(PyInterpreterState *interp) 1491 1492 Return the next interpreter state object after *interp* from the list of all 1493 such objects. 1494 1495 1496.. c:function:: PyThreadState * PyInterpreterState_ThreadHead(PyInterpreterState *interp) 1497 1498 Return the pointer to the first :c:type:`PyThreadState` object in the list of 1499 threads associated with the interpreter *interp*. 1500 1501 1502.. c:function:: PyThreadState* PyThreadState_Next(PyThreadState *tstate) 1503 1504 Return the next thread state object after *tstate* from the list of all such 1505 objects belonging to the same :c:type:`PyInterpreterState` object. 1506 1507 1508.. _thread-local-storage: 1509 1510Thread Local Storage Support 1511============================ 1512 1513.. sectionauthor:: Masayuki Yamamoto <ma3yuki.8mamo10@gmail.com> 1514 1515The Python interpreter provides low-level support for thread-local storage 1516(TLS) which wraps the underlying native TLS implementation to support the 1517Python-level thread local storage API (:class:`threading.local`). The 1518CPython C level APIs are similar to those offered by pthreads and Windows: 1519use a thread key and functions to associate a :c:type:`void\*` value per 1520thread. 1521 1522The GIL does *not* need to be held when calling these functions; they supply 1523their own locking. 1524 1525Note that :file:`Python.h` does not include the declaration of the TLS APIs, 1526you need to include :file:`pythread.h` to use thread-local storage. 1527 1528.. note:: 1529 None of these API functions handle memory management on behalf of the 1530 :c:type:`void\*` values. You need to allocate and deallocate them yourself. 1531 If the :c:type:`void\*` values happen to be :c:type:`PyObject\*`, these 1532 functions don't do refcount operations on them either. 1533 1534.. _thread-specific-storage-api: 1535 1536Thread Specific Storage (TSS) API 1537--------------------------------- 1538 1539TSS API is introduced to supersede the use of the existing TLS API within the 1540CPython interpreter. This API uses a new type :c:type:`Py_tss_t` instead of 1541:c:type:`int` to represent thread keys. 1542 1543.. versionadded:: 3.7 1544 1545.. seealso:: "A New C-API for Thread-Local Storage in CPython" (:pep:`539`) 1546 1547 1548.. c:type:: Py_tss_t 1549 1550 This data structure represents the state of a thread key, the definition of 1551 which may depend on the underlying TLS implementation, and it has an 1552 internal field representing the key's initialization state. There are no 1553 public members in this structure. 1554 1555 When :ref:`Py_LIMITED_API <stable>` is not defined, static allocation of 1556 this type by :c:macro:`Py_tss_NEEDS_INIT` is allowed. 1557 1558 1559.. c:macro:: Py_tss_NEEDS_INIT 1560 1561 This macro expands to the initializer for :c:type:`Py_tss_t` variables. 1562 Note that this macro won't be defined with :ref:`Py_LIMITED_API <stable>`. 1563 1564 1565Dynamic Allocation 1566~~~~~~~~~~~~~~~~~~ 1567 1568Dynamic allocation of the :c:type:`Py_tss_t`, required in extension modules 1569built with :ref:`Py_LIMITED_API <stable>`, where static allocation of this type 1570is not possible due to its implementation being opaque at build time. 1571 1572 1573.. c:function:: Py_tss_t* PyThread_tss_alloc() 1574 1575 Return a value which is the same state as a value initialized with 1576 :c:macro:`Py_tss_NEEDS_INIT`, or ``NULL`` in the case of dynamic allocation 1577 failure. 1578 1579 1580.. c:function:: void PyThread_tss_free(Py_tss_t *key) 1581 1582 Free the given *key* allocated by :c:func:`PyThread_tss_alloc`, after 1583 first calling :c:func:`PyThread_tss_delete` to ensure any associated 1584 thread locals have been unassigned. This is a no-op if the *key* 1585 argument is `NULL`. 1586 1587 .. note:: 1588 A freed key becomes a dangling pointer, you should reset the key to 1589 `NULL`. 1590 1591 1592Methods 1593~~~~~~~ 1594 1595The parameter *key* of these functions must not be ``NULL``. Moreover, the 1596behaviors of :c:func:`PyThread_tss_set` and :c:func:`PyThread_tss_get` are 1597undefined if the given :c:type:`Py_tss_t` has not been initialized by 1598:c:func:`PyThread_tss_create`. 1599 1600 1601.. c:function:: int PyThread_tss_is_created(Py_tss_t *key) 1602 1603 Return a non-zero value if the given :c:type:`Py_tss_t` has been initialized 1604 by :c:func:`PyThread_tss_create`. 1605 1606 1607.. c:function:: int PyThread_tss_create(Py_tss_t *key) 1608 1609 Return a zero value on successful initialization of a TSS key. The behavior 1610 is undefined if the value pointed to by the *key* argument is not 1611 initialized by :c:macro:`Py_tss_NEEDS_INIT`. This function can be called 1612 repeatedly on the same key -- calling it on an already initialized key is a 1613 no-op and immediately returns success. 1614 1615 1616.. c:function:: void PyThread_tss_delete(Py_tss_t *key) 1617 1618 Destroy a TSS key to forget the values associated with the key across all 1619 threads, and change the key's initialization state to uninitialized. A 1620 destroyed key is able to be initialized again by 1621 :c:func:`PyThread_tss_create`. This function can be called repeatedly on 1622 the same key -- calling it on an already destroyed key is a no-op. 1623 1624 1625.. c:function:: int PyThread_tss_set(Py_tss_t *key, void *value) 1626 1627 Return a zero value to indicate successfully associating a :c:type:`void\*` 1628 value with a TSS key in the current thread. Each thread has a distinct 1629 mapping of the key to a :c:type:`void\*` value. 1630 1631 1632.. c:function:: void* PyThread_tss_get(Py_tss_t *key) 1633 1634 Return the :c:type:`void\*` value associated with a TSS key in the current 1635 thread. This returns ``NULL`` if no value is associated with the key in the 1636 current thread. 1637 1638 1639.. _thread-local-storage-api: 1640 1641Thread Local Storage (TLS) API 1642------------------------------ 1643 1644.. deprecated:: 3.7 1645 This API is superseded by 1646 :ref:`Thread Specific Storage (TSS) API <thread-specific-storage-api>`. 1647 1648.. note:: 1649 This version of the API does not support platforms where the native TLS key 1650 is defined in a way that cannot be safely cast to ``int``. On such platforms, 1651 :c:func:`PyThread_create_key` will return immediately with a failure status, 1652 and the other TLS functions will all be no-ops on such platforms. 1653 1654Due to the compatibility problem noted above, this version of the API should not 1655be used in new code. 1656 1657.. c:function:: int PyThread_create_key() 1658.. c:function:: void PyThread_delete_key(int key) 1659.. c:function:: int PyThread_set_key_value(int key, void *value) 1660.. c:function:: void* PyThread_get_key_value(int key) 1661.. c:function:: void PyThread_delete_key_value(int key) 1662.. c:function:: void PyThread_ReInitTLS() 1663 1664