Lines Matching +full:publish +full:- +full:pypi
1 .. _logging-cookbook:
7 :Author: Vinay Sajip <vinay_sajip at red-dove dot com>
11 :ref:`cookbook-ref-links`.
16 ---------------------------------
39 formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
78 .. code-block:: none
80 2005-03-23 23:47:11,663 - spam_application - INFO -
82 2005-03-23 23:47:11,665 - spam_application.auxiliary.Auxiliary - INFO -
84 2005-03-23 23:47:11,665 - spam_application - INFO -
86 2005-03-23 23:47:11,668 - spam_application - INFO -
88 2005-03-23 23:47:11,668 - spam_application.auxiliary.Auxiliary - INFO -
90 2005-03-23 23:47:11,669 - spam_application.auxiliary.Auxiliary - INFO -
92 2005-03-23 23:47:11,670 - spam_application - INFO -
94 2005-03-23 23:47:11,671 - spam_application - INFO -
96 2005-03-23 23:47:11,672 - spam_application.auxiliary - INFO -
98 2005-03-23 23:47:11,673 - spam_application - INFO -
102 -----------------------------
135 .. code-block:: none
137 0 Thread-1 Hi from myfunc
139 505 Thread-1 Hi from myfunc
141 1007 Thread-1 Hi from myfunc
143 1508 Thread-1 Hi from myfunc
144 2010 Thread-1 Hi from myfunc
146 2512 Thread-1 Hi from myfunc
148 3013 Thread-1 Hi from myfunc
149 3515 Thread-1 Hi from myfunc
151 4017 Thread-1 Hi from myfunc
153 4518 Thread-1 Hi from myfunc
159 --------------------------------
167 previous simple module-based configuration example::
180 formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
197 The ability to create new handlers with higher- or lower-severity filters can be
205 .. _multiple-destinations:
208 --------------------------------
218 # set up logging to file - see previous section for more details
220 format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
221 datefmt='%m-%d %H:%M',
228 formatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s')
250 .. code-block:: none
259 .. code-block:: none
261 10-22 22:19 root INFO Jackdaws love my big sphinx of quartz.
262 10-22 22:19 myapp.area1 DEBUG Quick zephyrs blow, vexing daft Jim.
263 10-22 22:19 myapp.area1 INFO How quickly daft jumping zebras vex.
264 10-22 22:19 myapp.area2 WARNING Jail zesty vixen who grabbed pay from quack.
265 10-22 22:19 myapp.area2 ERROR The five boxing wizards jump quickly.
275 choose a different directory name for the log - just ensure that the directory exists
279 .. _custom-level-handling:
282 -------------------------
295 .. code-block:: json
302 "format": "%(levelname)-8s - %(message)s"
341 .. code-block:: json
354 .. code-block:: json
369 .. code-block:: python
383 so its module will be ``__main__`` - hence the ``__main__.filter_maker`` in the
389 .. code-block:: python
401 "format": "%(levelname)-8s - %(message)s"
459 .. code-block:: shell
465 .. code-block:: shell
471 DEBUG - A DEBUG message
472 INFO - An INFO message
473 WARNING - A WARNING message
474 ERROR - An ERROR message
475 CRITICAL - A CRITICAL message
479 ERROR - An ERROR message
480 CRITICAL - A CRITICAL message
484 INFO - An INFO message
485 WARNING - A WARNING message
489 ----------------------------
523 properly preceded with the binary-encoded length, as the new logging
544 .. _blocking-handlers:
547 --------------------------------
558 performing mail or network infrastructure). But almost any network-based
563 One solution is to use a two-part approach. For the first part, attach only a
565 performance-critical threads. They simply write to their queue, which can be
569 in your code. If you are a library developer who has performance-critical
584 resource-friendly than, say, having threaded versions of the existing handler
589 que = queue.Queue(-1) # no limit on size
607 .. code-block:: none
629 .. _network-logging:
632 -----------------------------------------------------
681 Handle multiple requests - each expected to be a 4-byte length,
692 chunk = chunk + self.connection.recv(slen - len(chunk))
709 # is normally called AFTER logger-level filtering. If you want
716 Simple TCP socket-based logging receiver suitable for testing.
742 format='%(relativeCreated)5d %(name)-15s %(levelname)-8s %(message)s')
753 .. code-block:: none
772 .. _socket-listener-gist: https://gist.github.com/vsajip/4b227eeec43817465ca835ca66f75e2b
775 process-management tool such as `Supervisor <http://supervisord.org/>`_.
776 `Here is a Gist <socket-listener-gist_>`__
777 which provides the bare-bones files to run the above functionality using
780 +-------------------------+----------------------------------------------------+
785 +-------------------------+----------------------------------------------------+
787 | | entries for the listener and a multi-process web |
789 +-------------------------+----------------------------------------------------+
792 +-------------------------+----------------------------------------------------+
795 +-------------------------+----------------------------------------------------+
798 +-------------------------+----------------------------------------------------+
800 +-------------------------+----------------------------------------------------+
802 +-------------------------+----------------------------------------------------+
807 without conflicting with one another --- they all go through the socket listener.
811 #. Download `the Gist <socket-listener-gist_>`__
817 This creates a :file:`run` subdirectory to contain Supervisor-related and
830 worker processes in a non-deterministic way.
833 ``venv/bin/supervisorctl -c supervisor.conf shutdown``.
838 .. _context-info:
841 ----------------------------------------------------
845 networked application, it may be desirable to log client-specific information
849 :class:`Logger` instances on a per-connection basis, this is not a good idea
869 :class:`Logger` instance and a dict-like object which contains your contextual
889 an 'extra' key in the keyword argument whose value is the dict-like object
893 The advantage of using 'extra' is that the values in the dict-like object are
896 the keys of the dict-like object. If you need a different method, e.g. if you
903 This example adapter expects the passed in dict-like object to have a
920 You don't need to pass an actual dict to a :class:`LoggerAdapter` - you could
926 .. _filters-contextual:
931 You can also add contextual information to log output using a user-defined
939 add, say, information from the request - say, the remote IP address and remote
940 user's username - to the ``LogRecord``, using the attribute names 'ip' and
968 … format='%(asctime)-15s %(name)-5s %(levelname)-8s IP: %(ip)-15s User: %(user)-8s %(message)s')
984 .. code-block:: none
986 2010-09-06 22:38:15,292 a.b.c DEBUG IP: 123.231.231.123 User: fred A debug message
987 …2010-09-06 22:38:15,300 a.b.c INFO IP: 192.168.0.1 User: sheila An info message with som…
988 …2010-09-06 22:38:15,300 d.e.f CRITICAL IP: 127.0.0.1 User: sheila A message at CRITICAL le…
989 …2010-09-06 22:38:15,300 d.e.f ERROR IP: 127.0.0.1 User: jim A message at ERROR level…
990 …2010-09-06 22:38:15,300 d.e.f DEBUG IP: 127.0.0.1 User: sheila A message at DEBUG level…
991 …2010-09-06 22:38:15,300 d.e.f ERROR IP: 123.231.231.123 User: fred A message at ERROR level…
992 …2010-09-06 22:38:15,300 d.e.f CRITICAL IP: 192.168.0.1 User: jim A message at CRITICAL le…
993 …2010-09-06 22:38:15,300 d.e.f CRITICAL IP: 127.0.0.1 User: sheila A message at CRITICAL le…
994 …2010-09-06 22:38:15,300 d.e.f DEBUG IP: 192.168.0.1 User: jim A message at DEBUG level…
995 …2010-09-06 22:38:15,301 d.e.f ERROR IP: 127.0.0.1 User: sheila A message at ERROR level…
996 …2010-09-06 22:38:15,301 d.e.f DEBUG IP: 123.231.231.123 User: fred A message at DEBUG level…
997 …2010-09-06 22:38:15,301 d.e.f INFO IP: 123.231.231.123 User: fred A message at INFO level …
1000 ----------------------
1002 Since Python 3.7, the :mod:`contextvars` module has provided context-local storage
1004 of storage may thus be generally preferable to thread-locals. The following example
1005 shows how, in a multi-threaded environment, logs can populated with contextual
1017 .. code-block:: python
1032 ``Request`` and ``WebApp``. These simulate how real threaded web applications work -
1035 .. code-block:: python
1060 # A dummy set of requests which will be used in the simulation - we'll just pick
1077 …formatter = logging.Formatter('%(threadName)-11s %(appName)s %(name)-9s %(user)-6s %(ip)s %(method…
1087 A filter which injects context-specific information into logs and ensures
1104 webapp-specific log.
1138 aa('--count', '-c', type=int, default=100, help='How many requests to simulate')
1177 logged to :file:`app.log`. Each webapp-specific log will contain only log entries for
1182 .. code-block:: shell
1184 ~/logging-contextual-webapp$ python main.py
1187 ~/logging-contextual-webapp$ wc -l *.log
1192 ~/logging-contextual-webapp$ head -3 app1.log
1193 Thread-3 (process_request) app1 __main__ jim 192.168.3.21 POST Request processing started
1194 Thread-3 (process_request) app1 webapplib jim 192.168.3.21 POST Hello from webapplib!
1195 Thread-5 (process_request) app1 __main__ jim 192.168.3.21 POST Request processing started
1196 ~/logging-contextual-webapp$ head -3 app2.log
1197 Thread-1 (process_request) app2 __main__ sheila 192.168.2.21 GET Request processing started
1198 Thread-1 (process_request) app2 webapplib sheila 192.168.2.21 GET Hello from webapplib!
1199 Thread-2 (process_request) app2 __main__ jim 192.168.2.20 GET Request processing started
1200 ~/logging-contextual-webapp$ head app.log
1201 Thread-1 (process_request) app2 __main__ sheila 192.168.2.21 GET Request processing started
1202 Thread-1 (process_request) app2 webapplib sheila 192.168.2.21 GET Hello from webapplib!
1203 Thread-2 (process_request) app2 __main__ jim 192.168.2.20 GET Request processing started
1204 Thread-3 (process_request) app1 __main__ jim 192.168.3.21 POST Request processing started
1205 Thread-2 (process_request) app2 webapplib jim 192.168.2.20 GET Hello from webapplib!
1206 Thread-3 (process_request) app1 webapplib jim 192.168.3.21 POST Hello from webapplib!
1207 Thread-4 (process_request) app2 __main__ fred 192.168.2.22 GET Request processing started
1208 Thread-5 (process_request) app1 __main__ jim 192.168.3.21 POST Request processing started
1209 Thread-4 (process_request) app2 webapplib fred 192.168.2.22 GET Hello from webapplib!
1210 Thread-6 (process_request) app1 __main__ jim 192.168.3.21 POST Request processing started
1211 ~/logging-contextual-webapp$ grep app1 app1.log | wc -l
1213 ~/logging-contextual-webapp$ grep app2 app2.log | wc -l
1215 ~/logging-contextual-webapp$ grep app1 app.log | wc -l
1217 ~/logging-contextual-webapp$ grep app2 app.log | wc -l
1222 --------------------------------------------
1227 a new :class:`~LogRecord` instead of modifying it in-place, as shown in the following script::
1241 formatter = logging.Formatter('%(message)s from %(user)-8s')
1248 .. _multiple-processes:
1251 ------------------------------------------------
1253 Although logging is thread-safe, and logging to a single file from multiple
1262 :ref:`This section <network-logging>` documents this approach in more detail and
1277 all logging events to one of the processes in your multi-process application.
1282 thread rather than a separate listener process -- the implementation would be
1311 f = logging.Formatter('%(asctime)s %(processName)-10s %(name)s %(levelname)-8s %(message)s')
1315 # This is the listener process top-level loop: wait for logging events
1326 logger.handle(record) # No level or filter logic applied - just do it!
1355 # This is the worker process top-level loop, which just logs ten events with
1374 queue = multiprocessing.Queue(-1)
1433 … 'format': '%(asctime)s %(name)-15s %(levelname)-8s %(processName)-10s %(message)s'
1449 'filename': 'mplog-foo.log',
1455 'filename': 'mplog-errors.log',
1488 - e.g. the ``foo`` logger has a special handler which stores all events in the
1489 ``foo`` subsystem in a file ``mplog-foo.log``. This will be used by the logging
1500 .. code-block:: python
1502 queue = multiprocessing.Queue(-1)
1506 .. code-block:: python
1508 queue = multiprocessing.Manager().Queue(-1) # also works with the examples above
1531 <https://uwsgi-docs.readthedocs.io/en/latest/>`_ (or similar), multiple worker
1533 file-based handlers directly in your web application. Instead, use a
1535 process. This can be set up using a process management tool such as Supervisor - see
1540 -------------------
1580 .. code-block:: none
1597 .. _format-styles:
1600 ------------------------------------
1603 formatting messages with variable content was to use the %-formatting
1618 .. code-block:: pycon
1630 2010-10-28 15:11:55,341 foo.bar DEBUG This is a DEBUG message
1632 2010-10-28 15:12:11,526 foo.bar CRITICAL This is a CRITICAL message
1637 2010-10-28 15:13:06,924 foo.bar DEBUG This is a DEBUG message
1639 2010-10-28 15:13:11,494 foo.bar CRITICAL This is a CRITICAL message
1644 That can still use %-formatting, as shown here::
1647 2010-10-28 15:19:29,833 foo.bar ERROR This is another, ERROR, message
1658 uses %-formatting to merge the format string and the variable arguments.
1660 all logging calls which are out there in existing code will be using %-format
1663 There is, however, a way that you can use {}- and $- formatting to construct
1687 Either of these can be used in place of a format string, to allow {}- or
1688 $-formatting to be used to build the actual "message" part which appears in the
1692 underscore --- not to be confused with _, the single underscore used as a
1699 .. code-block:: pycon
1766 .. _custom-logrecord:
1769 -------------------------
1812 top-level logger, but this would not be invoked if an application developer
1813 attached a handler to a lower-level library logger --- so output from that
1839 However, it should be borne in mind that each link in the chain adds run-time
1844 .. _zeromq-handlers:
1846 Subclassing QueueHandler - a ZeroMQ example
1847 -------------------------------------------
1850 of queues, for example a ZeroMQ 'publish' socket. In the example below,the
1885 Subclassing QueueListener - a ZeroMQ example
1886 --------------------------------------------
1915 :ref:`A basic logging tutorial <logging-basic-tutorial>`
1917 :ref:`A more advanced logging tutorial <logging-advanced-tutorial>`
1920 An example dictionary-based configuration
1921 -----------------------------------------
1923 Below is an example of a logging configuration dictionary - it's taken from
1924 …he Django project <https://docs.djangoproject.com/en/stable/topics/logging/#configuring-logging>`_.
1980 section <https://docs.djangoproject.com/en/stable/topics/logging/#configuring-logging>`_
1983 .. _cookbook-rotator-namer:
1986 --------------------------------------------------------------
2021 .. code-block:: shell-session
2027 2023-01-20 02:28:17,767 Message no. 996
2028 2023-01-20 02:28:17,767 Message no. 997
2029 2023-01-20 02:28:17,767 Message no. 998
2032 ----------------------------------------
2049 Here's the script - the docstrings and the comments hopefully explain how it
2098 # would appear - hence the "if posix" clause.
2129 # would appear - hence the "if posix" clause.
2185 … 'format': '%(asctime)s %(name)-15s %(levelname)-8s %(processName)-10s %(message)s'
2189 'format': '%(name)-15s %(levelname)-8s %(processName)-10s %(message)s'
2206 'filename': 'mplog-foo.log',
2212 'filename': 'mplog-errors.log',
2261 -----------------------------------------------------
2265 following structure: an optional pure-ASCII component, followed by a UTF-8 Byte
2266 Order Mark (BOM), followed by Unicode encoded using UTF-8. (See the
2267 :rfc:`relevant section of the specification <5424#section-6>`.)
2272 beginning of the message and hence not allowing any pure-ASCII component to
2277 want to produce :rfc:`5424`-compliant messages which include a BOM, an optional
2278 pure-ASCII sequence before it and arbitrary Unicode after it, encoded using
2279 UTF-8, then you need to do the following:
2287 The Unicode code point U+FEFF, when encoded using UTF-8, will be
2288 encoded as a UTF-8 BOM -- the byte-string ``b'\xef\xbb\xbf'``.
2292 way, it will remain unchanged after UTF-8 encoding).
2296 range, that's fine -- it will be encoded using UTF-8.
2298 The formatted message *will* be encoded using UTF-8 encoding by
2300 :rfc:`5424`-compliant messages. If you don't, logging may not complain, but your
2301 messages will not be RFC 5424-compliant, and your syslog daemon may complain.
2305 -------------------------------
2308 readily machine-parseable, there might be circumstances where you want to output
2313 which uses JSON to serialise the event in a machine-parseable manner::
2333 .. code-block:: none
2375 .. code-block:: none
2383 .. _custom-handlers:
2388 --------------------------------------------
2394 handlers in the stdlib don't offer built-in support. You can customize handler
2428 'encoding': 'utf-8',
2471 'encoding': 'utf-8',
2486 .. code-block:: shell-session
2490 2013-11-05 09:34:51,128 DEBUG mylogger A debug message
2491 $ ls -l chowntest.log
2492 -rw-r--r-- 1 pulse pulse 55 2013-11-05 09:34 chowntest.log
2496 supports :func:`dictConfig` - namely, Python 2.7, 3.2 or later. With pre-3.3
2500 In practice, the handler-creating function may be in a utility module somewhere
2515 types of file change - e.g. setting specific POSIX permission bits - in the
2519 :class:`~logging.FileHandler` - for example, one of the rotating file handlers,
2525 .. _formatting-styles:
2528 --------------------------------------------------------------
2544 because internally the logging package uses %-formatting to merge the format
2547 code will be using %-format strings.
2551 existing code could be using a given logger name and using %-formatting.
2553 For logging to work interoperably between any third-party libraries and your
2569 should be careful to support all formatting styles and allow %-formatting as
2580 There is another, perhaps simpler way that you can use {}- and $- formatting to
2582 :ref:`arbitrary-object-messages`) that when logging you can use an arbitrary
2605 Either of these can be used in place of a format string, to allow {}- or
2606 $-formatting to be used to build the actual "message" part which appears in the
2643 .. _filters-dictconfig:
2648 -------------------------------------------
2701 logging.debug('hello - noshow')
2707 .. code-block:: none
2718 in :ref:`logging-config-dict-externalobj`. For example, you could have used
2723 handlers and formatters. See :ref:`logging-config-dict-userdef` for more
2724 information on how logging supports using user-defined objects in its
2725 configuration, and see the other cookbook recipe :ref:`custom-handlers` above.
2728 .. _custom-format-exception:
2731 -------------------------------
2733 There might be times when you want to do customized exception formatting - for
2776 .. code-block:: none
2785 .. _spoken-messages:
2788 -------------------------
2792 text-to-speech (TTS) functionality available in your system, even if it doesn't have
2811 cmd = ['espeak', '-s150', '-ven+f3', msg]
2839 .. _buffered-logging:
2842 ------------------------------------------------------------
2855 - passed to another handler (the ``target`` handler) for processing. By default,
2864 parameter to ``foo`` which, if true, will log at ERROR and CRITICAL levels -
2940 .. code-block:: none
2979 .. _buffered-smtp:
2982 -------------------------------------------------
2989 send things via SMTP. (Run the downloaded script with the ``-h`` argument to see the
2992 .. code-block:: python
3011 self.setFormatter(logging.Formatter("%(asctime)s %(levelname)-5s %(message)s"))
3036 aa('--port', '-p', type=int, default=587, help='SMTP port')
3041 aa('--subject', '-s',
3061 .. _utc-formatting:
3064 --------------------------------------------------
3120 .. code-block:: none
3122 2015-10-17 12:53:29,501 The local time is Sat Oct 17 13:53:29 2015
3123 2015-10-17 13:53:29,501 The local time is Sat Oct 17 13:53:29 2015
3129 .. _context-manager:
3132 ---------------------------------------------
3171 block exit - you could do this if you don't need the handler any more.
3187 logger.debug('5. This should appear twice - once on stderr and once on stdout.')
3203 .. code-block:: shell-session
3208 5. This should appear twice - once on stderr and once on stdout.
3209 5. This should appear twice - once on stderr and once on stdout.
3215 .. code-block:: shell-session
3218 5. This should appear twice - once on stderr and once on stdout.
3222 .. code-block:: shell-session
3227 5. This should appear twice - once on stderr and once on stdout.
3237 .. _starter-template:
3240 ----------------------------------
3244 * Use a logging level based on command-line arguments
3249 Suppose we have a command-line application whose job is to stop, start or
3254 command-line argument, defaulting to ``logging.INFO``. Here's one way that
3267 parser.add_argument('--log-level', default='INFO', choices=levels)
3355 .. code-block:: shell-session
3372 .. code-block:: shell-session
3374 $ python app.py --log-level DEBUG start foo
3378 $ python app.py --log-level DEBUG stop foo bar
3382 $ python app.py --log-level DEBUG restart foo bar baz
3388 .. code-block:: shell-session
3390 $ python app.py --log-level WARNING start foo
3391 $ python app.py --log-level WARNING stop foo bar
3392 $ python app.py --log-level WARNING restart foo bar baz
3397 .. _qt-gui:
3400 --------------------
3404 cross-platform UI framework with Python bindings using `PySide2
3405 <https://pypi.org/project/PySide2/>`_ or `PyQt5
3406 <https://pypi.org/project/PyQt5/>`_ libraries.
3423 .. code-block:: python3
3457 # string is just a convenience - you could format a string for output any way
3475 # are named something like "Dummy-1". The function below gets the Qt name of the
3519 # * A read-only text edit window which holds formatted log messages
3549 fs = '%(asctime)s %(qThreadName)-12s %(levelname)-8s %(message)s'
3564 # Connect the non-worker slots and signals
3633 --------------------------------------
3654 tz_offset = re.compile(r'([+-]\d{2})(\d{2})$')
3678 hostname = '-'
3679 appname = self.appname or '-'
3681 msgid = '-'
3683 sdata = '-'
3686 # This should be a dict where the keys are SD-ID and the value is a
3687 # dict mapping PARAM-NAME to PARAM-VALUE (refer to the RFC for what these
3689 # There's no error checking here - it's purely for illustration, and you
3722 -------------------------------------------
3724 Sometimes, you need to interface to a third-party API which expects a file-like
3726 can do this using a class which wraps a logger with a file-like API.
3729 .. code-block:: python
3743 # doesn't actually do anything, but might be expected of a file-like
3744 # object - so optional depending on your situation
3748 # doesn't actually do anything, but might be expected of a file-like
3749 # object - so optional depending on your situation. You might want
3766 .. code-block:: text
3774 .. code-block:: python
3786 .. code-block:: pycon
3802 .. code-block:: python
3809 .. code-block:: text
3813 WARNING:demo: File "/home/runner/cookbook-loggerwriter/test.py", line 53, in <module>
3817 WARNING:demo: File "/home/runner/cookbook-loggerwriter/test.py", line 49, in main
3831 .. code-block:: python
3853 .. code-block:: text
3856 WARNING:demo: File "/home/runner/cookbook-loggerwriter/main.py", line 55, in <module>
3858 WARNING:demo: File "/home/runner/cookbook-loggerwriter/main.py", line 52, in main
3863 .. patterns-to-avoid:
3866 -----------------
3882 a copy/paste/forget-to-change error).
3902 and wasted debugging time - log entries end up in unexpected places, or are
3904 and grows in size unexpectedly despite size-based rotation being supposedly
3907 Use the techniques outlined in :ref:`multiple-processes` to circumvent such
3935 the :ref:`existing mechanisms <context-info>` for passing contextual
3938 more fine-grained than that).
3940 .. _cookbook-ref-links:
3943 ---------------
3956 :ref:`Basic Tutorial <logging-basic-tutorial>`
3958 :ref:`Advanced Tutorial <logging-advanced-tutorial>`