Lines Matching +full:- +full:- +full:log +full:- +full:level
1 .. _logging-cookbook:
7 :Author: Vinay Sajip <vinay_sajip at red-dove dot com>
11 :ref:`cookbook-ref-links`.
16 ---------------------------------
33 fh = logging.FileHandler('spam.log')
35 # create console handler with a higher log level
39 formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
78 .. code-block:: none
80 2005-03-23 23:47:11,663 - spam_application - INFO -
82 2005-03-23 23:47:11,665 - spam_application.auxiliary.Auxiliary - INFO -
84 2005-03-23 23:47:11,665 - spam_application - INFO -
86 2005-03-23 23:47:11,668 - spam_application - INFO -
88 2005-03-23 23:47:11,668 - spam_application.auxiliary.Auxiliary - INFO -
90 2005-03-23 23:47:11,669 - spam_application.auxiliary.Auxiliary - INFO -
92 2005-03-23 23:47:11,670 - spam_application - INFO -
94 2005-03-23 23:47:11,671 - spam_application - INFO -
96 2005-03-23 23:47:11,672 - spam_application.auxiliary - INFO -
98 2005-03-23 23:47:11,673 - spam_application - INFO -
102 -----------------------------
117 … logging.basicConfig(level=logging.DEBUG, format='%(relativeCreated)6d %(threadName)s %(message)s')
135 .. code-block:: none
137 0 Thread-1 Hi from myfunc
139 505 Thread-1 Hi from myfunc
141 1007 Thread-1 Hi from myfunc
143 1508 Thread-1 Hi from myfunc
144 2010 Thread-1 Hi from myfunc
146 2512 Thread-1 Hi from myfunc
148 3013 Thread-1 Hi from myfunc
149 3515 Thread-1 Hi from myfunc
151 4017 Thread-1 Hi from myfunc
153 4518 Thread-1 Hi from myfunc
159 --------------------------------
163 will be beneficial for an application to log all messages of all severities to a
167 previous simple module-based configuration example::
174 fh = logging.FileHandler('spam.log')
176 # create console handler with a higher log level
180 formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
197 The ability to create new handlers with higher- or lower-severity filters can be
203 modify the severity level of the logger and/or handler to debug.
205 .. _multiple-destinations:
208 --------------------------------
210 Let's say you want to log to console and file with different message formats and
211 in differing circumstances. Say you want to log messages with levels of DEBUG
212 and higher to file, and those messages at level INFO and higher to the console.
218 # set up logging to file - see previous section for more details
219 logging.basicConfig(level=logging.DEBUG,
220 format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
221 datefmt='%m-%d %H:%M',
222 filename='/tmp/myapp.log',
228 formatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s')
234 # Now, we can log to the root logger, or any other logger. First the root...
250 .. code-block:: none
259 .. code-block:: none
261 10-22 22:19 root INFO Jackdaws love my big sphinx of quartz.
262 10-22 22:19 myapp.area1 DEBUG Quick zephyrs blow, vexing daft Jim.
263 10-22 22:19 myapp.area1 INFO How quickly daft jumping zebras vex.
264 10-22 22:19 myapp.area2 WARNING Jail zesty vixen who grabbed pay from quack.
265 10-22 22:19 myapp.area2 ERROR The five boxing wizards jump quickly.
273 Note that the above choice of log filename ``/tmp/myapp.log`` implies use of a
275 choose a different directory name for the log - just ensure that the directory exists
279 .. _custom-level-handling:
282 -------------------------
291 * Send messages of severity ``DEBUG`` and above to file ``app.log``
295 .. code-block:: json
302 "format": "%(levelname)-8s - %(message)s"
308 "level": "INFO",
314 "level": "ERROR",
321 "filename": "app.log",
326 "level": "DEBUG",
341 .. code-block:: json
347 "level": "WARNING"
354 .. code-block:: json
359 "level": "INFO",
369 .. code-block:: python
371 def filter_maker(level):
372 level = getattr(logging, level)
375 return record.levelno <= level
379 This converts the string argument passed in to a numeric level, and returns a
380 function which only returns ``True`` if the level of the passed in record is
381 at or below the specified level. Note that in this example I have defined the
383 so its module will be ``__main__`` - hence the ``__main__.filter_maker`` in the
389 .. code-block:: python
401 "format": "%(levelname)-8s - %(message)s"
407 "level": "WARNING"
413 "level": "INFO",
420 "level": "ERROR",
427 "filename": "app.log",
432 "level": "DEBUG",
442 def filter_maker(level):
443 level = getattr(logging, level)
446 return record.levelno <= level
459 .. code-block:: shell
461 python main.py 2>stderr.log >stdout.log
465 .. code-block:: shell
467 $ more *.log
469 app.log
471 DEBUG - A DEBUG message
472 INFO - An INFO message
473 WARNING - A WARNING message
474 ERROR - An ERROR message
475 CRITICAL - A CRITICAL message
477 stderr.log
479 ERROR - An ERROR message
480 CRITICAL - A CRITICAL message
482 stdout.log
484 INFO - An INFO message
485 WARNING - A WARNING message
489 ----------------------------
523 properly preceded with the binary-encoded length, as the new logging
544 .. _blocking-handlers:
547 --------------------------------
558 performing mail or network infrastructure). But almost any network-based
563 One solution is to use a two-part approach. For the first part, attach only a
565 performance-critical threads. They simply write to their queue, which can be
569 in your code. If you are a library developer who has performance-critical
584 resource-friendly than, say, having threaded versions of the existing handler
589 que = queue.Queue(-1) # no limit on size
598 # The log output will display the thread which generated
607 .. code-block:: none
622 because it was assumed that level filtering was all done on the other side,
625 listener's constructor. When this is done, the listener compares the level
626 of each message with the handler's level, and only passes a message to a
629 .. _network-logging:
632 -----------------------------------------------------
648 # Now, we can log to the root logger, or any other logger. First the root...
681 Handle multiple requests - each expected to be a 4-byte length,
692 chunk = chunk + self.connection.recv(slen - len(chunk))
709 # is normally called AFTER logger-level filtering. If you want
716 Simple TCP socket-based logging receiver suitable for testing.
742 format='%(relativeCreated)5d %(name)-15s %(levelname)-8s %(message)s')
753 .. code-block:: none
772 .. _socket-listener-gist: https://gist.github.com/vsajip/4b227eeec43817465ca835ca66f75e2b
775 process-management tool such as `Supervisor <http://supervisord.org/>`_.
776 `Here is a Gist <socket-listener-gist_>`__
777 which provides the bare-bones files to run the above functionality using
780 +-------------------------+----------------------------------------------------+
785 +-------------------------+----------------------------------------------------+
787 | | entries for the listener and a multi-process web |
789 +-------------------------+----------------------------------------------------+
792 +-------------------------+----------------------------------------------------+
793 | :file:`log_listener.py` | The socket listener program which receives log |
795 +-------------------------+----------------------------------------------------+
798 +-------------------------+----------------------------------------------------+
800 +-------------------------+----------------------------------------------------+
802 +-------------------------+----------------------------------------------------+
806 requests. This example setup shows how the workers can write to the same log file
807 without conflicting with one another --- they all go through the socket listener.
811 #. Download `the Gist <socket-listener-gist_>`__
817 This creates a :file:`run` subdirectory to contain Supervisor-related and
818 log files, and a :file:`venv` subdirectory to contain a virtual environment
825 which will lead to records being written to the log.
827 #. Inspect the log files in the :file:`run` subdirectory. You should see the
828 most recent log lines in files matching the pattern :file:`app.log*`. They won't be in
830 worker processes in a non-deterministic way.
833 ``venv/bin/supervisorctl -c supervisor.conf shutdown``.
838 .. _context-info:
841 ----------------------------------------------------
845 networked application, it may be desirable to log client-specific information
846 in the log (e.g. remote client's username, or IP address). Although you could
849 :class:`Logger` instances on a per-connection basis, this is not a good idea
852 level of granularity you want to use in logging an application, it could
864 :meth:`exception`, :meth:`critical` and :meth:`log`. These methods have the
869 :class:`Logger` instance and a dict-like object which contains your contextual
889 an 'extra' key in the keyword argument whose value is the dict-like object
893 The advantage of using 'extra' is that the values in the dict-like object are
896 the keys of the dict-like object. If you need a different method, e.g. if you
903 This example adapter expects the passed in dict-like object to have a
904 'connid' key, whose value in brackets is prepended to the log message.
914 Then any events that you log to the adapter will have the value of
915 ``some_conn_id`` prepended to the log messages.
920 You don't need to pass an actual dict to a :class:`LoggerAdapter` - you could
926 .. _filters-contextual:
931 You can also add contextual information to log output using a user-defined
939 add, say, information from the request - say, the remote IP address and remote
940 user's username - to the ``LogRecord``, using the attribute names 'ip' and
950 This is a filter which injects contextual information into the log.
967 logging.basicConfig(level=logging.DEBUG,
968 … format='%(asctime)-15s %(name)-5s %(levelname)-8s IP: %(ip)-15s User: %(user)-8s %(message)s')
980 a2.log(lvl, 'A message at %s level with %d %s', lvlname, 2, 'parameters')
984 .. code-block:: none
986 2010-09-06 22:38:15,292 a.b.c DEBUG IP: 123.231.231.123 User: fred A debug message
987 …2010-09-06 22:38:15,300 a.b.c INFO IP: 192.168.0.1 User: sheila An info message with som…
988 …2010-09-06 22:38:15,300 d.e.f CRITICAL IP: 127.0.0.1 User: sheila A message at CRITICAL le…
989 …2010-09-06 22:38:15,300 d.e.f ERROR IP: 127.0.0.1 User: jim A message at ERROR level…
990 …2010-09-06 22:38:15,300 d.e.f DEBUG IP: 127.0.0.1 User: sheila A message at DEBUG level…
991 …2010-09-06 22:38:15,300 d.e.f ERROR IP: 123.231.231.123 User: fred A message at ERROR level…
992 …2010-09-06 22:38:15,300 d.e.f CRITICAL IP: 192.168.0.1 User: jim A message at CRITICAL le…
993 …2010-09-06 22:38:15,300 d.e.f CRITICAL IP: 127.0.0.1 User: sheila A message at CRITICAL le…
994 …2010-09-06 22:38:15,300 d.e.f DEBUG IP: 192.168.0.1 User: jim A message at DEBUG level…
995 …2010-09-06 22:38:15,301 d.e.f ERROR IP: 127.0.0.1 User: sheila A message at ERROR level…
996 …2010-09-06 22:38:15,301 d.e.f DEBUG IP: 123.231.231.123 User: fred A message at DEBUG level…
997 …2010-09-06 22:38:15,301 d.e.f INFO IP: 123.231.231.123 User: fred A message at INFO level …
1000 ----------------------
1002 Since Python 3.7, the :mod:`contextvars` module has provided context-local storage
1004 of storage may thus be generally preferable to thread-locals. The following example
1005 shows how, in a multi-threaded environment, logs can populated with contextual
1010 common to them. How can each of these applications have their own log, where all
1012 the appropriate application's log file, while including in the log additional
1017 .. code-block:: python
1032 ``Request`` and ``WebApp``. These simulate how real threaded web applications work -
1035 .. code-block:: python
1060 # A dummy set of requests which will be used in the simulation - we'll just pick
1077 …formatter = logging.Formatter('%(threadName)-11s %(appName)s %(name)-9s %(user)-6s %(ip)s %(method…
1087 A filter which injects context-specific information into logs and ensures
1088 that only information for a specific webapp is included in its log
1104 webapp-specific log.
1108 handler = logging.FileHandler(name + '.log', 'w')
1138 aa('--count', '-c', type=int, default=100, help='How many requests to simulate')
1148 handler = logging.FileHandler('app.log', 'w')
1176 into :file:`app1.log` and the rest into :file:`app2.log`, and the all the requests are
1177 logged to :file:`app.log`. Each webapp-specific log will contain only log entries for
1179 log (i.e. the information in each dummy request will always appear together in a log
1182 .. code-block:: shell
1184 ~/logging-contextual-webapp$ python main.py
1187 ~/logging-contextual-webapp$ wc -l *.log
1188 153 app1.log
1189 147 app2.log
1190 300 app.log
1192 ~/logging-contextual-webapp$ head -3 app1.log
1193 Thread-3 (process_request) app1 __main__ jim 192.168.3.21 POST Request processing started
1194 Thread-3 (process_request) app1 webapplib jim 192.168.3.21 POST Hello from webapplib!
1195 Thread-5 (process_request) app1 __main__ jim 192.168.3.21 POST Request processing started
1196 ~/logging-contextual-webapp$ head -3 app2.log
1197 Thread-1 (process_request) app2 __main__ sheila 192.168.2.21 GET Request processing started
1198 Thread-1 (process_request) app2 webapplib sheila 192.168.2.21 GET Hello from webapplib!
1199 Thread-2 (process_request) app2 __main__ jim 192.168.2.20 GET Request processing started
1200 ~/logging-contextual-webapp$ head app.log
1201 Thread-1 (process_request) app2 __main__ sheila 192.168.2.21 GET Request processing started
1202 Thread-1 (process_request) app2 webapplib sheila 192.168.2.21 GET Hello from webapplib!
1203 Thread-2 (process_request) app2 __main__ jim 192.168.2.20 GET Request processing started
1204 Thread-3 (process_request) app1 __main__ jim 192.168.3.21 POST Request processing started
1205 Thread-2 (process_request) app2 webapplib jim 192.168.2.20 GET Hello from webapplib!
1206 Thread-3 (process_request) app1 webapplib jim 192.168.3.21 POST Hello from webapplib!
1207 Thread-4 (process_request) app2 __main__ fred 192.168.2.22 GET Request processing started
1208 Thread-5 (process_request) app1 __main__ jim 192.168.3.21 POST Request processing started
1209 Thread-4 (process_request) app2 webapplib fred 192.168.2.22 GET Hello from webapplib!
1210 Thread-6 (process_request) app1 __main__ jim 192.168.3.21 POST Request processing started
1211 ~/logging-contextual-webapp$ grep app1 app1.log | wc -l
1213 ~/logging-contextual-webapp$ grep app2 app2.log | wc -l
1215 ~/logging-contextual-webapp$ grep app1 app.log | wc -l
1217 ~/logging-contextual-webapp$ grep app2 app.log | wc -l
1222 --------------------------------------------
1227 a new :class:`~LogRecord` instead of modifying it in-place, as shown in the following script::
1241 formatter = logging.Formatter('%(message)s from %(user)-8s')
1246 logger.info('A log message')
1248 .. _multiple-processes:
1251 ------------------------------------------------
1253 Although logging is thread-safe, and logging to a single file from multiple
1257 need to log to a single file from multiple processes, one way of doing this is
1258 to have all the processes log to a :class:`~handlers.SocketHandler`, and have a
1262 :ref:`This section <network-logging>` documents this approach in more detail and
1277 all logging events to one of the processes in your multi-process application.
1282 thread rather than a separate listener process -- the implementation would be
1303 # simple example, the listener does not apply level or filter logic to received records.
1310 h = logging.handlers.RotatingFileHandler('mptest.log', 'a', 300, 10)
1311 f = logging.Formatter('%(asctime)s %(processName)-10s %(name)s %(levelname)-8s %(message)s')
1315 # This is the listener process top-level loop: wait for logging events
1326 logger.handle(record) # No level or filter logic applied - just do it!
1352 # send all messages, for demo; no other level or filter logic applied.
1355 # This is the worker process top-level loop, which just logs ten events with
1365 level = choice(LEVELS)
1367 logger.log(level, message)
1374 queue = multiprocessing.Queue(-1)
1424 logger.log(lvl, 'Message no. %d', i)
1433 … 'format': '%(asctime)s %(name)-15s %(levelname)-8s %(processName)-10s %(message)s'
1439 'level': 'INFO',
1443 'filename': 'mplog.log',
1449 'filename': 'mplog-foo.log',
1455 'filename': 'mplog-errors.log',
1457 'level': 'ERROR',
1467 'level': 'DEBUG',
1488 - e.g. the ``foo`` logger has a special handler which stores all events in the
1489 ``foo`` subsystem in a file ``mplog-foo.log``. This will be used by the logging
1500 .. code-block:: python
1502 queue = multiprocessing.Queue(-1)
1506 .. code-block:: python
1508 queue = multiprocessing.Manager().Queue(-1) # also works with the examples above
1531 <https://uwsgi-docs.readthedocs.io/en/latest/>`_ (or similar), multiple worker
1533 file-based handlers directly in your web application. Instead, use a
1534 :class:`SocketHandler` to log from the web application to a listener in a separate
1535 process. This can be set up using a process management tool such as Supervisor - see
1540 -------------------
1545 Sometimes you want to let a log file grow to a certain size, then open a new
1546 file and log to that. You may want to keep a certain number of these files, and
1557 # Set up a specific logger with our desired output level
1561 # Add the log message handler to the logger
1567 # Log some messages
1577 The result should be 6 separate files, each with part of the log history for the
1580 .. code-block:: none
1594 Obviously this example sets the log length much too small as an extreme
1597 .. _format-styles:
1600 ------------------------------------
1603 formatting messages with variable content was to use the %-formatting
1618 .. code-block:: pycon
1630 2010-10-28 15:11:55,341 foo.bar DEBUG This is a DEBUG message
1632 2010-10-28 15:12:11,526 foo.bar CRITICAL This is a CRITICAL message
1637 2010-10-28 15:13:06,924 foo.bar DEBUG This is a DEBUG message
1639 2010-10-28 15:13:11,494 foo.bar CRITICAL This is a CRITICAL message
1644 That can still use %-formatting, as shown here::
1647 2010-10-28 15:19:29,833 foo.bar ERROR This is another, ERROR, message
1655 to indicate additional contextual information to be added to the log). So
1658 uses %-formatting to merge the format string and the variable arguments.
1660 all logging calls which are out there in existing code will be using %-format
1663 There is, however, a way that you can use {}- and $- formatting to construct
1664 your individual log messages. Recall that for a message you can use an
1687 Either of these can be used in place of a format string, to allow {}- or
1688 $-formatting to be used to build the actual "message" part which appears in the
1689 formatted log output in place of "%(message)s" or "{message}" or "$message".
1690 It's a little unwieldy to use the class names whenever you want to log
1692 underscore --- not to be confused with _, the single underscore used as a
1699 .. code-block:: pycon
1718 would of course use ``logger.debug()`` or similar to actually log using this
1723 when (and if) the logged message is actually about to be output to a log by a
1746 def log(self, level, msg, /, *args, **kwargs):
1747 if self.isEnabledFor(level):
1749 self.logger._log(level, Message(msg, args), (), **kwargs)
1757 logging.basicConfig(level=logging.DEBUG)
1760 The above script should log the message ``Hello, world!`` when run with
1766 .. _custom-logrecord:
1769 -------------------------
1772 When an event is logged and not filtered out by a logger's level, a
1810 at module level). It's probably one too many things to think about. Developers
1812 top-level logger, but this would not be invoked if an application developer
1813 attached a handler to a lower-level library logger --- so output from that
1839 However, it should be borne in mind that each link in the chain adds run-time
1844 .. _zeromq-handlers:
1846 Subclassing QueueHandler - a ZeroMQ example
1847 -------------------------------------------
1885 Subclassing QueueListener - a ZeroMQ example
1886 --------------------------------------------
1915 :ref:`A basic logging tutorial <logging-basic-tutorial>`
1917 :ref:`A more advanced logging tutorial <logging-advanced-tutorial>`
1920 An example dictionary-based configuration
1921 -----------------------------------------
1923 Below is an example of a logging configuration dictionary - it's taken from
1924 …he Django project <https://docs.djangoproject.com/en/stable/topics/logging/#configuring-logging>`_.
1946 'level':'DEBUG',
1947 'class':'django.utils.log.NullHandler',
1950 'level':'DEBUG',
1955 'level': 'ERROR',
1956 'class': 'django.utils.log.AdminEmailHandler',
1964 'level':'INFO',
1968 'level': 'ERROR',
1973 'level': 'INFO',
1980 section <https://docs.djangoproject.com/en/stable/topics/logging/#configuring-logging>`_
1983 .. _cookbook-rotator-namer:
1985 Using a rotator and namer to customize log rotation processing
1986 --------------------------------------------------------------
1989 runnable script, which shows gzip compression of the log file::
2007 rh = logging.handlers.RotatingFileHandler('rotated.log', maxBytes=128, backupCount=5)
2021 .. code-block:: shell-session
2023 $ ls rotated.log*
2024 rotated.log rotated.log.2.gz rotated.log.4.gz
2025 rotated.log.1.gz rotated.log.3.gz rotated.log.5.gz
2026 $ zcat rotated.log.1.gz
2027 2023-01-20 02:28:17,767 Message no. 996
2028 2023-01-20 02:28:17,767 Message no. 997
2029 2023-01-20 02:28:17,767 Message no. 998
2032 ----------------------------------------
2042 see logging in the main process, how the workers log to a QueueHandler and how
2049 Here's the script - the docstrings and the comments hopefully explain how it
2098 # would appear - hence the "if posix" clause.
2129 # would appear - hence the "if posix" clause.
2135 logger.log(lvl, 'Message no. %d', i)
2146 'level': 'INFO'
2151 'level': 'DEBUG'
2170 'level': 'DEBUG'
2185 … 'format': '%(asctime)s %(name)-15s %(levelname)-8s %(processName)-10s %(message)s'
2189 'format': '%(name)-15s %(levelname)-8s %(processName)-10s %(message)s'
2196 'level': 'INFO'
2200 'filename': 'mplog.log',
2206 'filename': 'mplog-foo.log',
2212 'filename': 'mplog-errors.log',
2215 'level': 'ERROR'
2225 'level': 'DEBUG'
2228 # Log some initial events, just to show that logging in the parent works
2261 -----------------------------------------------------
2265 following structure: an optional pure-ASCII component, followed by a UTF-8 Byte
2266 Order Mark (BOM), followed by Unicode encoded using UTF-8. (See the
2267 :rfc:`relevant section of the specification <5424#section-6>`.)
2272 beginning of the message and hence not allowing any pure-ASCII component to
2277 want to produce :rfc:`5424`-compliant messages which include a BOM, an optional
2278 pure-ASCII sequence before it and arbitrary Unicode after it, encoded using
2279 UTF-8, then you need to do the following:
2287 The Unicode code point U+FEFF, when encoded using UTF-8, will be
2288 encoded as a UTF-8 BOM -- the byte-string ``b'\xef\xbb\xbf'``.
2292 way, it will remain unchanged after UTF-8 encoding).
2296 range, that's fine -- it will be encoded using UTF-8.
2298 The formatted message *will* be encoded using UTF-8 encoding by
2300 :rfc:`5424`-compliant messages. If you don't, logging may not complain, but your
2301 messages will not be RFC 5424-compliant, and your syslog daemon may complain.
2305 -------------------------------
2308 readily machine-parseable, there might be circumstances where you want to output
2310 (without needing complex regular expressions to parse the log message). This is
2313 which uses JSON to serialise the event in a machine-parseable manner::
2328 logging.basicConfig(level=logging.INFO, format='%(message)s')
2333 .. code-block:: none
2367 logging.basicConfig(level=logging.INFO, format='%(message)s')
2375 .. code-block:: none
2383 .. _custom-handlers:
2388 --------------------------------------------
2393 log file. On POSIX, this is easily done using :func:`shutil.chown`, but the file
2394 handlers in the stdlib don't offer built-in support. You can customize handler
2418 # used to create the handler, set the handler's level and
2421 'level':'DEBUG',
2426 'filename': 'chowntest.log',
2428 'encoding': 'utf-8',
2433 'level': 'DEBUG',
2461 # used to create the handler, set the handler's level and
2464 'level':'DEBUG',
2469 'filename': 'chowntest.log',
2471 'encoding': 'utf-8',
2476 'level': 'DEBUG',
2486 .. code-block:: shell-session
2489 $ cat chowntest.log
2490 2013-11-05 09:34:51,128 DEBUG mylogger A debug message
2491 $ ls -l chowntest.log
2492 -rw-r--r-- 1 pulse pulse 55 2013-11-05 09:34 chowntest.log
2496 supports :func:`dictConfig` - namely, Python 2.7, 3.2 or later. With pre-3.3
2500 In practice, the handler-creating function may be in a utility module somewhere
2515 types of file change - e.g. setting specific POSIX permission bits - in the
2519 :class:`~logging.FileHandler` - for example, one of the rotating file handlers,
2525 .. _formatting-styles:
2528 --------------------------------------------------------------
2542 contextual information to be added to the log). So you cannot directly make
2544 because internally the logging package uses %-formatting to merge the format
2547 code will be using %-format strings.
2551 existing code could be using a given logger name and using %-formatting.
2553 For logging to work interoperably between any third-party libraries and your
2554 code, decisions about formatting need to be made at the level of the
2569 should be careful to support all formatting styles and allow %-formatting as
2580 There is another, perhaps simpler way that you can use {}- and $- formatting to
2581 construct your individual log messages. You may recall (from
2582 :ref:`arbitrary-object-messages`) that when logging you can use an arbitrary
2605 Either of these can be used in place of a format string, to allow {}- or
2606 $-formatting to be used to build the actual "message" part which appears in the
2607 formatted log output in place of “%(message)s” or “{message}” or “$message”.
2608 If you find it a little unwieldy to use the class names whenever you want to log
2636 when (and if) the logged message is actually about to be output to a log by a
2643 .. _filters-dictconfig:
2648 -------------------------------------------
2693 'level': 'DEBUG',
2701 logging.debug('hello - noshow')
2707 .. code-block:: none
2718 in :ref:`logging-config-dict-externalobj`. For example, you could have used
2723 handlers and formatters. See :ref:`logging-config-dict-userdef` for more
2724 information on how logging supports using user-defined objects in its
2725 configuration, and see the other cookbook recipe :ref:`custom-handlers` above.
2728 .. _custom-format-exception:
2731 -------------------------------
2733 There might be times when you want to do customized exception formatting - for
2776 .. code-block:: none
2785 .. _spoken-messages:
2788 -------------------------
2792 text-to-speech (TTS) functionality available in your system, even if it doesn't have
2811 cmd = ['espeak', '-s150', '-ven+f3', msg]
2839 .. _buffered-logging:
2842 ------------------------------------------------------------
2844 There might be situations where you want to log messages in a temporary area
2847 errors, you don't want to clutter the log with the collected debug information,
2855 - passed to another handler (the ``target`` handler) for processing. By default,
2857 level is greater than or equal to a specified threshold is seen. You can use this
2862 all the logging levels, writing to ``sys.stderr`` to say what level it's about
2863 to log at, and then actually logging a message at that level. You can pass a
2864 parameter to ``foo`` which, if true, will log at ERROR and CRITICAL levels -
2871 a level at which flushing should occur, and a capacity for the buffer (number of
2912 write_line('about to log at DEBUG ...')
2914 write_line('about to log at INFO ...')
2916 write_line('about to log at WARNING ...')
2919 write_line('about to log at ERROR ...')
2921 write_line('about to log at CRITICAL ...')
2940 .. code-block:: none
2943 about to log at DEBUG ...
2944 about to log at INFO ...
2945 about to log at WARNING ...
2947 about to log at DEBUG ...
2948 about to log at INFO ...
2949 about to log at WARNING ...
2950 about to log at ERROR ...
2951 about to log at CRITICAL ...
2953 about to log at DEBUG ...
2954 about to log at INFO ...
2955 about to log at WARNING ...
2957 about to log at DEBUG ...
2958 about to log at INFO ...
2959 about to log at WARNING ...
2960 about to log at ERROR ...
2965 about to log at CRITICAL ...
2979 .. _buffered-smtp:
2982 -------------------------------------------------
2984 To illustrate how you can send log messages via email, so that a set number of
2989 send things via SMTP. (Run the downloaded script with the ``-h`` argument to see the
2992 .. code-block:: python
3011 self.setFormatter(logging.Formatter("%(asctime)s %(levelname)-5s %(message)s"))
3036 aa('--port', '-p', type=int, default=587, help='SMTP port')
3041 aa('--subject', '-s',
3058 have ten log messages, and the eleventh will have two messages. That makes up 102
3061 .. _utc-formatting:
3064 --------------------------------------------------
3120 .. code-block:: none
3122 2015-10-17 12:53:29,501 The local time is Sat Oct 17 13:53:29 2015
3123 2015-10-17 13:53:29,501 The local time is Sat Oct 17 13:53:29 2015
3129 .. _context-manager:
3132 ---------------------------------------------
3138 optionally change the logging level and add a logging handler purely in the
3145 def __init__(self, logger, level=None, handler=None, close=True):
3147 self.level = level
3152 if self.level is not None:
3153 self.old_level = self.logger.level
3154 self.logger.setLevel(self.level)
3159 if self.level is not None:
3167 If you specify a level value, the logger's level is set to that value in the
3171 block exit - you could do this if you don't need the handler any more.
3182 with LoggingContext(logger, level=logging.DEBUG):
3186 with LoggingContext(logger, level=logging.DEBUG, handler=h, close=True):
3187 logger.debug('5. This should appear twice - once on stderr and once on stdout.')
3191 We initially set the logger's level to ``INFO``, so message #1 appears and
3192 message #2 doesn't. We then change the level to ``DEBUG`` temporarily in the
3194 logger's level is restored to ``INFO`` and so message #4 doesn't appear. In the
3195 next ``with`` block, we set the level to ``DEBUG`` again but also add a handler
3203 .. code-block:: shell-session
3208 5. This should appear twice - once on stderr and once on stdout.
3209 5. This should appear twice - once on stderr and once on stdout.
3215 .. code-block:: shell-session
3218 5. This should appear twice - once on stderr and once on stdout.
3222 .. code-block:: shell-session
3227 5. This should appear twice - once on stderr and once on stdout.
3237 .. _starter-template:
3240 ----------------------------------
3244 * Use a logging level based on command-line arguments
3246 level in a consistent way
3249 Suppose we have a command-line application whose job is to stop, start or
3254 command-line argument, defaulting to ``logging.INFO``. Here's one way that
3267 parser.add_argument('--log-level', default='INFO', choices=levels)
3291 logging.basicConfig(level=options.log_level,
3353 If we run this application with the default log level, we get output like this:
3355 .. code-block:: shell-session
3366 The first word is the logging level, and the second word is the module or
3369 If we change the logging level, then we can change the information sent to the
3370 log. For example, if we want more information:
3372 .. code-block:: shell-session
3374 $ python app.py --log-level DEBUG start foo
3378 $ python app.py --log-level DEBUG stop foo bar
3382 $ python app.py --log-level DEBUG restart foo bar baz
3388 .. code-block:: shell-session
3390 $ python app.py --log-level WARNING start foo
3391 $ python app.py --log-level WARNING stop foo bar
3392 $ python app.py --log-level WARNING restart foo bar baz
3395 at ``WARNING`` level or above is logged by them.
3397 .. _qt-gui:
3400 --------------------
3402 A question that comes up from time to time is about how to log to a GUI
3404 cross-platform UI framework with Python bindings using `PySide2
3408 The following example shows how to log to a Qt GUI. This introduces a simple
3411 can log to the GUI from both the UI itself (via a button for manual logging)
3423 .. code-block:: python3
3456 # formatted log message, and the log record which generated it. The formatted
3457 # string is just a convenience - you could format a string for output any way
3474 # This example uses QThreads, which means that the threads at the Python level
3475 # are named something like "Dummy-1". The function below gets the Qt name of the
3512 level = random.choice(LEVELS)
3513 logger.log(level, 'Message after delay of %3.1f: %d', delay, i, extra=extra)
3519 # * A read-only text edit window which holds formatted log messages
3520 # * A button to start work and log stuff in a separate thread
3521 # * A button to log something from the main thread
3522 # * A button to clear the log window
3545 self.log_button = PB('Log a message at a random level', self)
3546 self.clear_button = PB('Clear log window', self)
3549 fs = '%(asctime)s %(qThreadName)-12s %(levelname)-8s %(message)s'
3564 # Connect the non-worker slots and signals
3611 # color according to its severity (level).
3612 level = random.choice(LEVELS)
3614 logger.log(level, 'Manually logged!', extra=extra)
3633 --------------------------------------
3643 need to be able to log to a syslog server with support for it, you can do so with a
3654 tz_offset = re.compile(r'([+-]\d{2})(\d{2})$')
3678 hostname = '-'
3679 appname = self.appname or '-'
3681 msgid = '-'
3683 sdata = '-'
3686 # This should be a dict where the keys are SD-ID and the value is a
3687 # dict mapping PARAM-NAME to PARAM-VALUE (refer to the RFC for what these
3689 # There's no error checking here - it's purely for illustration, and you
3710 to the log). Nevertheless, the above should be adaptable to your speciric needs. With
3722 -------------------------------------------
3724 Sometimes, you need to interface to a third-party API which expects a file-like
3726 can do this using a class which wraps a logger with a file-like API.
3729 .. code-block:: python
3734 def __init__(self, logger, level):
3736 self.level = level
3740 self.logger.log(self.level, message)
3743 # doesn't actually do anything, but might be expected of a file-like
3744 # object - so optional depending on your situation
3748 # doesn't actually do anything, but might be expected of a file-like
3749 # object - so optional depending on your situation. You might want
3754 logging.basicConfig(level=logging.DEBUG)
3766 .. code-block:: text
3774 .. code-block:: python
3786 .. code-block:: pycon
3802 .. code-block:: python
3809 .. code-block:: text
3813 WARNING:demo: File "/home/runner/cookbook-loggerwriter/test.py", line 53, in <module>
3817 WARNING:demo: File "/home/runner/cookbook-loggerwriter/test.py", line 49, in main
3828 this problem, you need to buffer things and only output log lines when newlines
3831 .. code-block:: python
3834 def __init__(self, logger, level):
3835 super().__init__(logger, level)
3845 self.logger.log(self.level, s)
3848 self.logger.log(self.level, part)
3853 .. code-block:: text
3856 WARNING:demo: File "/home/runner/cookbook-loggerwriter/main.py", line 55, in <module>
3858 WARNING:demo: File "/home/runner/cookbook-loggerwriter/main.py", line 52, in main
3863 .. patterns-to-avoid:
3866 -----------------
3873 Opening the same log file multiple times
3882 a copy/paste/forget-to-change error).
3902 and wasted debugging time - log entries end up in unexpected places, or are
3904 and grows in size unexpectedly despite size-based rotation being supposedly
3907 Use the techniques outlined in :ref:`multiple-processes` to circumvent such
3935 the :ref:`existing mechanisms <context-info>` for passing contextual
3938 more fine-grained than that).
3940 .. _cookbook-ref-links:
3943 ---------------
3956 :ref:`Basic Tutorial <logging-basic-tutorial>`
3958 :ref:`Advanced Tutorial <logging-advanced-tutorial>`