Lines Matching +full:structured +full:- +full:clone
1 - Feature Name: `structured_logging`
2 - Start Date: 2019-03-11
3 - RFC PR: [log/rfcs#0296](https://github.com/rust-lang-nursery/log/pull/296)
8 …structured logging to the `log` crate in both `std` and `no_std` environments, allowing log record…
10 `log` will provide an API for capturing structured data that offloads complex serialization to de-f…
12 The API is heavily inspired by `slog` and `tokio-trace`.
17 - [Motivation](#motivation)
18 - [What is structured logging?](#what-is-structured-logging)
19 - [Why do we need structured logging in `log`?](#why-do-we-need-structured-logging-in-log)
20 - [Guide-level explanation](#guide-level-explanation)
21 - [Logging structured key-value pairs](#logging-structured-key-value-pairs)
22 …- [Supporting key-value pairs in `Log` implementations](#supporting-key-value-pairs-in-log-impleme…
23 - [Integrating log frameworks with `log`](#integrating-log-frameworks-with-log)
24 …- [How producers and consumers of structured values interact](#how-producers-and-consumers-of-stru…
25 - [Reference-level explanation](#reference-level-explanation)
26 - [Design considerations](#design-considerations)
27 - [Cargo features](#cargo-features)
28 - [A complete key-values API](#a-complete-key-values-api)
29 - [`Error`](#error)
30 - [`Value`](#value)
31 - [`ToValue`](#tovalue)
32 - [`Key`](#key)
33 - [`ToKey`](#tokey)
34 - [`Source`](#source)
35 - [`Visitor`](#visitor)
36 - [`Record` and `RecordBuilder`](#record-and-recordbuilder)
37 - [A minimal key-values API](#a-minimal-key-values-api)
38 - [The `log!` macros](#the-log-macros)
39 - [Drawbacks, rationale, and alternatives](#drawbacks-rationale-and-alternatives)
40 - [Prior art](#prior-art)
41 - [Unresolved questions](#unresolved-questions)
46 ## What is structured logging?
50 Structured logs can retain their original structure in a machine-readable format. They can be chang…
52 As an example of structured logging, a textual record like this:
55 [INF 2018-09-27T09:32:03Z basic] [service: database, correlation: 123] Operation completed successf…
58 could be represented as a structured record like this:
72 When log records are kept in a structured format like this, potentially interesting queries like _w…
74 Even when logging to a console for immediate consumption, the human-readable message can be present…
77 [INF 2018-09-27T09:32:03Z] Operation completed successfully in 18ms
84 …nal metadata is good for human-centric formats. Having a way to retain the structure of that metad…
86 ## Why do we need structured logging in `log`?
88 Why add structured logging support to the `log` crate when libraries like `slog` already exist and …
92 …-of-the-box to capture log records, define and implement the pieces of a logging pipeline, and pas…
94 …-box implementation of the `Log` trait. In this role, the `Log` trait can act as a common entry-po…
96 …ugh, `log` needs to support structured logging so that libraries and their consumers can take adva…
98 # Guide-level explanation
99 [guide-level-explanation]: #guide-level-explanation
101 This section introduces `log`'s structured logging API through a tour of how structured records can…
103 ## Logging structured key-value pairs
105 Structured logging is supported in `log` by allowing typed key-value pairs to be associated with a …
107 ### What can be captured as a structured value?
109 A value can be captured as a structured value in a log record if it implements the `ToValue` trait:
113 fn to_value<'v>(&'v self) -> Value<'v>;
117 where `Value` is a special container for structured data:
132 - Standard formats: `Arguments`
133 - Primitives: `bool`, `char`
134 - Unsigned integers: `u8`, `u16`, `u32`, `u64`, `u128`
135 - Signed integers: `i8`, `i16`, `i32`, `i64`, `i128`
136 - Strings: `&str`, `String`
137 - Paths: `&Path`, `PathBuf`
138 - Special types: `Option<T>`, `&T`, and `()`.
144 fn to_value(&self) -> Value {
150 The `Value::from_any` method accepts any type, `&T`, and an ad-hoc function that tells the `Value` …
154 pub fn from_any<T>(v: &'v T, from: fn(FromAny, &T) -> Result<(), Error>) -> Self {
162 pub fn debug(v: impl Debug) -> Result<(), Error> {
167 fn u64(v: u64) -> Result<(), Error> {
181 fn to_value(&self) -> Value {
191 fn to_value(&self) -> Value {
201 fn to_value(&self) -> Value {
211 …aque and unstructured string. It would be better represented as a map of key-value pairs. However,…
213 …-way integration with `log`'s `Value` type through optional Cargo features. Let's use `sval` as an…
229 fn to_value(&self) -> Value {
249 fn to_value(&self) -> Value {
255 ## Supporting key-value pairs in `Log` implementations
257 …structured logs is only half the story. Implementors of the `Log` trait also need to be able to wo…
261 pub fn key_values(&self) -> impl Source;
265 where `Source` is a trait for iterating over the individual key-value pairs:
270 fn get<'kvs, Q>(&'kvs self, key: Q) -> Option<Value<'kvs>>
277 // Run a function for each key-value pair
278 fn for_each<F>(self, f: F) -> Result<(), Error>
286 // Run a function for each key-value pair
287 fn try_for_each<F, E>(self, f: F) -> Result<(), Error>
290 F: FnMut(Key, Value) -> Result<(), E>,
296 // Serialize the source as a map of key-value pairs
297 fn as_map(self) -> AsMap<Self>
304 // Serialize the source as a sequence of key-value tuples
305 fn as_seq(self) -> AsSeq<Self>
316 ### Writing key-value pairs as text
321 [INF 2018-09-27T09:32:03Z] Operation completed successfully in 18ms
328 Each key-value pair, shown as a `$key: $value` line, can be formatted from the `Source` on a record…
333 fn log_record(w: impl Write, r: &Record) -> io::Result<()> {
337 // Write each key-value pair on a new line
346 In the above example, the `Source::for_each` method iterates over each key-value pair in the `Sourc…
348 ### Writing key-value pairs as JSON
350 Let's look at a structured example. Take the following JSON map:
376 fn log_record(w: impl Write, r: &Record) -> io::Result<()> {
399 …to get an adapter that implements `serde::Serialize` by serializing each key-value pair as an entr…
403 …structured key-value pairs that can be iterated or serialized. Other log frameworks that want to i…
405 …urce::as_map`. Both of those methods are provided on top of a required lower-level `Source::visit`…
409 fn visit<'kvs>(&'kvs self, visitor: &mut impl Visitor<'kvs>) -> Result<(), Error>;
415 where `Visitor` is another trait that accepts individual key-value pairs:
419 fn visit_pair(&mut self, k: Key<'kvs>, v: Value<'kvs>) -> Result<(), Error>;
423 …structured data we saw previously. The lifetime `'kvs` is threaded from the original borrow of the…
425 Let's implement a `Source`. As an example, let's say our log framework captures its key-value pairs…
439 …fn visit<'kvs>(&'kvs self, visitor: &mut impl source::Visitor<'kvs>) -> Result<(), source::Error> {
449 …-value pairs directly like `BTreeMap<String, serde_json::Value>` though. It could act like an adap…
460 …fn visit<'kvs>(&'kvs self, visitor: &mut impl source::Visitor<'kvs>) -> Result<(), source::Error> {
461 // `Seen` is a visitor that will capture key-value pairs
462 // in a `BTreeMap`. We use it internally to sort and de-duplicate
463 // the key-value pairs that `SortRetainLast` is wrapping.
467 …fn visit_pair<'vis>(&'vis mut self, k: source::Key<'kvs>, v: source::Value<'kvs>) -> Result<(), so…
474 // Visit the inner source and collect its key-value pairs into `seen`
478 // Iterate through the seen key-value pairs in order
489 ## How producers and consumers of structured values interact
491 The previous sections demonstrated some of the APIs for capturing and consuming structured data on …
495 # Reference-level explanation
496 [reference-level-explanation]: #reference-level-explanation
498 This section details the nuts-and-bolts of the structured logging API.
504 Allow structured logging to be added in the current `0.4.x` series of `log`. This gives us the opti…
508 …y library that wants their data types to be logged. Logging is a truly cross-cutting concern, so i…
512 …eworks to integrate through if they want. Producers of structured data and consumers of structured…
516 `log` is already designed to be object-safe so this new structured logging API needs to be object-s…
520 Once structured logging is available, there will be a lot of new ways to hold `log` and new concept…
524 Structured logging will be supported in either `std` or `no_std` contexts by default.
530 kv_serde = ["std", "serde", "erased-serde", "sval"]
535 Using default features, implementors of the `Log` trait will be able to format structured data (in …
537 …structured logging in mind. It's `no_std` and object-safe, but isn't stable and requires `rustc` `…
539 …-facto general-purpose serialization framework for Rust. It's widely supported and stable, but req…
541 ## A complete key-values API
543 …structured values in `log`, along with possible future extensions. Actual implementation details a…
547 Just about the only things you can do with a structured value are format it or serialize it. Serial…
553 pub fn msg(msg: &'static str) -> Self {
581 pub fn into_sval(self) -> sval::Error {
590 pub fn into_serde<E>(self) -> E
602 pub fn custom(err: impl fmt::Display) -> Self {
621 …he `Error` type doesn't try to be a general-purpose error management tool, it tries to make it eas…
627 …ed container for some type whose structure can be visited, with a potentially short-lived lifetime:
633 pub fn from_any<T>(v: &'v T, from: FromAnyFn<T>) -> Self {
637 pub fn from_debug(v: &'v impl Debug) -> Self {
642 pub fn from_sval(v: &'v (impl sval::Value + Debug)) -> Self {
647 pub fn from_serde(v: &'v (impl serde::Serialize + Debug)) -> Self {
670 type FromAnyFn<T> = fn(FromAny, &T) -> Result<(), Error>;
679 pub fn debug(self, v: impl Debug) -> Result<(), Error> {
684 pub fn sval(self, v: impl sval::Value + Debug) -> Result<(), Error> {
689 pub fn serde(self, v: impl serde::Serialize + Debug) -> Result<(), Error> {
693 fn value(self, v: Value) -> Result<(), Error> {
697 fn u64(self, v: u64) -> Result<(), Error> {
701 fn u128(self, v: u128) -> Result<(), Error> {
705 fn i64(self, v: i64) -> Result<(), Error> {
709 fn i128(self, v: i128) -> Result<(), Error> {
713 fn f64(self, v: f64) -> Result<(), Error> {
717 fn bool(self, v: bool) -> Result<(), Error> {
721 fn char(self, v: char) -> Result<(), Error> {
725 fn none(self) -> Result<(), Error> {
729 fn str(self, v: &str) -> Result<(), Error> {
743 pub fn from_any<T>(v: &'v T, from: FromAnyFn<T>) -> Self {
753 #[derive(Clone, Copy)]
759 type FromAnyFn<T> = fn(FromAny, &T) -> Result<(), Error>;
762 fn new<T>(data: &'a T, from: FromAnyFn<T>) -> Self {
772 fn visit(&self, backend: &mut dyn Backend) -> Result<(), Error> {
784 #### Thread-safety
794 fn to_value(&self) -> Value;
798 It's the generic trait bound that macros capturing structured values can require.
802 The `ToValue` trait is object-safe.
810 fn to_value(&self) -> Value {
819 fn to_value(&self) -> Value {
825 fn to_value(&self) -> Value {
831 fn to_value(&self) -> Value {
837 fn to_value(&self) -> Value {
843 fn to_value(&self) -> Value {
849 fn to_value(&self) -> Value {
855 fn to_value(&self) -> Value {
861 fn to_value(&self) -> Value {
867 fn to_value(&self) -> Value {
873 fn to_value(&self) -> Value {
879 fn to_value(&self) -> Value {
885 fn to_value(&self) -> Value {
891 fn to_value(&self) -> Value {
897 fn to_value(&self) -> Value {
903 fn to_value(&self) -> Value {
909 fn to_value(&self) -> Value {
918 fn to_value(&self) -> Value {
927 fn to_value(&self) -> Value {
933 fn to_value(&self) -> Value {
943 fn to_value(&self) -> Value {
949 fn to_value(&self) -> Value {
955 fn to_value(&self) -> Value {
961 fn to_value(&self) -> Value {
967 fn to_value(&self) -> Value {
973 fn to_value(&self) -> Value {
983 A `Key` is a short-lived structure that can be represented as a UTF-8 string:
989 pub fn from_str(key: &'k (impl Borrow<str> + ?Sized)) -> Self {
993 pub fn as_str(&self) -> &str {
1041 pub fn from_owned(key: impl Into<String>) -> Self {
1047 fn to_key(&self) -> Key {
1081 #### Thread-safety
1087 … optional index into a source. This could be used to retrieve a specific key-value pair more effic…
1095 fn to_key(&self) -> Key;
1101 The `ToKey` trait is object-safe.
1112 fn to_key(&self) -> Key {
1118 fn to_key(&self) -> Key {
1124 fn to_key(&self) -> Key {
1132 …:iter::Iterator` for key-value pairs. It gives us a way to inspect some arbitrary collection of ke…
1136 fn visit<'kvs>(&'kvs self, visitor: &mut impl Visitor<'kvs>) -> Result<(), Error>;
1138 fn erase(&self) -> ErasedSource
1145 fn get<'kvs, Q>(&'kvs self, key: Q) -> Option<Value<'kvs>>
1152 fn by_ref(&self) -> &Self {
1156 fn chain<KVS>(self, other: KVS) -> Chained<Self, KVS>
1163 fn for_each<F>(self, f: F) -> Result<(), Error>
1171 fn try_for_each<F, E>(self, f: F) -> Result<(), Error>
1174 F: FnMut(Key, Value) -> Result<(), E>,
1181 fn as_map(self) -> AsMap<Self>
1189 fn as_seq(self) -> AsSeq<Self>
1198 …lity point for structured logging in the `log` crate that's used by the `Record` type. It doesn't …
1204 - `by_ref` to get a reference to a `Source` within a method chain.
1205 - `chain` to concatenate one source with another. This is useful for composing implementations of `…
1206 - `get` to try find the value associated with a key. This is useful for well-defined key-value pair…
1207 - `for_each` to execute some closure over all key-value pairs. This is a convenient way to do somet…
1208 - `try_for_each` is like `for_each`, but takes a fallible closure.
1209 - `as_map` to get a serializable map. This is a convenient way to serialize key-value pairs without…
1210 - `as_seq` is like `as_map`, but for serializing as a sequence of tuples.
1212 None of these methods are required for the core API. They're helpful tools for working with key-val…
1216 …not object-safe because of the provided adapter methods not being object-safe. The only required m…
1219 #[derive(Clone, Copy)]
1223 pub fn erased(kvs: &'a impl Source) -> Self {
1227 pub fn empty() -> Self {
1233 fn visit<'kvs>(&'kvs self, visitor: &mut impl Visitor<'kvs>) -> Result<(), Error> {
1237 fn get<'kvs, Q>(&'kvs self, key: Q) -> Option<Value<'kvs>>
1246 fn erased_visit<'kvs>(&'kvs self, visitor: &mut impl Visitor<'kvs>) -> Result<(), Error>;
1247 fn erased_get<'kvs>(&'kvs self, key: Key) -> Option<Value<'kvs>>;
1254 fn erased_visit<'kvs>(&'kvs self, visitor: &mut impl Visitor<'kvs>) -> Result<(), Error> {
1258 fn erased_get<'kvs>(&'kvs self, key: Key) -> Option<Value<'kvs>> {
1266 A `Source` containing a single key-value pair is implemented for a tuple of a key and value:
1274 fn visit<'kvs>(&'kvs self, visitor: &mut impl Visitor<'kvs>) -> Result<(), Error>
1285 fn visit<'kvs>(&'kvs self, visitor: &mut impl Visitor<'kvs>) -> Result<(), Error> {
1299 fn visit<'kvs>(&'kvs self, visitor: &mut impl Visitor<'kvs>) -> Result<(), Error> {
1305 fn visit<'kvs>(&'kvs self, visitor: &mut impl Visitor<'kvs>) -> Result<(), Error> {
1311 fn visit<'kvs>(&'kvs self, visitor: &mut impl Visitor<'kvs>) -> Result<(), Error> {
1317 fn visit<'kvs>(&'kvs self, visitor: &mut impl Visitor<'kvs>) -> Result<(), Error> {
1327 fn visit<'kvs>(&'kvs self, visitor: &mut impl Visitor<'kvs>) -> Result<(), Error>
1336 fn get<'kvs, Q>(&'kvs self, key: Q) -> Option<Value<'kvs>>
1349 fn visit<'kvs>(&'kvs self, visitor: &mut impl Visitor<'kvs>) -> Result<(), Error>
1358 fn get<'kvs, Q>(&'kvs self, key: Q) -> Option<Value<'kvs>>
1371 …structured data. The `Source` trait is the point where having some way to convert from a borrowed …
1379 fn to_owned(&self) -> OwnedSource {
1384 #[derive(Clone)]
1388 pub fn new(impl Into<Arc<impl Source + Send + Sync>>) -> Self {
1392 pub fn collect(impl Source) -> Self {
1405 The `Visitor` trait used by `Source` can visit a single key-value pair:
1409 fn visit_pair(&mut self, k: Key<'kvs>, v: Value<'kvs>) -> Result<(), Error>;
1420 …ys and values as it sees them. It may also do other work, like sorting or de-duplicating them. Ope…
1424 There aren't any public implementors of `Visitor` in the `log` crate. Other crates that use key-val…
1428 The `Visitor` trait is object-safe.
1432 Structured key-value pairs can be set on a `RecordBuilder` as an implementation of a `Source`:
1436 pub fn key_values(&mut self, kvs: ErasedSource<'a>) -> &mut RecordBuilder<'a> {
1443 These key-value pairs can then be accessed on the built `Record`:
1446 #[derive(Clone, Debug)]
1454 pub fn key_values(&self) -> ErasedSource {
1455 self.kvs.clone()
1460 ## A minimal key-values API
1466 pub fn key_values(&mut self, kvs: ErasedSource<'a>) -> &mut RecordBuilder<'a> {
1472 pub fn key_values(&self) -> ErasedSource {
1480 pub fn msg(msg: &'static str) -> Self {
1503 pub fn custom(err: impl fmt::Display) -> Self {
1526 pub fn from_debug(value: &'v impl Debug) -> Self {
1542 pub fn from_str(key: &'k (impl Borrow<str> + ?Sized)) -> Self {
1546 pub fn as_str(&self) -> &str {
1552 fn visit<'kvs>(&'kvs self, visitor: &mut impl Visitor<'kvs>) -> Result<(), Error>;
1554 fn erase(&self) -> ErasedSource
1565 fn visit_pair(&mut self, k: Key<'kvs>, v: Value<'kvs>) -> Result<(), Error>;
1571 …structured logging. Instead, `log` will rely on new `log!` macro implementations and existing stru…
1573 …plementations of the `log!` macros that are structured by design, rather than attempting to graft …
1578 ## Supporting structured logging at all
1580 Structured logging is a non-trivial feature to support. It adds complexity and overhead to the `log…
1584 …th its underlying structure retained. This is done through an internal one-to-one integration from…
1586 …se it's purpose-built for serializing values in structured logging. The choice of `serde` as a sup…
1590 The one-to-one bridge between serialization frameworks within `log` makes the effort needed to supp…
1606 fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error>;
1610 fn u64(&mut self, v: u64) -> Result<(), Error>;
1611 fn i64(&mut self, v: i64) -> Result<(), Error>;
1619 …structured values is a complex, necessary, but not primary function of `log`, so it should avoid o…
1625 …t end-user experience for existing users of that framework when interacting with the `log!` macros…
1627 …-ground between baking in a specific framework and only offering a contract without any direct sup…
1630 [prior-art]: #prior-art
1632 Structured logging is a paradigm that's supported by logging frameworks in many language ecosystems.
1636 …structured logging framework for Rust. Its API predates a stable `serde` crate so it defines its o…
1638 …`, but doesn't directly support distinguishing between owned or borrowed key-value pairs. Everythi…
1642 …ogrus` library is a structured logging framework for Go. It uses a similar separation of the textu…
1646 …structured key-value pairs. Instead of logging a rendered message and separate bag of structured d…
1651 [unresolved-questions]: #unresolved-questions