# Upgrading to nom 4.0 The nom 4.0 is a nearly complete rewrite of nom's internal structures, along with a cleanup of a lot of parser and combinators whose semantics were unclear. Upgrading from previous nom versions can require a lot of changes, especially if you have a lot of unit tests. But most of those changes are pretty straightforward. ## Changes in internal structures Previous versions of nom all generated parsers with the following signature: ```rust fn parser(input: I) -> IResult { ... } ``` With the following definition for `IResult`: ```rust pub enum IResult { /// remaining input, result value Done(I,O), /// indicates the parser encountered an error. E is a custom error type you can redefine Error(Err), /// Incomplete contains a Needed, an enum that can represent a known quantity of input data, or unknown Incomplete(Needed) } pub enum Needed { /// needs more data, but we do not know how much Unknown, /// contains the required total data size Size(usize) } // if the "verbose-errors" feature is not active pub type Err = ErrorKind; // if the "verbose-errors" feature is active pub enum Err{ /// An error code, represented by an ErrorKind, which can contain a custom error code represented by E Code(ErrorKind), /// An error code, and the next error Node(ErrorKind, Vec>), /// An error code, and the input position Position(ErrorKind, P), /// An error code, the input position and the next error NodePosition(ErrorKind, P, Vec>) } ``` The new design uses the `Result` type from the standard library: ```rust pub type IResult = Result<(I, O), Err>; pub enum Err { /// There was not enough data Incomplete(Needed), /// The parser had an error (recoverable) Error(Context), /// The parser had an unrecoverable error Failure(Context), } pub enum Needed { /// needs more data, but we do not know how much Unknown, /// contains the required additional data size Size(usize) } // if the "verbose-errors" feature is inactive pub enum Context { Code(I, ErrorKind), } // if the "verbose-errors" feature is active pub enum Context { Code(I, ErrorKind), List(Vec<(I, ErrorKind)>), } ``` With this new design, the `Incomplete` case is now part of the error case, and we get a `Failure` case representing an unrecoverable error (combinators like `alt!` will not try another branch). The verbose error management is now a truly additive feature above the simple one (it adds a case to the `Context` enum). Error management types also get smaller and more efficient. We can now return the related input as part of the error in all cases. All of this will likely not affect your existing parsers, but require changes to the surrounding code that manipulates parser results. ## Faster parsers, new memory layout but with lower footprint These changes keep the same memory footprint in simple errors mode, and reduce it in verbose errors: | size of `IResult<&[u8], &[u8]>` | simple errors | verbose errors | |---|---|---| | nom 3 | 40 bytes | 64 bytes | | nom 4 | 40 bytes | 48 bytes | In addition, [parsers are faster in nom 4 than in nom 3](https://github.com/Geal/nom/issues/356#issuecomment-333816834). This change is justified. ## Replacing parser result matchers Whenever you use pattern matching on the result of a parser, or compare it to another parser result (like in a unit test), you will have to perform the following changes: For the correct result case: ```rust IResult::Done(i, o) // becomes Ok((i, o)) ``` For the error case (note that argument position for `error_position` and other sibling macros was changed for the sake of consistency with the rest of the code): ```rust IResult::Error(error_position!(ErrorKind::OneOf, input)), // becomes Err(Err::Error(error_position!(input, ErrorKind::OneOf))) ``` ```rust IResult::Incomplete(Needed::Size(1)) // becomes Err(Err::Incomplete(Needed::Size(1))) ``` For pattern matching, you now need to handle the `Failure` case as well, which works like the error case: ```rust match result { Ok((remaining, value)) => { ... }, Err(Err::Incomplete(needed)) => { ... }, Err(Err::Error(e)) | Err(Err::Failure(e)) => { ... } } ``` ## Errors on `Incomplete` data size calculation In previous versions, `Needed::Size(sz)` indicated the total needed data size (counting the actual input). Now it only returns the additional data needed, so the values will have changed. ## New trait for input types nom allows other input types than `&[u8]` and `&str`, as long as they implement a set of traits that are used everywhere in nom. This version introduces the `AtEof` trait: ```rust pub trait AtEof { fn at_eof(&self) -> bool; } ``` This trait allows the input value to indicate whether there can be more input coming later (buffering data from a file, or waiting for network data). ## Dealing with `Incomplete` usage nom's parsers are designed to work around streaming issues: if there is not enough data to decide, a parser will return `Incomplete` instead of returning a partial value that might be false. As an example, if you want to parse alphabetic characters then digits, when you get the whole input `abc123;`, the parser will return `abc` for alphabetic characters, and `123` for the digits, and `;` as remaining input. But if you get that input in chunks, like `ab` then `c123;`, the alphabetic characters parser will return `Incomplete`, because it does not know if there will be more matching characters afterwards. If it returned `ab` directly, the digit parser would fail on the rest of the input, even though the input had the valid format. For some users, though, the input will never be partial (everything could be loaded in memory at once), and the solution in nom 3 and before was to wrap parts of the parsers with the `complete!()` combinator that transforms `Incomplete` in `Error`. nom 4 is much stricter about the behaviour with partial data, but provides better tools to deal with it. Thanks to the new `AtEof` trait for input types, nom now provides the `CompleteByteSlice(&[u8])` and `CompleteStr(&str)` input types, for which the `at_eof()` method always returns true. With these types, no need to put a `complete!()` combinator everywhere, you can just apply those types like this: ```rust named!(parser<&str,ReturnType>, ... ); // becomes named!(parser, ... ); ``` ```rust named!(parser<&str,&str>, ... ); // becomes named!(parser, ... ); ``` ```rust named!(parser, ... ); // becomes named!(parser, ... ); ``` And as an example, for a unit test: ```rust assert_eq!(parser("abcd123"), Ok(("123", "abcd")); // becomes assert_eq!(parser(CompleteStr("abcd123")), Ok((CompleteStr("123"), CompleteStr("abcd"))); ``` These types allow you to correctly handle cases like text formats for which there might be a last empty line or not, as seen in [one of the examples](https://github.com/Geal/nom/blob/87d837006467aebcdb0c37621da874a56c8562b5/tests/multiline.rs). If those types feel a bit long to write everywhere in the parsers, it's possible to alias them like this: ```rust use nom::types::CompleteByteSlice as Input; ``` ## Custom error types Custom error types caused a lot of type inference issues in previous nom versions. Now error types are automatically converted as needed. If you want to set up a custom error type, you now need to implement `std::convert::From` for this type. ## Producers and consumers Producers and consumers were removed in nom 4. That feature was too hard to integrate in code that deals with IO.