readme.md
1# asn1crypto
2
3A fast, pure Python library for parsing and serializing ASN.1 structures.
4
5 - [Features](#features)
6 - [Why Another Python ASN.1 Library?](#why-another-python-asn1-library)
7 - [Related Crypto Libraries](#related-crypto-libraries)
8 - [Current Release](#current-release)
9 - [Dependencies](#dependencies)
10 - [Installation](#installation)
11 - [License](#license)
12 - [Security Policy](#security-policy)
13 - [Documentation](#documentation)
14 - [Continuous Integration](#continuous-integration)
15 - [Testing](#testing)
16 - [Development](#development)
17 - [CI Tasks](#ci-tasks)
18
19[](https://github.com/wbond/asn1crypto/actions?workflow=CI)
20[](https://circleci.com/gh/wbond/asn1crypto)
21[](https://pypi.org/project/asn1crypto/)
22
23## Features
24
25In addition to an ASN.1 BER/DER decoder and DER serializer, the project includes
26a bunch of ASN.1 structures for use with various common cryptography standards:
27
28| Standard | Module | Source |
29| ---------------------- | ------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
30| X.509 | [`asn1crypto.x509`](asn1crypto/x509.py) | [RFC 5280](https://tools.ietf.org/html/rfc5280) |
31| CRL | [`asn1crypto.crl`](asn1crypto/crl.py) | [RFC 5280](https://tools.ietf.org/html/rfc5280) |
32| CSR | [`asn1crypto.csr`](asn1crypto/csr.py) | [RFC 2986](https://tools.ietf.org/html/rfc2986), [RFC 2985](https://tools.ietf.org/html/rfc2985) |
33| OCSP | [`asn1crypto.ocsp`](asn1crypto/ocsp.py) | [RFC 6960](https://tools.ietf.org/html/rfc6960) |
34| PKCS#12 | [`asn1crypto.pkcs12`](asn1crypto/pkcs12.py) | [RFC 7292](https://tools.ietf.org/html/rfc7292) |
35| PKCS#8 | [`asn1crypto.keys`](asn1crypto/keys.py) | [RFC 5208](https://tools.ietf.org/html/rfc5208) |
36| PKCS#1 v2.1 (RSA keys) | [`asn1crypto.keys`](asn1crypto/keys.py) | [RFC 3447](https://tools.ietf.org/html/rfc3447) |
37| DSA keys | [`asn1crypto.keys`](asn1crypto/keys.py) | [RFC 3279](https://tools.ietf.org/html/rfc3279) |
38| Elliptic curve keys | [`asn1crypto.keys`](asn1crypto/keys.py) | [SECG SEC1 V2](http://www.secg.org/sec1-v2.pdf) |
39| PKCS#3 v1.4 | [`asn1crypto.algos`](asn1crypto/algos.py) | [PKCS#3 v1.4](ftp://ftp.rsasecurity.com/pub/pkcs/ascii/pkcs-3.asc) |
40| PKCS#5 v2.1 | [`asn1crypto.algos`](asn1crypto/algos.py) | [PKCS#5 v2.1](http://www.emc.com/collateral/white-papers/h11302-pkcs5v2-1-password-based-cryptography-standard-wp.pdf) |
41| CMS (and PKCS#7) | [`asn1crypto.cms`](asn1crypto/cms.py) | [RFC 5652](https://tools.ietf.org/html/rfc5652), [RFC 2315](https://tools.ietf.org/html/rfc2315) |
42| TSP | [`asn1crypto.tsp`](asn1crypto/tsp.py) | [RFC 3161](https://tools.ietf.org/html/rfc3161) |
43| PDF signatures | [`asn1crypto.pdf`](asn1crypto/pdf.py) | [PDF 1.7](http://wwwimages.adobe.com/content/dam/Adobe/en/devnet/pdf/pdfs/PDF32000_2008.pdf) |
44
45## Why Another Python ASN.1 Library?
46
47Python has long had the [pyasn1](https://pypi.org/project/pyasn1/) and
48[pyasn1_modules](https://pypi.org/project/pyasn1-modules/) available for
49parsing and serializing ASN.1 structures. While the project does include a
50comprehensive set of tools for parsing and serializing, the performance of the
51library can be very poor, especially when dealing with bit fields and parsing
52large structures such as CRLs.
53
54After spending extensive time using *pyasn1*, the following issues were
55identified:
56
57 1. Poor performance
58 2. Verbose, non-pythonic API
59 3. Out-dated and incomplete definitions in *pyasn1-modules*
60 4. No simple way to map data to native Python data structures
61 5. No mechanism for overridden universal ASN.1 types
62
63The *pyasn1* API is largely method driven, and uses extensive configuration
64objects and lowerCamelCase names. There were no consistent options for
65converting types of native Python data structures. Since the project supports
66out-dated versions of Python, many newer language features are unavailable
67for use.
68
69Time was spent trying to profile issues with the performance, however the
70architecture made it hard to pin down the primary source of the poor
71performance. Attempts were made to improve performance by utilizing unreleased
72patches and delaying parsing using the `Any` type. Even with such changes, the
73performance was still unacceptably slow.
74
75Finally, a number of structures in the cryptographic space use universal data
76types such as `BitString` and `OctetString`, but interpret the data as other
77types. For instance, signatures are really byte strings, but are encoded as
78`BitString`. Elliptic curve keys use both `BitString` and `OctetString` to
79represent integers. Parsing these structures as the base universal types and
80then re-interpreting them wastes computation.
81
82*asn1crypto* uses the following techniques to improve performance, especially
83when extracting one or two fields from large, complex structures:
84
85 - Delayed parsing of byte string values
86 - Persistence of original ASN.1 encoded data until a value is changed
87 - Lazy loading of child fields
88 - Utilization of high-level Python stdlib modules
89
90While there is no extensive performance test suite, the
91`CRLTests.test_parse_crl` test case was used to parse a 21MB CRL file on a
92late 2013 rMBP. *asn1crypto* parsed the certificate serial numbers in just
93under 8 seconds. With *pyasn1*, using definitions from *pyasn1-modules*, the
94same parsing took over 4,100 seconds.
95
96For smaller structures the performance difference can range from a few times
97faster to an order of magnitude or more.
98
99## Related Crypto Libraries
100
101*asn1crypto* is part of the modularcrypto family of Python packages:
102
103 - [asn1crypto](https://github.com/wbond/asn1crypto)
104 - [oscrypto](https://github.com/wbond/oscrypto)
105 - [csrbuilder](https://github.com/wbond/csrbuilder)
106 - [certbuilder](https://github.com/wbond/certbuilder)
107 - [crlbuilder](https://github.com/wbond/crlbuilder)
108 - [ocspbuilder](https://github.com/wbond/ocspbuilder)
109 - [certvalidator](https://github.com/wbond/certvalidator)
110
111## Current Release
112
1131.5.0 - [changelog](changelog.md)
114
115## Dependencies
116
117Python 2.6, 2.7, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8, 3.9, 3.10 or pypy. *No third-party
118packages required.*
119
120## Installation
121
122```bash
123pip install asn1crypto
124```
125
126## License
127
128*asn1crypto* is licensed under the terms of the MIT license. See the
129[LICENSE](LICENSE) file for the exact license text.
130
131## Security Policy
132
133The security policies for this project are covered in
134[SECURITY.md](https://github.com/wbond/asn1crypto/blob/master/SECURITY.md).
135
136## Documentation
137
138The documentation for *asn1crypto* is composed of tutorials on basic usage and
139links to the source for the various pre-defined type classes.
140
141### Tutorials
142
143 - [Universal Types with BER/DER Decoder and DER Encoder](docs/universal_types.md)
144 - [PEM Encoder and Decoder](docs/pem.md)
145
146### Reference
147
148 - [Universal types](asn1crypto/core.py), `asn1crypto.core`
149 - [Digest, HMAC, signed digest and encryption algorithms](asn1crypto/algos.py), `asn1crypto.algos`
150 - [Private and public keys](asn1crypto/keys.py), `asn1crypto.keys`
151 - [X509 certificates](asn1crypto/x509.py), `asn1crypto.x509`
152 - [Certificate revocation lists (CRLs)](asn1crypto/crl.py), `asn1crypto.crl`
153 - [Online certificate status protocol (OCSP)](asn1crypto/ocsp.py), `asn1crypto.ocsp`
154 - [Certificate signing requests (CSRs)](asn1crypto/csr.py), `asn1crypto.csr`
155 - [Private key/certificate containers (PKCS#12)](asn1crypto/pkcs12.py), `asn1crypto.pkcs12`
156 - [Cryptographic message syntax (CMS, PKCS#7)](asn1crypto/cms.py), `asn1crypto.cms`
157 - [Time stamp protocol (TSP)](asn1crypto/tsp.py), `asn1crypto.tsp`
158 - [PDF signatures](asn1crypto/pdf.py), `asn1crypto.pdf`
159
160## Continuous Integration
161
162Various combinations of platforms and versions of Python are tested via:
163
164 - [macOS, Linux, Windows](https://github.com/wbond/asn1crypto/actions/workflows/ci.yml) via GitHub Actions
165 - [arm64](https://circleci.com/gh/wbond/asn1crypto) via CircleCI
166
167## Testing
168
169Tests are written using `unittest` and require no third-party packages.
170
171Depending on what type of source is available for the package, the following
172commands can be used to run the test suite.
173
174### Git Repository
175
176When working within a Git working copy, or an archive of the Git repository,
177the full test suite is run via:
178
179```bash
180python run.py tests
181```
182
183To run only some tests, pass a regular expression as a parameter to `tests`.
184
185```bash
186python run.py tests ocsp
187```
188
189### PyPi Source Distribution
190
191When working within an extracted source distribution (aka `.tar.gz`) from
192PyPi, the full test suite is run via:
193
194```bash
195python setup.py test
196```
197
198### Package
199
200When the package has been installed via pip (or another method), the package
201`asn1crypto_tests` may be installed and invoked to run the full test suite:
202
203```bash
204pip install asn1crypto_tests
205python -m asn1crypto_tests
206```
207
208## Development
209
210To install the package used for linting, execute:
211
212```bash
213pip install --user -r requires/lint
214```
215
216The following command will run the linter:
217
218```bash
219python run.py lint
220```
221
222Support for code coverage can be installed via:
223
224```bash
225pip install --user -r requires/coverage
226```
227
228Coverage is measured by running:
229
230```bash
231python run.py coverage
232```
233
234To change the version number of the package, run:
235
236```bash
237python run.py version {pep440_version}
238```
239
240To install the necessary packages for releasing a new version on PyPI, run:
241
242```bash
243pip install --user -r requires/release
244```
245
246Releases are created by:
247
248 - Making a git tag in [PEP 440](https://www.python.org/dev/peps/pep-0440/#examples-of-compliant-version-schemes) format
249 - Running the command:
250
251 ```bash
252 python run.py release
253 ```
254
255Existing releases can be found at https://pypi.org/project/asn1crypto/.
256
257## CI Tasks
258
259A task named `deps` exists to download and stage all necessary testing
260dependencies. On posix platforms, `curl` is used for downloads and on Windows
261PowerShell with `Net.WebClient` is used. This configuration sidesteps issues
262related to getting pip to work properly and messing with `site-packages` for
263the version of Python being used.
264
265The `ci` task runs `lint` (if flake8 is available for the version of Python) and
266`coverage` (or `tests` if coverage is not available for the version of Python).
267If the current directory is a clean git working copy, the coverage data is
268submitted to codecov.io.
269
270```bash
271python run.py deps
272python run.py ci
273```
274
){kind=link}
){kind=link}
){kind=link}