1 2=========================== 3Grant Proposal for CFFI 1.0 4=========================== 5 6*Accepted by the PSF board on April 4, 2015* 7 8This Grant Proposal is to give a boost towards "CFFI 1.0". Two main 9issues with the current CFFI need to be solved: the difficulties of 10installation, and the potentially large time taken at import. 11 121. The difficulties of installation can be seen from outside by looking 13at various workarounds and 3rd-party documentation that have grown into 14existence. For example, the `setup.py` of projects like cryptography, 15PyNaCl and bcrypt deploys workarounds that are explicitly documented in 16https://caremad.io/2014/11/distributing-a-cffi-project/. 17 182. The time taken at import is excessive in some cases. For example, 19importing `pygame-cffi` on a Raspberry Pi ARM board takes on the order 20of 10 to 20 seconds (and this is the "fast" case where the compiler 21doesn't need to be invoked any more). 22 23 24Technical Overview 25------------------ 26 27"CFFI" is an existing Python project which complements the ctypes, 28SWIG and Cython approaches to ease writing C Extension Modules for 29Python. It has several advantages over the previous approaches, which 30are presented at the start of the documentation at 31http://cffi.readthedocs.org/en/latest/ . It has been very successful 32so far: http://pypi-ranking.info/alltime records almost 7 million 33downloads (for comparison, the #1 of all packages has almost 36 34million downloads). CFFI works on any Python >= 2.6, including 3.x, 35as well as on PyPy. 36 37One problem is that while getting started with CFFI is very easy, the 38installation process of a package that uses CFFI has got its rough 39edges. CFFI (at least in its "verify()" mode) is based on calling the 40C compiler to get information about the exact C types, structures, 41argument types to functions, and so on. The C compiler is invoked 42transparently at run-time, and the results cached. A 43correctly-installed package using CFFI should cache the results at 44installation time, but it can be difficult to ensure that no more 45run-time compiler invocation is needed; doing so requires following 46some extra guidelines or understanding some internal details. (The 47problem is particularly acute on Windows where a typical user might 48not have a proper C compiler installed.) 49 50To fix this, we have in mind adding a different CFFI mode (replacing 51"verify()"), while keeping the access to the underlying C library 52unmodified. In this mode, the code containing the cdef() and verify() 53invocations would be moved to a separate Python source file. Running 54that Python file would produce a dynamically-linked library. There 55would be no caching logic involved; you would need to run it 56explicitly during development whenever you made changes to it, to 57re-generate and re-compile the dynamically-linked library. 58 59When distributed, the same file would be run (once) during 60installation. This can be fully automated in setuptools-based 61setup.py files; alternatively, it can be done in distutils-based 62setup.py files by requiring prior manual installation of CFFI itself. 63 64A major difference with the existing verify() approach would be that 65the ``.so/.dll/.dylib`` file would not be immediately loaded into the 66process; you would load it only from the installed program at 67run-time, and get the ``ffi`` and ``lib`` objects in this way (these 68are the two objects that you use so far to access a C library with 69verify()). 70 71Additionally, this would solve another issue: every import of a large 72CFFI-using package takes a while so far. This is caused by CFFI 73needing to parse again the C source code given in the cdef() (adding a 74run-time dependency to the ``pycparser`` and ``ply`` packages). CFFI 75also computes a CRC to know if it can reuse its cache. In the 76proposed change, all the cdef() code would be pre-parsed and stored in 77the dynamically-linked library, and no CRC would be needed. This 78would massively reduce the import times. 79 80 81Grant objective 82--------------- 83 84The objective is to give a boost towards "CFFI 1.0", which needs to have 85the functionalities described above in order to solve the two main 86issues with the current CFFI: the difficulties of installation, and the 87time taken at import. 88 89Included in the objective: the internal refactorings of CFFI that are 90needed to get it done cleanly. The goal is to avoid simply adding 91another layer on top of the old unchanged CFFI. 92 93This work may happen eventually in any case, but support from the PSF 94would help make it happen sooner rather than later. 95 96 97Grant size 98---------- 99 1002'500 US$ for supporting the development time. This would cover 2.5 101weeks of full-time work at the part-time cost of 25 US$ per hour. 102 103The estimated work time until the CFFI 1.0 release is a bit larger 104than that (I estimate it at roughly 4 weeks), but 2.5 weeks should 105cover all the basics. An extended grant size of 4'000 US$ would be 106appreciated but not required ``:-)`` 107 108 109Grant beneficiaries 110------------------- 111 112Armin Rigo, main author of CFFI, committing 2.5 weeks of full-time 113work. 114 115 116Grant follow-up 117--------------- 118 119I will report on the success of the grant on the CFFI mailing list and 120on the blog I usually post to (the PyPy blog) and mention the PSF as 121providing the grant. The PSF will receive an email pointing to these 122postings once they are out. Moreover a full CFFI 1.0 release should 123follow (likely starting with beta versions); the PSF will receive 124another email pointing to it. 125