Python Misc subdirectory
========================

This directory contains files that wouldn't fit in elsewhere.  Some
documents are only of historic importance.

Files found here
----------------

ACKS                    Acknowledgements
gdbinit                 Handy stuff to put in your .gdbinit file, if you use gdb
HISTORY                 News from previous releases -- oldest last
indent.pro              GNU indent profile approximating my C style
NEWS                    News for this release (for some meaning of "this")
Porting                 Mini-FAQ on porting to new platforms
python-config.in        Python script template for python-config
python.man              UNIX man page for the python interpreter
python.pc.in            Package configuration info template for pkg-config
python-wing*.wpr        Wing IDE project file
README                  The file you're reading now
README.AIX              Information about using Python on AIX
README.coverity         Information about running Coverity's Prevent on Python
README.valgrind         Information for Valgrind users, see valgrind-python.supp
SpecialBuilds.txt       Describes extra symbols you can set for debug builds
svnmap.txt              Map of old SVN revs and branches to hg changeset ids,
                        help history-digging
valgrind-python.supp    Valgrind suppression file, see README.valgrind
vgrindefs               Python configuration for vgrind (a generic pretty printer)

This documentation tries to help people who intend to use Python on
AIX.

There used to be many issues with Python on AIX, but the major ones
have been corrected for version 3.2, so that Python should now work
rather well on this platform. The remaining known issues are listed in
this document.


======================================================================
			   Compiling Python
----------------------------------------------------------------------

You can compile Python with gcc or the native AIX compiler. The native
compiler used to give better performances on this system with older
versions of Python.  With Python 3.2 it may not be the case anymore,
as this compiler does not allow compiling Python with computed gotos.
Some benchmarks need to be done.

Compiling with gcc:

cd Python-3.2
CC=gcc OPT="-O2" ./configure --enable-shared
make

There are various aliases for the native compiler.  The recommended
alias for compiling Python is 'xlc_r', which provides a better level of
compatibility and handles thread initialization properly.

It is a good idea to add the '-qmaxmem=70000' option, otherwise the
compiler considers various files too complex to optimize.

Compiling with xlc:

cd Python-3.2
CC=xlc_r OPT="-O2 -qmaxmem=70000" ./configure --without-computed-gotos --enable-shared
make


======================================================================
			  Memory Limitations
----------------------------------------------------------------------

Note: this section may not apply when compiling Python as a 64 bit
application.

By default on AIX each program gets one segment register for its data
segment. As each segment register covers 256 MiB, a Python program that
would use more than 256 MiB will raise a MemoryError.  The standard
Python test suite is one such application.

To allocate more segment registers to Python, you must use the linker
option -bmaxdata or the ldedit tool to specify the number of bytes you
need in the data segment.

For example, if you want to allow 512 MiB of memory for Python (this is
enough for the test suite to run without MemoryErrors), you should run
the following command at the end of compilation:

ldedit -b maxdata:0x20000000 ./python

You can allow up to 2 GiB of memory for Python by using the value
0x80000000 for maxdata.

It is also possible to go beyond 2 GiB of memory by activating Large
Page Use. You should consult the IBM documentation if you need to use
this option. You can also follow the discussion of this problem
in issue 11212 at bugs.python.org.

http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=/com.ibm.aix.cmds/doc/aixcmds3/ldedit.htm


======================================================================
			     Known issues
----------------------------------------------------------------------

Those issues are currently affecting Python on AIX:

* Python has not been fully tested on AIX when compiled as a 64 bit
  application.

* issue 3526: the memory used by a Python process will never be
  released to the system. If you have a Python application on AIX that
  uses a lot of memory, you should read this issue and you may
  consider using the provided patch that implements a custom malloc
  implementation

* issue 11192: test_socket fails

* issue 11190: test_locale fails

* issue 11193: test_subprocess fails

* issue 9920: minor arithmetic issues in cmath

* issue 11215: test_fileio fails



======================================================================
		Implementation details for developers
----------------------------------------------------------------------

Python and python modules can now be built as shared libraries on AIX
as usual.

AIX shared libraries require that an "export" and "import" file be
provided at compile time to list all extern symbols which may be
shared between modules.  The "export" file (named python.exp) for the
modules and the libraries that belong to the Python core is created by
the "makexp_aix" script before performing the link of the python
binary. It lists all global symbols (exported during the link) of the
modules and the libraries that make up the python executable.

When shared library modules (.so files) are made, a second shell
script is invoked.  This script is named "ld_so_aix" and is also
provided with the distribution in the Modules subdirectory.  This
script acts as an "ld" wrapper which hides the explicit management of
"export" and "import" files; it adds the appropriate arguments (in the
appropriate order) to the link command that creates the shared module.
Among other things, it specifies that the "python.exp" file is an
"import" file for the shared module.

This mechanism should be transparent.

Coverity has a static analysis tool (Prevent) which is similar to Klocwork.
They run their tool on the Python source code (SVN head) on a daily basis.
The results are available at:

     http://scan.coverity.com/

About 20 people have access to the analysis reports.  Other
people can be added by request.

Prevent was first run on the Python 2.5 source code in March 2006.
There were originally about 100 defects reported.  Some of these
were false positives.  Over 70 issues were uncovered.

Each warning has a unique id and comments that can be made on it.
When checking in changes due to a warning, the unique id
as reported by the tool was added to the SVN commit message.

False positives were annotated so that the comments can
be reviewed and reversed if the analysis was incorrect.

Contact python-dev@python.org for more information.

This document describes some caveats about the use of Valgrind with
Python.  Valgrind is used periodically by Python developers to try
to ensure there are no memory leaks or invalid memory reads/writes.

If you want to enable valgrind support in Python, you will need to
configure Python --with-valgrind option or an older option
--without-pymalloc.

UPDATE: Python 3.6 now supports PYTHONMALLOC=malloc environment variable which
can be used to force the usage of the malloc() allocator of the C library.

If you don't want to read about the details of using Valgrind, there
are still two things you must do to suppress the warnings.  First,
you must use a suppressions file.  One is supplied in
Misc/valgrind-python.supp.  Second, you must uncomment the lines in
Misc/valgrind-python.supp that suppress the warnings for PyObject_Free and
PyObject_Realloc.

If you want to use Valgrind more effectively and catch even more
memory leaks, you will need to configure python --without-pymalloc.
PyMalloc allocates a few blocks in big chunks and most object
allocations don't call malloc, they use chunks doled about by PyMalloc
from the big blocks.  This means Valgrind can't detect
many allocations (and frees), except for those that are forwarded
to the system malloc.  Note: configuring python --without-pymalloc
makes Python run much slower, especially when running under Valgrind.
You may need to run the tests in batches under Valgrind to keep
the memory usage down to allow the tests to complete.  It seems to take
about 5 times longer to run --without-pymalloc.

Apr 15, 2006:
  test_ctypes causes Valgrind 3.1.1 to fail (crash).
  test_socket_ssl should be skipped when running valgrind.
	The reason is that it purposely uses uninitialized memory.
	This causes many spurious warnings, so it's easier to just skip it.


Details:
--------
Python uses its own small-object allocation scheme on top of malloc,
called PyMalloc.

Valgrind may show some unexpected results when PyMalloc is used.
Starting with Python 2.3, PyMalloc is used by default.  You can disable
PyMalloc when configuring python by adding the --without-pymalloc option.
If you disable PyMalloc, most of the information in this document and
the supplied suppressions file will not be useful.  As discussed above,
disabling PyMalloc can catch more problems.

PyMalloc uses 256KB chunks of memory, so it can't detect anything
wrong within these blocks.  For that reason, compiling Python
--without-pymalloc usually increases the usefulness of other tools.

If you use valgrind on a default build of Python,  you will see
many errors like:

        ==6399== Use of uninitialised value of size 4
        ==6399== at 0x4A9BDE7E: PyObject_Free (obmalloc.c:711)
        ==6399== by 0x4A9B8198: dictresize (dictobject.c:477)

These are expected and not a problem.  Tim Peters explains
the situation:

        PyMalloc needs to know whether an arbitrary address is one
	that's managed by it, or is managed by the system malloc.
	The current scheme allows this to be determined in constant
	time, regardless of how many memory areas are under pymalloc's
	control.

        The memory pymalloc manages itself is in one or more "arenas",
	each a large contiguous memory area obtained from malloc.
	The base address of each arena is saved by pymalloc
	in a vector.  Each arena is carved into "pools", and a field at
	the start of each pool contains the index of that pool's arena's
	base address in that vector.

        Given an arbitrary address, pymalloc computes the pool base
	address corresponding to it, then looks at "the index" stored
	near there.  If the index read up is out of bounds for the
	vector of arena base addresses pymalloc maintains, then
	pymalloc knows for certain that this address is not under
	pymalloc's control.  Otherwise the index is in bounds, and
	pymalloc compares

            the arena base address stored at that index in the vector

        to

            the arbitrary address pymalloc is investigating

        pymalloc controls this arbitrary address if and only if it lies
        in the arena the address's pool's index claims it lies in.

        It doesn't matter whether the memory pymalloc reads up ("the
	index") is initialized.  If it's not initialized, then
	whatever trash gets read up will lead pymalloc to conclude
	(correctly) that the address isn't controlled by it, either
	because the index is out of bounds, or the index is in bounds
	but the arena it represents doesn't contain the address.

        This determination has to be made on every call to one of
	pymalloc's free/realloc entry points, so its speed is critical
	(Python allocates and frees dynamic memory at a ferocious rate
	-- everything in Python, from integers to "stack frames",
	lives in the heap).