Release notes for Valgrind
~~~~~~~~~~~~~~~~~~~~~~~~~~
If you are building a binary package of Valgrind for distribution,
please read README_PACKAGERS.  It contains some important information.

If you are developing Valgrind, please read README_DEVELOPERS.  It contains
some useful information.

For instructions on how to build/install, see the end of this file.

If you have problems, consult the FAQ to see if there are workarounds.


Executive Summary
~~~~~~~~~~~~~~~~~
Valgrind is a framework for building dynamic analysis tools. There are
Valgrind tools that can automatically detect many memory management
and threading bugs, and profile your programs in detail. You can also
use Valgrind to build new tools.

The Valgrind distribution currently includes six production-quality
tools: a memory error detector, two thread error detectors, a cache
and branch-prediction profiler, a call-graph generating cache abd
branch-prediction profiler, and a heap profiler. It also includes
three experimental tools: a heap/stack/global array overrun detector,
a different kind of heap profiler, and a SimPoint basic block vector
generator.

Valgrind is closely tied to details of the CPU, operating system and to
a lesser extent, compiler and basic C libraries. This makes it difficult
to make it portable.  Nonetheless, it is available for the following
platforms:

- X86/Linux
- AMD64/Linux
- PPC32/Linux
- PPC64/Linux
- ARM/Linux
- x86/MacOSX
- AMD64/MacOSX
- S390X/Linux
- MIPS32/Linux
- MIPS64/Linux

Note that AMD64 is just another name for x86_64, and Valgrind runs fine
on Intel processors.  Also note that the core of MacOSX is called
"Darwin" and this name is used sometimes.

Valgrind is licensed under the GNU General Public License, version 2.
Read the file COPYING in the source distribution for details.

However: if you contribute code, you need to make it available as GPL
version 2 or later, and not 2-only.


Documentation
~~~~~~~~~~~~~
A comprehensive user guide is supplied.  Point your browser at
$PREFIX/share/doc/valgrind/manual.html, where $PREFIX is whatever you
specified with --prefix= when building.


Building and installing it
~~~~~~~~~~~~~~~~~~~~~~~~~~
To install from the Subversion repository :

  0. Check out the code from SVN, following the instructions at
     http://www.valgrind.org/downloads/repository.html.

  1. cd into the source directory.

  2. Run ./autogen.sh to setup the environment (you need the standard
     autoconf tools to do so).

  3. Continue with the following instructions...

To install from a tar.bz2 distribution:

  4. Run ./configure, with some options if you wish.  The only interesting
     one is the usual --prefix=/where/you/want/it/installed.

  5. Run "make".

  6. Run "make install", possibly as root if the destination permissions
     require that.

  7. See if it works.  Try "valgrind ls -l".  Either this works, or it
     bombs out with some complaint.  In that case, please let us know
     (see www.valgrind.org).

Important!  Do not move the valgrind installation into a place
different from that specified by --prefix at build time.  This will
cause things to break in subtle ways, mostly when Valgrind handles
fork/exec calls.


The Valgrind Developers

Status
~~~~~~

As of Jan 2014 the trunk contains a port to AArch64 ARMv8 -- loosely,
the 64-bit ARM architecture.  Currently it supports integer and FP
instructions and can run anything generated by gcc-4.8.2 -O3.  The
port is under active development.

Current limitations, as of mid-May 2014.

* limited support of vector (SIMD) instructions.  Initial target is
  support for instructions created by gcc-4.8.2 -O3
  (via autovectorisation).  This is complete.

* Integration with the built in GDB server:
   - works ok (breakpoint, attach to a process blocked in a syscall, ...)
   - still to do:
      arm64 xml register description files (allowing shadow registers
                                            to be looked at).
      cpsr transfer to/from gdb to be looked at (see also arm equivalent code)

* limited syscall support

There has been extensive testing of the baseline simulation of integer
and FP instructions.  Memcheck is also believed to work, at least for
small examples.  Other tools appear to at least not crash when running
/bin/date.

Enough syscalls and instructions are supported for substantial
programs to work.  Firefox 26 is able to start up and quit.  The noise
level from Memcheck is low enough to make it practical to use for real
debugging.


Building
~~~~~~~~

You could probably build it directly on a target OS, using the normal
non-cross scheme

  ./autogen.sh ; ./configure --prefix=.. ; make ; make install

Development so far was however done by cross compiling, viz:

  export CC=aarch64-linux-gnu-gcc
  export LD=aarch64-linux-gnu-ld
  export AR=aarch64-linux-gnu-ar

  ./autogen.sh
  ./configure --prefix=`pwd`/Inst --host=aarch64-unknown-linux \
              --enable-only64bit
  make -j4
  make -j4 install

Doing this assumes that the install path (`pwd`/Inst) is valid on
both host and target, which isn't normally the case.  To avoid
this limitation, do instead:

  ./configure --prefix=/install/path/on/target \
              --host=aarch64-unknown-linux \
              --enable-only64bit
  make -j4
  make -j4 install DESTDIR=/a/temp/dir/on/host
  # and then copy the contents of DESTDIR to the target.

See README.android for more examples of cross-compile building.


Implementation tidying-up/TODO notes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

UnwindStartRegs -- what should that contain?


vki-arm64-linux.h: vki_sigaction_base
I really don't think that __vki_sigrestore_t sa_restorer
should be present.  Adding it surely puts sa_mask at a wrong
offset compared to (kernel) reality.  But not having it causes
compilation of m_signals.c to fail in hard to understand ways,
so adding it temporarily.


m_trampoline.S: what's the unexecutable-insn value? 0xFFFFFFFF
is there at the moment, but 0x00000000 is probably what it should be.
Also, fix indentation/tab-vs-space stuff


./include/vki/vki-arm64-linux.h: uses __uint128_t.  Should change
it to __vki_uint128_t, but what's the defn of that?


m_debuginfo/priv_storage.h: need proper defn of DiCfSI


readdwarf.c: is this correct?
#elif defined(VGP_arm64_linux)
#  define FP_REG         29    //???
#  define SP_REG         31    //???
#  define RA_REG_DEFAULT 30    //???


vki-arm64-linux.h:
re linux-3.10.5/include/uapi/asm-generic/sembuf.h
I'd say the amd64 version has padding it shouldn't have.  Check?


syswrap-linux.c run_a_thread_NORETURN assembly sections
seems like tst->os_state.exitcode has word type
in which case the ppc64_linux use of lwz to read it, is wrong


syswrap-linux.c ML_(do_fork_clone)
assuming that VGP_arm64_linux is the same as VGP_arm_linux here


dispatch-arm64-linux.S: FIXME: set up FP control state before
entering generated code.  Also fix screwy indentation.


dispatcher-ery general: what's a good (predictor-friendly) way to
branch to a register?


in vki-arm64-scnums.h
//#if __BITS_PER_LONG == 64 && !defined(__SYSCALL_COMPAT)
Probably want to reenable that and clean up accordingly


putIRegXXorZR: figure out a way that the computed value is actually
used, so as to keep any memory reads that might generate it, alive.
(else the simulation can lose exceptions).  At least, for writes to
the zero register generated by loads .. or .. can anything other
integer instructions, that write to a register, cause exceptions?


loads/stores: generate stack alignment checks as necessary


fix barrier insns: ISB, DMB


fix atomic loads/stores


FMADD/FMSUB/FNMADD/FNMSUB: generate and use the relevant fused
IROps so as to avoid double rounding


ARM64Instr_Call getRegUsage: re-check relative to what
getAllocableRegs_ARM64 makes available


Make dispatch-arm64-linux.S save any callee-saved Q regs
I think what is required is to save D8-D15 and nothing more than that.


wrapper for __NR3264_fstat -- correct?


PRE(sys_clone): get rid of references to vki_modify_ldt_t and the
definition of it in vki-arm64-linux.h.  Ditto for 32 bit arm.


sigframe-arm64-linux.c: build_sigframe: references to nonexistent
siguc->uc_mcontext.trap_no, siguc->uc_mcontext.error_code have been
replaced by zero.  Also in synth_ucontext.


m_debugger.c:
uregs.pstate   = LibVEX_GuestARM64_get_nzcv(vex); /* is this correct? */
Is that remotely correct?


host_arm64_defs.c: emit_ARM64INstr:
ARM64in_VDfromX and ARM64in_VQfromXX: use simple top-half zeroing
MOVs to vector registers instead of INS Vd.D[0], Xreg, to avoid false
dependencies on the top half of the register.  (Or at least check
the semantics of INS Vd.D[0] to see if it zeroes out the top.)


preferredVectorSubTypeFromSize: review perf effects and decide
on a types-for-subparts policy


fold_IRExpr_Unop: add a reduction rule for this
1Sto64(CmpNEZ64( Or64(GET:I64(1192),GET:I64(1184)) ))
vis 1Sto64(CmpNEZ64(x)) --> CmpwNEZ64(x)


check insn selection for memcheck-only primops:
Left64 CmpwNEZ64 V128to64 V128HIto64 1Sto64 CmpNEZ64 CmpNEZ32
widen_z_8_to_64 1Sto32 Left32 32HLto64 CmpwNEZ32 CmpNEZ8


isel: get rid of various cases where zero is put into a register
and just use xzr instead.  Especially for CmpNEZ64/32.  And for
writing zeroes into the CC thunk fields.


/* Keep this list in sync with that in iselNext below */
/* Keep this list in sync with that for Ist_Exit above */
uh .. they are not in sync


very stupid:
imm64  x23, 0xFFFFFFFFFFFFFFA0
17 F4 9F D2 F7 FF BF F2 F7 FF DF F2 F7 FF FF F2


valgrind.h: fix VALGRIND_ALIGN_STACK/VALGRIND_RESTORE_STACK,
also add CFI annotations


could possibly bring r29 into use, which be useful as it is
callee saved


ubfm/sbfm etc: special case cases that are simple shifts, as iropt
can't always simplify the general-case IR to a shift in such cases.


LDP,STP (immediate, simm7) (FP&VEC)
should zero out hi parts of dst registers in the LDP case


DUP insns: use Iop_Dup8x16, Iop_Dup16x8, Iop_Dup32x4
rather than doing it "by hand"


Any place where ZeroHI64ofV128 is used in conjunction with
FP vector IROps: find a way to make sure that arithmetic on
the upper half of the values is "harmless."


math_MINMAXV: use real Iop_Cat{Odd,Even}Lanes ops rather than
inline scalar code


chainXDirect_ARM64: use direct jump forms when possible

How to cross-compile for Android.  These notes were last updated on
17 Feb 2012, for Valgrind SVN revision 12390/2257.

This is known to work at least for :
ARM:
  Android 4.0.3 running on a (rooted, AOSP build) Nexus S.
  Android 4.0.3 running on Motorola Xoom.
  Android 4.0.3 running on android arm emulator.
  Android 4.1   running on android emulator.
  Android 2.3.4 on Nexus S worked at some time in the past.

x86:
  Android 4.0.3 running on android x86 emulator.

mips32:
  Android 4.1.2 running on android mips emulator.
  Android 4.2.2 running on android mips emulator.
  Android 4.3   running on android mips emulator.
  Android 4.0.4 running on BROADCOM bcm7425

On android-arm, GDBserver might insert breaks at wrong addresses.
Feedback on this welcome.

Other configurations and toolchains might work, but haven't been tested.
Feedback is welcome.


You need the android-ndk-r6 native development kit.  r6b and r7
give a non-completely-working build; see
http://code.google.com/p/android/issues/detail?id=23203
For the android emulator, the versions needed and how to
install them are described in README.android_emulator.

You can get android-ndk-r6 from
http://dl.google.com/android/ndk/android-ndk-r6-linux-x86.tar.bz2
Install it somewhere.  Doesn't matter where.  Then do this:


# Modify this (obviously).  Note, this "export" command is only done
# so as to reduce the amount of typing required.  None of the commands
# below read it as part of their operation.
#
export NDKROOT=/path/to/android-ndk-r6


# Modify this too.  Tell the build system which Android hardware you
# are building for.  It needs to know this so it can compile in
# support for the right Android-hw-specific ioctls.  (sigh.)  As with
# NDKROOT above, this is merely to avoid repeated typing; none of the
# commands read it.
#
# Currently the supported values are:  nexus_s pandaboard
# So choose one of the below:
#
export HWKIND=nexus_s         # Samsung Nexus S; also Xoom (for now)
export HWKIND=generic         # A generic Android device. eg, Pandaboard
export HWKIND=emulator        # Android emulator

# Then cd to the root of your Valgrind source tree.
#
cd /path/to/valgrind/source/tree


# After this point, you don't need to modify anything; just copy and
# paste the commands below.


# Set up toolchain paths.
#
# For ARM
export AR=$NDKROOT/toolchains/arm-linux-androideabi-4.4.3/prebuilt/linux-x86/bin/arm-linux-androideabi-ar
export LD=$NDKROOT/toolchains/arm-linux-androideabi-4.4.3/prebuilt/linux-x86/bin/arm-linux-androideabi-ld
export CC=$NDKROOT/toolchains/arm-linux-androideabi-4.4.3/prebuilt/linux-x86/bin/arm-linux-androideabi-gcc

# For x86
export AR=$NDKROOT/toolchains/x86-4.4.3/prebuilt/linux-x86/bin/i686-android-linux-ar
export LD=$NDKROOT/toolchains/x86-4.4.3/prebuilt/linux-x86/bin/i686-android-linux-ld
export CC=$NDKROOT/toolchains/x86-4.4.3/prebuilt/linux-x86/bin/i686-android-linux-gcc

# For MIPS32
export AR=$NDKROOT/toolchains/mipsel-linux-android-4.8/prebuilt/linux-x86_64/bin/mipsel-linux-android-ar
export LD=$NDKROOT/toolchains/mipsel-linux-android-4.8/prebuilt/linux-x86_64/bin/mipsel-linux-android-ld
export CC=$NDKROOT/toolchains/mipsel-linux-android-4.8/prebuilt/linux-x86_64/bin/mipsel-linux-android-gcc

# Do configuration stuff.  Don't mess with the --prefix in the
# configure command below, even if you think it's wrong.
# You may need to set the --with-tmpdir path to something
# different if /sdcard doesn't work on the device -- this is
# a known cause of difficulties.

# The below re-generates configure, Makefiles, ...
# This is not needed if you start from a release tarball.
./autogen.sh

# for ARM
CPPFLAGS="--sysroot=$NDKROOT/platforms/android-3/arch-arm -DANDROID_HARDWARE_$HWKIND" \
   CFLAGS="--sysroot=$NDKROOT/platforms/android-3/arch-arm" \
   ./configure --prefix=/data/local/Inst \
   --host=armv7-unknown-linux --target=armv7-unknown-linux \
   --with-tmpdir=/sdcard
# note: on android emulator, android-14 platform was also tested and works.
# It is not clear what this platform nr really is.

# for x86
CPPFLAGS="--sysroot=$NDKROOT/platforms/android-9/arch-x86 -DANDROID_HARDWARE_$HWKIND" \
   CFLAGS="--sysroot=$NDKROOT/platforms/android-9/arch-x86 -fno-pic" \
   ./configure --prefix=/data/local/Inst \
   --host=i686-android-linux --target=i686-android-linux \
   --with-tmpdir=/sdcard

# for MIPS32
CPPFLAGS="--sysroot=$NDKROOT/platforms/android-18/arch-mips -DANDROID_HARDWARE_$HWKIND" \
   CFLAGS="--sysroot=$NDKROOT/platforms/android-18/arch-mips" \
   ./configure --prefix=/data/local/Inst \
   --host=mipsel-linux-android --target=mipsel-linux-android \
   --with-tmpdir=/sdcard

# At the end of the configure run, a few lines of details
# are printed.  Make sure that you see these two lines:
#
# For ARM:
#          Platform variant: android
#     Primary -DVGPV string: -DVGPV_arm_linux_android=1
#
# For x86:
#          Platform variant: android
#     Primary -DVGPV string: -DVGPV_x86_linux_android=1
#
# For mips32:
#          Platform variant: android
#     Primary -DVGPV string: -DVGPV_mips32_linux_android=1
#
# If you see anything else at this point, something is wrong, and
# either the build will fail, or will succeed but you'll get something
# which won't work.


# Build, and park the install tree in `pwd`/Inst
#
make -j2
make -j2 install DESTDIR=`pwd`/Inst


# To get the install tree onto the device:
# (I don't know why it's not "adb push Inst /data/local", but this
# formulation does appear to put the result in /data/local/Inst.)
#
adb push Inst /

# To run (on the device)
/data/local/Inst/bin/valgrind [the usual args etc]


# Once you're up and running, a handy modify-V-rebuild-reinstall
# command line (on the host, of course) is
#
mq -j2 && mq -j2 install DESTDIR=`pwd`/Inst && adb push Inst /
#
# where 'mq' is an alias for 'make --quiet'.


# One common cause of runs failing at startup is the inability of
# Valgrind to find a suitable temporary directory.  On the device,
# there doesn't seem to be any one location which we always have
# permission to write to.  The instructions above use /sdcard.  If
# that doesn't work for you, and you're Valgrinding one specific
# application which is already installed, you could try using its
# temporary directory, in /data/data, for example
# /data/data/org.mozilla.firefox_beta.
#
# Using /system/bin/logcat on the device is helpful for diagnosing
# these kinds of problems.

How to install and run an android emulator.

mkdir android # or any other place you prefer
cd android

# download java JDK
# http://www.oracle.com/technetwork/java/javase/downloads/index.html
# download android SDK
# http://developer.android.com/sdk/index.html
# download android NDK
# http://developer.android.com/sdk/ndk/index.html

# versions I used:
#  jdk-7u4-linux-i586.tar.gz
#  android-ndk-r8-linux-x86.tar.bz2
#  android-sdk_r18-linux.tgz

# install jdk
tar xzf jdk-7u4-linux-i586.tar.gz

# install sdk
tar xzf android-sdk_r18-linux.tgz

# install ndk
tar xjf android-ndk-r8-linux-x86.tar.bz2


# setup PATH to use the installed software:
export SDKROOT=$HOME/android/android-sdk-linux
export PATH=$PATH:$SDKROOT/tools:$SDKROOT/platform-tools
export NDKROOT=$HOME/android/android-ndk-r8

# install android platforms you want by starting:
android
# (from $SDKROOT/tools)

# select the platforms you need
# I selected and installed:
#   Android 4.0.3 (API 15)
# Upgraded then to the newer version available:
#     Android sdk 20
#     Android platform tools 12

# then define a virtual device:
Tools -> Manage AVDs...
# I define an AVD Name with 64 Mb SD Card, (4.0.3, api 15)
# rest is default


# compile and make install Valgrind, following README.android


# Start your android emulator (it takes some time).
# You can use adb shell to get a shell on the device
# and see it is working. Note that I usually get
# one or two time out from adb shell before it works
adb shell

# Once the emulator is ready, push your Valgrind to the emulator:
adb push Inst /


# if you need to debug:
# You have on the android side a gdbserver
# on the device side:
gdbserver :1234 your_exe

# on the host side:
adb forward tcp:1234 tcp:1234
$HOME/android/android-ndk-r8/toolchains/arm-linux-androideabi-4.4.3/prebuilt/linux-x86/bin/arm-linux-androideabi-gdb your_exe
target remote :1234

Supported platforms
-------------------
- MIPS32 and MIPS64 platforms are currently supported.
- Both little-endian and big-endian cores are supported.
- MIPS DSP ASE on MIPS32 platforms is supported.


Building V for MIPS
-------------------
- Native build is available for all supported platforms. The build system
expects that native GCC is configured correctly and optimized for the platform.
Yet, this may not be the case with some Debian distributions which configure
GCC to compile to "mips1" by default. Depending on a target platform, using
CFLAGS="-mips32r2", CFLAGS="-mips32" or CFLAGS="-mips64" or
CFLAGS="-mips64 -mabi=64" will do the trick and compile Valgrind correctly.

- Use of cross-toolchain is supported as well.
- Example of configure line and additional configure options:

   $ ./configure --host=mipsel-linux-gnu --prefix=<path_to_install_directory>
        [--with-pagesize=<4|16|64>]

 * --host=mips-linux-gnu is necessary only if Valgrind is built on platform
   other then MIPS, tools for building MIPS application have to be in PATH.

 * --with-pagesize option is used to set default PAGE SIZE. If option is not
   used, PAGE SIZE is set to value default for platform on which Valgrind is
   built on. Possible values are 4, 16 of 64 and represent size in kilobytes.

 * --host=mips-linux-gnu is necessary if you compile it with cross toolchain
   compiler for big endian platform.

 * --host=mipsel-linux-gnu is necessary if you compile it with cross toolchain
   compiler for little endian platform.

 * --build=mips-linux is needed if you want to build it for MIPS32 on 64-bit
   MIPS system.

 * If you are compiling Valgrind for mips32 with gcc version older then
   gcc (GCC) 4.5.1, you must specify CFLAGS="-mips32r2 -mplt", e.g.

   ./configure --prefix=<path_to_install_directory>
   CFLAGS="-mips32r2 -mplt"


Limitations
-----------
- Some gdb tests will fail when gdb (GDB) older than 7.5 is used and gdb is
  not compiled with '--with-expat=yes'.
- You can not compile tests for DSP ASE if you are using gcc (GCC) older
  then 4.6.1 due to a bug in the toolchain.
- Older GCC may have issues with some inline assembly blocks. Get a toolchain
  based on newer GCC versions, if possible.

Requirements
------------
- You need GCC 3.4 or later to compile the s390 port.
- To run valgrind a z10 machine or any later model is needed.
  Older machine models down to and including z900 may work but have
  not been tested extensively.


Limitations
-----------
- 31-bit client programs are not supported.
- Hexadecimal floating point is not supported.
- memcheck, cachegrind, drd, helgrind, massif, lackey, and none are
  supported.
- On machine models predating z10, cachegrind will assume a z10 cache
  architecture. Otherwise, cachegrind will query the hosts cache system
  and use those parameters.
- callgrind and all experimental tools are currently not supported.
- Some gcc versions use mvc to copy 4/8 byte values. This will affect
  certain debug messages. For example, memcheck will complain about
  4 one-byte reads/writes instead of just a single read/write.


Hardware facilities
-------------------
Valgrind does not require that the host machine has the same hardware
facilities as the machine for which the client program was compiled.
This is convenient. The JIT compiler will translate the client instructions
according to the facilities available on the host.
This means, though, that probing for hardware facilities by issuing
instructions from that facility and observing whether SIGILL is thrown
may not work. As a consequence, programs that attempt to do so may
behave differently. It is believed that this is a rare use case.


Recommendations
---------------
Applications should be compiled with -fno-builtin to avoid
false positives due to builtin string operations when running memcheck.


Reading Material
----------------
(1) Linux for zSeries ELF ABI Supplement
    http://refspecs.linuxfoundation.org/ELF/zSeries/index.html
(2) z/Architecture Principles of Operation
    http://publibfi.boulder.ibm.com/epubs/pdf/dz9zr009.pdf
(3) z/Architecture Reference Summary
    http://publibfi.boulder.ibm.com/epubs/pdf/dz9zs007.pdf

Building and not installing it
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To run Valgrind without having to install it, run coregrind/valgrind
with the VALGRIND_LIB environment variable set, where <dir> is the root
of the source tree (and must be an absolute path).  Eg:

  VALGRIND_LIB=~/grind/head4/.in_place ~/grind/head4/coregrind/valgrind

This allows you to compile and run with "make" instead of "make install",
saving you time.

Or, you can use the 'vg-in-place' script which does that for you.

I recommend compiling with "make --quiet" to further reduce the amount of
output spewed out during compilation, letting you actually see any errors,
warnings, etc.


Building a distribution tarball
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To build a distribution tarball from the valgrind sources:

  make dist

In addition to compiling, linking and packaging everything up, the command
will also build the documentation. Even if all required tools for building the
documentation are installed, this step may not succeed because of hidden
dependencies. E.g. on Ubuntu you must have "docbook-xsl" installed.
Additionally, specific tool versions maybe needed.

If you only want to test whether the generated tarball is complete and runs
regression tests successfully, building documentation is not needed.
Edit docs/Makefile.am, search for BUILD_ALL_DOCS and follow instructions there.


Running the regression tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To build and run all the regression tests, run "make [--quiet] regtest".

To run a subset of the regression tests, execute:

  perl tests/vg_regtest <name>

where <name> is a directory (all tests within will be run) or a single
.vgtest test file, or the name of a program which has a like-named .vgtest
file.  Eg:

  perl tests/vg_regtest memcheck
  perl tests/vg_regtest memcheck/tests/badfree.vgtest
  perl tests/vg_regtest memcheck/tests/badfree


Running the performance tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To build and run all the performance tests, run "make [--quiet] perf".

To run a subset of the performance suite, execute:

  perl perf/vg_perf <name>

where <name> is a directory (all tests within will be run) or a single
.vgperf test file, or the name of a program which has a like-named .vgperf
file.  Eg:

  perl perf/vg_perf perf/
  perl perf/vg_perf perf/bz2.vgperf
  perl perf/vg_perf perf/bz2

To compare multiple versions of Valgrind, use the --vg= option multiple
times.  For example, if you have two Valgrinds next to each other, one in
trunk1/ and one in trunk2/, from within either trunk1/ or trunk2/ do this to
compare them on all the performance tests:

  perl perf/vg_perf --vg=../trunk1 --vg=../trunk2 perf/


Debugging Valgrind with GDB
~~~~~~~~~~~~~~~~~~~~~~~~~~~
To debug the valgrind launcher program (<prefix>/bin/valgrind) just
run it under gdb in the normal way.

Debugging the main body of the valgrind code (and/or the code for
a particular tool) requires a bit more trickery but can be achieved
without too much problem by following these steps:

(1) Set VALGRIND_LAUNCHER to point to the valgrind executable.  Eg:

      export VALGRIND_LAUNCHER=/usr/local/bin/valgrind

    or for an uninstalled version in a source directory $DIR:

      export VALGRIND_LAUNCHER=$DIR/coregrind/valgrind

(2) Run gdb on the tool executable.  Eg:

      gdb /usr/local/lib/valgrind/ppc32-linux/lackey

    or

      gdb $DIR/.in_place/x86-linux/memcheck

(3) Do "handle SIGSEGV SIGILL nostop noprint" in GDB to prevent GDB from
    stopping on a SIGSEGV or SIGILL:

    (gdb) handle SIGILL SIGSEGV nostop noprint

(4) Set any breakpoints you want and proceed as normal for gdb. The
    macro VG_(FUNC) is expanded to vgPlain_FUNC, so If you want to set
    a breakpoint VG_(do_exec), you could do like this in GDB:

    (gdb) b vgPlain_do_exec

(5) Run the tool with required options (the --tool option is required
    for correct setup), e.g.

    (gdb) run --tool=lackey pwd

Steps (1)--(3) can be put in a .gdbinit file, but any directory names must
be fully expanded (ie. not an environment variable).

A different and possibly easier way is as follows:

(1) Run Valgrind as normal, but add the flag --wait-for-gdb=yes.  This
    puts the tool executable into a wait loop soon after it gains
    control.  This delays startup for a few seconds.

(2) In a different shell, do "gdb /proc/<pid>/exe <pid>", where
    <pid> you read from the output printed by (1).  This attaches
    GDB to the tool executable, which should be in the abovementioned
    wait loop.

(3) Do "cont" to continue.  After the loop finishes spinning, startup
    will continue as normal.  Note that comment (3) above re passing
    signals applies here too.


Self-hosting
~~~~~~~~~~~~
This section explains :
  (A) How to configure Valgrind to run under Valgrind.
      Such a setup is called self hosting, or outer/inner setup.
  (B) How to run Valgrind regression tests in a 'self-hosting' mode,
      e.g. to verify Valgrind has no bugs such as memory leaks.
  (C) How to run Valgrind performance tests in a 'self-hosting' mode,
      to analyse and optimise the performance of Valgrind and its tools.

(A) How to configure Valgrind to run under Valgrind:

(1) Check out 2 trees, "Inner" and "Outer".  Inner runs the app
    directly.  Outer runs Inner.

(2) Configure inner with --enable-inner and build/install as usual.

(3) Configure Outer normally and build/install as usual.

(4) Choose a very simple program (date) and try

    outer/.../bin/valgrind --sim-hints=enable-outer --trace-children=yes  \
       --smc-check=all-non-file \
       --run-libc-freeres=no --tool=cachegrind -v \
       inner/.../bin/valgrind --vgdb-prefix=./inner --tool=none -v prog

Note: You must use a "make install"-ed valgrind.
Do *not* use vg-in-place for the outer valgrind.

If you omit the --trace-children=yes, you'll only monitor Inner's launcher
program, not its stage2. Outer needs --run-libc-freeres=no, as otherwise
it will try to find and run __libc_freeres in the inner, while libc is not
used by the inner. Inner needs --vgdb-prefix=./inner to avoid inner
gdbserver colliding with outer gdbserver.
Currently, inner does *not* use the client request
VALGRIND_DISCARD_TRANSLATIONS for the JITted code or the code patched for
translation chaining. So the outer needs --smc-check=all-non-file to
detect the modified code.

Debugging the whole thing might imply to use up to 3 GDB:
  * a GDB attached to the Outer valgrind, allowing
    to examine the state of Outer.
  * a GDB using Outer gdbserver, allowing to
    examine the state of Inner.
  * a GDB using Inner gdbserver, allowing to
    examine the state of prog.

The whole thing is fragile, confusing and slow, but it does work well enough
for you to get some useful performance data.  Inner has most of
its output (ie. those lines beginning with "==<pid>==") prefixed with a '>',
which helps a lot. However, when running regression tests in an Outer/Inner
setup, this prefix causes the reg test diff to fail. Give
--sim-hints=no-inner-prefix to the Inner to disable the production
of the prefix in the stdout/stderr output of Inner.

The allocator (coregrind/m_mallocfree.c) is annotated with client requests
so Memcheck can be used to find leaks and use after free in an Inner
Valgrind.

The Valgrind "big lock" is annotated with helgrind client requests
so helgrind and drd can be used to find race conditions in an Inner
Valgrind.

All this has not been tested much, so don't be surprised if you hit problems.

When using self-hosting with an outer Callgrind tool, use '--pop-on-jump'
(on the outer). Otherwise, Callgrind has much higher memory requirements.

(B) Regression tests in an outer/inner setup:

 To run all the regression tests with an outer memcheck, do :
   perl tests/vg_regtest --outer-valgrind=../outer/.../bin/valgrind \
                         --all

 To run a specific regression tests with an outer memcheck, do:
   perl tests/vg_regtest --outer-valgrind=../outer/.../bin/valgrind \
                         none/tests/args.vgtest

 To run regression tests with another outer tool:
   perl tests/vg_regtest --outer-valgrind=../outer/.../bin/valgrind \
                         --outer-tool=helgrind --all

 --outer-args allows to give specific arguments to the outer tool,
 replacing the default one provided by vg_regtest.

Note: --outer-valgrind must be a "make install"-ed valgrind.
Do *not* use vg-in-place.

When an outer valgrind runs an inner valgrind, a regression test
produces one additional file <testname>.outer.log which contains the
errors detected by the outer valgrind.  E.g. for an outer memcheck, it
contains the leaks found in the inner, for an outer helgrind or drd,
it contains the detected race conditions.

The file tests/outer_inner.supp contains suppressions for
the irrelevant or benign errors found in the inner.

(C) Performance tests in an outer/inner setup:

 To run all the performance tests with an outer cachegrind, do :
    perl perf/vg_perf --outer-valgrind=../outer/.../bin/valgrind perf

 To run a specific perf test (e.g. bz2) in this setup, do :
    perl perf/vg_perf --outer-valgrind=../outer/.../bin/valgrind perf/bz2

 To run all the performance tests with an outer callgrind, do :
    perl perf/vg_perf --outer-valgrind=../outer/.../bin/valgrind \
                      --outer-tool=callgrind perf

Note: --outer-valgrind must be a "make install"-ed valgrind.
Do *not* use vg-in-place.

 To compare the performance of multiple Valgrind versions, do :
    perl perf/vg_perf --outer-valgrind=../outer/.../bin/valgrind \
      --vg=../inner_xxxx --vg=../inner_yyyy perf
  (where inner_xxxx and inner_yyyy are the toplevel directories of
  the versions to compare).
  Cachegrind and cg_diff are particularly handy to obtain a delta
  between the two versions.

When the outer tool is callgrind or cachegrind, the following
output files will be created for each test:
   <outertoolname>.out.<inner_valgrind_dir>.<tt>.<perftestname>.<pid>
   <outertoolname>.outer.log.<inner_valgrind_dir>.<tt>.<perftestname>.<pid>
 (where tt is the two letters abbreviation for the inner tool(s) run).

For example, the command
    perl perf/vg_perf \
      --outer-valgrind=../outer_trunk/install/bin/valgrind \
      --outer-tool=callgrind \
      --vg=../inner_tchain --vg=../inner_trunk perf/many-loss-records

produces the files
    callgrind.out.inner_tchain.no.many-loss-records.18465
    callgrind.outer.log.inner_tchain.no.many-loss-records.18465
    callgrind.out.inner_tchain.me.many-loss-records.21899
    callgrind.outer.log.inner_tchain.me.many-loss-records.21899
    callgrind.out.inner_trunk.no.many-loss-records.21224
    callgrind.outer.log.inner_trunk.no.many-loss-records.21224
    callgrind.out.inner_trunk.me.many-loss-records.22916
    callgrind.outer.log.inner_trunk.me.many-loss-records.22916


Printing out problematic blocks
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you want to print out a disassembly of a particular block that
causes a crash, do the following.

Try running with "--vex-guest-chase-thresh=0 --trace-flags=10000000
--trace-notbelow=999999".  This should print one line for each block
translated, and that includes the address.

Then re-run with 999999 changed to the highest bb number shown.
This will print the one line per block, and also will print a
disassembly of the block in which the fault occurred.

This file documents various "processes" that are used by Valgrind
developers for development and release activities.
This file contains one section for each process.
A summary of each process is given here. Each process is described
more in details afterwards.


* Update of the NEWS file: NEWS describes fixed bugs and new features.
  It is updated and committed together with the code fixing the bug/implementing
  the feature.

* Major release production:
  See docs/internals/release-HOWTO.txt (currently a bit out of date)

* Minor/correction release production: TBD


Processes detailed descriptions:

Update of the NEWS file.
========================
  The NEWS file gives for each release:
    - the list of fixed bugs,
    - a short description of each functional change,
    - a short description of each technical change impacting the users.

  The update of the NEWS file should be committed together with the
  code change (or as part of the last committed change) that fixes the
  bug or implements the new feature/technical change.
  The documentation (e.g. user manual) should also be committed as part of
  the code change.

  Fixing a bug
  ------------
  When fixing a bug, add a line in the 'FIXED BUGS' section of
  the NEWS file.  Keep the list of bugs sorted by bugzilla entry number.

  Once you have commit the change, update the bug status in bugzilla,
  adding in the comment the revision number of the commit fixing the bug.

  If a bug is not entered in bugzilla (not encouraged), use "n-i-bz"
  and add the bug line at the end of the bug list.

  The file docs/internals/X_Y_BUGSTATUS.txt (where X_Y is the last
  major release e.g. 3_9) contains information/statuses for some bugs.
  If a bug is fixed, remove the (possible) bug info from this file.

  Implementing a change
  ---------------------
  When implementing a functional or 'user impacting' technical change,
  add a short description of the change in the relevant sub-section
  (e.g. TOOL CHANGES, PLATFORM CHANGES, ...).


  Some special cases:
  -------------------
  Some bugs or changes only touch the VEX SVN repository, so it is not
  possible to commit the NEWS change together with the code changes.
  In such a case, first commit the VEX change. Then just after, commit
  the NEWS change. In the bugzilla status, reference (at least) the Valgrind
  revision number.

  Some changes or bug fixes are very big and might be implemented over
  a significant period. In such a case, update the NEWS as part of the
  final commit.
  If relevant, you might already update the NEWS file as part of
  earlier commits, using the word 'PARTIAL' to indicate that the change or
  bug fix is not complete yet.

  Some bugs are reported more than once in bugzilla.
  Also document in NEWS that such duplicated bugs have been fixed, using a line
  such as:
     308333 == 307106
  to indicate that the bug 308333 is a duplicate of 307106, and was thus
  fixed in the commit that fixed 307106.
  Change also the status of the duplicated bug  in bugzilla,
  indicating in the comment the commit revision that fixed the 'master bug'.



Minor/correction release:
=========================
Describe here how to do changes and bug fixed in a minor (correction) release
and how/when to merge the branch to the trunk.

Proposal to be discussed:
When a bug is fixed on the branch, the NEWS file is updated on the branch
(i.e. a 3.9.1 section is created if needed).

When often to merge the branch to trunk ?
  after each fix ?
  just after the correction release is produced ?

How is the branch merged to the trunk ?

Dealing with missing system call or ioctl wrappers in Valgrind
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You're probably reading this because Valgrind bombed out whilst
running your program, and advised you to read this file.  The good
news is that, in general, it's easy to write the missing syscall or
ioctl wrappers you need, so that you can continue your debugging.  If
you send the resulting patches to me, then you'll be doing a favour to
all future Valgrind users too.

Note that an "ioctl" is just a special kind of system call, really; so
there's not a lot of need to distinguish them (at least conceptually)
in the discussion that follows.

All this machinery is in coregrind/m_syswrap.


What are syscall/ioctl wrappers?  What do they do?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Valgrind does what it does, in part, by keeping track of everything your
program does.  When a system call happens, for example a request to read
part of a file, control passes to the Linux kernel, which fulfills the
request, and returns control to your program.  The problem is that the
kernel will often change the status of some part of your program's memory
as a result, and tools (instrumentation plug-ins) may need to know about
this.

Syscall and ioctl wrappers have two jobs:

1. Tell a tool what's about to happen, before the syscall takes place.  A
   tool could perform checks beforehand, eg. if memory about to be written
   is actually writeable.  This part is useful, but not strictly
   essential.

2. Tell a tool what just happened, after a syscall takes place.  This is
   so it can update its view of the program's state, eg. that memory has
   just been written to.  This step is essential.

The "happenings" mostly involve reading/writing of memory.

So, let's look at an example of a wrapper for a system call which
should be familiar to many Unix programmers.


The syscall wrapper for time()
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The wrapper for the time system call looks like this:

  PRE(sys_time)
  {
     /* time_t time(time_t *t); */
     PRINT("sys_time ( %p )",ARG1);
     PRE_REG_READ1(long, "time", int *, t);
     if (ARG1 != 0) {
        PRE_MEM_WRITE( "time(t)", ARG1, sizeof(vki_time_t) );
     }
  }

  POST(sys_time)
  {
     if (ARG1 != 0) {
        POST_MEM_WRITE( ARG1, sizeof(vki_time_t) );
     }
  }

The first thing we do happens before the syscall occurs, in the PRE() function.
The PRE() function typically starts with invoking to the PRINT() macro. This
PRINT() macro implements support for the --trace-syscalls command line option.
Next, the tool is told the return type of the syscall, that the syscall has
one argument, the type of the syscall argument and that the argument is being
read from a register:

     PRE_REG_READ1(long, "time", int *, t);

Next, if a non-NULL buffer is passed in as the argument, tell the tool that the
buffer is about to be written to:

     if (ARG1 != 0) {
        PRE_MEM_WRITE( "time", ARG1, sizeof(vki_time_t) );
     }

Finally, the really important bit, after the syscall occurs, in the POST()
function:  if, and only if, the system call was successful, tell the tool that
the memory was written:

     if (ARG1 != 0) {
        POST_MEM_WRITE( ARG1, sizeof(vki_time_t) );
     }

The POST() function won't be called if the syscall failed, so you
don't need to worry about checking that in the POST() function.
(Note: this is sometimes a bug; some syscalls do return results when
they "fail" - for example, nanosleep returns the amount of unslept
time if interrupted. TODO: add another per-syscall flag for this
case.)

Note that we use the type 'vki_time_t'.  This is a copy of the kernel
type, with 'vki_' prefixed.  Our copies of such types are kept in the
appropriate vki*.h file(s).  We don't include kernel headers or glibc headers
directly.


Writing your own syscall wrappers (see below for ioctl wrappers)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If Valgrind tells you that system call NNN is unimplemented, do the
following:

1.  Find out the name of the system call:

       grep NNN /usr/include/asm/unistd*.h

    This should tell you something like  __NR_mysyscallname.
    Copy this entry to include/vki/vki-scnums-$(VG_PLATFORM).h.


2.  Do 'man 2 mysyscallname' to get some idea of what the syscall
    does.  Note that the actual kernel interface can differ from this,
    so you might also want to check a version of the Linux kernel
    source.

    NOTE: any syscall which has something to do with signals or
    threads is probably "special", and needs more careful handling.
    Post something to valgrind-developers if you aren't sure.


3.  Add a case to the already-huge collection of wrappers in
    the coregrind/m_syswrap/syswrap-*.c files.
    For each in-memory parameter which is read or written by
    the syscall, do one of

      PRE_MEM_READ( ... )
      PRE_MEM_RASCIIZ( ... )
      PRE_MEM_WRITE( ... )

    for  that parameter.  Then do the syscall.  Then, if the syscall
    succeeds, issue suitable POST_MEM_WRITE( ... ) calls.
    (There's no need for POST_MEM_READ calls.)

    Also, add it to the syscall_table[] array; use one of GENX_, GENXY
    LINX_, LINXY, PLAX_, PLAXY.
    GEN* for generic syscalls (in syswrap-generic.c), LIN* for linux
    specific ones (in syswrap-linux.c) and PLA* for the platform
    dependant ones (in syswrap-$(PLATFORM)-linux.c).
    The *XY variant if it requires a PRE() and POST() function, and
    the *X_ variant if it only requires a PRE()
    function.

    If you find this difficult, read the wrappers for other syscalls
    for ideas.  A good tip is to look for the wrapper for a syscall
    which has a similar behaviour to yours, and use it as a
    starting point.

    If you need structure definitions and/or constants for your syscall,
    copy them from the kernel headers into include/vki.h and co., with
    the appropriate vki_*/VKI_* name mangling.  Don't #include any
    kernel headers.  And certainly don't #include any glibc headers.

    Test it.

    Note that a common error is to call POST_MEM_WRITE( ... )
    with 0 (NULL) as the first (address) argument.  This usually means
    your logic is slightly inadequate.  It's a sufficiently common bug
    that there's a built-in check for it, and you'll get a "probably
    sanity check failure" for the syscall wrapper you just made, if this
    is the case.


4.  Once happy, send us the patch.  Pretty please.




Writing your own ioctl wrappers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Is pretty much the same as writing syscall wrappers, except that all
the action happens within PRE(ioctl) and POST(ioctl).

There's a default case, sometimes it isn't correct and you have to write a
more specific case to get the right behaviour.

As above, please create a bug report and attach the patch as described
on http://www.valgrind.org.

Greetings, packaging person!  This information is aimed at people
building binary distributions of Valgrind.

Thanks for taking the time and effort to make a binary distribution of
Valgrind.  The following notes may save you some trouble.


-- Do not ship your Linux distro with a completely stripped
   /lib/ld.so.  At least leave the debugging symbol names on -- line
   number info isn't necessary.  If you don't want to leave symbols on
   ld.so, alternatively you can have your distro install ld.so's
   debuginfo package by default, or make ld.so.debuginfo be a
   requirement of your Valgrind RPM/DEB/whatever.

   Reason for this is that Valgrind's Memcheck tool needs to intercept
   calls to, and provide replacements for, some symbols in ld.so at
   startup (most importantly strlen).  If it cannot do that, Memcheck
   shows a large number of false positives due to the highly optimised
   strlen (etc) routines in ld.so.  This has caused some trouble in
   the past.  As of version 3.3.0, on some targets (ppc32-linux,
   ppc64-linux), Memcheck will simply stop at startup (and print an
   error message) if such symbols are not present, because it is
   infeasible to continue.

   It's not like this is going to cost you much space.  We only need
   the symbols for ld.so (a few K at most).  Not the debug info and
   not any debuginfo or extra symbols for any other libraries.


-- (Unfortunate but true) When you configure to build with the
   --prefix=/foo/bar/xyzzy option, the prefix /foo/bar/xyzzy gets
   baked into valgrind.  The consequence is that you _must_ install
   valgrind at the location specified in the prefix.  If you don't,
   it may appear to work, but will break doing some obscure things,
   particularly doing fork() and exec().

   So you can't build a relocatable RPM / whatever from Valgrind.


-- Don't strip the debug info off lib/valgrind/$platform/vgpreload*.so
   in the installation tree.  Either Valgrind won't work at all, or it
   will still work if you do, but will generate less helpful error
   messages.  Here's an example:

   Mismatched free() / delete / delete []
      at 0x40043249: free (vg_clientfuncs.c:171)
      by 0x4102BB4E: QGArray::~QGArray(void) (tools/qgarray.cpp:149)
      by 0x4C261C41: PptDoc::~PptDoc(void) (include/qmemarray.h:60)
      by 0x4C261F0E: PptXml::~PptXml(void) (pptxml.cc:44)
      Address 0x4BB292A8 is 0 bytes inside a block of size 64 alloc'd
      at 0x4004318C: __builtin_vec_new (vg_clientfuncs.c:152)
      by 0x4C21BC15: KLaola::readSBStream(int) const (klaola.cc:314)
      by 0x4C21C155: KLaola::stream(KLaola::OLENode const *) (klaola.cc:416)
      by 0x4C21788F: OLEFilter::convert(QCString const &) (olefilter.cc:272)

   This tells you that some memory allocated with new[] was freed with
   free().

   Mismatched free() / delete / delete []
      at 0x40043249: (inside vgpreload_memcheck.so)
      by 0x4102BB4E: QGArray::~QGArray(void) (tools/qgarray.cpp:149)
      by 0x4C261C41: PptDoc::~PptDoc(void) (include/qmemarray.h:60)
      by 0x4C261F0E: PptXml::~PptXml(void) (pptxml.cc:44)
      Address 0x4BB292A8 is 0 bytes inside a block of size 64 alloc'd
      at 0x4004318C: (inside vgpreload_memcheck.so)
      by 0x4C21BC15: KLaola::readSBStream(int) const (klaola.cc:314)
      by 0x4C21C155: KLaola::stream(KLaola::OLENode const *) (klaola.cc:416)
      by 0x4C21788F: OLEFilter::convert(QCString const &) (olefilter.cc:272)

   This isn't so helpful.  Although you can tell there is a mismatch,
   the names of the allocating and deallocating functions are no longer
   visible.  The same kind of thing occurs in various other messages
   from valgrind.


-- Don't strip symbols from lib/valgrind/* in the installation tree.
   Doing so will likely cause problems.  Removing the line number info is
   probably OK (at least for some of the files in that directory), although
   that has not been tested by the Valgrind developers.


-- Please test the final installation works by running it on something
   huge.  I suggest checking that it can start and exit successfully
   both Firefox and OpenOffice.org.  I use these as test programs, and I
   know they fairly thoroughly exercise Valgrind.  The command lines to use
   are:

   valgrind -v --trace-children=yes firefox

   valgrind -v --trace-children=yes soffice


If you find any more hints/tips for packaging, please report
it as a bugreport. See http://www.valgrind.org for details.