plpa-1.3.2/README

Copyright (c) 2004-2006 The Trustees of Indiana University and Indiana
                        University Research and Technology
                        Corporation.  All rights reserved.
Copyright (c) 2004-2005 The Regents of the University of California.
                        All rights reserved.
Copyright (c) 2006-2008 Cisco Systems, Inc.  All rights reserved.
$COPYRIGHT$

See LICENSE file for a rollup of all copyright notices.

$HEADER$

===========================================================================

This is the Portable Linux Processor Affinity (PLPA) package
(pronounced "pli-pa").  The PLPA has evolved over time to provide the
following capabilities:

1. Provide a stable API on Linux for processor affinity (Linux has
   provided three different API signatures over time).
2. Provide a simple API that translates between Linux processor ID and
   (socket ID, core ID) tuples, and allows querying processor topology
   information on supported platforms.
3. Provide a command-line executable (plpa-taskset(1)) that provides
   all the same functionality as the venerable taskset(1) command, and
   several extensions, including the ability to bind processes to
   specific (socket, core) tuples on supported platforms.

Note that the PLPA is fully embeddable, meaning that it can be wholly
contained in larger software packages that wish to have a single,
stable version of processor affinity API functionality.  See below for
more details on embedding.

Also note that PLPA's socket/core and other topology information is
only available on certain platforms.  Specifically, PLPA reads the
/sys filesystem to glean its information; if your system does not
export processor topology information through /sys, the PLPA cannot
provide that information.  For example, AMD/Intel processor topology
support was included in Linux kernel v2.6.16, but POWER processor
topology information is not yet supported as of Linux kernel v2.6.26.

In a world where the processor counts in hosts are [again] increasing,
particularly where at least some of them are NUMA-based architectures,
processor affinity is becoming more important.  We hope that the PLPA
is helpful to you.  Enjoy.

Note that if you're looking into processor affinity, and if you're on
a NUMA machine, you probably also want to look into libnuma:

    ftp://ftp.suse.com/pub/people/ak/numa/

If you are a developer, keep reading.  If you are a system
administrator or other end-user, you're probably more interested in
using the plpa-info(1) and plpa-taskset(1) executable commands; see
the output of "plpa-info" and "plpa-taskset --help" for more
information.

===========================================================================

The following text is specific technical information about the
original problem that PLPA Was created to solve.

The original intent for the PLPA was for developers who wished to use
Linux processor affinity via the sched_setaffinity() and
sched_getaffinity() library calls, but don't want to wade through the
morass of 3 different APIs that have been offered through the life of
these calls in various Linux distributions and glibc versions.

Specifically, to compile for any given Linux system, you need some
complex compile-time tests to figure out which of the 3 APIs to use.
And if you want your application to be binary portable across
different Linux distributions, more complex run-time tests (and horrid
compile-time trickery) are required to figure out which API the system
you are running on uses.  

These problems all stem from the fact that the same 2 symbols have had
three different APIs (with different numbers and types of
parameters) throughout their life in Linux.  Ick.

The PLPA is an attempt to solve this problem by providing a single API
that developers can write to.  It provides three things:

1. A single API that developers can write to, regardless of what
   back-end API the system you are compiling on has.
2. A run-time test and dispatch that will invoke the Right back-end
   API depending on what back-end API the system you are running on
   has.
3. Mapping information between (socket ID, core ID) tuples and Linux
   virtual processor IDs.

===========================================================================

What, exactly, is the problem?  History.
----------------------------------------

There are at least 3 different ways that sched_setaffinity is
implemented in glibc (only one of which is documented in the
sched_setaffinity(2) man page), and some corresponding changes
to what the kernel considers to be valid arguments:

1. int sched_setaffinity(pid_t pid, unsigned int len, unsigned
                         long *mask);

This originated in the time period of 2.5 kernels and some distros
back-ported it to their 2.4 kernels and libraries.  It's unknown if
this version was ever packaged with any 2.6 kernels.

2. int sched_setaffinity (pid_t __pid, size_t __cpusetsize,
                          const cpu_set_t *__cpuset);

This appears to be in recent distros using 2.6 kernels.  We don't
know exactly when #1 changed into #2.  However, this prototype is nice
because the cpu_set_t type is accompanied by fdset-like CPU_ZERO(),
CPU_SET(), CPU_ISSET(), etc. macros.

3. int sched_setaffinity (pid_t __pid, const cpu_set_t *__mask);

(note the missing len parameter) This is in at least some Linux
distros (e.g., MDK 10.0 with a 2.6.3 kernel, and SGI Altix, even
though the Altix uses a 2.4-based kernel and therefore likely
back-ported the 2.5 work or originated it in the first place).
Similar to #2, the cpu_set_t type is accompanied by fdset-like
CPU_ZERO(), CPU_SET(), CPU_ISSET(), etc. macros.

But wait, it gets worse.

Remember that getting/setting processor affinity has to involve the
kernel.  The sched_[sg]etaffinity() glibc functions typically do a
little error checking and then make a syscall down into the kernel to
actually do the work.  There are multiple possibilities for problems
here as the amount of checking has changed:

1. The glibc may support the affinity functions, but the kernel may
   not (and vice versa).

   This is typically only an issue with slightly older Linux distributions.
   Mandrake 9.2 is an example of this.  PLPA can detect this at run-time
   and turn its internal functions into no-ops and return appropriate error
   codes (ENOSYS).

2. The glibc affinity functions may be buggy (i.e., they pass bad data
   down to the syscall).

   This is fortunately restricted to some older versions of glibc, and
   is relatively easy to check for at run-time.  PLPA reliably detects
   this situation at run-time and returns appropriate error codes
   (ENOSYS).

   The original SuSE 9.1 version seems to have this problem, but it was
   fixed it somewhere in the SuSE patching history (it is unknown exactly
   when).  Specifically, updating to the latest SuSE 9.1 patch level
   (as of Dec 2005) seems to fix the problem.

3. The CPU_* macros for manipulating cpu_set_t bitmasks may not
   compile because of typo bugs in system header files.

   PLPA avoids this problem by providing its own PLPA_CPU_* macros for
   manipulating CPU bitmasks.  See "How do I use PLPA?", below, for
   more details.

The PLPA avoids all the glibc issues by using syscall() to directly
access the kernel set and get affinity functions.  This is described
below.

===========================================================================

How does PLPA work?
-------------------

Jeff Squyres initially sent a mail to the Open MPI developer's mailing
list explaining the Linux processor affinity problems and asking for
help coming up with a solution (particularly for binary
compatibility):

    http://www.open-mpi.org/community/lists/devel/2005/11/0558.php

Discussion on that thread and others eventually resulted in the
run-time tests that form the heart of the PLPA.  Many thanks to Paul
Hargrove and Bogdan Costescu for their time and effort to get these
tests right.

PLPA was written so that other developers who want to use processor
affinity in Linux don't have to go through this mess.  The PLPA
provides a single interface that can be used on any platform,
regardless of which back-end API variant it has.  This includes both
the sched_setaffinity() and sched_getaffinity() calls as well as the
CPU_*() macros.

The PLPA avoids glibc altogether -- although tests were developed that
could *usually* figure out which glibc variant to use at run time,
there were still some cases where it was either impossible to
determine or the glibc interface itself was buggy.  Hence, it was
decided that a simpler approach was simply to use syscall() to invoke
the back-end kernel functions directly.

The kernel functions have gone through a few changes as well, so the
PLPA does a few run-time tests to determine which variant to use
before actually invoking the back-end functions with the
user-specified arguments.

NOTE: The run-time tests that the PLPA performs involve getting the
current affinity for the process in question and then attempting to
set them back to the same value.  By definition, this introduces a
race condition (there is no atomic get-and-set functionality for
processor affinity).  The PLPA cannot guarantee consistent results if
multiple entities (such as multiple threads or multiple processes) are
setting the affinity for a process at the same time.  In a worst case
scenario, the PLPA may actually determine that it cannot determine the
kernel variant at run time if another entity modifies a process'
affinity while PLPA is executing its run-time tests.

===========================================================================

Does PLPA make truly portable binaries?
---------------------------------------

As much as Linux binaries are portable, yes.  That is, if you have
within your power to make a binary that is runnable on several
different Linux distributions/versions/etc., then you may run into
problems with the Linux processor affinity functions.  PLPA attempts
to solve this problem for you by *also* making the Linux processor
affinity calls be binary portable.

Hence, you need to start with something that is already binary
portable (perhaps linking everything statically) -- then PLPA will be
of help to you.  Do not fall into the misconception that PLPA will
magically make your executable be binary portable between different
Linux variants.

===========================================================================

How do I use PLPA?
------------------

There are three main uses of the PLPA:

1. Using the plpa-info(1) executable to check if your system supports
   processor affinity and the PLPA can determine which to use at
   run-time.
2. Developers using the PLPA library both to enable source and binary
   Linux processor affinity portability, and to write
   processor-topology-aware applications.
3. Using the plpa-taskset(1) executable to bind arbitrary executables
   to Linux virtual processor IDs and/or specific socket/core tuples.

In more detail:

1. The plpa-info(1) executable is a few simple calls into the PLPA
   library that checks which API variant the system it is running on
   has.  If the kernel supports processor affinity and the PLPA is
   able to figure out which API variant to use, it prints "Kernel
   affinity support: no".  Other responses indicate an error.  The
   "--topo" switch will print out basic topology information about
   your system, if supported.

   Since the PLPA library abstracts this kind of problem away, this is
   more a diagnostic tool than anything else.  

   See "plpa-info --help" for more information.  A man page does not
   yet exist, unfortunately.

   Note that plpa-info is *only* compiled and installed if PLPA is
   installed as a standalone package (see below).

2. Developers can use this package by including the <plpa.h> header
   file and using the following prototypes for setting and getting
   processor affinity:

   int plpa_sched_setaffinity(pid_t pid, size_t cpusetsize,
                              const plpa_cpu_set_t *cpuset);

   int plpa_sched_getaffinity(pid_t pid, size_t cpusetsize,
                              const plpa_cpu_set_t *cpuset)

   These functions perform run-time tests to determine which back-end
   API variant exists on the system and then dispatch to it correctly.
   The units of cpusetsize is number of bytes.  This should normally
   just be sizeof(*cpuset), but is made available as a parameter to
   allow for future expansion of the PLPA (stay tuned).

   The observant reader will notice that this is remarkably similar to
   the one of the Linux API's (the function names are different and
   the CPU set type is different).  PLPA also provides several macros
   for manipulating the plpa_cpu_set_t bitmask, quite similar to FDSET
   macros (see "What, Exactly, Is the Problem?" above for a
   description of problems with the native CPU_* macros):

   - PLPA_CPU_ZERO(&cpuset): Sets all bits in a plpa_cpu_set_t to
     zero.
   - PLPA_CPU_SET(num, &cpuset): Sets bit <num> of <cpuset> to one.
   - PLPA_CPU_CLR(num, &cpuset): Sets bit <num> of <cpuset> to zero.
   - PLPA_CPU_ISSET(num, &cpuset): Returns one if bit <num> of
     <cpuset> is one; returns zero otherwise.

   Note that all four macros take a *pointer* to a plpa_cpu_set_t, as
   denoted by "&cpuset" in the descriptions above.

   Also note that he PLPA distinguishes between Linux processor,
   socket, and core IDs and processor, socket, and core numbers.  The
   *Linux IDs* are kernel-assigned integer values that do not
   necessarily start with zero and are not necessarily contiguous.
   The *numbers* start with 0 and are contiguous to (N-1).  The
   numbers are therefore mainly a human convenience; they may or may
   not exactly correspond to the Linux IDs; it is safest to assume
   that they do not.

   The following API functions are also available on supported
   platforms with kernels that support topology information (e.g.,
   AMD/Intel platforms with Linux kernel v2.6.16 or later).  The list
   below is a summary only; see plpa.h for a specific list of function
   signatures:

   - plpa_have_topology_information() 
     Will return 1 if the PLPA is able to provide topology
     information, 0 otherwise.  If 0 is returned, all the functions
     below will return a negative value to signify a graceful failure.

   - plpa_map_to_processor_id()
     Take a (socket ID, core ID) tuple and map it to a Linux processor
     ID

   - plpa_map_to_socket_core()
     Take a Linux processor ID and map it to a (socket ID, core ID)
     tuple

   - plpa_get_processor_info()
     Return the number of processors and the max Linux processor ID

   - plpa_get_processor_id()
     Return the Linux processor ID for the Nth processor (starting
     with 0)

   - plpa_get_processor_flags()
     Return whether a Linux processor ID exists, and if so, if it is
     online

   - plpa_get_socket_info()
     Return the number of sockets and the max Linux socket ID

   - plpa_get_socket_id()
     Return the Linux socket ID for the Nth socket (starting with 0)

   - plpa_get_core_info()
     For a given socket ID, return the number of cores and the max
     Linux core ID

   - plpa_get_core_id()
     For a given socket ID, return the Linux core ID of the Nth core
     (starting with 0)

   - plpa_get_core_flags()
     Return whether a (socket ID,core ID) tuple exists, and if so, if
     it is online

   - plpa_set_cache_behavior()
     Tell PLPA to use (or not) a local cache for the topology
     information, or to refresh the cache right now

   - plpa_finalize()
     Release all internal resources allocated and maintained by the
     PLPA.  It is permissible to invoke other PLPA functions after
     plpa_finalize(), but if you want to release PLPA's resources, you
     will need to invoke plpa_finalize() again.  Note that it is not
     necessary (but harmless) to invoke plpa_finalize() on systems
     where plpa_have_topology_information() returns that the topology
     information is not supported.

   *** NOTE: Topology information (i.e., (socket ID, core ID) tuples)
       may not be reported for offline processors.  Hence, if any
       processors are offline, the socket/core values returned by PLPA
       will likely change once the processor is brought back online.
       Sorry; this is how the Linux kernel works -- there's nothing
       PLPA can do about it.

   The above functions are slightly more documented in plpa.h.
   Contributions of real man pages would be greatly appreciated.

3. The plpa-taskset(1) executable represents an evolution of the
   venerable "taskset(1)" command.  It allows binding of arbitrary
   processes to specific Linux processor IDs and/or specific (socket
   ID, core ID) tuples.  It supports all the same command line syntax
   of the taskset(1) command, but also supports additional syntax for
   specifying socket and core IDs.  Hence, you can launch
   processor-bound jobs without needing to modify their source code to
   call the PLPA library.  See "plpa-taskset --help" for more
   information on the command line options available, and brief
   examples of usage.  A man page does not yet exist, unfortunately.

===========================================================================

How do I compile / install the PLPA as a standalone package?
------------------------------------------------------------

The PLPA uses the standard GNU Autoconf/Automake/Libtool toolset to
build and install itself.  This means that generally, the following
works:

shell$ ./configure --prefix=/where/you/want/to/install
[...lots of output...]
shell$ make all
[...lots of output...]
shell$ make install

Depending on your --prefix, you may need to run the "make install"
step as root or some other privileged user.

There are a few noteworthy configure options listed below.  The
enable/disable options are shown in their non-default form.  For
example, if --enable-foo is shown below, it is because --disable-foo
is the default.

--enable-emulate: allow using PLPA on platforms that do not have
  __NR_sched_setaffinity (e.g., OS X); usually only useful in
  development / testing scenarios.

--disable-executables: do not build the PLPA executables; only build
  the library.

--enable-included-mode: build PLPA in the "included" mode (see
  below).

--enable-debug: this option is probably only helpful for PLPA
  developers.

--with-plpa-symbol-prefix=STRING: a string prefix to add to all public
  PLPA symbols.  This is usually only useful in included mode (see
  below).

--with-valgrind(=DIR): require building PLPA with Valgrind support
  (requires finding include/valgrind/memcheck.h).  This will add a
  small number of Valgrind annotations in the PLPA code base that
  remove false/irrelevant Valgrind warnings.  The =DIR clause is only
  necessary if Valgrind's header files cannot be found by the
  preprocessor's default search path.

"make install" will install the following:

- <plpa.h> in $includedir (typically $prefix/include)
- libplpa.la and libplpa.a and/or libplpa.so in $libdir (typically
  $prefix/lib)
- plpa-info(1) executable in $bindir (typically $prefix/bin)
- plpa-taskset(1) executable in $bindir (typically $prefix/bin)

Note that since PLPA builds itself with GNU Libtool, it can be built
as a static or shared library (or both).  The default is to build a
shared library.  You can enable building a static library by supplying
the "--enable-static" argument to configure; you can disable building
the shared library by supplying the "--disable-shared" argument to
configure.  "make install" will install whichever library was built
(or both).

"make uninstall" will fully uninstall PLPA from the prefix directory
(again, depending in filesystem permissions, you may need to run this
as root or some privileged user).

===========================================================================

How do I include/embed PLPA in my software package?
---------------------------------------------------

It can be desirable to include PLPA in a larger software package
(be sure to check out the LICENSE file) so that users don't have to
separately download and install it before installing your software
(after all, PLPA is a tiny little project -- why make users bother
with it?).

When used in "included" mode, PLPA will:

- not install any header files
- not build or install any executables
- not build libplpa.* -- instead, it will build libplpa_included.*

There are two ways to put PLPA into "included" mode.  From the
configure command line:

shell$ ./configure --enable-included-mode ...

Or by directly integrating PLPA's m4 configure macro in your configure
script and invoking a specific macro to enable the included mode.  

Every project is different, and there are many different ways of
integrating PLPA into yours.  What follows is *one* example of how to
do it.

Copy the PLPA directory in your source tree and include the plpa.m4
file in your configure script -- perhaps with the following line in
acinclude.m4 (assuming the use of Automake):

m4_include(path/to/plpa.m4)

The following macros can then be used from your configure script (only
PLPA_INIT *must* be invoked if using the m4 macros):

- PLPA_STANDALONE
  Force the building of PLPA in standalone mode.  Overrides the
  --enable-included-mode command line switch. 

- PLPA_INCLUDED
  Force the building of PLPA in included mode.

- PLPA_SET_SYMBOL_PREFIX(foo)
  Tells the PLPA to prefix all types and public symbols with "foo"
  instead of "plpa_".  This is recommended behavior if you are
  including PLPA in a larger project -- it is possible that your
  software will be combined with other software that also includes
  PLPA.  If you both use different symbol prefixes, there will be no
  type/symbol clashes, and everything will compile and link
  successfully.  If you both include PLPA and do not change the symbol
  prefix, it is likely that you will get multiple symbol definitions
  when linking if an external PLPA is linked against your library /
  application.  Note that the PLPA_CPU_*() macros are *NOT* prefixed
  (because they are only used when compiling and therefore present no
  link/run-time conflicts), but all other types, enum values, and
  symbols are.  Enum values are prefixed with an upper-case
  translation if the prefix supplied.  For example,
  PLPA_SET_SYMBOL_PREFIX(foo_) will result in foo_init() and
  FOO_PROBE_OK.  Tip: It might be good to include "plpa" in the
  prefix, just for clarity.

- PLPA_DISABLE_EXECUTABLES
  Provides the same result as the --disable-executables configure
  flag, and is implicit in included mode.

- PLPA_ENABLE_EXECUTABLES
  Provides the same result as the --enable-executables configure flag.
  If used in conjunction with PLPA_INCLUDED, it must be specified
  *after* PLPA_INLCLUDED to have effect, as PLPA_INCLUDED *disables*
  executables.

- PLPA_INIT(config-prefix, action-upon-success, action-upon-failure)
  Invoke the PLPA tests and setup the PLPA to build.  A traversal of
  "make" into the PLPA directory should build everything (it is safe
  to list the PLPA directory in the SUBDIRS of a higher-level
  Makefile.am, for example).  ***PLPA_INIT must be invoked after the
  STANDALONE, INCLUDED, SET_SYMBOL_PREFIX, DISABLE_EXECUTABLES, and
  ENABLE_EXECUTABLES macros.*** The first argument is the prefix to
  use for AC_OUTPUT files.  Hence, if your embedded PLPA is located in
  the source tree at contrib/plpa, you should pass [contrib/plpa] as
  the first argument.

- PLPA_DO_AM_CONDITIONALS
  If you embed PLPA in a larger project and build it conditionally
  (e.g., if PLPA_INIT is in a conditional), you must unconditionally
  invoke PLPA_DO_AM_CONDITIONALS to avoid warnings from Automake (for
  the cases where PLPA is not selected to be built).  This macro is
  necessary because PLPA uses some AM_CONDITIONALs to build itself;
  AM_CONDITIONALs cannot be defined conditionally.  It is safe (but
  unnecessary) to call PLPA_DO_AM_CONDITIONALS even if PLPA_INIT is
  invoked unconditionally.

Here's an example of integrating with a larger project named sandbox:

----------
shell$ cd sandbox
shell$ cp -r /somewhere/else/plpa-<version> plpa
shell$ edit acinclude.m4
...add the line "m4_include(plpa/config/plpa.m4)"...
shell$ edit Makefile.am
...add "plpa" to SUBDIRS...
...add "$(top_builddir)/plpa/src/libplpa/libplpa_included.la" to
   my executable's LDADD line...
...add "-I$(top_builddir)/plpa/src/libplpa" to AM_CPPFLAGS
shell$ edit configure.ac
...add "PLPA_INCLUDED" line...
...add "PLPA_SET_SYMBOL_PREFIX(sandbox_plpa_)" line...
...add "PLPA_INIT([./plpa], [plpa_happy=yes], [plpa_happy=no])" line...
...add error checking for plpa_happy=no case...
shell$ edit src/my_program.c
...add #include <plpa.h>...
...add calls to sandbox_plpa_sched_setaffinity()...
shell$ aclocal
shell$ autoconf
shell$ libtoolize --automake
shell$ automake -a
shell$ ./configure
...lots of output...
shell$ make
...lots of output...
----------

===========================================================================

How can I tell if PLPA is working?
----------------------------------

Run plpa-info; if it says "Kernel affinity support: yes", then PLPA is
working properly.

If you want to compile your own test program to verify it, try
compiling and running the following:

---------------------------------------------------------------------------
#include <stdio.h>
#include <plpa.h>

int main(int argc, char* argv[]) {
    plpa_api_type_t p;
    if (0 == plpa_api_probe(&p) && PLPA_PROBE_OK == p) {
        printf("All is good!\n");
    } else {
        printf("Looks like PLPA is not working\n");
    }
    return 0;
}
---------------------------------------------------------------------------

You may need to supply appropriate -I and -L arguments to the
compiler/linker, respectively, to tell it where to find the PLPA
header and library files.  Also don't forget to supply -lplpa to link
in the PLPA library itself.  For example, if you configured PLPA with:

shell$ ./configure --prefix=$HOME/my-plpa-install

Then you would compile the above program with:

shell$ gcc my-plpa-test.c \
       -I$HOME/my-plpa-install/include \
       -L$HOME/my-plpa-install/lib -lplpa \
       -o my-plpa-test
shell$ ./my-plpa-test

If it compiles, links, runs, and prints "All is good!", then all
should be well.

===========================================================================

What license does PLPA use?
---------------------------

This package is distributed under the BSD license (see the LICENSE
file in the top-level directory of a PLPA distribution).  The
copyrights of several institutions appear throughout the code base
because some of the code was directly derived from the Open MPI
project (http://www.open-mpi.org/), which is also distributed under
the BSD license.

===========================================================================

How do I get involved in PLPA?
------------------------------

The PLPA continues to evolve, particularly as core counts increase and
internal host topology becomes more important.  We want to hear your
opinions.

The best way to report bugs, send comments, or ask questions is to
sign up on the user's mailing list:

        plpa-users@open-mpi.org

Because of spam, only subscribers are allowed to post to this list
(ensure that you subscribe with and post from exactly the same e-mail
address -- joe@example.com is considered different than
joe@mycomputer.example.com!).  Visit this page to subscribe to the
list:

     http://www.open-mpi.org/mailman/listinfo.cgi/plpa-users

Thanks for your time.