libc/kernel/README.TXT

Bionic comes with a set of 'clean' Linux kernel headers that can safely be
included by userland applications and libraries without fear of hideous
conflicts. for more information why this is needed, see the "RATIONALE"
section at the end of this document.

these clean headers are automatically generated by several scripts located
in the 'bionic/kernel/tools' directory, which process a set of original
and unmodified kernel headers in order to get rid of many annoying
declarations and constructs that usually result in compilation failure.

the 'clean headers' only contain type and macro definitions, with the
exception of a couple static inline functions used for performance
reason (e.g. optimized CPU-specific byte-swapping routines)

they can be included from C++, or when compiling code in strict ANSI mode.
they can be also included before or after any Bionic C library header.

the generation process works as follows:

  * 'external/kernel-headers/original/'
    contains a set of kernel headers as normally found in the 'include'
    directory of a normal Linux kernel source tree. note that this should
    only contain the files that are really needed by Android (use
    'find_headers.py' to find these automatically).

  * 'bionic/libc/kernel/common'
    contains the non-arch-specific clean headers and directories
    (e.g. linux, asm-generic and mtd)

  * 'bionic/libc/kernel/arch-arm/'
    contains the ARM-specific directory tree of clean headers.

  * 'bionic/libc/kernel/arch-arm/asm'
    contains the real ARM-specific headers

  * 'bionic/libc/kernel/arch-x86'
    'bionic/libc/kernel/arch-x86/asm'
    similarly contains all headers and symlinks to be used on x86

  * 'bionic/libc/kernel/tools' contains various Python and shell scripts used
    to manage and re-generate the headers

the tools you can use are:

  * tools/find_users.py
    scans a list of source files or directories and prints which ones do
    include Linux headers.

  * tools/find_headers.py
    scans a list of source files or directories and recursively finds all
    the original kernel headers they need.

  * tools/clean_header.py
    prints the clean version of a given kernel header. with the -u option,
    this will also update the corresponding clean header file if its
    content has changed. you can also process more than one file with -u

  * tools/update_all.py
    automatically update all clean headers from the content of 
    'external/kernel-headers/original'. this is the script you're likely going to
    run whenever you update the original headers.


HOW TO BUILD BIONIC AND OTHER PROGRAMS WITH THE CLEAN HEADERS:
==============================================================

add bionic/kernel/common and bionic/kernel/arch-<yourarch> to your C
include path. that should be enough. Note that Bionic will not compile properly 
if you don't.


HOW TO SUPPORT ANOTHER ARCHITECTURE:
====================================

see the content of tools/defaults.py, you will need to make a few updates
here:

  - add a new item to the 'kernel_archs' list of supported architectures

  - add a proper definition for 'kernel_known_<arch>_statics' with
    relevant definitions.

  - update 'kernel_known_statics' to map "<arch>" to
    'kernel_known_<arch>_statics'

then, add the new architecture-specific headers to original/asm-<arch>.
(please ensure that these are really needed, e.g. with tools/find_headers.py)

finally, run tools/update_all.py



HOW TO UPDATE THE HEADERS WHEN NEEDED:
======================================

IMPORTANT IMPORTANT:

  WHEN UPDATING THE HEADERS, ALWAYS CHECK THAT THE NEW CLEAN HEADERS DO
  NOT BREAK THE KERNEL <-> USER ABI, FOR EXAMPLE BY CHANGING THE SIZE
  OF A GIVEN TYPE. THIS TASK CANNOT BE EASILY AUTOMATED AT THE MOMENT

copy any updated kernel header into the corresponding location under
'bionic/kernel/original'.

for any new kernel header you want to add, first run tools/find_headers.py to be
sure that it is really needed by the Android sources. then add it to
'bionic/kernel/original'

then, run tools/update_all.py to re-run the auto-cleaning



HOW THE CLEANUP PROCESS WORKS:
==============================

this section describes the action performed by the cleanup program(s) when they
process the original kernel headers into clean ones:

1. Optimize well-known macros (e.g. __KERNEL__, __KERNEL_STRICT_NAMES)

    this pass gets rid of everything that is guarded by a well-known macro
    definition. this means that a block like

       #ifdef __KERNEL__
       ....
       #endif

    will be totally omitted from the output. the optimizer is smart enough to
    handle all complex C-preprocessor conditional expression appropriately.
    this means that, for example:

       #if defined(__KERNEL__) || defined(FOO)
       ...
       #endif

    will be transformed into:

       #ifdef FOO
       ...
       #endif

    see tools/defaults.py for the list of well-known macros used in this pass,
    in case you need to update it in the future.

    note that this also remove any reference to a kernel-specific configuration
    macro like CONFIG_FOO from the clean headers.


2. remove variable and function declarations:

  this pass scans non-directive text and only keeps things that look like a
  typedef/struct/union/enum declaration. this allows to get rid of any variable
  or function declaration that should only be used within the kernel anyway
  (and which normally *should* be guarded in a #ifdef __KERNEL__ ... #endif
  block, if the kernel writers were not so messy)

  there are however a few exceptions: it is seldom useful to keep the definition
  of some static inline functions performing very simple operations. a good
  example is the optimized 32-bit byte-swap function found in
  arch-arm/asm/byteorder.h

  the list of exceptions is in tools/defaults.py in case you need to update it
  in the future.

  note that we do *not* remove macro definitions, including these macro that
  perform a call to one of these kernel-header functions, or even define other
  functions. we consider it safe since userland applications have no business
  using them anyway.


3. whitespace cleanup:

  the final pass remove any comments and empty lines from the final headers.


4. add a standard disclaimer:

  prepended to each generated header, contains a message like
  "do not edit directly - file was auto-generated by ...."


RATIONALE:
==========

OVERVIEW OF THE CURRENT KERNEL HEADER MESS:
-------------------------------------------

The original kernel headers are not easily usable from userland applications.
they contain many declarations and construct that will result in a compilation
failure or even worse, incorrect behaviour. for example:

- some headers try to define Posix types (e.g. size_t, ssize_t) that can
  conflict with the corresponding definitions provided by your C library.

- some headers use constructs that cannot be compiled in ANSI C mode.

- some headers use constructs do not compile with C++ at all.

- some headers contain invalid "legacy" definitions for the benefit of old
  C libraries (e.g. glibc5) but result in incorrect behaviour if used
  directly.

  e.g. gid_t being defined in <linux/types.h> as a 16-bit type while the
  kernel uses 32-bit ids. this results in problems when getgroups() or
  setgroups() are called, since they operate on gid_t arrays.

unfortunately, these headers are also the only source of some really extensive
constant and type definitions that are required by userland applications.
think any library/program that need to access ALSA, or Video4Linux, or
anything related to a specific device or Linux-specific system interface
(e.g. IOCTLS, etc...)

As a consequence, every Linux distribution provides a set of patched kernel
headers to be used by userland applications (which installs in
/usr/include/linux/, /usr/include/asm/, etc...). these are manually maintained
by distribution packagers, and generated either manually or with various
scripts. these headers are also tailored to GNU LibC and cannot be reused
easily by Bionic.

for a really long period, the kernel authors have stated that they don't want
to fix the problem, even when someone proposed a patch to start cleaning the
official headers. from their point of view this is purely a library author
problem.

fortunately, enlightnment happened, and the kernel now provides a way to
install a set of "user-friendly" headers that are generated from the official
ones by stripping the __KERNEL__ protected declarations.

unfortunately, this is not enough for Bionic because the result still contains
a few broken declarations that are difficult to route around. (see below for
a little bit of details).

we plan to be able to support these kernel-generated user-land headers in the
future, but the priority on this issue is very low.


WHAT WE DO:
-----------

so we're doomed to repeat the same effort than anyone else. the big difference
here is that we want to automate as much as possible the generation of the
clean headers to easily support additional architectures in the future,
and keep current with upstream changes in the header definitions with the
least possible hassle.

of course, this is only a race to the bottom. the kernel maintainers still
feel free to randomly break the structure of their headers (e.g. moving the
location of some files) occasionally, so we'll need to keep up with that by
updating our build script/original headers as these cases happen.

what we do is keep a set of "original" kernel headers, and process them
automatically to generate a set of "clean" headers that can be used from
userland and the C library.

note that the "original" headers can be tweaked a little to avoid some subtle
issues. for example:

- when the location of various USB-related headers changes in the kernel
  source tree, we want to keep them at the same location in our generated
  headers (there is no reason to break the userland API for something
  like that).

- sometimes, we prefer to take certain things out of blocks guarded by a
  #ifdef __KERNEL__ .. #endif. for example, on recent kernels <linux/wireless.h>
  only includes <linux/if.h> when in kernel mode. we make it available to
  userland as well since some code out there assumes that this is the case.

- sometimes, the header is simply incorrect (e.g. it uses a type without
  including the header that defines it before-hand)