DEVEL

Information for developpers

OVERVIEW

   Although the sources of camlp5 are written in camlp5 syntaxes, the
   distribution contain no 'magic' binary file for the first compilation.

     This first compilation can be done thanks to a version of the core of
   camlp5 written in pure ocaml syntax. Once this 'core' is compiled, all
   sources in revised syntax with syntax extensions can be compiled.

COMPATIBILITY

   This distribution of camlp5 is compatible with several versions of
   ocaml. The problem is about the definition of ocaml syntax trees which
   may change from a version of ocaml to another. Since ocaml does not
   install the sources nor the compiled versions of its syntax tree, a
   copy of the necessary source files, borrowed from the source of the
   ocaml compiler has been done in the directory 'ocaml_stuff', in
   subdirectories with the ocaml version number.

     If the present distribution of camlp5 is not compatible with the
   version of ocaml you have (the command 'configure' tells you), it
   is possible to add it. For that, you need the sources of the ocaml
   distribution you have. Once done, after a 'configure' telling you
   that it is not compatible, do:
       make steal OCAML_SRC=<path-to-ocaml-sources>

     This creates a new directory in 'ocaml_stuff' with sources of the
   syntax tree of your ocaml compiler.

     If you want to check that the sources of the syntax tree of ocaml
   are up-to-date (e.g. if this is the current ocaml developpement), do:
       make compare_stolen OCAML_SRC=<path-to-ocaml-sources>

     The compatibility is also done with the file 'lib/versdep.ml',
   which is a module containing miscellaneous features depending to
   the version of OCaml.

     In the directory 'ocaml_src' which contains the pure OCaml sources
   of the Camlp5 core (see chapter TREE STRUCTURE below), there are as
   many versions of this files as versions of OCaml. They are named
   '<version>.ml' in the directory 'lib/versdep'. If you are adding
   a new version of OCaml, you need this file. As a first step, make a
   copy from a close version:
         cd ocaml_src/lib/versdep
         cp <close_version>.ml <version>.ml

     Then, you can redo "configure" and do "make core". If the file
   'ocaml_src/lib/versdep.ml' has a compilation problems, fix them and
   to 'make core' again.

     Later, the same file 'lib/versdep.ml' in camlp5 syntax may have
   similar compilation problem. This file is in one examplary, thanks to
   IFDEF used here or there.

     While compiling with some specific version of ocaml, this file is
   compiled with 'OCAML_vers' defined where 'vers' is the version number
   form the beginning to the first space or charcter '+' with all dots
   converted into underscores. For example, if your ocaml version is
   7.04.2+dev35, you can see in the compilation process of versdep.ml
   that OCAML_7_04_2 is defined, and you can add statements defined
   by the syntax extension 'pa_macro.cmo', for example IFDEF OCAML_7_04_2.
   Add statements like that in 'lib/versdep.ml' to make it compile.

TREE STRUCTURE

   The directory 'ocaml_src' contains images in pure ocaml syntax of the
   directories odyl lib main and meta. This allow to create a core version
   of camlp5 from only the ocaml compiler installed.

     You can decompose the building of the camlp5 core into:
       1. make library_cold
            just makes the directory 'ocaml_src/lib' and copy the cmo and cmi
            files into the directory 'boot'
       2. make compile_cold
            makes the other directories of ocaml_src
       3. make promote_cold
            copies the executables "camlp5", "camlp5r" and the syntax
            extensions (cmo files) into the directory 'boot'

     From that point, the core camlp5 is in directory 'boot'. The real
   sources in the top directories odyl lib main and meta, which are
   written in revised syntax with some syntax extensions (grammars,
   quotations) can be compiled. To achieve their compilation, you can
   do:
       make core

     Or to compile everything do:
       make all
     or just:
       make

     Notice that doing "make core" or "make all" from scratch (after a
   make clean), automatically start by making the core files from their
   pure ocaml versions.

FAST COMPILATION FROM SCRATCH

   ./configure
   make clean core compare
   make coreboot
   make all opt opt.opt

TESTING CHANGES

   1. do your changes
   2. do:
         make core compare
      if it says that the bootstrap is ok, you can do:
         make all
         make opt
         make opt.opt
      otherwise, to make sure everything is ok, first do:
         make coreboot
      sometimes two bootstraps ('make coreboot' twice) are necessary,
      in particular if you change things in the directory 'lib'. It is
      even possible that three bootstraps are necessary.

        If things goes wrong, it is possible to return to the previous
      version by typing:
         make restore clean_hot
      then you can change what is necessary and continue by typing:
         make core
      and test the bootstrap again:
         make coreboot

        After several bootstraps (by 'make coreboot' or 'make bootstrap'),
      many versions are pushed in the directory 'boot' (you can type
      'find boot -type d -print' to see that). If your system correctly
      bootstraps, you can clean that by typing:
         make cleanboot
      which keeps only two versions. (The command 'make clean' also removes
      these stack of versions.)

BEFORE COMMITTING YOUR CHANGES

   Make sure that the cold start with pure ocaml sources work. For that, do:
       make compare_sources | less

     This shows you the changes that would be done in the ocaml pure sources
   of the directory ocaml_src.

     To make the new versions, do:
       make new_sources
       make promote_sources

     Notice that these pure ocaml sources are not supposed to be modified
   by hand, but only created by the above commands. Besides, their sources,
   although pretty printed, are sometimes not easy to read, particularly for
   expanded grammars (of the statement 'EXTEND').
     However, if these sources do not compile, due to changes in the ocaml
   compiler, it is possible to edit them. In this case, similar changes
   may have to be done in the normal sources in revised syntax.

     After doing 'make new_sources' above, and before doing 'make
   promote_sources' below, it is possible to do 'make untouch_sources'
   which change the dates of the new created files with the dates of the
   old files if they are not modified. This way, the "svn commit" will not
   have to compare these files, which may have some importance if you
   network is not fast.

     The 'make new_sources' builds a directory named 'ocaml_src.new'.
   If this directory still exists, due to a previous 'make new_sources',
   the command fails. In this case, just delete it (rm -rf ocaml_src.new)
   without problem: this directory is not part of the distribution, it is
   just temporary.

     The 'make clean_sources' deletes old versions of ocaml_src, keeping
   only the last and the before last ones.

     The command:
       make bootstrap_sources
   is a shortcut for:
       make new_sources
       make untouch_sources
       make promote_sources
       make clean_sources

     If there are changes in the specific file 'lib/versdep.ml', do
   also:
       make compare_all_versdep
   and possibly:
       make bootstrap_all_versdep
   because this file, in 'ocaml_src/lib/versdep' directory has different
   versions according to the ocaml version.

     After having rebuilt the pure ocaml sources, check they work by
   rebuilding everything from scratch, starting with "configure".

IF YOU CHANGE THE MAIN PARSER

   If you change the main parser 'meta/pa_r.ml', you may check that the
   quotations expanders of syntax tree 'meta/q_MLast.ml' match the new
   version. For that, do:
       cd meta
       make compare_q_MLast

     If no differences are displayed, it means that 'q_MLast.ml' is ok,
   relatively to 'pa_r.ml'.

     Otherwise, if the displayed differences seem reasonable, update the
   version by typing:
       make bootstrap_q_MLast

     Then returning to the top directory, do 'make core compare' and
   possibly 'make coreboot' (one of several times) to check the
   correctness of the file.

     And don't forget, if you want to commit, to re-create the pure ocaml
   sources like indicated above.