README.md 6.47 KB
Newer Older
1 2 3 4 5
# OCaml-migrate-parsetree
Convert OCaml parsetrees between different major versions

This library converts between parsetrees of different OCaml versions.

6
Supported versions are 4.02, 4.03, 4.04, 4.05, 4.06, 4.07, 4.08 and 4.09.
7 8 9 10 11 12
For each version, there is a snapshot of the parsetree and conversion functions
to the next and/or previous version.

## Asts

```ocaml
13
module Ast_402, Ast_403, Ast_404, Ast_405, Ast_406, Ast_407, Ast_408, Ast_409 : sig
14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80

  (* These two modules didn't change between compiler versions.
     Just share the ones from compiler-libs. *)
  module Location = Location
  module Longident = Longident

  (* Version specific copy of AST *)
  module Asttypes
  module Parsetree
  module Outcometree

  (* Other modules that are useful for implementing PPX.

     Docstrings and Ast_mapper only contain general definitions
     In particular, the internal state used by compiler-libs has been
     removed.
     Also equalities are lost for abstract types (Docstring.docstring).  *)
  module Docstrings
  module Ast_helper
  module Ast_mapper

  (* Magic numbers used for marshalling *)
  module Config : sig
    val ast_impl_magic_number : string
    val ast_intf_magic_number : string
  end
end
```

These embed copies of AST definitions for each supported OCaml major version.

The AST matching the version of the OCaml toolchain will contain equalities
relating the copy of types to the definitions from compiler-libs.  For
instance, when installed with OCaml 4.04.x, `Ast_404.Parsetree` looks
like.

## Migration modules

For each pair of versions `$(n)` and `$(n+1)`, the two modules
`Migrate_parsetree_$(n)_$(n+1)` and `Migrate_parsetree_$(n+1)_$(n)` convert the AST forward and backward.

The forward conversion is total while the backward conversion is partial: when
a feature is not available in a previous version of the parsetree, a
`Migrate_parsetree_def.Migration_error` exception is raised detailing the
failure case.

`Migrate_parsetree_versions` abstract versions of the compiler. Each version is
represented as a module with `OCaml_version` signature.  Instances are named
`OCaml_402`, `OCaml_403`, ... `OCaml_current` is an alias to the version of the
current compiler.
The `Convert` functor takes two versions of OCaml and produce conversion
functions.

Finally, the `Migrate_parsetree_ast_io` provides an easy interface for
marshalling/unmarshalling.

## Migrate_parsetree.Driver

The `Migrate_parsetree.Driver` provides an API for ppx rewriters to
register OCaml AST rewriters. Ppx rewriters using this API can be used
as standalone rewriter executable or as part of a _driver_ including
several rewriters.

Using a single driver for several rewritings has the advantage that it
is faster. Especially when using many ppx rewriters, it can speed up
compilation a lot.

81 82 83
If using [Dune](https://github.com/ocaml/dune), you can
consult the dune manual to see how to define and use ppx
rewriters. Dune automatically creates drivers based on
84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120
ocaml-migrate-parsetree on demand.

The rest of this section describes how to do things manually or with
[ocamlbuild](https://github.com/ocaml/ocamlbuild).

## Building a custom driver using ocamlfind

To build a custom driver using ocamlfind, simply link all the ppx
rewriter libraries together with the
`ocaml-migrate-parsetree.driver-main` package at the end:

    ocamlfind ocamlopt -predicates ppx_driver -o ppx -linkpkg \
      -package ppx_sexp_conv -package ppx_bin_prot \
      -package ocaml-migrate-parsetree.driver-main

Normally, ocaml-migrate-parsetree based rewriters should be build with
the approriate `-linkall` option on individual libraries. If one is
missing this option, the rewriter might not get linked in. If this is
the case, a workaround is to pass `-linkall` when linking the custom
driver.

The resulting `ppx` program can be used as follow:

- `./ppx file.ml` to print the transformed code
- `ocamlc -pp './ppx --as-pp' ...` to use it as a pre-processor
- `ocamlc -ppx './ppx --as-ppx' ...` to use it as a `-ppx` rewriter

# Development

It started from the work of Alain Frisch in
[ppx\_tools](https://github.com/alainfrisch/ppx_tools).

The library is distributed under LGPL 2.1 and is copyright INRIA.

## Adding a new OCaml version

We use [Cinaps](https://github.com/janestreet/cinaps) to generate boilerplate.
121
You can install it via opam: `opam install cinaps`.
122 123

Add the new version in
124
[src/cinaps_helpers](https://github.com/ocaml-ppx/ocaml-migrate-parsetree/blob/master/src/cinaps_helpers)
125 126 127 128 129 130 131 132 133 134 135 136 137 138
`supported_versions`.

Snapshot the ast in file "asts/ast\_NEW.ml".
* Define the modules `Location` and `Longident` as aliases to corresponding
  modules from compiler-libs.
* Copy `Asttypes`, `Parsetree`, `Outcometree`, `Docstrings`, `Ast_helper` and
  `Ast_mapper` from the upstream files in `parsing/` directory.
* Global state and definitions referencing external values should be removed
  from `Docstrings` and `Ast_mapper`. Take a look at existing snapshots.
* Create a `Config` module containing `ast_impl_magic_number`
  `ast_impl_magic_number` from upstream `Config`
* Call `tools/add_special_comments.native` on the file

Add migration functions:
139 140
- Manually compile the asts (`ocamlc -c src/ast_{NEW,OLD}.ml -I +compiler-libs -I _build/default/src/.migrate_parsetree.objs/byte/ -open Migrate_parsetree__`)
- Using `tools/gencopy.exe` (`dune build tools/gencopy.exe`), generate copy code to and from previous version (assuming it is 408):
141
```
142 143
_build/default/tools/gencopy.exe -I . -I src/ -I +compiler-libs -map Ast_409:Ast_408 Ast_409.Parsetree.{expression,expr,pattern,pat,core_type,typ,toplevel_phrase} Ast_409.Outcometree.{out_phrase,out_type_extension} > src/migrate_parsetree_409_408_migrate.ml
_build/default/tools/gencopy.exe -I . -I src/ -I +compiler-libs -map Ast_408:Ast_409 Ast_408.Parsetree.{expression,expr,pattern,pat,core_type,typ,toplevel_phrase} Ast_408.Outcometree.{out_phrase,out_type_extension} > src/migrate_parsetree_408_409_migrate.ml
144 145
```
- Fix the generated code by implementing new cases
146
- The migration functor expects specific names, look at `Migrate_parsetree_versions` interface.
147 148 149

*TODO*: specialize and improve gencopy for these cases

150 151
Add mapper lifting functions in the files `migrate_parsetree_NEW_408.ml` and
`migrate_parsetree_408_NEW.ml`:
152 153 154 155 156 157 158
- include the corresponding `Migrate_parsetree_40x_40y_migrate` module
- define `copy_mapper` function, look at existing `Migrate_parsetree_40x_40y`
  for guidance.

At any time, you can expand boilerplate code by running `make cinaps`.

Update build system:
Andy Li's avatar
Andy Li committed
159
- make sure `make cinaps` reaches a fixed point :)
160
- `make` should succeed