Porting GHCJS Template Haskell to GHC

GHCJS uses an external process to run Template Haskell code. It compiles each splice to JavaScript, collects its dependencies and runs it on a node.js server. The server stays active between splices to support persistent state in the Quasi monad.

The goal is to get out of process Template Haskell merged into GHC 7.10, so that cross compilers can use it. If performance is good enough, we can make it the normal way that TH is run, since it has some additional advantages.

There are basically two steps. The first is to make compiling the splices to libraries for the target platform work correctly, and to make sure that the libraries can be loaded. Once this works, we should merge the serialization and implement communication between the compiler and server.

Build and load TH code for the target platform:

1. build a program that can load a function from a dylib and run it:

   the loader:

   -- load a library and use a (String -> IO String) function from that library
   runAction :: FilePath -> IO ()
   runAction file = do
     l <- loadDynamicLibrary file
     f <- loadFunction l -- f :: String -> IO String
     print =<< f "hello, world"

   the dylib:

   -- export this somehow
   exportedToUpper :: String -> IO String
   exportedToUpper = return . map toUpper

2. Load multiple dynamic libraries with different dependencies than the loader program. Make sure that the libraries share state with the loader (i.e. mutation in one library is visible everywhere).

   main program looks like:

   -- load (IO ()) actions from the dynamic libraries and run them
   main = do
     someAction1 <- loadLibraryAndGetSymbol "test1.dylib" -- someAction1 :: IO ()
     someAction1
     someAction2 <- loadLibraryAndGetSymbol "test2.dylib" -- someAction2 :: IO ()
     someAction2

   Make sure that the libraries have different but partially overlapping dependencies and that all mutation is consistent everywhere. For example some top-level mutable value for the base package that you can check is the locale encoding:

   -- example export to check that base is loaded correctly
   import GHC.IO.Encoding
   
   testLocaleEncoding :: IO ()
   testLocaleEncoding = do
     putStrLn "locale encoding library 1, before/after:"
     print =<< getLocaleEncoding
     setLocaleEncoding latin1 -- use a different encoding for each lib
     print =<< getLocaleEncoding
     -- change should now be visible in the Runner and other loaded libraries

3. Write a small test code loader that loads some TH code from libraries and runs it.

   main = forever $ do
     file   <- waitForNewLibraryFile
     action <- loadTemplateHaskellSpliceLibrary file -- action :: Quasi m => m Exp
     print =<< Language.Haskell.TH.runQ action

4. Use libraries loading code from GHC hook

   The goal is to make sure that the splices and their dependencies are loaded correctly. The server does not actually return a result to the compiler here yet, so we use a hardcoded result.

   • Either edit ghc/Main.hs in the GHC repository or copy the file into your own package to have a standalone executable. GHCJS uses a customized version of this file, see Compiler.Program in the ghcjs package.

   • Use the GHC Hooks to set a hook for hscCompileCoreExpr. Your hook will be called whenever a splice needs to be compiled and run. The result is returned as an HValue, you just use unsafeCoerce to convert your actual value to that. See Compiler.Program and Compiler.GhcjsHooks for an example of how to set up hooks.

   • Test the hook first by printing a message and immediately returning a hardcoded result of type Quasi m => m Exp (unsafeCoerced to an HValue). Only test with Template Haskell splices that produce an Exp, so we can just assume that this is the expected type (see Gen2.TH.ghcjsCompileCoreExpr and Gen2.TH.runTh for how GHCJS handles multiple types, it really is a hack currently)

     -- example hardcoded return value, returning Nothing (make sure that the result typechecks)
     result :: HValue
     result = unsafeCoerce exampleExpr
     
     exampleExpr :: Quasi m => m Expr
     exampleExpr = return (ConE (mkName "Nothing"))

   • Extend the hook to compile the core expr to a splice dylib. Use some unique module and package name for each splice, do not use the name of the package you're compiling. See Gen2.TH.ghcjsCompileCoreExpr for and example of how to construct a binding and module for the expression.

   • Load the splice dylib on the server manually, make sure that it runs correctly (use qRunIO to run IO actions) and produces the correct result.

   • Automate loading the dylibs from the hscCompileCoreExpr hook and make sure that running multiple splices for a module works correctly without restarting the server. Don't bother getting the result to GHC yet, just return a hardcoded result like before.

   • Create a test package with modules A, B, C where all contain TH splices, B depends on A, C depends on A and B. This will require code to be loaded from an incomplete package. You'll need to build a dylib each time you need to run TH for a new module. Your compiled splices will depend on these libraries.

     • Ideally you build a dylib with only the new modules since the last dylib you built, but this isn't absolutely necessary.
     • Contrary to code for splices, which gets passed explicitly to hscCompileCoreExpr and compiled to a special module name, the modules A, B, C are just part of the package GHC is compiling. You may need to query DynFlags to find their object files or use other hooks (like Hooks.runPhaseHook) to catch them as they are being built.

After loading shared libraries works (for regular and cross compilers), support for static linking should be added for platforms that do not have dynamic libs (or GHC installations that do not have dynamic libs built):

• In the hscCompileCoreExpr hook, when the first splice is requested, compile a Template Haskell runner program, statically linked with all dependencies.
• The runner program should load the object files from the current package as they come in, using the GHCi linker (Linker module).
• You may want to split off the parts of the GHC library that need to be built for the target environment to a separate package.

Merge template-haskell changes and implement communication between the server and client.

• Binary instances in template-haskell
• Adjust GHC API code to get rid of HValue and unsafeCoerce