Huge rewrite, many fixes

.editorconfig

@@ -0,0 +1,18 @@
+# EditorConfig is awesome: https://EditorConfig.org
+
+# top-most EditorConfig file
+root = true
+
+[*]
+indent_style = tab
+indent_size = 4
+end_of_line = lf
+charset = utf-8
+trim_trailing_whitespace = true
+insert_final_newline = false
+
+[*.py]
+indent_style = space
+
+[*.nix]
+indent_style = space

.gitignore

@@ -1,3 +1,8 @@
 __pycache__/
-/state
-/zip2tarindex/target/
+/scripts/zip2tarindex/target/
+
+/build
+/tests/build
+
+*.ignore
+*.iso

README.md

@@ -1,142 +1,145 @@
-# Tectonic TeXLive Bundle Builder
+# Tectonic Bundle Builder
+
+This repository contains scripts for building bundles for
+[Tectonic](https://tectonic-typesetting.github.io), each of which is a complete TeX distribution.
+
+**You do not need this repository to build Tectonic.** \
+You only need these scripts if you want to make your own bundles of TeX files.
+
+**Warning:** The `./tests` do not work yet, they still need to be reworked for the new bundle spec!
+
+
+
+
 
-This repository contains scripts for building “bundles” for
-[Tectonic](https://tectonic-typesetting.github.io) based on [Norbert Preining’s
-Git mirror](http://git.texlive.info/texlive/) of [the TeXLive Subversion
-repository](http://tug.org/svn/texlive/).
 
-*You do not need this repository to build Tectonic.* You only need these scripts
-if you want to make your own bundle of TeX files based on the TeXLive sources.
 
 
 ## Prerequisites
 
 To use these tools, you will need:
 
+- Bash
+- Python 3.11 & Python standard packages
+- GNU `patch` and `diff`
 - An installation of [Docker](https://www.docker.com/).
-- A checkout of the Preining TeXLive Git repository
-  (`git://git.texlive.info/texlive.git`), placed or symlinked in a subdirectory
-  named `state/repo` below this file. Be aware that this repository currently
-  weighs in at **40 gigabytes**!
-- A Rust toolchain if you want to create “indexed tar” bundles. (So, you don’t
-  need Rust if you want to create a bundle and test it locally.)
+- A [TeXlive iso](https://tug.org/texlive/acquire-iso.html). Different bundles need different TeXlive versions.
+- A Rust toolchain if you want to create “indexed tar” bundles. You don’t
+  need Rust if you want to create a bundle and test it locally.
+
+This repo also contains a `shell.nix` with pinned versions that contains all dependencies.
+
+
+
+
+
+
+
+
+## Bundles:
+Each directory in `./bundles` is a *bundle specification* which contains everything we need to reproducibly build a bundle.\
+See [`./bundles/README.md`](./bundles/README.md) for details.
+
+The following bundles are available:
+ - `texlive/2022.0r0`: directly copied from the bundle in `master`. \
+ Uses `texlive-2022.0r0`, and is probably broken.
 
-Data files associated with the staging process will land in other subdirectories
-of `state/`.
+ - `texlive2023-nopatch`: based on `texlive2023-20230313`.
 
 
-## Getting started: creating the bundler image
 
-The first step is to create a Docker container that will host most of the
-computations — this promotes reproducibility and portability, regardless of what
-kind of system you are using. To create this container, run:
 
-```
-./driver.sh build-image
-```
 
 
-## Creating TeXLive containers
 
-The next step is to create TeXLive “containers” — which are different than
-Docker containers. A *Docker* container is an encapsulated Linux machine that
-provides a reproducible build environment. *TeXLive* containers are archives
-containing the files associated with the various TeXLive packages.
 
-To create TeXLive container files for all of the packages associated with your
-TeXLive checkout, run:
 
-```
-./driver.sh update-containers
-```
+## Build Process:
+Before building any bundles, acquire a [TeXlive iso](https://tug.org/texlive/acquire-iso.html) with a version that matches the bundle you want to build. `build.sh` checks the hash of this file when you run `install`.
 
-This will use the Docker container to generate TeXLive container files in
-`state/containers`. *The results of this step will depend on what version of the
-TeXLive tree you currently have checked out in `state/repo`.*
+To build a bundle, run the following jobs. This **must** be run in order!
 
+ - `./build container`: builds the docker container from `./docker`
+ - `./build <bundle> install <iso>`: installs TeXLive to `./build/install/`
+ - `./build <bundle> content`: assemble all files into a bundle at `./build/output/content`.\
+  This will delete all bundles in `output/<bundle>/`, move them elsewhere if you still need them.
 
-## Creating a TeXLive installation tree
+Once `./build/output/content` has been created, run any of the following commands to package the bundle:
 
-Run:
+ - `./build <bundle> zip`: create a zip bundle from the content directory.\
+  Zip bundles can only be used locally, they may not be hosted on the web.
 
-```
-./driver.sh make-installation bundles/tlextras
-./driver.sh install-packages bundles/tlextras
-```
+ - `./build <bundle> itar`: create an indexed tar bundle from the content directory. \
+ These cannot be used locally, itar bundles must be used as web bundles. \
+ If you want to host your own, you'll need to put `bundle.tar` and `<bundle>.tar.sha256sum` under the same url.
 
-(In the future, we might add more specifications to the `bundles` directory for
-creating specialized bundles. The `tlextras` bundle is the one-size-fits-all
-default bundle.)
+`build.sh` also provides a few shortcuts:
+ - `./build <bundle> all <iso>`: Runs all the above jobs, *including* a full re-install.
+ - `./build <bundle> most <iso>`: Runs all jobs except `container` and `install`.
+ - `./build <bundle> package`: Runs `zip` and `itar`. Assumes `content` has already been run.
 
 
-## Updating patches
 
-As of TeXLive 2021, we have bitten the bullet and decided to maintain some
-patches against the TeXLive tree.
+### Build Notes & Troubleshooting:
+ - The `install` job could take a while. `tail -f` its log file to watch progress.
+ - `install` will fail if your iso hash does not match the hash of the iso the bundle was designed for.\
+ This may be overriden by replacing `./build.sh <bundle> install <iso>` with `./build.sh <bundle> forceinstall <iso>`.
+ - the `install` job occasionally throws the following error: `mount: /iso-mount: failed to setup loop device for /iso.img.`\
+ Run the job again, it should work. We don't yet know why this happens.
+ - You do not need to run `install` every time you change a bundle. In fact, the contents of TeXlive installations should NEVER change. You only need to install each version of TeXlive once.\
+ If you're building multiple bundles from the same TeXLive version, you could install once then copy & rename that installation to save time. Automating this would add a bit of needless complexity to the build process, but we may implement it later.
 
-Maintaining long-lived patches is never fun, but Git makes life a lot easier
-than it could be. We use a secondary branch named `vendor-pristine` to help
-maintain our patches. The way we do that is to copy the “vendor” (TeXLive
-original) files into branch, then use `git merge` to update the main branch with
-whatever changes have been introduced between TeXLive updates.
 
-First, bump the version of your bundle and run the standard update steps through
-the `install-packages` step described above. Make sure that the current branch
-is clean with no changes in the working tree or index. Then run:
 
-```
-./driver.sh get-vendor-pristine bundles/tlextras
-```
 
-Then follow the suggested workflow as printed out by that command. The basic
-plan is to commit the vendor files into the bundle’s `patched/` directory *on
-the vendor-pristine* branch, then merge them back into the main branch.
 
 
-## Exporting to a Zip-format bundle
+## Output Files
 
-Run:
 
-```
-./driver.sh make-zipfile bundles/tlextras
-```
+**`./build.sh <bundle> content` produces the following:**
+ - `./build/output/<bundle>/content`: contains all bundle files. This directory also contains some metadata:
+   - `content/INDEX`: each line of this file maps a filename in the bundle to a relative path.
+   - `content/SHA256SUM`: a hash of this bundle's contents.
+   - `content/TEXLIVE-SHA265SUM`: a hash of the TeXlive image used to build this bundle.
+ - `listing`: a sorted list of all files in the bundle
+ - `clash-report`: debug file. did any files have the same name? (if any)
+ - `file-hashes`: debug file. Indexes the contents of the bundle. Used to find which files differ between two builds.
+  `file-hashes` and `content/SHA265SUM` are generated in roughly the same way, so the `file-hashes` files from two different bundles should match if and only if the two bundles have the same sha256sum
 
-This will create a large Zip-format bundle file with a name something like
-`state/artifacts/tlextras-2020.0r0/tlextras-2020.0r0.zip`. Such a bundle file
-can be used with the `tectonic` command-line program with the `-b` argument.
 
+**`./build.sh <bundle> zip` produces the following:**
+ - `<bundle>.zip`: the main zip bundle
 
-## Converting to an “indexed tar” bundle
 
-This step is needed to create a bundle that will be hosted on the web. Run:
 
-```
-./driver.sh make-itar bundles/tlextras
-```
+**`./build.sh <bundle> itar` produces the following:**\
+Note that both `<bundle>.tar` and `<bundle>.tar.index.gz` are required to host a web bundle.
+ - `<bundle>.tar`: the tar bundle
+ - `<bundle>.tar.index.gz`: the (compressed) tar index, with format `<file> <start> <len>`\
+ This tells us that the first bit of `<file>` is at `<start>`, and the last is at `<start> + <len> - 1`.\
+ You can extract a file from a local bundle using `dd if=file.tar ibs=1 skip=<start> count=<len>`\
+ Or from a web bundle with `curl -r <start>-<start>+<len> https://url.tar`
 
-This will create both the `.tar` and the `.tar.index.gz` files that need to be
-uploaded for use as a web bundle.
 
 
-## Testing
 
-Bundle definitions come with testing information. To test a bundle, you need the
-`tectonic` command-line program to be in your $PATH, as well as a Python 3
-interpreter and the [toml] package.
 
-[toml]: https://pypi.org/project/toml/
 
-Test scripts are located in the `tests` directory. Currently available:
 
-- `tests/classes.py`: basic compilation smoketest of the documentclasses in a bundle
-- `tests/formats.py`: test generation of the format files defined in the bundle
-- `tests/packages.py`: test loading if the package (style) files defined in the
-  bundle. There are thousands of style files in a typical bundle, so this
-  program uses a framework to run a random-but-reproducible subset of the tests.
-  See the header comment in the Python file for more information.
+## Reproducibility
+The `SHA256HASH` stored in each bundle should stay the same between builds. \
+Below is a list of "problem files" that have made bit-perfect rebuilds difficult in the past:
 
+ - The following contain a timestamp:
+   - `fmtutil.cnf`
+   - `mf.base`
+   - `updmap.cfg`
+ - The following contain a UUID: (Most of these have a UUID *and* a timestamp)
+   - `357744afc7b3a35aafa10e21352f18c5.luc`
+   - `929f6dbc83f6d3b65dab91f1efa4aacb.luc`
+   - `b4a1d8ccc0c60e24e909f01c247f0a0f.luc`
 
-#### Copyright and Licensing
+Fortunately, installing TeXlive with `faketime -f` seems to pin both UUIDs and timestamps.\
+The date each bundle is built on is defined in its specification.
 
-The infrastructure scripts in this repository are licensed under the MIT
-License. Their copyright is assigned to the Tectonic Project.