-
Notifications
You must be signed in to change notification settings - Fork 4
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #14 from rm-dr/rework
Huge rewrite, many fixes
- Loading branch information
Showing
79 changed files
with
2,307 additions
and
55,556 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
# EditorConfig is awesome: https://EditorConfig.org | ||
|
||
# top-most EditorConfig file | ||
root = true | ||
|
||
[*] | ||
indent_style = space | ||
indent_size = 4 | ||
end_of_line = lf | ||
charset = utf-8 | ||
trim_trailing_whitespace = true | ||
insert_final_newline = false | ||
|
||
[*.sh] | ||
indent_style = tab |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,3 +1,9 @@ | ||
__pycache__/ | ||
/state | ||
/zip2tarindex/target/ | ||
/scripts/*/target/ | ||
|
||
/build | ||
/tests/build | ||
|
||
*.ignore | ||
*.tar | ||
*.tar.xz |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,142 +1,115 @@ | ||
# Tectonic TeXLive Bundle Builder | ||
# Tectonic Bundles | ||
|
||
This repository contains scripts for building bundles for | ||
[Tectonic](https://tectonic-typesetting.github.io), each of which is a complete TeX distribution. | ||
|
||
**You do not need this repository to build Tectonic.** \ | ||
You only need these scripts if you want to make your own bundles of TeX files. | ||
|
||
**Warning:** The `./tests` do not work yet, they need to be reworked for the new bundle spec! | ||
|
||
|
||
This repository contains scripts for building “bundles” for | ||
[Tectonic](https://tectonic-typesetting.github.io) based on [Norbert Preining’s | ||
Git mirror](http://git.texlive.info/texlive/) of [the TeXLive Subversion | ||
repository](http://tug.org/svn/texlive/). | ||
|
||
*You do not need this repository to build Tectonic.* You only need these scripts | ||
if you want to make your own bundle of TeX files based on the TeXLive sources. | ||
|
||
|
||
## Prerequisites | ||
|
||
To use these tools, you will need: | ||
|
||
- An installation of [Docker](https://www.docker.com/). | ||
- A checkout of the Preining TeXLive Git repository | ||
(`git://git.texlive.info/texlive.git`), placed or symlinked in a subdirectory | ||
named `state/repo` below this file. Be aware that this repository currently | ||
weighs in at **40 gigabytes**! | ||
- A Rust toolchain if you want to create “indexed tar” bundles. (So, you don’t | ||
need Rust if you want to create a bundle and test it locally.) | ||
- Bash, `pv`, GNU `patch` and `diff` | ||
- A [TeXlive tarball](https://tug.org/texlive/acquire-tar.html). Different bundles need different TeXlive versions. | ||
- A Rust toolchain (`cargo`). | ||
|
||
Data files associated with the staging process will land in other subdirectories | ||
of `state/`. | ||
|
||
|
||
## Getting started: creating the bundler image | ||
|
||
The first step is to create a Docker container that will host most of the | ||
computations — this promotes reproducibility and portability, regardless of what | ||
kind of system you are using. To create this container, run: | ||
## Bundles: | ||
Each directory in `./bundles` is a bundle specification, which contains everything we need to reproducibly build a bundle.\ | ||
See [`./bundles/README.md`](./bundles/README.md) for details. | ||
|
||
``` | ||
./driver.sh build-image | ||
``` | ||
The following bundles are available: | ||
- `texlive2023-nopatch`: based on `texlive2023-20230313`. | ||
|
||
|
||
## Creating TeXLive containers | ||
|
||
The next step is to create TeXLive “containers” — which are different than | ||
Docker containers. A *Docker* container is an encapsulated Linux machine that | ||
provides a reproducible build environment. *TeXLive* containers are archives | ||
containing the files associated with the various TeXLive packages. | ||
|
||
To create TeXLive container files for all of the packages associated with your | ||
TeXLive checkout, run: | ||
|
||
``` | ||
./driver.sh update-containers | ||
``` | ||
|
||
This will use the Docker container to generate TeXLive container files in | ||
`state/containers`. *The results of this step will depend on what version of the | ||
TeXLive tree you currently have checked out in `state/repo`.* | ||
## Build Process: | ||
Before building any bundles, acquire a [TeXlive tarball](https://tug.org/texlive/acquire-tar.html) with a version that matches the bundle you want to build. These are usually distributed as compressed tarballs, which you'll have to manually decompress. **`build.sh` expects an uncompressed `.tar` file!** It checks the hash of this file, and will refuse to work if that hash doesn't match. | ||
|
||
The tarball you end up with should have a name of the form `texlive-YYYYMMDD-texmf.tar`. **Do not rename this file!** | ||
|
||
## Creating a TeXLive installation tree | ||
|
||
Run: | ||
|
||
``` | ||
./driver.sh make-installation bundles/tlextras | ||
./driver.sh install-packages bundles/tlextras | ||
``` | ||
To build a bundle, run the following jobs. These **must** be run in order! | ||
|
||
(In the future, we might add more specifications to the `bundles` directory for | ||
creating specialized bundles. The `tlextras` bundle is the one-size-fits-all | ||
default bundle.) | ||
- `./build.sh <tarball> extract`: extracts texlive into `./build/texlive/<version>`\ | ||
This also generates `TEXLIVE-SHA256SUM` in the texlive version directory. | ||
|
||
- `./build.sh <bundle> content`: assemble all files into a bundle at `./build/output/<bundle>content`.\ | ||
This will delete all bundles in `output/<bundle>/`, move them elsewhere if you still need them. | ||
|
||
## Updating patches | ||
Once `./build/output/content` has been created, run any of the following commands to package the bundle.\ | ||
See [`builder/README.md`](./scripts/builder/README.md) for details. | ||
|
||
As of TeXLive 2021, we have bitten the bullet and decided to maintain some | ||
patches against the TeXLive tree. | ||
- `./build.sh <bundle> ttbv1`: create a ttb (version 1) bundle from the content directory.\ | ||
TTB bundles may be used locally or hosted on the web. | ||
|
||
Maintaining long-lived patches is never fun, but Git makes life a lot easier | ||
than it could be. We use a secondary branch named `vendor-pristine` to help | ||
maintain our patches. The way we do that is to copy the “vendor” (TeXLive | ||
original) files into branch, then use `git merge` to update the main branch with | ||
whatever changes have been introduced between TeXLive updates. | ||
|
||
First, bump the version of your bundle and run the standard update steps through | ||
the `install-packages` step described above. Make sure that the current branch | ||
is clean with no changes in the working tree or index. Then run: | ||
|
||
``` | ||
./driver.sh get-vendor-pristine bundles/tlextras | ||
``` | ||
|
||
Then follow the suggested workflow as printed out by that command. The basic | ||
plan is to commit the vendor files into the bundle’s `patched/` directory *on | ||
the vendor-pristine* branch, then merge them back into the main branch. | ||
|
||
|
||
## Exporting to a Zip-format bundle | ||
## Output Files | ||
|
||
Run: | ||
|
||
``` | ||
./driver.sh make-zipfile bundles/tlextras | ||
``` | ||
**`./build.sh <bundle> content` produces the following:** | ||
- `./build/output/<bundle>/content`: contains all bundle files. It is organized by source: files from the bundle's `include` dir will be under `./include`, texlive files will be under `./texlive`, and so on. See `main.rs` of `scripts/select`. | ||
This directory also contains some metadata: | ||
- `content/FILES`: each line of this file is `<path> <hash>`, sorted by file name.\ | ||
Files with identical names are included.\ | ||
Files not in any search path are also included.\ | ||
`<hash>` is either a hex sha256 of that file's contents, or `nohash` for a few special files. | ||
- `content/SHA256SUM`: The sha256sum of `content/FILES`. This string uniquely defines this bundle. | ||
- `content/SEARCH`: File search order for this bundle. See bundle spec documentation. | ||
- `search-report`: debug file. Lists all directories that will not be searched by the rules in `search-order`.\ | ||
The entries in this file are non-recursive: If `search-report` contains a line with `/texlive`, this means that direct children of `/texlive` (like `/texlive/file.tex`) will not be found, but files in *subdirectories* (like `/texlive/tex/file.tex`) may be. | ||
|
||
This will create a large Zip-format bundle file with a name something like | ||
`state/artifacts/tlextras-2020.0r0/tlextras-2020.0r0.zip`. Such a bundle file | ||
can be used with the `tectonic` command-line program with the `-b` argument. | ||
|
||
**`./build.sh <bundle> ttbv1` produces the following:** | ||
- `<bundle>.ttb`: the bundle. Note that the ttb version is *not* included in the extension. | ||
- Index location and length are printed once this job completes. | ||
- You can extract files from this bundle by running `dd if=file.ttb ibs=1 skip=<start> count=<len> | gunzip` | ||
|
||
## Converting to an “indexed tar” bundle | ||
|
||
This step is needed to create a bundle that will be hosted on the web. Run: | ||
|
||
``` | ||
./driver.sh make-itar bundles/tlextras | ||
``` | ||
## Testing | ||
|
||
This will create both the `.tar` and the `.tar.index.gz` files that need to be | ||
uploaded for use as a web bundle. | ||
Tests are under `./tests`, and are currently a work in progress. All tests are run through `test.sh` as follows: `./test.sh <path-to-ttb> <test set>`. | ||
|
||
Tests require the following: | ||
- a `ttb` bundle (local or remote) | ||
- a recent installation of Tectonic | ||
|
||
## Testing | ||
### Test Sets | ||
The following test sets are avaiable: | ||
- `files`, which tries to compile all files under `tests/files` and `tests/formats` | ||
- `classes`, which tries to compile a simple document using `tests/classes.list` | ||
|
||
Bundle definitions come with testing information. To test a bundle, you need the | ||
`tectonic` command-line program to be in your $PATH, as well as a Python 3 | ||
interpreter and the [toml] package. | ||
Note that most test files contain comments explaining the reason and expected outcome of the test. | ||
|
||
[toml]: https://pypi.org/project/toml/ | ||
|
||
Test scripts are located in the `tests` directory. Currently available: | ||
|
||
- `tests/classes.py`: basic compilation smoketest of the documentclasses in a bundle | ||
- `tests/formats.py`: test generation of the format files defined in the bundle | ||
- `tests/packages.py`: test loading if the package (style) files defined in the | ||
bundle. There are thousands of style files in a typical bundle, so this | ||
program uses a framework to run a random-but-reproducible subset of the tests. | ||
See the header comment in the Python file for more information. | ||
### Test Output | ||
All test output ends up under `tests/build` | ||
|
||
**Output for `files`:** | ||
- `files/logs`: log files for all builds (passed or failed) | ||
- `files/*.{pdf,fmt,etc}`: output files for each build | ||
|
||
#### Copyright and Licensing | ||
|
||
The infrastructure scripts in this repository are licensed under the MIT | ||
License. Their copyright is assigned to the Tectonic Project. | ||
**Output for `classes` | ||
- `failed`: classes that failed to compile | ||
- `passed`: classes that complied without error | ||
- `logs`: log files for all compile jobs |
Oops, something went wrong.