Editor and Compile Process

This document provides a broad overview for the handling of documents and the compile process.

Note: This document described the compile process with sandboxed compiles as available in Server Pro only. In Server CE, the compile process uses simple sub-processes -- replace the items referencing a "container" with a single item "run compile in sub-process".

Components/Actors:

• user, a user of the Server CE/Pro application
• editor, the client application running in the browser
• clsi, the micro service used for compiling PDFs
• document-updater, the micro service used for processing document updates
• filestore, the micro service handling binary files
• real-time, the micro service used for handling web sockets
• web, the (not so) micro service used for handling API requests

Redis caching:

• user: loads editor page

• editor: opens web socket

• editor: sends request to open a document via web socket

  • real-time -> document-updater: document is loaded from mongo into redis

• editor: sends document update via web socket

  • real-time -> document-updater: document is updated in redis

• editor: sends more compile requests

  • 5min passed since last flush (per doc):
    • document-updater: flush doc from redis to mongo

• editor: sends more updates

  • every 100 updates (per doc):
    • document-updater: flush doc history from redis to mongo

• user: leaves the editor/closes browser tab

  • 5min later
    • real-time: check for other collaborators, in case there are none:
      • real-time -> document-updater: flush docs from redis to mongo

Reading from mongo into redis:

• document-updater -> web -> docstore: read from mongo

Flushing from redis into mongo

• document-updater -> web -> docstore: write into mongo

Compile - "full" sync-mode

• editor: sends compile request with sync-mode set to "full" compile
• web -> document-updater: any documents are flushed from redis into mongo
• web -> docstore: all documents are downloaded from mongo
• web -> clsi: compile request is sent to clsi, including:
  • the sync-mode
  • a hash of the file tree -> the "project state"
  • all docs with their content -> subject to 7MB request body limit
  • binary file URLs for separate downloading
• clsi: check on-disk state with sync-mode and "project state"
  • this is a full sync, so we can ignore what was previously written
• clsi: cleanup compile dir
• clsi: write all docs into compile dir
• clsi: write all binary files into compile dir
  • clsi copies the files from a per project local cache
  • on cache miss:
    • clsi->filestore: download files
• clsi: write the "project state"
• clsi: ensure docker container exists with desired config
  • build container options, includes texlive version
  • hash options
  • container name: project-<project-id>-<user-id>-<hash>
• clsi: start container and stream stdout/stderr into memory -> limit 2MB
• clsi: leave stopped container behind -> cleaned up after 24h
• clsi: write stdout/stderr to disk
• clsi: copy output files into unique output directory
  • build-id composed of 8 random bytes plus timestamp in ms precision
  • delete all but last 3 (anonymous)/ 1 (logged in user) build folders
• clsi: compile was failure/timeout
  • delete compile cache - it may have partial files/corrupted cache
• editor: downloads output.log and output.pdf

Compile - "incremental" sync-mode

• editor: sends compile request with sync-mode set to "incremental" compile
• web -> document-updater: get any documents from redis
  • the "project state" hash is also stored in redis
  • web sends the hash of the file tree to document-updater and document-updater can turn the incremental compile into a full compile on mismatch
    • see compile process as performed when 'editor requested "full" compile'
• web -> clsi: compile request is sent to clsi, including:
  • the sync-mode
  • a hash of the file tree -> the "project state"
  • all docs from redis with their content -> subject to 7MB request body limit
  • no binary fines
• clsi: check on-disk state with sync-mode and "project state"
  • this is an incremental sync, so the "project state" must match
  • on mismatch: respond with 409, let web retry with "full" sync
    • see compile process as performed when 'editor requested "full" compile'
• clsi: write updated docs into compile dir
• clsi: ensure docker container exists with desired config
  • build container options, includes texlive version
  • hash options
  • container name: project-<project-id>-<user-id>-<hash>
• clsi: start container and stream stdout/stderr into memory -> limit 2MB
• clsi: leave stopped container behind -> cleaned up after 24h
• clsi: write stdout/stderr to disk
• clsi: copy output files into unique output directory
  • build-id composed of 8 random bytes plus timestamp in ms precision
  • delete all but last 3 (anonymous)/ 1 (logged in user) build folders
• clsi: compile was failure/timeout
  • delete compile cache - it may have partial files/corrupted cache
• editor: downloads output.log and output.pdf

Compile - switching between modes

• editor: observes a compile failure, next compile is a "full" compile
• editor: observes a compile success, next compile is an "incremental" compile