Crowler

This repo/app is not very useful, but it is a good example that showcases how much it takes to build a fully automated website.

Also:

• is a good example of nim<->python interop.
• has nim<->rust bindings for the rust libraries imageflow and minifyhtml that could be useful elsewhere.
• is a good example of automatic amp-ification of web-pages (not perfect)
• implements translation utilities for html (that wrap web apis) that could be useful elsewhere
• there might be some other stuff that could made into separate nim packages…

Build

css:

For $sites assume the name of the python Site object or `$CONFIG_NAME` in nim.

• If new icons have to be added, a use the `index.test.html` and the vite server to access the unocss dashboard, and copy the css from the dashboard into `src/css/icons.scss`.
• run the nim server `nim r src/nim/server.nim` on port `:5050`
• `scripts/copy.sh -css $sites` (ensure that `npm run build` is working correctly by checking files in `dist/*.css` to be fresh.)
• scripts/docker.sh -c $sites

leveldb:

The leveldb binary is included in the repository, because we don’t want many mmapped files. Leveldb keeps up to 1000 mmaped files in memory (where the file size is the block size of the database). For default block sizes this means that leveldb can take >2G memory on 64bit. To rebuild:

cd vendor/leveldb
git submodule init
git submodule update

Change `kDefaultMmapLimit` to a very small value e.g. 10. In file `util/env_posix.c` (and `env_windows.c`)

constexpr const int kDefaultMmapLimit = (sizeof(void*) >= 8) ? 10: 0;

Deploy

• Ensure /mnt is a mounted storage on remove server
• create /mnt/hosts directory
• copy `data` directory to /mnt/hosts/data
• ensure docker is running
• make sure the start script includes dns option for configuring timeout (important for proxies)
• run docker image using script in `scripts/start.sh`

NIM compilation

• `panics:on` can’t be enabled, the codebase does large use of indexdefects