python-template

Are you tired of dealing with messy code, debugging errors, and fixing bugs that could have been prevented with proper tools and practices? Do you want to improve the overall quality of your projects while saving time in the long run? Then this template is perfect for you!

This is a template repository dedicated to ML projects in Python. It provides some basic tools and configs necessary to kickstart the development. It includes linting, type checking, package management, testing and CI.

Prerequisites

In order to use this template, you need to have both poetry and make installed.

Poetry

poetry is a package manager we're using. You can check the installation instructions here. You should avoid installing poetry through pip as it is a system-level tool, separated from the Python runtime you're currently using. It enables us to treat Python as any other dependency and pin its version in pyproject.toml.

curl -sSL https://install.python-poetry.org | python3 -

make

make is a utility tool that enables us to create aliases for commands we type in the console to run programs. It's much more efficient to use make so you don't have to memorize console commands. You can also expose your project's CLI via Makefile.

make should be available on most Linux distributions.

If you're using Windows, I would get choco first, then get make from choco package gallery.

choco install make

Mac users can pull it using Homebrew.

brew install make

Getting started

To start your work, you need to set up your local environment and hooks.

make install_dev

Later you will define aliases for your CLI here. Makefile already contains calls to build tools and checks. Use build to run them all.

build: pre_commit mypy test

Please note that build does not encompass the coverage and dependencies updates. These need to be run separately. Coverage may not always be necessary and you're gonna be running build often while working locally. Updating dependencies should be approached with the utmost care, no reason to bump the versions all the time when code does the work it's intended for. Better use some automation tool like dependabot.

To quickly format your repo while coding run:

make format  # isort black flake8

Before you commit, always run a full check. It's much faster to run it locally than to wait for CI build.

make build  # pre_commit mypy test

Structure

• .github - CI/CD pipelines, usually named after repository host (.github, .azure, .gitlab, etc)
• src - here goes your project's code
• src/cli - project's entry points should be wrapped with CLI and exposed via Makefile, good idea to store them separately
• .codespell - whitelist for project-related terms
• .coveragearc - coverage config, usually you don't want to report coverage on CLI, tests and some expressions
• .env.example - here you should state any environment variables your project uses
• .pre-commit-config.yaml - pre-commit pipeline configuration
• Makefile - tasks definitions, much simpler to call make than writing whole commands in the terminal; it's also easy to check what project-specific functialities you're exposing
• mypy.ini - mypy config, usually some of your dependencies won't be hinted so you're gonna ignore them here
• poetry.lock - compiled dependencies
• poetry.toml - poetry config, as you shouldn't enforce other devs where to put their virtual environment this is must be a separate config file
• pyproject.toml - repo config

Tools

When developing a project there's a need to automate some tedious stuff that helps you keep the repo clean and check it against common standards. These include managing the environment, dependencies, syntax, etc.

• poetry
• flake8
• isort
• ruff
• black
• pre-commit
• mypy
• coverage

pre-commit

This hook is here to prevent you from committing any nasty code to your repository.

• miscellaneous syntax checks and fixers
• spell checks (codespell)
• sort imports isort
• upgrade old syntax to newer Python version (pyupgrade)
• remove unnecessary #noqa comments (yesqa)
• autoformatting black
• finally some linter to be sure flake8

Why are we using the local environment to run pre-commit for us? This is rather unusual, isn't it? Yes - it is. You want to use the same versions of libs (flake8, black) and have them specified in a single file. Otherwise, you need to keep track of pre-commit dependencies as well. If you want a truly separate environment for the pre-commit - use dependency groups in your package manager.

Closing remarks

Hopefully, this template helps you jump off your project. If any of these tools are unfamiliar to you, follow the links for more info on them. Feel free to customize it, this is a template after all. The point I often make while doing workshops is you being able to make your toolset. This is but an example. I often look up how my favorite libraries are developed and take what I like. Just try not to overdo it. Don't use tools that are not necessary in for project. The main idea is to spend time actually coding, solving problems, and not thinking about code quality whatsoever.