Data For Good - Batch 11 - E-cartomobile

This project aims at encouraging and planning electric mobility in French territories by developing a dashboard leveraging Open-Data for decision makers.

The technical interface is made with Streamlit for quick iterations, directly in the application. The final interface is made in https://github.com/Thopiax/batch11_e_cartomobile_ui

Index

• I want to contribute! Where do I start?
• Development
  • File Structure
  • Setting up the environment

🔧 Development

📁 Repo structure

-.github/workflows --------------------- ochestrate GH actions jobs
- e_cartomobile ------------------------ all methods needed to serve the dashboard
        content ------------------------ documentation
        data_analytics ----------------- methods and figures for final dashboard
        data_extracts ------------------ methods to extract raw data & some actual extracts
        data_load ---------------------- methods dealing with complete workflow and upload to database
        data_transform ----------------- methods related to process raw and aggregated data
        infra -------------------------- methods related to database connections
        utils -------------------------- common in-memory data (Constants, colors, ...)
        tests -------------------------- unit tests
- notebooks ---------------------------- R&D notebooks
- pages -------------------------------- the different pages making the dashboard
app.py --------------------------------- run dashboard

🔩 Setting up the environment

Doing the following step will enable your local environement to be aligned with the one of any other collaborator.

First install pyenv:

OS	Command
MacOS	cd - brew install pyenv # pyenv itself brew install pyenv-virtualenv # integration with Python virtualenvsec
Ubuntu	sudo apt-get update; sudo apt-get install make build-essential libssl-dev zlib1g-dev \ libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm \ libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev curl https://pyenv.run | bash
Windows	An installation using miniconda is generally simpler than a pyenv one on Windows.

Make the shell pyenv aware:

OS	Command
MacOS	eval "$(pyenv init --path)" eval "$(pyenv init -)" eval "$(pyenv virtualenv-init -)"
Ubuntu	export PYENV_ROOT="$HOME/.pyenv" command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH" eval "$(pyenv init -)" eval "$(pyenv virtualenv-init -)"
Windows	🇫🇷 Dans Propriétés systèmes > Paramètres système avancés > Variables d'environnement... Choisissez la variable "Path" > Modifier... et ajoutez le chemin de votre installation python, où se trouve le python.exe. (par défaut, C:\Users\username\AppData\Roaming\Python\Scripts\ ) 🇬🇧 In System Properties > Advanced > Environment Variables... Choose the variable "Path" > Edit... et add the path to your python's installation, where is located the pyhton.exe (by default, this should be at C:\Users\username\AppData\Roaming\Python\Scripts\ )

Let's install a python version (for windows, this step has been done with miniconda):

pyenv install 3.9.16 # this will take time

Supported python versions for this project: ">=3.8.1,<3.9.7 || >3.9.7,<3.10"

Check if it works properly, this command:

pyenv versions

should return:

  system
  3.9.16

Then you are ready to create a virtual environment. Go in the project folder, and run:

  pyenv virtualenv 3.9.16 batch11_e_cartomobile
  pyenv local batch11_e_cartomobile

A nice tutorial to know more about pyenv: pyenv tuto.

You now need a tool to manage dependencies. Let's use poetry: Poetry documentation On windows, if not already installed, you will need a VS installation.

Link : https://wiki.python.org/moin/WindowsCompilers#Microsoft_Visual_C.2B-.2B-_14.x_with_Visual_Studio_2022_.28x86.2C_x64.2C_ARM.2C_ARM64.29

pip install poetry
poetry --version

In the console, you can now try :

poetry update

The virtual environment will be installed in .venv folder.

If the dependency resolution takes forever, you can use this workaround:

poetry export -f requirements.txt > requirements.txt
python -m pip install -r requirements.txt
poetry install

It takes a lower time space to install the package locally since all deps are already installed.

Make sure to run poetry shell before to access the created virtual environment and install on it instead of on user/global path.

When you need to install a new dependency that is not in the pyproject.toml (use a new package, e.g. nltk), run

poetry add ntlk

After commiting to the repo, other team members will be able to use the exact same environment you are using. Please be sure that you do not break the CI or other important stuff when merging your PR to the main branch.

All Pull requests should be validated by 1 member before being merged. See below for more details on pull requests review.

Disclaimer: The virtual environment and CI are not perfect and need collaboration to be stable. Please comment any surprising things you see. Also, your PR may be bloked due to Black, Flake8 or isort. Pre-commit hooks are here to help and apply some fixes to your code when you commit stuff to follow PEP8 standards. But you may have other surprises with a CI failing. Please take the time to analyze errors in "Actions" on Github and try to fix them. Try not to ignore too many errors. We will tweak flake8 and other libs parameters iteratively to have the best experience.

A few details about pre-commit

The pre-commit library is installed in the virtual environment. It runs isort on your files (check if your imports are well sorted), then black (formatter following PEP8 standards) and eventually runs flake8 to check if everything is good. This suite runs automatically when you do a git commit. You can also run this pipeline with the following command:

pre-commit run --all-files

If you want more details, here is a small tutorial on pre-commit : tuto

Run the dashboard

Run:

make run_streamlit

Or full commands:

poetry run streamlit run app.py

On Windows, you may need instead of above solutions:

poetry run python -m streamlit run app.py

Depending on your installation process and version, "python" can also be "python3" or "py".

Fix linting & unit tests

Before committing, make sure that the line of codes you wrote are conform to PEP8 standard by running:

poetry run black e_cartomobile --check
poetry run flake8 e_cartomobile
poetry run isort e_cartomobile

You can also test your code using pytest:

poetry run pytest e_cartomobile/tests

Please try to develop unit tests for your code. See this tuto.

Use the virtual environment in VS Code notebooks or Jupyter notebooks

TO BE VALIDATED, like all README. Please edit/comment if anything is wrong.

You can select the virtual environment created in VSCode to run the notebooks (in python evironments, .venv should be available). If you prefer to run your notebooks with Jupyter in the browser:

poetry run jupyter notebook

Then, open your notebook, you should be inside your virtual environment. If something is weird, trop running poetry install and relaunch poetry run jupyter notebook.

Code Review Guidelines

1. Look for anti-patterns that the linter and CI/CD do not already pick up on, for example too many magic strings or typos in column names that are only evaluated at runtime.

2. Be detail oriented. Consider off-by-one indexing errors, spelling mistakes in docstrings, and other easy-to-miss errors.

3. Evaluate if the unit tests cover most major corner cases.

4. Consider readability. If a section of code is difficult to understand ask for it to be refactored or better commented.

Data used

TODO: Details on data set used after convergence