Skip to content

Commit

Permalink
Update python projects notebook.
Browse files Browse the repository at this point in the history
  • Loading branch information
@ivastar
ivastar committed Feb 8, 2024
1 parent 772ea99 commit 10f0b49
Showing 1 changed file with 110 additions and 95 deletions.
205 changes: 110 additions & 95 deletions files/python_projects.ipynb
View file
Expand Up @@ -53,7 +53,7 @@
"\n",
"This workshop will use a test repository which is available here:\n",
"\n",
"https://github.com/ivastar/structrure_test_repo\n",
"https://github.com/mpi-astronomy/mpia-python-template\n",
"\n",
"### Create a Virtual Environment\n",
"\n",
Expand Down Expand Up @@ -164,7 +164,9 @@
{
"cell_type": "markdown",
"id": "49ec2abb",
"metadata": {},
"metadata": {
"jp-MarkdownHeadingCollapsed": true
},
"source": [
"### Packages in Python\n",
"\n",
Expand Down Expand Up @@ -196,19 +198,25 @@
" |-- sub_module.py\n",
"```\n",
"\n",
"__Excercise__: Let's reorganize the repo and combine all the code in a single place.\n",
"__Exercise__: Follow the instructions for the template here and make a copy under your own GitHub account: https://github.com/mpi-astronomy/mpia-python-template and clone it locally.\n",
"If you decide not to use GitHub, you can download the .ZIP file for the __static_version__ branch.\n",
"\n",
"The standard structure is to have a `src` directory and then have the directory with the package name inside that:\n",
"\n",
"```\n",
"mkdir src\n",
"cd src\n",
"mkdir poet\n",
"touch poet/__init__.py\n",
"import poet\n",
"```\n",
"my_package\n",
"|-- src\n",
" |-- my_package\n",
" |-- __init__.py\n",
" |-- code.py\n",
" |-- more_code.py\n",
"\n",
"You can set up this for your own code or take a look at the template.\n",
"\n",
"__Exercise__: Rename the `my_package` subdirectory to something else.\n",
"\n",
"Setting up the python path is an outdated method, does not translate to other people’s directory structure and operating systems. Also, does not install all the dependencies. Better, use the same methods as installation. \n",
"Setting up the python path is an outdated method, does not translate to other people’s directory structure and operating systems. Also, does not install all the dependencies. Better, use the same methods as other installation. \n",
"\n",
"```\n",
"pip install numpy\n",
Expand All @@ -217,84 +225,84 @@
"\n",
"How do you get your project to behave that way?\n",
"\n",
"You need to provide some additional files that would allow the built-in Python tools to treat your code as a package:\n",
"You need to provide some additional files that would allow the built-in Python tools to treat your code as a package. The instructions on how to turn your code into a library are contained in the `pyproject.toml` file. Let's review the contents of this file to understand what these instructions are. A minumum `pyproject.toml` is already included in the template repository. At this point it would be helpful to open the template direcotry in a text editor. \n",
"\n",
"```\n",
"touch setup.cfg\n",
"\n",
"[metadata]\n",
"name = my-package\n",
"Version = 0.0.1\n",
"description = My package description\n",
"long_description = file: README.rst\n",
"author = Your Name\n",
"author_email = your@email.com\n",
"url = https://link-to-your-project\n",
"license = BSD 3-Clause License\n",
"The first section of the file contains some basic information about your project. Edit it to correspond to your project. \n",
"\n",
"[options]\n",
"package_dir =\n",
" =src\n",
"packages = find:\n",
"python_requires= >=3.7\n",
"```\n",
"[project]\n",
"name = \"my_package\"\n",
"authors = [{name = \"Example Author\", email = \"author@mpia.de\"}]\n",
"description = \"An example package\"\n",
"readme = \"README.md\"\n",
"license = { file = 'LICENSE' }\n",
"classifiers = [\n",
" \"Programming Language :: Python :: 3\",\n",
" \"License :: OSI Approved :: BSD License\",\n",
" \"Operating System :: OS Independent\",\n",
"]\n",
"\n",
"dynamic = ['version']\n",
"\n",
"# Example of how to add dependencies:\n",
"install_requires =\n",
" importlib-metadata >= 2.0 ; python_version < \"3.8\"\n",
" \n",
"[options.packages.find]\n",
"where = src\n",
"[project.urls]\n",
"\"Bug Tracker\" = \"https://github.com/mpi-astronomy/snowblind/issues\"\n",
"\"Source Code\" = \"https://github.com/mpi-astronomy/snowblind\"\n",
"\n",
"```\n",
"\n",
"In setup.cfg the install_requires option is used to declare run-time dependencies for the package, which are dependencies that are needed for the package to import and run correctly. However, your package may have dependencies that are needed to build the package in the first place. For example, the setup.py file shown previously will only run correctly if both setuptools and setuptools_scm are installed.\n",
"The recommended way to specify build-time dependencies is to create a file called pyproject.toml which contains:\n",
"There are dependencies that are needed to build the package in the first place. These are specified in the `[build-system]` block. At minimum `setuptools` and `wheel` are needed. In addition, if you want to do the dynamic versioning, you should also include `setuptools_sc`. If your package has C or Fortran extensions that interface with `numpy`, you may also need to add `numpy` to the list below. Look up documentation for more details. The recommended way to specify build-time dependencies is to create a file called pyproject.toml which contains:\n",
"\n",
"```\n",
"[build-system]\n",
"requires = [\"setuptools>=42\", \"wheel\"]\n",
"requires = [\n",
" \"setuptools>=60\",\n",
" \"setuptools_scm>=8.0\",\n",
" \"wheel\",\n",
"]\n",
"build-backend = \"setuptools.build_meta\"\n",
"```\n",
"\n",
"If your package has C extensions that interface with Numpy, you may also need to add Numpy to the above list - see Compiled C/Cython extensions for more details.\n",
"Your package also has dependencies that need to be installed in order for it to run. There are specified in the `dependencies` block like this. Extend the list as needed.\n",
"\n",
"```\n",
"dependencies = [\n",
" \"numpy\",\n",
"]\n",
"```\n",
"\n",
"Finally, there is a block of `[project.optional-dependencies]` which specifiy packages that are only needed for tests and documentation. Users generally should not need to install these. You can also create some special installation setups here, for example if your code supports several different instruments, or different compute environments, you can specify different sets or versions of dependencies. In the example repo we have `[test]` and `[docs]` dependencies specified.\n",
"\n",
"Now you can install your library. Try the commands below. \n",
"\n",
"```\n",
"pip install -e .\n",
"pip show poet\n",
"pip show my_package\n",
"pip uninstall \n",
"pip install .\n",
"pip show poet\n",
"pip show my_package\n",
"```\n",
"Try install installing with -e and without is and showing where the files are located. Look what is in the directory. Note when you do the latter `pip` installation, only the files in the `src/my_package/` directory are being copied to `site-packages`. Within python you can do the following:\n",
"\n",
"```{python}\n",
"import sys\n",
"sys.modules['poet']\n",
"poet?\n",
"sys.modules['my_package']\n",
"my_package?\n",
"```\n",
"\n",
"\n",
"\n",
"Try install installing with -e and without is and showing where the files are located\n",
"\n",
"Push any changes to GitHub. Now you can try installing directly form GitHub:\n",
"\n",
"```\n",
"pip install git+https://\n",
"```\n",
"or\n",
"\n",
"And any one else can clone your repo and install it locally:\n",
"\n",
"```\n",
"git clone ....\n",
"pip install -e .\n",
"```\n",
"\n",
"\n",
"Try importing some of the `poet` modules and looking inside:\n",
"\n",
"```\n",
"Poet.ballardip.ballardip?\n",
"```\n",
"\n",
"To have a well-behaved function we can also add our own:\n",
"__Exercise__: Add your own function to the package. Install. Commit & push. \n",
"\n",
"```\n",
"touch my_func.py\n",
Expand Down Expand Up @@ -369,7 +377,9 @@
{
"cell_type": "markdown",
"id": "c9861940",
"metadata": {},
"metadata": {
"jp-MarkdownHeadingCollapsed": true
},
"source": [
"## 2. Data and Configuration Files\n",
"\n",
Expand Down Expand Up @@ -399,23 +409,32 @@
"\n",
"\n",
"```{code-block} python\n",
"% cd src/poet\n",
"% mkdir data\n",
"% mkdir src/my_package/data\n",
"```\n",
"\n",
"Now let's move some data in here to see how it works. \n",
"Now let's put some data here to see how this works. \n",
"\n",
"```{code-block} python\n",
"% ls data\n",
"\n",
"TEP\t\t\teg00-initvals.txt\tleapseconds\t\tpsf\n",
"config\t\t\tfilter\t\t\tmastermaps\n",
"% touch src/my_package/data/data.txt\n",
"% pip uninstall my_package\n",
"% pip install .\n",
"% pip show my_package\n",
"% ls ...../site-packages/my_package/data/\n",
"```\n",
"\n",
"% ls data/filter\n",
"By default, data put in this `data` directory will be included. You should now have your `data.txt` file included. But subdirectories will not be. Let's test this:\n",
"\n",
"irac_tr1_2004-08-09.dat\tirac_tr2_2004-08-09.dat\tirac_tr3_2004-08-09.dat\tirac_tr4_2004-08-09.dat\n",
"```\n",
"% mkdir src/my_package/data/filter\n",
"% touch src/my_package/data/filter/filter.flt\n",
"% pip uninstall my_package\n",
"% pip install .\n",
"% pip show my_package\n",
"% ls ...../site-packages/my_package/data/filter/\n",
"```\n",
"\n",
"This is probably no there. In order to include this you need an additional block in the `pyproject.toml` file that directs it to include the subdirectories:\n",
"\n",
"But just because the data is here, that does not automatically include it in the installed repository. \n",
"\n",
"Try it:\n",
Expand All @@ -427,30 +446,27 @@
"pip uninstall poet\n",
"```\n",
"\n",
"The information about which files should be included is again in the `setup.cfg` file, where the following section needs to be added:\n",
"The information about which files should be included is again in the `pyproject.toml` file, where the following section needs to be added:\n",
"\n",
"```\n",
"[options.package_data]\n",
"poet=\n",
" data/config/*\n",
" data/filters/*\n",
" data/eg00-initvals.txt\n",
"[tool.setuptools.package-data]\n",
"my_package = [\"data/filters/*\"]\n",
"```\n",
"\n",
"Try the installation again:\n",
"\n",
"```\n",
"% pip install .\n",
"% show poet\n",
"% ls .........../site-packages/poet/data\n",
"% show my_package\n",
"% ls .........../site-packages/my_package/data\n",
"```\n",
"\n",
"And now, since this not some random directry set via an environmental variable, you can always find the location of the data via the installation directory of the code:\n",
"\n",
"```\n",
"> import pkg_resources\n",
"> poet_dir = pkg_resources.resource_filename(\"poet\",\"\")\n",
"> data_dir = os.path.join(poet_dir, 'data')\n",
"> my_package_dir = pkg_resources.resource_filename(\"my_package\",\"\")\n",
"> data_dir = os.path.join(my_package_dir, 'data')\n",
"```\n",
"\n",
"One generally occasional legitimate reason to include data is to have an example configuration file. It is much better to code the generation up so your code can just create a fresh one on demand. But if you must. Do not create your own configuration file format. Use [`configparser`](https://docs.python.org/3/library/configparser.html). \n",
Expand Down Expand Up @@ -496,17 +512,18 @@
"% touch test_my_func.py\n",
"```\n",
"\n",
"Tests can have their own dependencies that are different from the code. For example, if the tests need to generate or get data, those can be dependencies to JUST the tests and not the main code. Thereore you can specify a separate list of dependencies in the `setup.cfg` file like so:\n",
"Tests can have their own dependencies that are different from the code. For example, if the tests need to generate or get data, those can be dependencies to JUST the tests and not the main code. Thereore you can specify a separate list of dependencies in the `pyproject.toml` file like so:\n",
"\n",
"```\n",
"[options.extra.require]\n",
"test =\n",
" pytest\n",
" pytest-doctestplus\n",
" flake8\n",
" codecov\n",
" pytest-cov\n",
"\n",
"[project.optional-dependencies]\n",
"test = [\n",
" \"pytest\",\n",
" \"pytest-doctestplus\",\n",
" \"flake8\",\n",
" \"flake8-pyproject\",\n",
" \"codecov\",\n",
" \"pytest-cov\",\n",
"]\n",
"```\n",
"\n",
"To install both the main and the test dependancies, you can use pip as follows:\n",
Expand All @@ -518,7 +535,7 @@
"Let's write a really simple test, just for example purposes. \n",
"\n",
"```\n",
"from poet import my_func\n",
"from my_package import my_func\n",
"import numpy as np\n",
"\n",
"def test_add():\n",
Expand Down Expand Up @@ -667,15 +684,15 @@
"```\n",
"[Sphinx](https://www.sphinx-doc.org/en/master/usage/quickstart.html#setting-up-the-documentation-sources) uses restructured text which is just enough different from markdown to be confusing. Look at the docs. \n",
"\n",
"The docs can have their own dependencies that need to be documented in the `setup.cfg` file:\n",
"The docs can have their own dependencies that need to be documented in the `pyproject.toml` file:\n",
"\n",
"```\n",
"Documentation:\n",
"[options.extras_require]\n",
"docs =\n",
" sphinx\n",
" sphinx-automodapi\n",
" numpydoc\n",
"[project.optional-dependencies]\n",
"docs = [\n",
" \"sphinx\",\n",
" \"sphinx-automodapi\",\n",
" \"numpydoc\",\n",
"]\n",
"```\n",
"\n",
"\n",
Expand Down Expand Up @@ -744,9 +761,7 @@
"| LICENSE | defines rules on how your code can be used/modified/distributed |\n",
"| MANIFEST.in | (optional?) sets files to be distributed with the source distribution (e.g. LICENSE, README) |\n",
"| README.md | At least contains a project name and a description |\n",
"| pyproject.toml | is the specified file format of [PEP 518](https://www.python.org/dev/peps/pep-0518/) which contains the build system requirements of Python projects. |\n",
"| setup.py | (not always) contains some specific configuration of the project that the toml file cannot handle (compiled codes etc) |\n",
"| setup.cfg | specifies package metadata, dependencies, etc. |\n",
"| pyproject.toml | is the specified file format of [PEP 518](https://www.python.org/dev/peps/pep-0518/) which contains the build system requirements, dependencies and metadata of Python projects. |\n",
"\n",
"\n",
"Template: https://github.com/mpi-astronomy/mpia-python-template\n"
Expand Down Expand Up @@ -799,7 +814,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.16"
"version": "3.11.6"
}
},
"nbformat": 4,
Expand Down

0 comments on commit 10f0b49

Please sign in to comment.