Version 0.4.0

Version 0.4.0 focuses around being able to manipulate data already within the dataregistry, i.e., adding the ability to delete and modify previous datasets.

Changelog for developers:

Registrar now has a class for each table. They inherit from a BaseTable class, this means that shared functions, like deleting entries, are available for all tables. (#92)
Working with tables via the python interface has slightly different syntax (see user changelog below). (#92)
is_valid is removed as a dataset property. It has been replaced with status which is a bitmask (bit 0="valid", bit 1= "deleted" and bit 2="archived"), so now datasets can a combination of multiple states. (#93)
archive_date, archive_path, delete_date, delete_uid and move_date have been added as new dataset fields. (#93)
Database version bumped to 2.0.1 (#93)
dataset entries can be deleted (see below) (#94)
The CI for the CLI is now pure Python (i.e., there is no more bash script to ingest dummy entries into the registry for testing).
Can no longer "bump" a dataset that has a version suffix (trying to do so will raise an error). If a user wants to make a new version of a dataset with a suffix they can still do so by manually specifying the version and suffix (#97 ).
Dataset entries can be modified (see below, #100)

Changelog for users:

All database tables (dataset, execution, etc) have a more universal syntax. The functionality is still accessed via the Registrar class, but now for example to register a dataset it's Registrar.dataset.register(), similarly for an execution Registrar.execution.register() (#92). The docs and tutorials have been updated (#95).
dataset entries can now be deleted using the Registrar.dataset.delete(dataset_id=...) function. This will also delete the raw data within the root_dir. Note that the entry in the database will always remain (with an updated status field to indicate it has been deleted). (#94)
Documentation has been updated to make things a bit clearer. Now split into more focused tutorials (#95).
Certain dataset quantities can be modified after registration (#100). Documentation has been updated with examples.

Assets 2

07 Dec 21:12

stuartmcalpine

v0.3.0

bb23167

Release v0.3.0

v0.3.0 Release notes

For users

access_API_configuration field added to the dataset table. When registering a dataset you can now specify a text configuration file that will be read in as raw text and associated with the dataset. This is similar to the configuration column in the execution table.
The data registry path to where the data is stored now includes the schema name, i.e., <root_dir>/<schema>/<owner_type>/<owner>/<relative_path>
There is now a check to make sure copying data was successful. If there is an entry in the dataset table with is_valid=False it means that the data copying from old_location failed for some reason. If the data failed to copy when the dataset was intended to be overwritten, it will not corrupt the original data, the original data is kept.
Can now specify a "site" rather than a manual root_dir. For now the only site is "nersc", which points to the common shared space on NERSC for the dataregistry. If root_dir=None for the DataRegistry class, "nersc" is assumed to be the default.

For developers

Database version is updated to version 2.0.0
Implemented production schema.
Can now link dependencies to datasets in other schemas (i.e., to production).
Schema has been moved to a yaml file, which is called upon in the code. This saves dataset description duplication.
Use pytest fixtures to create temporary files and directories for copying dummy files into the registry during CI and for creating dummy config files.
The original root_dir a dataset was ingested into is now stored as a field in the dataset table.