Embedding factory script looping through each MGRS tile #125
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Jupyter notebook script to generate GeoParquet embedding files on a per MGRS tile basis. The script first generates an mgrs_world.txt file with a list of MGRS code names like 12ABC. A for-loop then goes through each MGRS tile, with the model running the prediction to generate GeoParquet files that are uploaded to s3. There are about 947019 rows of embeddings generated from the clay-small-70MT-1100T-10E.ckpt model checkpoint in Dec 2023.
weiji14
changed the title
weiji14
commented
Jan 17, 2024
Comment on lines
+81
to
+84
| "# !aws s3 cp s3://clay-model-ckpt/v0/clay-small-70MT-1100T-10E.ckpt checkpoints/\n", | ||
| "trainer = L.Trainer(precision=\"bf16-mixed\", logger=False)\n", | ||
| "model: L.LightningModule = CLAYModule.load_from_checkpoint(\n", | ||
| " checkpoint_path=\"checkpoints/clay-small-70MT-1100T-10E.ckpt\"\n", |
There was a problem hiding this comment.
Should be possible to load the checkpoint directly from HuggingFace now, instead of manually downloading from the s3 bucket, as mentioned at #116 (comment)
Suggested change
| "# !aws s3 cp s3://clay-model-ckpt/v0/clay-small-70MT-1100T-10E.ckpt checkpoints/\n", | |
| "trainer = L.Trainer(precision=\"bf16-mixed\", logger=False)\n", | |
| "model: L.LightningModule = CLAYModule.load_from_checkpoint(\n", | |
| " checkpoint_path=\"checkpoints/clay-small-70MT-1100T-10E.ckpt\"\n", | |
| "trainer = L.Trainer(precision=\"bf16-mixed\", logger=False)\n", | |
| "model: L.LightningModule = CLAYModule.load_from_checkpoint(\n", | |
| " checkpoint_path=\"https://huggingface.co/made-with-clay/Clay/resolve/main/Clay_v0.1_epoch-24_val-loss-0.46.ckpt\"\n", |
weiji14
commented
Jan 17, 2024
There was a problem hiding this comment.
Might want to move this into the scripts/ folder, and will need to update the paths below accordingly.
Comment on lines
+86
to
+87
| "#!mamba install triton\n", | ||
| "# model.model.encoder = torch.compile(model=model.model.encoder)" |
There was a problem hiding this comment.
Can probably remove this torch.compile line. Was trying to speed up the model by compiling it, but there were some layers that didn't work.
| "import os\n", | ||
| "import warnings\n", | ||
| "\n", | ||
| "import duckdb\n", |
There was a problem hiding this comment.
DuckDB is optional and can be removed, but it's nice to get a quick count of all the rows across the GeoParquet files (see last cell).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Jupyter notebook script to generate GeoParquet embedding files on a per MGRS tile basis.
Steps:
Notes:
g5.4xlargeEC2 instance with 1 NVIDIA A10G GPU that allows for bfloat16 dtype calculations.Closes #120