Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CML GitHub Action #7

Draft
wants to merge 72 commits into
base: master
Choose a base branch
from

Conversation

gchhablani
Copy link

@gchhablani gchhablani commented Feb 24, 2022

This PR is a work-in-progress. Current action items include:

  • Use the container from moja global's repo directly instead of pulling and creating a container.
  • Look into self-hosted runners on CML. Current CML run takes 2+ hrs.
  • Run post-process scripts and get results in the report.

@gchhablani gchhablani changed the title Add cml test Add CML action Feb 24, 2022
@gchhablani gchhablani changed the title Add CML action Add CML GitHub Action Feb 24, 2022

jobs:
run:
runs-on: [ubuntu-latest]
Copy link

@aornugent aornugent Feb 25, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@HarshCasper - I think we can add the FLINTcloud containers here. Docs on Running Jobs in a Container

-runs-on: [ubuntu-latest]
+container: ghcr.io/moja-global/rest_api_gcbm:master

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I will add the container for the workflow.

@aornugent
Copy link

I've been tinkering with the post-processing tools to get some nice summaries. We can add this step to our CML Action:

# from the gcbm_project/ folder
python3 ../tools/CompileGCBMResults/compileresults.py sqlite:///output/gcbm_output.db --output_db sqlite:///output/compiled_gcbm_output.db

Then we want to pull out annual summaries. Here's an example that returns the total forest biomass in megatonnes:

import sqlite3
import pandas

conn = sqlite3.connect("output/compiled_gcbm_output.db")

query = f"""
        SELECT years.year, COALESCE(SUM(i.pool_tc), 0) / 1e6 AS total_biomass_mt
        FROM (SELECT DISTINCT year FROM v_age_indicators ORDER BY year) AS years
        LEFT JOIN v_pool_indicators i
            ON years.year = i.year
        WHERE i.indicator = 'Total Biomass'
            AND (years.year BETWEEN 1920 AND 2020)
        GROUP BY years.year
        ORDER BY years.year
        """

df = pd.read_sql_query(query, conn)
ax = df.plot.line("year") # x-axis
ax.figure.savefig('output/total_biomass_mt.png', dpi=300)

total_biomass_mt

@yodavo might have some good suggestions for relevant results summaries!

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants