init sankey #770
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||
|---|---|---|---|---|
|
|
@@ -141,6 +141,7 @@ | |||
| "message::costs", | ||||
| "message::emissions", | ||||
| ), | ||||
| ("message::sankey", "concat", "out::pyam", "in::pyam"), | ||||
|
There was a problem hiding this comment. #''Sankey''
import logging
from functools import partial
from typing import Tuple, Mapping, List
from genno.operator import broadcast_map
from ixmp.report import Key
from .pyam import collapse_message_cols
# Assuming TASKS1 was where Sankey tasks were defined:
TASKS1 = (
("message::sankey", "concat", "out::pyam", "in::pyam"),
)
def get_sankey_tasks() -> List[Tuple[Tuple, Mapping]]:
"""Return a list of tasks for Sankey diagram reporting."""
to_add: List[Tuple[Tuple, Mapping]] = []
strict = dict(strict=True)
# This might include specific Sankey diagram configuration or additional tasks.
for t in TASKS1:
to_add.append((t, strict))
return to_add
class SankeyReporter:
"""A specialized reporter for generating Sankey diagrams."""
@staticmethod
def add_tasks(reporter, fail_action: str = "raise") -> None:
"""Add Sankey-related tasks to a given reporter."""
reporter.add_queue(get_sankey_tasks(), fail=fail_action)
```suggestion
("message::sankey", "concat", "out::pyam", "in::pyam"),
```There was a problem hiding this comment. Thanks for this suggestion, @daymontas1 :)
Suggested change
|
||||
| ) | ||||
|
|
||||
|
|
||||
|
|
||||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| from typing import Any, Optional | ||
|
|
||
| import pandas as pd | ||
|
|
||
| try: | ||
| from pyam.str import get_variable_components as gvc | ||
| except ImportError: # Python < 3.10, pandas < 2.0 | ||
| from pyam.utils import get_variable_components as gvc | ||
|
|
||
|
|
||
| def sankey_mapper( | ||
| df: pd.DataFrame, | ||
| year: int, | ||
| region: str, | ||
| exclude: list[Optional[str]] = [], | ||
| ) -> dict[str, Any]: | ||
| mapping = {} | ||
|
|
||
| for var in df.filter(region=region + "*", year=year).variable: | ||
| is_input = gvc(var, 0) == "in" | ||
| (start_idx, end_idx) = ([1, 2], [3, 4]) if is_input else ([3, 4], [1, 2]) | ||
| source = gvc(var, start_idx, join=True) | ||
| target = gvc(var, end_idx, join=True) | ||
| if source in exclude or target in exclude: | ||
| continue | ||
| mapping[var] = (source, target) | ||
|
|
||
| return mapping |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,199 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "# Westeros Tutorial - Introducing Sankey diagrams\n", | ||
| "\n", | ||
| "Sankey diagrams are a useful technique to visualize energy flow accounts.\n", | ||
| "\n", | ||
| "This tutorial introduces the sankey feature provided by the ``pyam`` packages.\n", | ||
| "\n", | ||
| "\n", | ||
| "**Pre-requisites**\n", | ||
| "- You have the *MESSAGEix* framework installed and working\n", | ||
| " In particular, you should have installed ``message_ix``, ``pyam``, and ``plotly``\n", | ||
| "- Complete tutorial Part 1 (``westeros_baseline.ipynb``) and Introducing Reporting (``westeros_report.ipynb``)" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "import ixmp\n", | ||
| "\n", | ||
| "from message_ix import Scenario\n", | ||
| "\n", | ||
| "mp = ixmp.Platform()\n", | ||
| "scenario = Scenario(mp, model=\"Westeros Electrified\", scenario=\"baseline\")" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# Remove any existing solution\n", | ||
| "try:\n", | ||
| " scenario.remove_solution()\n", | ||
| "except ValueError:\n", | ||
| " pass\n", | ||
| "\n", | ||
| "scenario.solve()" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Create the reporter object. (Since ``\"-\"`` is not a unit, we replace it by ``\"\"``.)" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "from message_ix.report import Reporter\n", | ||
| "\n", | ||
| "rep = Reporter.from_scenario(scenario)\n", | ||
| "\n", | ||
| "rep.configure(units={\"replace\": {\"-\": \"\"}})" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Use the `message::sankey` reporter option to generate a pyam.dataframe including the reqiured input (`in::pyam`) and output flows (`out::pyam`) in iamc format.\n" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "df = rep.get(\"message::sankey\")" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "The utility function `sankey_mapper(df, year, region, exclude=[])` can be used to create the required mapping for the `figures.sankey()` function of the `pyam` package.\n", | ||
| "\n", | ||
| "In some models it might be necessary to exclude variables and flow to get meaningful sankey diagrams. But let´s try with all!" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "from message_ix.util.sankey import sankey_mapper\n", | ||
| "\n", | ||
| "mapping = sankey_mapper(df, year=700, region=\"Westeros\")" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "The pyam function `plot.sankey(mapping)`returns a plotly sankey figure object that can be further modified.\n", | ||
| "\n", | ||
| "To plot it as an interactive diagram in your web browser, you can do the following." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "from pyam.figures import sankey\n", | ||
| "\n", | ||
| "fig = sankey(df=df.filter(year=700), mapping=mapping)\n", | ||
| "fig.show()" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "mapping_without_final_electricity = sankey_mapper(\n", | ||
| " df, year=700, region=\"Westeros\", exclude=[\"final|electricity\"]\n", | ||
| ")" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "fig = sankey(df=df.filter(year=700), mapping=mapping_without_final_electricity)\n", | ||
| "fig.show()" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "mapping_without_wind_ppl_standard = sankey_mapper(\n", | ||
| " df, year=700, region=\"Westeros\", exclude=[\"wind_ppl|standard\"]\n", | ||
| ")\n", | ||
| "fig = sankey(df=df.filter(year=700), mapping=mapping_without_wind_ppl_standard)\n", | ||
| "fig.show()" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Do not forget to close the database ;-) " | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "mp.close_db()" | ||
| ] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "kernelspec": { | ||
| "display_name": "messageix", | ||
| "language": "python", | ||
| "name": "python3" | ||
| }, | ||
| "language_info": { | ||
| "codemirror_mode": { | ||
| "name": "ipython", | ||
| "version": 3 | ||
| }, | ||
| "file_extension": ".py", | ||
| "mimetype": "text/x-python", | ||
| "name": "python", | ||
| "nbconvert_exporter": "python", | ||
| "pygments_lexer": "ipython3", | ||
| "version": "3.10.12" | ||
| }, | ||
| "orig_nbformat": 4 | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 2 | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would get rid of this, and simply do the concatenation outside of reporter (e.g., in the jupyter notebook here)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I kept this for now as I don't know what the Reporter actually does and it seems useful to just get the dataframe format we need from the Reporter. But please elaborate :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have created a new file, sankey.py, within the report directory, which extracts Sankey diagram functionality from init.py. Fridolin (@glatterf42), could you please confirm if this is what we want? Additionally, I have updated the westeros_sankey.ipynb file to ensure compatibility with the new sankey.py file in the report directory. I have annotated these changes in the westeros_sankey.ipynb below. Also, in this case, the following line inside the init.py file must be either removed or commented out:
("message::sankey", "concat", "out::pyam", "in::pyam")
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, @daymontas1, these changes sound good to me. For the line that has become redundant, please remove it rather than commenting it out, this keeps the code clean.
Could you please push your changes to the branch so that we can take a look and see how the tests are doing?
If you don't have write access to @mabudz's fork, please let us know so that we can figure out a solution :)