Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Checklist]: Upgrade all parsers with event classes #6011

Open
55 of 96 tasks
kruschk opened this issue Oct 15, 2023 · 1 comment
Open
55 of 96 tasks

[Checklist]: Upgrade all parsers with event classes #6011

kruschk opened this issue Oct 15, 2023 · 1 comment

Comments

@kruschk
Copy link
Contributor

kruschk commented Oct 15, 2023 •
edited by VIKTORVAV99

I noticed on the Building a new parser wiki page that several classes have been introduced to ensure parsed data is structured correctly; however, only about a third of all the parsers have been updated to use them. As promised in my comment on #5986, I've compiled the following list to keep track of which parsers have been upgraded and which have not:

To determine which parsers have been upgraded, I used the command below to find the modules using any one of the event classes. This assumes that if the module instantiates any one of those objects, then all of the functions in that parser have been upgraded accordingly.

grep -El '(Exchange|Price|ProductionBreakdown|TotalConsumption|TotalProduction)(List)?\(' parsers/*.py

Let me know if I've missed anything!
kruschk added a commit to kruschk/electricitymap-contrib that referenced this issue Oct 15, 2023
@kruschk
refactor: upgrade CA-SK with event classes
64e175a 
- Upgrade the parser to use the ProductionBreakdown, ProductionMix, and
  TotalConsumption, and ZoneKey classes.
- Conform to Black's 88-column default line length limit.
- Factor out duplicate code.

Refs: electricitymaps#6011
kruschk added a commit to kruschk/electricitymap-contrib that referenced this issue Oct 15, 2023
@kruschk
refactor: upgrade CA-SK with event classes
3577bfa 
Upgrade the parser to use the ProductionBreakdown, ProductionMix, and
TotalConsumption, and ZoneKey classes. This should yield no functional
change to the parser. Note that this does not resolve the current issue
with the parser's fetch_consumption function. Additionally:

- Conform to Black's 88-column default line length limit.
- Factor out duplicate code.

Refs: electricitymaps#5986, electricitymaps#6011
kruschk added a commit to kruschk/electricitymap-contrib that referenced this issue Oct 15, 2023
@kruschk
refactor: upgrade CA-SK with event classes
7c646ce 
Upgrade the parser to use the ProductionBreakdown, ProductionMix,
TotalConsumption, and ZoneKey classes. This should yield no functional
change. Note that this does not resolve the current issue with the
fetch_consumption function. Additionally:

- Conform to Black's 88-column default line length limit.
- Factor out duplicate code.

Refs: electricitymaps#5986, electricitymaps#6011
kruschk added a commit to kruschk/electricitymap-contrib that referenced this issue Oct 15, 2023
@kruschk
refactor: upgrade CA-SK with event classes
168f392 
Upgrade the parser to use the ProductionBreakdown, ProductionMix,
TotalConsumption, and ZoneKey classes. This should yield no functional
change. Note that this does not resolve the current issue with the
fetch_consumption function. Additionally:

- Conform to Black's 88-column default line length limit.
- Factor out duplicate code.

Refs: electricitymaps#5986, electricitymaps#6011
@kruschk kruschk mentioned this issue Oct 15, 2023
Merged
2 tasks
@kruschk kruschk changed the title Upgrade all parsers with data-validating classes Upgrade all parsers with event classes Oct 16, 2023
@VIKTORVAV99
Copy link
Member

@unitrium FYI

@VIKTORVAV99 VIKTORVAV99 pinned this issue Oct 19, 2023
@VIKTORVAV99 VIKTORVAV99 changed the title Upgrade all parsers with event classes [Checklist]: Upgrade all parsers with event classes Oct 19, 2023
kruschk added a commit to kruschk/electricitymap-contrib that referenced this issue Oct 24, 2023
@kruschk
refactor: upgrade CA-SK with event classes
a23c827 
Upgrade the parser to use the ProductionBreakdown, ProductionMix,
TotalConsumption, and ZoneKey classes. This should yield no functional
change. Note that this does not resolve the current issue with the
fetch_consumption function. Additionally:

- Comply with Black's 88-column default line length limit.
- Factor out duplicate code.

Refs: electricitymaps#5986, electricitymaps#6011
kruschk added a commit to kruschk/electricitymap-contrib that referenced this issue Oct 24, 2023
@kruschk
refactor: address review comments
0015de9 
- Be more specific about the return type of the _request function.
- Import the Any type directly.
- Swap out pytz in exchange for zoneinfo.
- Use the ProductionMix.add_value method instead of a dictionary
  comprehension.

Refs: electricitymaps#5986, electricitymaps#6011
kruschk added a commit to kruschk/electricitymap-contrib that referenced this issue Oct 24, 2023
@kruschk
refactor: switch to TotalConsumptionList
5b162d0 
Swap out TotalConsumption for TotalConsumptionList, as the former is
considered an implementation detail while the latter is part of the
public API.

Refs: electricitymaps#5986, electricitymaps#6011
kruschk added a commit to kruschk/electricitymap-contrib that referenced this issue Oct 24, 2023
@kruschk
refactor: narrow down payload types
37bc7d6 
Refs: electricitymaps#5986, electricitymaps#6011
kruschk added a commit to kruschk/electricitymap-contrib that referenced this issue Oct 24, 2023
@kruschk
refactor: upgrade CA-SK with event classes
33678c0 
Upgrade the parser to use the ProductionBreakdown, ProductionMix,
TotalConsumption, and ZoneKey classes. This should yield no functional
change. Note that this does not resolve the current issue with the
fetch_consumption function. Additionally:

- Comply with Black's 88-column default line length limit.
- Factor out duplicate code.

Refs: electricitymaps#5986, electricitymaps#6011, electricitymaps#6013
kruschk added a commit to kruschk/electricitymap-contrib that referenced this issue Oct 24, 2023
@kruschk
refactor: address review comments
617bc4a 
- Be more specific about the return type of the _request function.
- Import the Any type directly.
- Swap out pytz in exchange for zoneinfo.
- Use the ProductionMix.add_value method instead of a dictionary
  comprehension.

Refs: electricitymaps#5986, electricitymaps#6011, electricitymaps#6013
kruschk added a commit to kruschk/electricitymap-contrib that referenced this issue Oct 24, 2023
@kruschk
refactor: switch to TotalConsumptionList
455a35e 
Swap out TotalConsumption for TotalConsumptionList, as the former is
considered an implementation detail while the latter is part of the
public API.

Refs: electricitymaps#5986, electricitymaps#6011, electricitymaps#6013
kruschk added a commit to kruschk/electricitymap-contrib that referenced this issue Oct 24, 2023
@kruschk
refactor: narrow down payload types
4a60f16 
Refs: electricitymaps#5986, electricitymaps#6011, electricitymaps#6013
kruschk added a commit to kruschk/electricitymap-contrib that referenced this issue Oct 25, 2023
@kruschk
refactor: upgrade CA-NS with event classes
a0d61d6 
Upgrade the parser to use the ExchangeList, ProductionBreakdownList,
ProductionMix, and ZoneKey classes. This should yield no functional
change. Additionally:

- Comply with Black's 88-column default line length limit.
- Define global constants to replace a number of local variables and
  literals.
- Try to use more consistent names throughout.

Refs: electricitymaps#6011, electricitymaps#6050
@kruschk kruschk mentioned this issue Oct 25, 2023
Merged
2 tasks
kruschk added a commit to kruschk/electricitymap-contrib that referenced this issue Oct 25, 2023
@kruschk
refactor: upgrade CA-NS with event classes
459e1cf 
Upgrade the parser to use the ExchangeList, ProductionBreakdownList,
ProductionMix, and ZoneKey classes. This should yield no functional
change. Additionally:

- Comply with Black's 88-column default line length limit.
- Define global constants to replace a number of local variables and
  literals.
- Try to use more consistent names throughout.

Refs: electricitymaps#6011, electricitymaps#6050
kruschk added a commit to kruschk/electricitymap-contrib that referenced this issue Oct 25, 2023
@kruschk
refactor: zip arrays together instead of searching
0de7079 
The JSON arrays a and b returned by the two endpoints appear to be
parallel in the sense that element a[i] corresponds with element b[i]
for all i of interest. Zip these arrays together and iterate over the
resulting pairs instead of iterating over one and performing a linear
search to find the corresponding element in the other. Additionally:

- Check whether we've been bitten by a race condition while requesting
  the above array data and bail out if so.
- Skip the first element of each array because the reported base load is
  always 0 MW.

Refs: electricitymaps#6011, electricitymaps#6050
VIKTORVAV99 added a commit that referenced this issue Oct 25, 2023
@kruschk @VIKTORVAV99
refactor: upgrade CA-SK with event classes (#6013)
266eff0 
* refactor: upgrade CA-SK with event classes

Upgrade the parser to use the ProductionBreakdown, ProductionMix,
TotalConsumption, and ZoneKey classes. This should yield no functional
change. Note that this does not resolve the current issue with the
fetch_consumption function. Additionally:

- Comply with Black's 88-column default line length limit.
- Factor out duplicate code.

Refs: #5986, #6011, #6013

* refactor: address review comments

- Be more specific about the return type of the _request function.
- Import the Any type directly.
- Swap out pytz in exchange for zoneinfo.
- Use the ProductionMix.add_value method instead of a dictionary
  comprehension.

Refs: #5986, #6011, #6013

* refactor: switch to TotalConsumptionList

Swap out TotalConsumption for TotalConsumptionList, as the former is
considered an implementation detail while the latter is part of the
public API.

Refs: #5986, #6011, #6013

* refactor: narrow down payload types

Refs: #5986, #6011, #6013

---------

Co-authored-by: Viktor Andersson <30777521+VIKTORVAV99@users.noreply.github.com>
kruschk added a commit to kruschk/electricitymap-contrib that referenced this issue Oct 26, 2023
@kruschk
feat: retry requests and simplify validation
9be429a 
- When a pair of requests have failed, retry them until they succeed or
  the maximum number of attempts is exceeded.
- Simplify the validation logic by making better use of the event
  classes instead of introducing extraneous data structures.

Refs: electricitymaps#6011, electricitymaps#6050
kruschk added a commit to kruschk/electricitymap-contrib that referenced this issue Oct 28, 2023
@kruschk
refactor: upgrade CA-YT with event classes
49586f0 
Upgrade the parser to use the ProductionBreakdownList, ProductionMix,
StorageMix, and ZoneKey classes. This should yield no functional change.
Additionally:

- Comply with Black's 88-column default line length limit.
- Define global constants to replace magic values.
- Eliminate a few unnecessary variables.

Refs: electricitymaps#6011, electricitymaps#6062
@kruschk kruschk mentioned this issue Oct 28, 2023
Merged
2 tasks
kruschk added a commit to kruschk/electricitymap-contrib that referenced this issue Oct 29, 2023
@kruschk
refactor: upgrade CA-PE with event classes
150941c 
Upgrade the parser to use the ExchangeList, ProductionBreakdownList,
ProductionMix, and ZoneKey classes. This should yield no functional
change. Additionally:

- Comply with Black's 88-column default line length limit.
- Define global constants to replace local variables and literals.
- Simplify the logic that retrieves event data from the source by
  transforming the response payload into a more convenient data
  structure.

Refs: electricitymaps#6011
@kruschk kruschk mentioned this issue Oct 29, 2023
Merged
2 tasks
kruschk added a commit to kruschk/electricitymap-contrib that referenced this issue Nov 2, 2023
@kruschk
refactor: upgrade CA-YT with event classes
358a14d 
Upgrade the parser to use the ProductionBreakdownList, ProductionMix,
StorageMix, and ZoneKey classes. This should yield no functional change.
Additionally:

- Comply with Black's 88-column default line length limit.
- Define global constants to replace magic values.
- Eliminate a few unnecessary variables.

Refs: electricitymaps#6011, electricitymaps#6062
kruschk added a commit to kruschk/electricitymap-contrib that referenced this issue Nov 2, 2023
@kruschk
refactor: switch to ProductionMix.add_value method
103b077 
Pass data to the ProductionMix object using the add_value method instead
of during object instantiaton.

Refs: electricitymaps#6011, electricitymaps#6062
kruschk added a commit to kruschk/electricitymap-contrib that referenced this issue Nov 2, 2023
@kruschk
refactor: upgrade CA-PE with event classes
718d998 
Upgrade the parser to use the ExchangeList, ProductionBreakdownList,
ProductionMix, and ZoneKey classes. This should yield no functional
change. Additionally:

- Comply with Black's 88-column default line length limit.
- Define global constants to replace local variables and literals.
- Simplify the logic that retrieves event data from the source by
  transforming the response payload into a more convenient data
  structure.

Refs: electricitymaps#6011
kruschk added a commit to kruschk/electricitymap-contrib that referenced this issue Feb 6, 2024
@kruschk
refactor: switch log messages back to f-strings
65a4769 
Refs: electricitymaps#6011
This was referenced Feb 23, 2024
Merged
Merged
Merged
AlisCode added a commit to AlisCode/electricitymaps-contrib that referenced this issue Mar 4, 2024
@AlisCode
(electricitymaps#6011) Refactor AX parser to use electricitymaps even…
f0b5459 
…t classes
AlisCode added a commit to AlisCode/electricitymaps-contrib that referenced this issue Mar 4, 2024
@AlisCode
(electricitymaps#6011) Refactor AX parser to use electricitymaps even…
2800000 
…t classes
@AlisCode AlisCode mentioned this issue Mar 4, 2024
Merged
2 tasks
AlisCode added a commit to AlisCode/electricitymaps-contrib that referenced this issue Mar 6, 2024
@AlisCode
(electricitymaps#6011) Refactor AX parser to use electricitymaps even…
b8dc9b4 
…t classes
This was referenced Mar 10, 2024
Merged
Merged
AlisCode added a commit to AlisCode/electricitymaps-contrib that referenced this issue Mar 10, 2024
@AlisCode
(electricitymaps#6011) Refactor AX parser to use electricitymaps even…
3963b3b 
…t classes
AlisCode added a commit to AlisCode/electricitymaps-contrib that referenced this issue Mar 10, 2024
@AlisCode
(electricitymaps#6011) Fix type errors for zone_key
7a26db7
VIKTORVAV99 pushed a commit that referenced this issue Mar 10, 2024
@AlisCode
(#6011) Refactor AX parser to use electricitymaps event classes (#6544)
7326207 
* (#6011) Refactor AX parser to use electricitymaps event classes

* (#6011) Fix type errors for zone_key
@yujia21 yujia21 mentioned this issue Mar 11, 2024
Open
2 tasks
AlisCode added a commit to AlisCode/electricitymaps-contrib that referenced this issue Mar 18, 2024
@AlisCode
(electricitymaps#6011) Update CZ parser
a074ec3
@AlisCode AlisCode mentioned this issue Mar 18, 2024
Merged
2 tasks
AlisCode added a commit to AlisCode/electricitymaps-contrib that referenced this issue Mar 19, 2024
@AlisCode
(electricitymaps#6011) Use .add_value() instead of indexing values
7edf886
VIKTORVAV99 added a commit that referenced this issue Mar 20, 2024
@AlisCode @VIKTORVAV99
(#6011) Update CZ parser to use event classes (#6564)
b3ec71f 
* (#6011) Update CZ parser

* (#6011) Use .add_value() instead of indexing values

---------

Co-authored-by: Viktor Andersson <30777521+VIKTORVAV99@users.noreply.github.com>
@gianantoniopini gianantoniopini mentioned this issue Mar 21, 2024
Merged
2 tasks
This was referenced Mar 27, 2024
Merged
Merged
Merged
Merged
Draft
This was referenced Apr 5, 2024
Open
Merged
Merged
Merged
@amv213 amv213 mentioned this issue Apr 14, 2024
Merged
2 tasks
@amv213 amv213 mentioned this issue May 6, 2024
Merged
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kruschk @VIKTORVAV99