Flex model fields on data model

[Flix6x]

We aim to support flex-model and flex-context fields on the data model, at least the ones where that makes sense to the user (up until now, they are expected to be sent via the API - that is highly flexible, but with the number of fields we have grows to be less user-friendly. It will always remain a power user feature, but the data model is a more convenient place for many).

In this discussion, we collaborate on which fields to start with and how to represent values.
It also serves as a page to track compatibility of dedicated fallback attributes for API and CLI flex model fields. That is, which fields already support falling back to a sensor attribute or asset attribute.

For flex-model fields, we look for fallback attributes on the sensor first, then on the asset that the sensor belongs to:

Field in flex model	Fallback attribute	API support since	CLI support since	References
soc-at-start	soc_in_mwh + soc_datetime
soc-unit	-	-	-	not relevant
soc-min	min_soc_in_mwh 1
soc-max	max_soc_in_mwh 1
soc-minima
soc-maxima
soc-targets
soc-gain
soc-usage
roundtrip-efficiency	roundtrip_efficiency 1
charging-efficiency	charging-efficiency
discharging-efficiency	discharging-efficiency
storage-efficiency	storage_efficiency 1
power-capacity	capacity_in_mw 1
consumption-capacity	consumption_capacity 1
production-capacity	production_capacity 1

---

For flex-context fields, we have fallback attributes on the asset:

Field in flex context	Fallback attribute	API support since	CLI support since	References
consumption-price-sensor	market_id 1 2
production-price-sensor	market_id 1 2
inflexible-device-sensors
site-power-capacity	capacity_in_mw 1
site-production-capacity	production_capacity_in_mw 1
site-consumption-capacity	consumption_capacity_in_mw 1

Related discussion: #827

Footnotes

1. We have plans to replace dashes with underscores and move units from the attribute's key to the attribute's value: see https://github.com/FlexMeasures/flexmeasures/issues/911 ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹ ↩¹²

2. We have plans to rename these: see https://github.com/FlexMeasures/flexmeasures/issues/774 ↩ ↩²

[nhoening]

For my understanding: API and CLI support mean that you can in fact leave these fields away, and they will look for the fallback?

[Flix6x]

Yes.

[nhoening]

I'd like to add a column for UI support if you agree, in that it should be possible to edit these fallback fields in the UI. In the end, editing the attributes dict in one text input field is not really cutting it anymore (it's already cumbersome now, if we support more attributes it will be worse).
Whether the fields will live in the data model or on attributes for the time being (a good discussion to have), we should start adding support for the most relevant fields. E.g. sensors_to_show would be great, and then the relevant ones from here, see my next comment.

[Flix6x]

What are the requirements that would need to be met to warrant a check mark in the UI support column, Marshmallow validation? I feel better UX for attributes editing is definitely desirable, but a non-field-specific improvement that doesn't have to be a qualification criterion for UI support in this table. See also #477 for an idea to improve the UX (specifically, the comment about the JSON accordion).

[nhoening]

Good question. Marshmallow support if we keep them as attributes. Or we move them to data model fields. That is a decision we should think about.

I personally don't believe attributes is a long-term place for fields we actually use in production code, other than serving us as an agile way for developing new features without needing to touch the data model right away.
There is no good argument I know of for supporting fields we know we are using for core functionality in a dictionary, including extra effort to make editing the dictionary a little nicer.

sensors_to_show is established. And the same we can say about flex-model and flex-context fields.

The argument one can make is about their structure. sensors_to_show is a JSON field (list with sublists of integers), flex-model fields are floats, or they represent a sensor (e.g. {"sensor": 3}, which essentially means they are JSON, as well.

If we define these "fallback" fields as data model fields, the UI support question is (to me) more straightforward - there should be a form field and support to pick values. Maybe the main page should show the values or link to sensors. Maybe we have some logic which only shows these fields on assets where it makes sense to show them (the rules for this I don't know right now).

[nhoening]

To me, it seems the relevant ones are (I mark the set I propose to tackle first with a star, looking into a typical battery problem):

Flex-model:

• soc-min (*)
• soc-max (*)
• roundtrip-efficiency (*)
• charging-efficiency
• discharging-efficiency
• storage-efficiency
• power-capacity
• consumption-capacity
• production-capacity

Flex-Context:

• consumption-price-sensor (*)
• production-price-sensor
• inflexible-device-sensors (*)
• site-power-capacity (*)
• site-production-capacity
• site-consumption-capacity

[nhoening]

Also an important question: Where do these live?

• I suggest the flex-model fallback fields live on the asset with the power sensor.
• The flex-context fields should be on the site asset

(I guess sometimes these can be the same asset)

Of course we might need some UX ideas to make this stuff a bit obvious. Navigating the asset structure is already important help.

[nhoening]

Also, all of these fields can be relevant for having a fallback, but some only make sense as a sensor (e.g. soc-minima)

It would be great value to see which they are, and to quickly browse that data.

[Flix6x]

all of these fields can be relevant for having a fallback, but some only make sense as a sensor (e.g. soc-minima)

Right, so you mean that "soc-minima": {"sensor": 65} could be a relevant fallback attribute, and you'd be able to override it by passing something else in the flex-model.

And on the other hand, "soc-minima": {"datetime": "2025-01-01T00:00+02", "value": "10 kWh"} doesn't make sense as a fallback attribute, because the information is too fleeting to be considered a sensible fallback, while it does makes sense as a flex-model field of a single scheduling trigger, e.g. via an API call.

[nhoening]

Yes, I would even call these fields "model attributes" or so, rather than "fallback".

I expect a reality where they will be the norm, and sending fleeting values in the API will be a power-user/pilot feature.