Stronger concepts section in documentation (#65)

* moving things around, adding some content to benefits of flex

* remove flex info from portfolio if not in demo mode, still has a problem (no data in figure)

* much novel content on flex opportunities and algorithms, and some reorganization

* energy flexibility activations, not flexible actions

* British english in docs (American english in code)

Co-authored-by: nhoening <nhoening@users.noreply.github.com>
Co-authored-by: Nicolas Höning <nicolas@seita.nl>
Co-authored-by: Nicolas Höning <iam@nicolashoening.de>
Co-authored-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com>
Co-authored-by: F.N. Claessen <felix@seita.nl>

documentation/api/introduction.rst

@@ -257,7 +257,7 @@ In case of a single group of connections, the message may be flattened to:
 Timeseries
 ^^^^^^^^^^
 
-Timestamps and durations are consistent with the ISO 8601 standard. All timestamps in requests to the API must be timezone aware. The timezone indication "Z" indicates a zero offset from UTC. Additionally, we use the following shorthand for sequential values within a time interval:
+Timestamps and durations are consistent with the ISO 8601 standard. All timestamps in requests to the API must be timezone-aware. The timezone indication "Z" indicates a zero offset from UTC. Additionally, we use the following shorthand for sequential values within a time interval:
 
 .. code-block:: json
 

documentation/changelog.rst

@@ -22,13 +22,15 @@ Bugfixes
 
 Infrastructure / Support
 ----------------------
+* Added concept pages to documentation [see `PR #65 <http://www.github.com/SeitaBV/flexmeasures/pull/65>`_]
 * Dump and restore postgres database as CLI commands [see `PR #68 <https://github.com/SeitaBV/flexmeasures/pull/68>`_]
 * Improved installation tutorial as part of [`PR #54 <http://www.github.com/SeitaBV/flexmeasures/pull/54>`_]
 * Moved developer docs from Readmes into the main documentation  [see `PR #73 <https://github.com/SeitaBV/flexmeasures/pull/73>`_]
 * Ensured unique sensor ids for all sensors [see `PR #70 <https://github.com/SeitaBV/flexmeasures/pull/70>`_ and (fix) `PR #77 <https://github.com/SeitaBV/flexmeasures/pull/77>`_]
 
 
 
+
 v0.2.3 | February 27, 2021
 ===========================
 

documentation/qa/algorithms.rst → documentation/concepts/algorithms.rst

@@ -1,19 +1,29 @@
 .. _algorithms:
 
-What algorithms does the platform use?
+
+Algorithms
 ==========================================
 
+.. contents::
+    :local:
+    :depth: 2
+
 
 Forecasting
 -----------
 
 Forecasting algorithms are used by FlexMeasures to assess the likelihood of future consumption/production and prices.
-Weather forecasting is included in the platform, but is not the result of an internal algorithm (see :ref:`weather`).
+Weather forecasting is included in the platform, but is usually not the result of an internal algorithm (weather forecast services are being used by import scripts, e.g. with `this tool <https://github.com/SeitaBV/weatherforecaststorage>`_).
+
+FlexMeasures uses linear regression and falls back to naive forecasting of the last known value if errors happen. 
+What might be even more important than the type of algorithm is the features handed to the model ― lagged values (e.g. value of the same time yesterday) and regressors (e.g. wind speed prediction to forecast wind power production).
+
+
 The performance of our algorithms is indicated by the mean absolute error (MAE) and the weighted absolute percentage error (WAPE).
 Power profiles on an asset level often include zero values, such that the mean absolute percentage error (MAPE), a common statistical measure of forecasting accuracy, is undefined.
 For such profiles, it is more useful to report the WAPE, which is also known as the volume weighted MAPE.
 The MAE of a power profile gives an indication of the size of the uncertainty in consumption and production.
-This allows the user to compare an asset's predictability to its flexibility, i.e. to the size of possible flexibility actions.
+This allows the user to compare an asset's predictability to its flexibility, i.e. to the size of possible flexibility activations.
 
 Example benchmarks per asset type are listed in the table below for various assets and forecasting horizons.
 FlexMeasures updates the benchmarks automatically for the data currently selected by the user.
@@ -54,44 +64,70 @@ Defaults:
 - Lagged outcome variables are selected based on the periodicity of the asset (e.g. daily and/or weekly).
 - Common external variables are weather forecasts of temperature, wind speed and irradiation.
 - Timeseries data with frequent zero values are transformed using a customised Box-Cox transformation.
-- To avoid overfitting, cross-validation is used.
-- Before fitting, explicit annotations of expert knowledge to the model (like the definition of asset-specific seasonalities and special time events) are possible.
+- To avoid over-fitting, cross-validation is used.
+- Before fitting, explicit annotations of expert knowledge to the model (like the definition of asset-specific seasonality and special time events) are possible.
 - The model is currently fit each day for each asset and for each horizon.
 
 Improvements:
 
-- Most assets have yearly seasonalities (e.g. wind, solar) and therefore forecasts would benefit from >= 2 years of history.
+- Most assets have yearly seasonality (e.g. wind, solar) and therefore forecasts would benefit from >= 2 years of history.
 
 
-Broker
-------
+Scheduling 
+------------
 
-A broker algorithm is used by the Aggregator to analyse flexibility in the Supplier's portfolio of assets, and to suggest the most valuable DR actions to take for each time slot.
-The actions are presented to the Supplier as flexibility offers in the form of an order book.
+Given price conditions or other conditions of relevance, a scheduling algorithm is used by the Aggregator (in case of explicit DR) or by the Energy Service Company (in case of implicit DR) to form a recommended schedule for the Prosumer's flexible assets.
 
-Defaults:
 
--
+Storage devices
+^^^^^^^^^^^^^^^
 
-Trading
--------
+So far, FlexMeasures provides algorithms for storage ― for batteries (e.g. home batteries or EVs) and car charging stations.
+We thus cover the asset types "battery", "one-way_evse" and "two-way_evse".
 
-A trading algorithm is used to assist the Supplier with its decision-making across time slots, based on the order books made by the broker (see above).
-The algorithm suggests which offers should be accepted next, and the Supplier may automate its decision-making by letting the algorithm place orders on its behalf.
+These algorithms schedule the storage assets based directly on the latest beliefs regarding market prices, within the specified time window.
+They are mixed integer linear programs, which are configured in FlexMeasures and then handed to a dedicated solver.
 
-Defaults:
+For all scheduling algorithms, a starting state of charge (SOC) as well as a set of SOC targets can be given. If no SOC is available, we set the starting SOC to 0. 
 
-- (Myopic greedy strategy) Order all flexibility with a positive expected value in the first available timeslot, then those in the second available timeslot, and so on.
+Also, per default we incentivise the algorithms to prefer scheduling charging now rather than later, and discharging later rather than now.
+We achieve this by adding a tiny artificial price slope. We penalise the future with at most 1 per thousand times the price spread. This behaviour can be turned off with the `prefer_charging_sooner` parameter set to `False`.
 
+.. note:: For the resulting consumption schedule, consumption is defined as positive values.
+
 
+Possible future work on algorithms
+-----------------------------------
 
+Enabling more algorithmic expression in FlexMeasures is crucial. This are a few ideas for future work. Some of them are excellent topics for Bachelor or Master theses. so get in touch if that is of interest to you.
 
-Planning
---------
+More configurable forecasting
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+On the roadmap for FlexMeasures is to make features easier to configure, especially regressors.
+Furthermore, we plan to add more types of forecasting algorithms, like random forest or even LSTM.
 
-Based on decisions about control actions, a planning algorithm is used by the Aggregator (in case of explicit DR) or by the Energy Service Company (in case of implicit DR)
-to form instructions for the Prosumer's flexible assets.
 
-Defaults:
+Other optimisation goals for scheduling
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Next to market prices, optimisation goals like reduced CO₂ emissions are sometimes required. There are multiple ways to measure this, e.g. against the CO₂ mix in the grid, or the use of fossil fuels.
+
+
+Scheduling of other flexible asset types
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Next to storage, there are other interesting flexible assets which can require specific implementations.
+For shifting, there are heat pumps and other buffers. For curtailment, there are wind turbines and solar panels.
+
+.. note:: See :ref:`flexibility_types` for more info on shifting and curtailment.
+
+Broker algorithm
+^^^^^^^^^^^^^^^^^
+A broker algorithm is used by the Aggregator to analyse flexibility in the Supplier's portfolio of assets, and to suggest the most valuable flexibility activations to take for each time slot.
+The differences to single-asset scheduling are that these activations are based on a helicopter perspective (the Aggregator optimises a portfolio, not a single asset) and that the flexibility offers are presented to the Supplier in the form of an order book.
+
+
+Trading algorithm
+^^^^^^^^^^^^^^^^^^
+A trading algorithm is used to assist the Supplier with its decision-making across time slots, based on the order books made by the broker (see above).
+The algorithm suggests which offers should be accepted next, and the Supplier may automate its decision-making by letting the algorithm place orders on its behalf.
 
-- 
+A default approach would be a myopic greedy strategy ― order all flexibility opportunities with a positive expected value in the first available timeslot, then those in the second available timeslot, and so on.

documentation/benefits.rst → documentation/concepts/benefits.rst

@@ -3,34 +3,35 @@
 .. _benefits:
 
 **************************************************
-Benefits of the FlexMeasures Platform
+Benefits
 **************************************************
 
 Automation
 -------------
 
-FlexMeasures provides decision-making support so that the platform operator can schedule flexibility actions.
-It forecasts the state of assets and proposes the best flexibility actions (shifting or curtailment)
-for future periods that need actions. This is done with modern forecasting and scheduling intelligence.
+FlexMeasures provides decision-making support so that the platform operator can schedule flexibility activations.
+It forecasts the state of assets and proposes the best flexibility activations (shifting or curtailment)
+for future periods. This is done with modern forecasting and scheduling intelligence.
 
 Insight
 --------------
 Both platform operator and asset owners can monitor the assets - past and current states as well as forecasts are displayed numerically in plots and tables.
-Flexibility actions ordered in the past can be reviewed.
-Proposed and scheduled flexibility actions show their expected effects (on imbalance as well as on financial returns).
+Activations of flexibility which were ordered in the past can be reviewed.
+Proposed and scheduled flexibility activations show their expected effects (on imbalance as well as on financial returns).
 
 Autonomy
 --------------
 The companies connected to FlexMeasures only give up as much control as necessary. The asset owners still control the main behaviour of their assets.
-The owners allow the platform operator to schedule flexibility actions within limits they can set.
+The owners allow the platform operator to schedule flexibility activations within limits they can set.
 
 Also the platform operator stays in charge:
-They can choose to approve all proposed flexibility actions manually or to let FlexMeasures automatically schedule them.
+They can choose to approve all proposed flexibility activations manually or to let FlexMeasures automatically schedule them.
 As FlexMeasures is open source, they can choose to host it themselves or let a third party (like Seita BV) do that.
 
 Profit sharing
 ---------------
-The platform operator and asset owners share the profit made from flexibility actions between them.
-FlexMeasures provides basic accounting.
+The platform operator (as ESCo or Aggregator) and asset owners can share the profit made from flexibility activations between them.
+FlexMeasures plans on providing basic accounting for this.
 
 
+.. note:: Read more on flexibility opportunities and activations, as well as profit sharing on :ref:`benefits_of_flex`