Parallelize sensor search

[Flix6x]

Tech spike to increase chart loading speed through parallelization. I'm using n=4 parallel processes here, but that could become a config setting.

@Ahmad-Wahid could you do a speed benchmark on this branch (compared to main) for me? Some manual tests of mine (locally) suggested an asset chart with 4 sensors and 1 week of data loads about twice as fast. I'm interested to see whether you can reproduce that result on a cloud server, and how the speed-up factor would change with a larger time frame and with more sensors.

[Ahmad-Wahid]

sure, I will do.

[Ahmad-Wahid]

Docker image cannot be build, as failing in the test. Locally, I could build it but making it live causing issue.

[Ahmad-Wahid]

[FLEXMEASURES][2024-01-16 14:59:19,356] ERROR: ConnectTimeout:"HTTPConnectionPool(host='*******', port=80): Max retries exceeded with url: //api/v3_0/assets?account_id=4 (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7fc41d9d2200>, 'Connection to **** timed out. (connect timeout=None)'))" [occurred at /usr/local/lib/python3.10/dist-packages/requests/adapters.py(send):507,URL was: /assets/]
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 203, in _new_conn
sock = connection.create_connection(
File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 85, in create_connection
raise err
File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 73, in create_connection
sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 790, in urlopen
response = self._make_request(
File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 496, in _make_request
conn.request(
File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 395, in request
self.endheaders()
File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output
self.send(msg)
File "/usr/lib/python3.10/http/client.py", line 976, in send
self.connect()
File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 243, in connect
self.sock = self._new_conn()
File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 212, in _new_conn
raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7fc41d9d2200>, 'Connection to **** timed out. (connect timeout=None)')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 844, in urlopen
retries = retries.increment(
File "/usr/local/lib/python3.10/dist-packages/urllib3/util/retry.py", line 515, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='****', port=80): Max retries exceeded with url: //api/v3_0/assets?account_id=4 (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7fc41d9d2200>, 'Connection to **** timed out. (connect timeout=None)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1823, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1799, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File "/usr/local/lib/python3.10/dist-packages/flask_classful.py", line 303, in proxy
response = view(**request.view_args)
File "/usr/local/lib/python3.10/dist-packages/flask_classful.py", line 271, in inner
return fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/flask_login/utils.py", line 290, in decorated_view
return current_app.ensure_sync(func)(*args, **kwargs)
File "/app/flexmeasures/ui/crud/assets.py", line 213, in index
assets += get_assets_by_account(account.id)
File "/app/flexmeasures/ui/crud/assets.py", line 182, in get_assets_by_account
get_assets_response = InternalApi().get(
File "/app/flexmeasures/ui/crud/api_wrapper.py", line 53, in get
response = requests.get(
File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 507, in send
raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='****l', port=80): Max retries exceeded with url: //api/v3_0/assets?account_id=4 (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7fc41d9d2200>, 'Connection to **** timed out. (connect timeout=None)'))