Distribution of Processing #481

tfarago · 2022-04-07T11:23:48Z

This will enable us to acquire data on a remote server and push it there via a ZMQ stream over network to processing nodes which can also be on different machines. Live preview will be able to subscribe to a stream of data and there will also be a stream splitter able to create a whole topology of processing nodes. The control will be done via concert, but only control, meaning mere synchronization of devices and data processing. All the rest will be done remotely, most probably via remote addons implemented as Tango servers for easy RPC access to them. More detailed info and diagrams to follow...

Todo

tfarago · 2022-04-07T11:26:22Z

This first commit enables us to have local and remote acquisitions and consumers. It's mostly about interfaces and defines a way how to create acquisitions and addons. It also introduces an explicit _Consumer class which encapsulates the consuming coroutine function which is way more explicit than the weird arbitrary passing of functions before.

concert/tests/integration/test_grating_interferometry_experiments.py

MarcusZuber · 2022-04-08T14:49:13Z

How about introducing a RemoteCamera, that sends to the Consumers instead of the RemoteAcquisition? Then the Experiments would not be different for local and remote acquisition.
But splitting it like you did looks nicer from the code structure (but most probable I have to create a lot of Experiment classes with all the combinations of remote/local etc. then).

tfarago · 2022-04-11T14:21:21Z

Why are you doing this to me?! :-) I will try.

codecov · 2022-07-01T13:36:12Z

Codecov Report

Attention: 642 lines in your changes are missing coverage. Please review.

Comparison is base (75c881f) 87.81% compared to head (2b5740d) 82.01%.
Report is 46 commits behind head on master.

❗ Current head 2b5740d differs from pull request most recent head 4f8ef8e. Consider uploading reports for the commit 4f8ef8e to get more accurate results

Files	Patch %	Lines
concert/networking/base.py	0.00%	184 Missing ⚠️
concert/experiments/addons/tango.py	0.00%	102 Missing ⚠️
concert/experiments/dummy.py	0.00%	99 Missing ⚠️
concert/experiments/addons/base.py	69.80%	61 Missing ⚠️
concert/devices/cameras/dummy.py	52.47%	48 Missing ⚠️
concert/experiments/base.py	54.87%	37 Missing ⚠️
concert/experiments/addons/local.py	81.28%	35 Missing ⚠️
concert/devices/cameras/base.py	67.60%	23 Missing ⚠️
concert/_ipython_setup.py	0.00%	21 Missing ⚠️
concert/session/utils.py	34.61%	17 Missing ⚠️
... and 3 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #481      +/-   ##
==========================================
- Coverage   87.81%   82.01%   -5.80%     
==========================================
  Files         115      115              
  Lines        7966     8222     +256     
==========================================
- Hits         6995     6743     -252     
- Misses        971     1479     +508

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

MarcusZuber · 2022-07-04T07:39:12Z

concert/ext/ufo.py

+                )
+            await self._params[arg].set(kwargs[arg])
+
+    @property


Could you make all 'properties' concert Quantities/Parameters? Otherwise this will be confusing (since there is no async access, which is mandatory for the others to use).

MarcusZuber · 2022-07-04T07:42:21Z

concert/ext/ufo.py

+        'retrieval_padded_width': q.px,
+        'retrieval_padded_height': q.px,
+        'projection_margin': q.px,
+        'x_region': q.px,


Off-topic: Since years I'm thinking about a elegant way to use vectors/lists in Quantities/Properties. At some point somebody tries to do args.x_region[2] = 10*q.px ...

Doesn't that work like it should automatically?

No, the getter will give you the vector (or whatever), but the setter will not be called, since only the []-operator is called on the result.

I am not sure I know what you mean, but this works:

concert > args = [1, 2 ,3] * q.px concert > args array([1, 2, 3]) <Unit('pixel')> concert > args[0] = 10 * q.px concert > args[0] 10 <Unit('pixel')> concert > args array([10, 2, 3]) <Unit('pixel')>

But this not (independent of quantity etc.):

from concert.base import Parameterizable class MyParameterizable(Parameterizable): my_vector = Parameter() async def __ainit__(self): self._vec = [1, 2, 3] async def _get_my_vector(self): return self._my_vector async def _set_my_vector(self, vec): self._vec = vec p = await MyParameterizable() p.my_vector[1] = 5

Ah that, well numpy managed to override []...

I already thought to do this for the QuantityValue...

Good, then I leave it to you 😉

MarcusZuber · 2022-07-04T07:45:34Z

concert/ext/ufo.py

@@ -232,26 +233,39 @@ async def __call__(self, producer):


 class GeneralBackprojectArgs(object):


Since you already break (slightly) the interface to the reco-args: I always found it a bit artificial that the reco-args and the Reco were splitted (and not the args where properties of the reco-class). What do you think about merging both classes?

I will keep it in mind, let me finish up the reco and the remote version of it and then I'll see if it will still be reasonable to merge.

tfarago · 2022-07-08T08:30:16Z

concert/ext/tangoservers/benchmarking.py

+
+def run_server():
+    TangoBenchmarker.run_server(
+        args=['name', '-ORBendPoint', 'giop:tcp::1235', '-v4', '-nodb', '-dlist', 'a/b/c'],


Change this everywhere to a proper DB-based start.

tfarago · 2022-07-13T09:15:21Z

concert/ext/tangoservers/camera.py

+    Tango device for the dummy concert camera sending images over a zmq socket.
+    """
+
+    endpoint = attribute(


@MarcusZuber do you know why Tango doesn't like inheriting attributes? If you run the TangoDummyCamera server you can set the endpoint without any problem, but try reading it and it just crashes.

tfarago · 2022-07-13T09:16:28Z

concert/ext/tangoservers/camera.py

+            self._params_initialized = True
+
+    def dynamic_getter(self, attr):
+        # If this were async def Tango would complain about it never being awaited


This is one of the reasons I will not change the format strings + eval in the reco server right now.

tfarago · 2022-07-29T11:09:18Z

@MarcusZuber in experiments.base and addons package is a draft of how I see things could go, if you are too bored in the next three weeks you can take a look, I will get back to it then.

which will be dealt with in PR #481.

MarcusZuber · 2023-08-14T10:17:39Z

This is slightly connected with this and libuca:

So far we have only one grab()-function, which will give in a buffered mode the oldest not read frames and in an unbuffered mode most probable the last recorded frame.
(I'm currently working on a detector that features the possibility to get the most recent frame as well to use a buffer)

Should we introduce a 2nd grab function for "live images" in libuca and concert?

What is related to this PR: It would be nice to stream the live_images to the concert.experiments.addons.Consumer (aka live_view), but the buffered data to writer/online-reco etc.

tfarago · 2023-08-23T09:41:15Z

I don't see why not having a camera.grab_live method. In the remote implementation one would then probably set up a PUB-SUB socket pair to disseminate the images.

MarcusZuber · 2023-10-31T05:46:30Z

We really need the current implementation of the image walker/writer that handles metadata in this also to work.
The metadata can be sent via a pipe, just the re-combination with the frame needs some thinking.

tfarago · 2023-10-31T09:18:47Z

We really need the current implementation of the image walker/writer that handles metadata in this also to work. The metadata can be sent via a pipe, just the re-combination with the frame needs some thinking.

Can you give an example why? Is it the motor positions?

MarcusZuber · 2023-10-31T17:18:26Z

I add a lot of stuff to the metadata, e.g. synchrotron current, motor positions, x-ray tube parameters, timestamps... That was the whole point of introducing the metadata.

tfarago · 2023-11-02T08:48:55Z

That's not so critical, it will start to be problematic when you really need some synchronization between metadata and a concrete image. What you mentioned here is easy to add, just another stream(s) with the metadata.

MarcusZuber · 2023-11-21T09:23:05Z

I thought a bit more about the whole logging/metadata stuff.

I think in the real use cases it would be ideal to attach metadata to a single image when a camera is run in software-trigger-source (e.g. a special stepped scan of which I have many) and to log time-coded metadata otherwise.

So my suggestion would be that we maybe introduce a trigger_with_metadata(dict) function to the camera, that takes care of the synchronization (so ucad would need to take care to combine the stuff correctly) and a log_metadata to the (remote)walker, that basically does what is basically this

concert/concert/writers.py

Line 32 in 9336224

def _write_metadata(self, image):

but with a timestamp instead of the frame number.

tfarago · 2023-11-22T19:40:02Z

I think this breaks encapsulation a little bit and would rather make the writer a bit more clever. I will come up with a concept and get back to this.

MarcusZuber · 2023-12-18T10:21:06Z

I thought a bit more about the whole logging/metadata stuff.

I think in the real use cases it would be ideal to attach metadata to a single image when a camera is run in software-trigger-source (e.g. a special stepped scan of which I have many) and to log time-coded metadata otherwise.

So my suggestion would be that we maybe introduce a trigger_with_metadata(dict) function to the camera, that takes care of the synchronization (so ucad would need to take care to combine the stuff correctly) and a log_metadata to the (remote)walker, that basically does what is basically this

concert/concert/writers.py

Line 32 in 9336224

def _write_metadata(self, image):

but with a timestamp instead of the frame number.

cf933cb would be a simple implementation of the 'log_metadata'. Open for discussions.

concert: Implement remote walker and logger

tfarago added feature enhancement in progress labels Apr 7, 2022

tfarago force-pushed the decentralize branch 2 times, most recently from b909e95 to ff891b3 Compare April 8, 2022 14:11

MarcusZuber reviewed Apr 8, 2022

View reviewed changes

concert/tests/integration/test_grating_interferometry_experiments.py Outdated Show resolved Hide resolved

tfarago force-pushed the decentralize branch from ff891b3 to 6baa29c Compare July 1, 2022 13:32

tfarago force-pushed the decentralize branch from 6baa29c to e44b60c Compare July 1, 2022 14:45

MarcusZuber reviewed Jul 4, 2022

View reviewed changes

tfarago force-pushed the decentralize branch from e44b60c to 7b7515f Compare July 5, 2022 09:45

tfarago commented Jul 8, 2022

View reviewed changes

tfarago force-pushed the decentralize branch 3 times, most recently from 0c99603 to 3da5890 Compare July 8, 2022 12:11

tfarago marked this pull request as draft July 12, 2022 06:48

tfarago commented Jul 13, 2022

View reviewed changes

tfarago force-pushed the decentralize branch from 113e055 to 2561612 Compare July 25, 2022 12:42

tfarago force-pushed the decentralize branch from 654cd0a to 8a67145 Compare August 25, 2022 13:14

tfarago added a commit that referenced this pull request Aug 26, 2022

experiment: remove KeyboardInterrupt

660a047

which will be dealt with in PR #481.

tfarago mentioned this pull request Aug 26, 2022

Fix KeyboardInterrupt #505

Open

tfarago force-pushed the decentralize branch from 8a67145 to c96ed85 Compare September 8, 2022 12:02

tfarago added a commit that referenced this pull request Sep 12, 2022

experiment: remove KeyboardInterrupt

44f498c

which will be dealt with in PR #481.

tfarago added 9 commits August 30, 2023 15:52

networking: return everything in receive_image

b8a5215

cameras: simplify remote API

e0d1105

remote uca: enable int64 in num_frames

00e92c8

uca remote: enable endpoint registration

c2e53eb

uca remote: enable endpoint type specification

c5c674b

SQUASH uca: update register/unregister

0ffe11e

SQUASH to "cameras: simplify remote API" tango camera

180ceea

SQUASH: tangodummycamera: add missing simulate parameter

70baf16

SQUASH docs: some updates

0d95bd9

sarkarchandan and others added 12 commits January 18, 2024 11:07

concert: Implement remote walker and logger

41a1884

concert: Unify async walker base class

2b78335

fix coverage upload

7094331

concert: Remove irrelevant changes

184776d

concert: Add walker device server to tango command

af186d5

concert: Plug remote walker to base experiment

1928f5a

concert: Fix broken tests for decentralized adaption

76a8b0e

concert: Cleanup unrelated changes

450dc18

concert: Refactor logging utility

fceea42

concert: Fix log handler io

a5c2919

concert: Refactor logfile handling

2619f03

Merge pull request #530 from ufo-kit/chandan/remote_walker_backup

4f8ef8e

concert: Implement remote walker and logger

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Distribution of Processing #481

Distribution of Processing #481

tfarago commented Apr 7, 2022 •

edited

tfarago commented Apr 7, 2022

MarcusZuber commented Apr 8, 2022

tfarago commented Apr 11, 2022

codecov bot commented Jul 1, 2022 •

edited

MarcusZuber Jul 4, 2022 •

edited

MarcusZuber Jul 4, 2022

tfarago Jul 4, 2022

MarcusZuber Jul 4, 2022 •

edited

tfarago Jul 4, 2022

MarcusZuber Jul 4, 2022

tfarago Jul 4, 2022

MarcusZuber Jul 4, 2022

tfarago Jul 4, 2022 •

edited

MarcusZuber Jul 4, 2022 •

edited

tfarago Jul 4, 2022

tfarago Jul 8, 2022

tfarago Jul 13, 2022

tfarago Jul 13, 2022

tfarago commented Jul 29, 2022

MarcusZuber commented Aug 14, 2023

tfarago commented Aug 23, 2023

MarcusZuber commented Oct 31, 2023

tfarago commented Oct 31, 2023

MarcusZuber commented Oct 31, 2023

tfarago commented Nov 2, 2023

MarcusZuber commented Nov 21, 2023

tfarago commented Nov 22, 2023

MarcusZuber commented Dec 18, 2023

		@@ -232,26 +233,39 @@ async def __call__(self, producer):


		class GeneralBackprojectArgs(object):

Distribution of Processing #481

Are you sure you want to change the base?

Distribution of Processing #481

Conversation

tfarago commented Apr 7, 2022 • edited

tfarago commented Apr 7, 2022

MarcusZuber commented Apr 8, 2022

tfarago commented Apr 11, 2022

codecov bot commented Jul 1, 2022 • edited

Codecov Report

MarcusZuber Jul 4, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MarcusZuber Jul 4, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tfarago Jul 4, 2022 • edited

Choose a reason for hiding this comment

MarcusZuber Jul 4, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tfarago commented Jul 29, 2022

MarcusZuber commented Aug 14, 2023

tfarago commented Aug 23, 2023

MarcusZuber commented Oct 31, 2023

tfarago commented Oct 31, 2023

MarcusZuber commented Oct 31, 2023

tfarago commented Nov 2, 2023

MarcusZuber commented Nov 21, 2023

tfarago commented Nov 22, 2023

MarcusZuber commented Dec 18, 2023

tfarago commented Apr 7, 2022 •

edited

codecov bot commented Jul 1, 2022 •

edited

MarcusZuber Jul 4, 2022 •

edited

MarcusZuber Jul 4, 2022 •

edited

tfarago Jul 4, 2022 •

edited

MarcusZuber Jul 4, 2022 •

edited