Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

gatherer raises KeyError in terminator when default config section lacks publish_topic and metadata lacks format #74

Open
gerritholl opened this issue Jan 14, 2021 · 0 comments
Open

gatherer raises KeyError in terminator when default config section lacks publish_topic and metadata lacks format #74

gerritholl opened this issue Jan 14, 2021 · 0 comments

Comments

@gerritholl
Copy link
Contributor

When for some reason in gatherer.py there is a call to terminator() with publish_topic=None, and the metadata (containing a message sent via posttroll?) does not contain the format key, gatherer.py raises a KeyError: 'format'.

MCVE

The true minimum would be a Python script calling terminator() with certain arguments, but I think that's not the most useful way to reproduce it, because the bug may be upstream (why does terminator() get called this way). My setup:

  • nameserver is running
  • trollstalker is running with trollstalker -c trollstalker.ini -C avhrrl0 -n localhost — contents of trollstalker.ini:
[avhrrl0]
topic=/file/poes/avhrr
directory=/data/pytroll/IN/HRPT/
posttroll_port=0
publish_port=
event_names=IN_CLOSE_WRITE,IN_MOVED_TO
loglevel=DEBUG
stalker_log_config=/opt/pytroll/pytroll_inst/config/trollstalker_logging.ini
filepattern={path}AVHR_HRP_00_{platform_name}_{start_time:%Y%m%d%H%M%S}Z_{end_time:%Y%m%d%H%M%S}Z_N_O_{processing_time:%Y%m%d%H%M%S}Z
instruments=avhrr/3
alias_platform_name=M01:Metop-B|M02:Metop-A|M03:Metop-C|noaa18:NOAA-18|noaa19:NOAA-19
history=0
nameservers=localhost
  • contents of gatherer.ini:
[default]
regions = eurol
timeliness = 180
service =
topics = /file/poes/avhrr

[metopa-avhrr-hrpt]
regions = eurol
service =
pattern = /data/pytroll/IN/HRPT/AVHR_HRP_00_M02_{start_time:%Y%m%d%H%M%S}Z_{end_time:%Y%m%d%H%M%S}Z_N_O_{processing_time:%Y%m%d%H%M%S}Z
sensor = avhrr
topics = /file/poes/avhrr
publish_topic = /collection/metopa/avhrr
timeliness = 180
duration = 300
platform_name = Metop-A
format = HRPT
type = binary
variant = DR
level = 0
orbit_type = polar
nameserver = localhost

Running gatherer.py -v gatherer.ini. In another terminal, move some matching files to the directory watched by trollstalker (mv AVHR_HRP_00_M02_20210114080* /tmp/foo/ && mv /tmp/foo/AVHR_HRP_00* .).

Expected output

I don't understand gatherer.py well enough yet to know what to expect, but I suspect it should publish a message and continue running.

Actual output

Output from gatherer.py (I added a print(metadata) in the terminator function for debugging purposes):

Output from `gatherer.py -v gatherer.ini` 
[DEBUG: 2021-01-14 16:37:41 : gatherer] Using posttroll for default
[DEBUG: 2021-01-14 16:37:41 : pytroll_collectors.trigger] Nameserver: localhost
[DEBUG: 2021-01-14 16:37:41 : gatherer] Using posttroll for metopa-avhrr-hrpt
[DEBUG: 2021-01-14 16:37:41 : pytroll_collectors.trigger] Nameserver: localhost
[INFO: 2021-01-14 16:37:41 : posttroll.publisher] publisher started on port 33687
[INFO: 2021-01-14 16:37:41 : posttroll.subscriber] Subscriber adding SUB hook tcp://localhost:16543 for topics ['pytroll://address']
[INFO: 2021-01-14 16:37:41 : posttroll.subscriber] Subscriber adding address tcp://141.38.37.143:16543 with topics ['pytroll://file/poes/avhrr']
[INFO: 2021-01-14 16:37:41 : posttroll.subscriber] Subscriber adding address tcp://141.38.37.143:32883 with topics ['pytroll://file/poes/avhrr']
[INFO: 2021-01-14 16:37:41 : posttroll.subscriber] Subscriber adding SUB hook tcp://localhost:16543 for topics ['pytroll://address']
[INFO: 2021-01-14 16:37:41 : posttroll.subscriber] Subscriber adding address tcp://141.38.37.143:16543 with topics ['pytroll://file/poes/avhrr']
[INFO: 2021-01-14 16:37:41 : posttroll.subscriber] Subscriber adding address tcp://141.38.37.143:32883 with topics ['pytroll://file/poes/avhrr']
[DEBUG: 2021-01-14 16:37:47 : pytroll_collectors.trigger] mda: {'path': '', 'platform_name': 'Metop-A', 'start_time': datetime.datetime(2021, 1, 14, 8, 0, 12), 'end_time': datetime.datetime(2021, 1, 14, 8, 3, 12), 'processing_time': datetime.datetime(2021, 1, 14, 8, 0, 12), 'uri': '/data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080012Z_20210114080312Z_N_O_20210114080012Z', 'uid': 'AVHR_HRP_00_M02_20210114080012Z_20210114080312Z_N_O_20210114080012Z', 'sensor': ['avhrr/3'], 'orig_platform_name': 'M02'}
[DEBUG: 2021-01-14 16:37:47 : pytroll_collectors.region_collector] Adding area ID to metadata: eurol
[INFO: 2021-01-14 16:37:47 : pytroll_collectors.region_collector] Platform name Metop-A and sensor ['avhrr/3']: Start and end times = 20210114 08:00:12 20210114 08:03:12
[DEBUG: 2021-01-14 16:37:47 : pytroll_collectors.trigger] mda: {'path': '', 'platform_name': 'Metop-A', 'start_time': datetime.datetime(2021, 1, 14, 8, 0, 12), 'end_time': datetime.datetime(2021, 1, 14, 8, 3, 12), 'processing_time': datetime.datetime(2021, 1, 14, 8, 0, 12), 'uri': '/data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080012Z_20210114080312Z_N_O_20210114080012Z', 'uid': 'AVHR_HRP_00_M02_20210114080012Z_20210114080312Z_N_O_20210114080012Z', 'sensor': ['avhrr/3'], 'orig_platform_name': 'M02'}
[DEBUG: 2021-01-14 16:37:47 : pytroll_collectors.region_collector] Adding area ID to metadata: eurol
[DEBUG: 2021-01-14 16:37:47 : pytroll_collectors.region_collector] Estimated granule duration to 0:03:00
[INFO: 2021-01-14 16:37:47 : pytroll_collectors.region_collector] Platform name Metop-A and sensor ['avhrr/3']: Start and end times = 20210114 08:00:12 20210114 08:03:12
[DEBUG: 2021-01-14 16:37:47 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'}
[DEBUG: 2021-01-14 16:37:47 : trollsched.satpass] instrument: avhrr/3
[DEBUG: 2021-01-14 16:37:47 : pyorbital.tlefile] Fetch TLE from the internet.
[DEBUG: 2021-01-14 16:37:47 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'}
[DEBUG: 2021-01-14 16:37:47 : trollsched.satpass] instrument: avhrr/3
[DEBUG: 2021-01-14 16:37:47 : pyorbital.tlefile] Fetch TLE from the internet.
[DEBUG: 2021-01-14 16:37:48 : trollsched.boundary] Instrument = avhrr/3
[INFO: 2021-01-14 16:37:49 : pytroll_collectors.region_collector] Added Metop-A (2021-01-14 08:00:12) granule to area eurol
[DEBUG: 2021-01-14 16:37:49 : pytroll_collectors.region_collector] Predicting granules covering eurol
[DEBUG: 2021-01-14 16:37:49 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'}
[DEBUG: 2021-01-14 16:37:49 : trollsched.satpass] instrument: avhrr/3
[DEBUG: 2021-01-14 16:37:49 : pyorbital.tlefile] Fetch TLE from the internet.
[DEBUG: 2021-01-14 16:37:50 : trollsched.boundary] Instrument = avhrr/3
[DEBUG: 2021-01-14 16:37:51 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'}
[DEBUG: 2021-01-14 16:37:51 : trollsched.satpass] instrument: avhrr/3
[DEBUG: 2021-01-14 16:37:51 : pyorbital.tlefile] Fetch TLE from the internet.
[DEBUG: 2021-01-14 16:37:52 : trollsched.boundary] Instrument = avhrr/3
[DEBUG: 2021-01-14 16:37:53 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'}
[DEBUG: 2021-01-14 16:37:53 : trollsched.satpass] instrument: avhrr/3
[DEBUG: 2021-01-14 16:37:53 : pyorbital.tlefile] Fetch TLE from the internet.
[DEBUG: 2021-01-14 16:37:53 : trollsched.boundary] Instrument = avhrr/3
[INFO: 2021-01-14 16:37:53 : pytroll_collectors.region_collector] Added Metop-A (2021-01-14 08:00:12) granule to area eurol
[DEBUG: 2021-01-14 16:37:53 : pytroll_collectors.region_collector] Predicting granules covering eurol
[DEBUG: 2021-01-14 16:37:53 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'}
[DEBUG: 2021-01-14 16:37:53 : trollsched.satpass] instrument: avhrr/3
[DEBUG: 2021-01-14 16:37:53 : pyorbital.tlefile] Fetch TLE from the internet.
[DEBUG: 2021-01-14 16:37:54 : trollsched.boundary] Instrument = avhrr/3
[DEBUG: 2021-01-14 16:37:54 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'}
[DEBUG: 2021-01-14 16:37:54 : trollsched.satpass] instrument: avhrr/3
[DEBUG: 2021-01-14 16:37:54 : pyorbital.tlefile] Fetch TLE from the internet.
[DEBUG: 2021-01-14 16:37:59 : trollsched.boundary] Instrument = avhrr/3
[DEBUG: 2021-01-14 16:37:59 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'}
[DEBUG: 2021-01-14 16:37:59 : trollsched.satpass] instrument: avhrr/3
[DEBUG: 2021-01-14 16:37:59 : pyorbital.tlefile] Fetch TLE from the internet.
[DEBUG: 2021-01-14 16:38:01 : trollsched.boundary] Instrument = avhrr/3
[DEBUG: 2021-01-14 16:38:01 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'}
[DEBUG: 2021-01-14 16:38:01 : trollsched.satpass] instrument: avhrr/3
[DEBUG: 2021-01-14 16:38:01 : pyorbital.tlefile] Fetch TLE from the internet.
[DEBUG: 2021-01-14 16:38:02 : trollsched.boundary] Instrument = avhrr/3
[DEBUG: 2021-01-14 16:38:02 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'}
[DEBUG: 2021-01-14 16:38:02 : trollsched.satpass] instrument: avhrr/3
[DEBUG: 2021-01-14 16:38:02 : pyorbital.tlefile] Fetch TLE from the internet.
[DEBUG: 2021-01-14 16:38:06 : trollsched.boundary] Instrument = avhrr/3
[DEBUG: 2021-01-14 16:38:07 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'}
[DEBUG: 2021-01-14 16:38:07 : trollsched.satpass] instrument: avhrr/3
[DEBUG: 2021-01-14 16:38:07 : pyorbital.tlefile] Fetch TLE from the internet.
[DEBUG: 2021-01-14 16:38:07 : trollsched.boundary] Instrument = avhrr/3
[INFO: 2021-01-14 16:38:08 : pytroll_collectors.region_collector] Planned granules for Euro 3.0km area - Europe: [datetime.datetime(2021, 1, 14, 7, 50, 12), datetime.datetime(2021, 1, 14, 7, 55, 12), datetime.datetime(2021, 1, 14, 8, 0, 12), datetime.datetime(2021, 1, 14, 8, 5, 12)]
[INFO: 2021-01-14 16:38:08 : pytroll_collectors.region_collector] Planned timeout for Euro 3.0km area - Europe: 2021-01-14T11:10:12
[WARNING: 2021-01-14 16:38:08 : pytroll_collectors.trigger] Timeout detected, terminating collector
[DEBUG: 2021-01-14 16:38:08 : pytroll_collectors.trigger] mda: {'path': '', 'platform_name': 'Metop-A', 'start_time': datetime.datetime(2021, 1, 14, 8, 3, 12), 'end_time': datetime.datetime(2021, 1, 14, 8, 6, 12), 'processing_time': datetime.datetime(2021, 1, 14, 8, 3, 12), 'uri': '/data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080312Z_20210114080612Z_N_O_20210114080312Z', 'uid': 'AVHR_HRP_00_M02_20210114080312Z_20210114080612Z_N_O_20210114080312Z', 'sensor': ['avhrr/3'], 'orig_platform_name': 'M02'}
/opt/pytroll/pytroll_inst/miniconda3/envs/pytroll-py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems
  proj_string = self.to_proj4()
[DEBUG: 2021-01-14 16:38:08 : pytroll_collectors.trigger] Area: Area ID: eurol
Description: Euro 3.0km area - Europe
Projection: {'ellps': 'WGS84', 'lat_0': '90', 'lat_ts': '60', 'lon_0': '0', 'no_defs': 'None', 'proj': 'stere', 'type': 'crs', 'units': 'm', 'x_0': '0', 'y_0': '0'}
Number of columns: 2560
Number of rows: 2048
Area extent: (-3780000.0, -7644000.0, 3900000.0, -1500000.0), timeout: 2021-01-14 11:10:12
[DEBUG: 2021-01-14 16:38:08 : pytroll_collectors.region_collector] Adding area ID to metadata: eurol
[INFO: 2021-01-14 16:38:08 : gatherer] Composing topic.
[INFO: 2021-01-14 16:38:08 : pytroll_collectors.region_collector] Platform name Metop-A and sensor ['avhrr/3']: Start and end times = 20210114 08:03:12 20210114 08:06:12
[DEBUG: 2021-01-14 16:38:08 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'}
[DEBUG: 2021-01-14 16:38:08 : trollsched.satpass] instrument: avhrr/3
[DEBUG: 2021-01-14 16:38:08 : pyorbital.tlefile] Fetch TLE from the internet.
[INFO: 2021-01-14 16:38:08 : gatherer] sending pytroll://collection/metopa/avhrr collection pytroll@oflks333.dwd.de 2021-01-14T16:38:08.450784 v1.01 application/json {"path": "", "platform_name": "Metop-A", "start_time": "2021-01-14T08:00:12", "end_time": "2021-01-14T08:03:12", "processing_time": "2021-01-14T08:00:12", "sensor": ["avhrr/3"], "orig_platform_name": "M02", "collection_area_id": "eurol", "collection": [{"start_time": "2021-01-14T08:00:12", "end_time": "2021-01-14T08:03:12", "uri": "/data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080012Z_20210114080312Z_N_O_20210114080012Z", "uid": "AVHR_HRP_00_M02_20210114080012Z_20210114080312Z_N_O_20210114080012Z"}]}
[DEBUG: 2021-01-14 16:38:08 : trollsched.boundary] Instrument = avhrr/3
[DEBUG: 2021-01-14 16:38:09 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'}
[DEBUG: 2021-01-14 16:38:09 : trollsched.satpass] instrument: avhrr/3
[DEBUG: 2021-01-14 16:38:09 : pyorbital.tlefile] Fetch TLE from the internet.
[DEBUG: 2021-01-14 16:38:12 : trollsched.boundary] Instrument = avhrr/3
[DEBUG: 2021-01-14 16:38:13 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'}
[DEBUG: 2021-01-14 16:38:13 : trollsched.satpass] instrument: avhrr/3
[DEBUG: 2021-01-14 16:38:13 : pyorbital.tlefile] Fetch TLE from the internet.
[DEBUG: 2021-01-14 16:38:13 : trollsched.boundary] Instrument = avhrr/3
[INFO: 2021-01-14 16:38:14 : pytroll_collectors.region_collector] Added Metop-A (2021-01-14 08:03:12) granule to area eurol
[DEBUG: 2021-01-14 16:38:14 : pytroll_collectors.region_collector] Predicting granules covering eurol
[DEBUG: 2021-01-14 16:38:14 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'}
[DEBUG: 2021-01-14 16:38:14 : trollsched.satpass] instrument: avhrr/3
[DEBUG: 2021-01-14 16:38:14 : pyorbital.tlefile] Fetch TLE from the internet.
[DEBUG: 2021-01-14 16:38:15 : trollsched.boundary] Instrument = avhrr/3
[DEBUG: 2021-01-14 16:38:15 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'}
[DEBUG: 2021-01-14 16:38:15 : trollsched.satpass] instrument: avhrr/3
[DEBUG: 2021-01-14 16:38:15 : pyorbital.tlefile] Fetch TLE from the internet.
[DEBUG: 2021-01-14 16:38:18 : trollsched.boundary] Instrument = avhrr/3
[DEBUG: 2021-01-14 16:38:18 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'}
[DEBUG: 2021-01-14 16:38:18 : trollsched.satpass] instrument: avhrr/3
[DEBUG: 2021-01-14 16:38:18 : pyorbital.tlefile] Fetch TLE from the internet.
[DEBUG: 2021-01-14 16:38:20 : trollsched.boundary] Instrument = avhrr/3
[DEBUG: 2021-01-14 16:38:21 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'}
[DEBUG: 2021-01-14 16:38:21 : trollsched.satpass] instrument: avhrr/3
[DEBUG: 2021-01-14 16:38:21 : pyorbital.tlefile] Fetch TLE from the internet.
[DEBUG: 2021-01-14 16:38:21 : trollsched.boundary] Instrument = avhrr/3
[INFO: 2021-01-14 16:38:22 : pytroll_collectors.region_collector] Planned granules for Euro 3.0km area - Europe: [datetime.datetime(2021, 1, 14, 7, 51, 12), datetime.datetime(2021, 1, 14, 7, 54, 12), datetime.datetime(2021, 1, 14, 7, 57, 12), datetime.datetime(2021, 1, 14, 8, 0, 12), datetime.datetime(2021, 1, 14, 8, 3, 12), datetime.datetime(2021, 1, 14, 8, 6, 12)]
[INFO: 2021-01-14 16:38:22 : pytroll_collectors.region_collector] Planned timeout for Euro 3.0km area - Europe: 2021-01-14T11:09:12
[DEBUG: 2021-01-14 16:38:22 : pytroll_collectors.trigger] mda: {'path': '', 'platform_name': 'Metop-A', 'start_time': datetime.datetime(2021, 1, 14, 8, 3, 12), 'end_time': datetime.datetime(2021, 1, 14, 8, 6, 12), 'processing_time': datetime.datetime(2021, 1, 14, 8, 3, 12), 'uri': '/data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080312Z_20210114080612Z_N_O_20210114080312Z', 'uid': 'AVHR_HRP_00_M02_20210114080312Z_20210114080612Z_N_O_20210114080312Z', 'sensor': ['avhrr/3'], 'orig_platform_name': 'M02'}
[DEBUG: 2021-01-14 16:38:22 : pytroll_collectors.region_collector] Adding area ID to metadata: eurol
[INFO: 2021-01-14 16:38:22 : pytroll_collectors.region_collector] Added Metop-A (2021-01-14 08:03:12) granule to area eurol
[DEBUG: 2021-01-14 16:38:22 : pytroll_collectors.trigger] mda: {'path': '', 'platform_name': 'Metop-A', 'start_time': datetime.datetime(2021, 1, 14, 8, 6, 12), 'end_time': datetime.datetime(2021, 1, 14, 8, 6, 41), 'processing_time': datetime.datetime(2021, 1, 14, 8, 6, 12), 'uri': '/data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080612Z_20210114080641Z_N_O_20210114080612Z', 'uid': 'AVHR_HRP_00_M02_20210114080612Z_20210114080641Z_N_O_20210114080612Z', 'sensor': ['avhrr/3'], 'orig_platform_name': 'M02'}
[DEBUG: 2021-01-14 16:38:22 : pytroll_collectors.region_collector] Adding area ID to metadata: eurol
[INFO: 2021-01-14 16:38:22 : pytroll_collectors.region_collector] Added Metop-A (2021-01-14 08:06:12) granule to area eurol
[INFO: 2021-01-14 16:38:22 : pytroll_collectors.region_collector] Adjusted timeout: 2021-01-14T11:00:12
[WARNING: 2021-01-14 16:38:22 : pytroll_collectors.trigger] Timeout detected, terminating collector
/opt/pytroll/pytroll_inst/miniconda3/envs/pytroll-py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems
  proj_string = self.to_proj4()
[DEBUG: 2021-01-14 16:38:22 : pytroll_collectors.trigger] Area: Area ID: eurol
Description: Euro 3.0km area - Europe
Projection: {'ellps': 'WGS84', 'lat_0': '90', 'lat_ts': '60', 'lon_0': '0', 'no_defs': 'None', 'proj': 'stere', 'type': 'crs', 'units': 'm', 'x_0': '0', 'y_0': '0'}
Number of columns: 2560
Number of rows: 2048
Area extent: (-3780000.0, -7644000.0, 3900000.0, -1500000.0), timeout: 2021-01-14 11:00:12
[INFO: 2021-01-14 16:38:22 : gatherer] Using default topic.
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/opt/pytroll/pytroll_inst/miniconda3/envs/pytroll-py38/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/opt/pytroll/pytroll_inst/miniconda3/envs/pytroll-py38/lib/python3.8/site-packages/pytroll_collectors/trigger.py", line 141, in run
    self.terminator(next_timeout[0].finish(),
  File "/opt/pytroll/pytroll_inst/miniconda3/envs/pytroll-py38/bin/gatherer.py", line 90, in terminator
    subject = "/".join(("", mda["format"], mda["data_processing_level"],
KeyError: 'format'
[CRITICAL: 2021-01-14 16:38:22 : gatherer] Something went wrong!
[WARNING: 2021-01-14 16:38:22 : gatherer] Ending publication the gathering of granules...
Setting timezone to UTC
[{'path': '', 'platform_name': 'Metop-A', 'start_time': datetime.datetime(2021, 1, 14, 8, 0, 12), 'end_time': datetime.datetime(2021, 1, 14, 8, 3, 12), 'processing_time': datetime.datetime(2021, 1, 14, 8, 0, 12), 'uri': '/data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080012Z_20210114080312Z_N_O_20210114080012Z', 'uid': 'AVHR_HRP_00_M02_20210114080012Z_20210114080312Z_N_O_20210114080012Z', 'sensor': ['avhrr/3'], 'orig_platform_name': 'M02', 'collection_area_id': 'eurol'}] /collection/metopa/avhrr
[{'path': '', 'platform_name': 'Metop-A', 'start_time': datetime.datetime(2021, 1, 14, 8, 0, 12), 'end_time': datetime.datetime(2021, 1, 14, 8, 3, 12), 'processing_time': datetime.datetime(2021, 1, 14, 8, 0, 12), 'uri': '/data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080012Z_20210114080312Z_N_O_20210114080012Z', 'uid': 'AVHR_HRP_00_M02_20210114080012Z_20210114080312Z_N_O_20210114080012Z', 'sensor': ['avhrr/3'], 'orig_platform_name': 'M02', 'collection_area_id': 'eurol'}, {'path': '', 'platform_name': 'Metop-A', 'start_time': datetime.datetime(2021, 1, 14, 8, 3, 12), 'end_time': datetime.datetime(2021, 1, 14, 8, 6, 12), 'processing_time': datetime.datetime(2021, 1, 14, 8, 3, 12), 'uri': '/data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080312Z_20210114080612Z_N_O_20210114080312Z', 'uid': 'AVHR_HRP_00_M02_20210114080312Z_20210114080612Z_N_O_20210114080312Z', 'sensor': ['avhrr/3'], 'orig_platform_name': 'M02', 'collection_area_id': 'eurol'}, {'path': '', 'platform_name': 'Metop-A', 'start_time': datetime.datetime(2021, 1, 14, 8, 6, 12), 'end_time': datetime.datetime(2021, 1, 14, 8, 6, 41), 'processing_time': datetime.datetime(2021, 1, 14, 8, 6, 12), 'uri': '/data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080612Z_20210114080641Z_N_O_20210114080612Z', 'uid': 'AVHR_HRP_00_M02_20210114080612Z_20210114080641Z_N_O_20210114080612Z', 'sensor': ['avhrr/3'], 'orig_platform_name': 'M02', 'collection_area_id': 'eurol'}] None
[DEBUG: 2021-01-14 16:38:25 : trollsched.boundary] Instrument = avhrr/3
[DEBUG: 2021-01-14 16:38:26 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'}
[DEBUG: 2021-01-14 16:38:26 : trollsched.satpass] instrument: avhrr/3
[DEBUG: 2021-01-14 16:38:26 : pyorbital.tlefile] Fetch TLE from the internet.
[DEBUG: 2021-01-14 16:38:28 : trollsched.boundary] Instrument = avhrr/3
[DEBUG: 2021-01-14 16:38:29 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'}
[DEBUG: 2021-01-14 16:38:29 : trollsched.satpass] instrument: avhrr/3
[DEBUG: 2021-01-14 16:38:29 : pyorbital.tlefile] Fetch TLE from the internet.
[DEBUG: 2021-01-14 16:38:34 : trollsched.boundary] Instrument = avhrr/3
[INFO: 2021-01-14 16:38:35 : pytroll_collectors.region_collector] Planned granules for Euro 3.0km area - Europe: [datetime.datetime(2021, 1, 14, 7, 48, 12), datetime.datetime(2021, 1, 14, 7, 53, 12), datetime.datetime(2021, 1, 14, 7, 58, 12), datetime.datetime(2021, 1, 14, 8, 3, 12)]
[INFO: 2021-01-14 16:38:35 : pytroll_collectors.region_collector] Planned timeout for Euro 3.0km area - Europe: 2021-01-14T11:08:12
[INFO: 2021-01-14 16:38:35 : pytroll_collectors.region_collector] Adjusted timeout: 2021-01-14T11:03:12

At this point, gatherer.py quits. I'm not sure if it quits normally; it quits with exit code 0. The logging output from trollstalker in the meantime:

Logging output from trollstalker 
[DEBUG: 2021-01-14 16:37:47,858: trollstalker] trigger: IN_CLOSE_WRITE
[DEBUG: 2021-01-14 16:37:47,858: trollstalker] processing /data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080012Z_20210114080312Z_N_O_20210114080012Z
[DEBUG: 2021-01-14 16:37:47,858: trollstalker] filter: {path}AVHR_HRP_00_{platform_name}_{start_time:%Y%m%d%H%M%S}Z_{end_time:%Y%m%d%H%M%S}Z_N_O_{processing_time:%Y%m%d%H%M%S}Z         event: /data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080012Z_20210114080312Z_N_O_20210114080012Z
[DEBUG: 2021-01-14 16:37:47,858: trollstalker] No origin_inotify_base_dir_skip_levels in self.custom_vars
[DEBUG: 2021-01-14 16:37:47,859: trollstalker] Extracted: OrderedDict([('path', ''), ('platform_name', 'M02'), ('start_time', datetime.datetime(2021, 1, 14, 8, 0, 12)), ('end_time', datetime.datetime(2021, 1, 14, 8, 3, 12)), ('processing_time', datetime.datetime(2021, 1, 14, 8, 0, 12))])
[DEBUG: 2021-01-14 16:37:47,859: trollstalker] self.info['sensor']: ['avhrr/3']
[INFO: 2021-01-14 16:37:47,860: trollstalker] Publishing message pytroll://file/poes/avhrr file pytroll@oflks333.dwd.de 2021-01-14T16:37:47.859793 v1.01 application/json {"path": "", "platform_name": "Metop-A", "start_time": "2021-01-14T08:00:12", "end_t
ime": "2021-01-14T08:03:12", "processing_time": "2021-01-14T08:00:12", "uri": "/data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080012Z_20210114080312Z_N_O_20210114080012Z", "uid": "AVHR_HRP_00_M02_20210114080012Z_20210114080312Z_N_O_20210114080012Z", "sensor": ["avhrr/3"], "orig_platform_name": "M02"}
[DEBUG: 2021-01-14 16:37:47,943: trollstalker] trigger: IN_CLOSE_WRITE
[DEBUG: 2021-01-14 16:37:47,943: trollstalker] processing /data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080312Z_20210114080612Z_N_O_20210114080312Z
[DEBUG: 2021-01-14 16:37:47,943: trollstalker] filter: {path}AVHR_HRP_00_{platform_name}_{start_time:%Y%m%d%H%M%S}Z_{end_time:%Y%m%d%H%M%S}Z_N_O_{processing_time:%Y%m%d%H%M%S}Z         event: /data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080312Z_20210114080612Z_N_O_20210114080312Z
[DEBUG: 2021-01-14 16:37:47,944: trollstalker] No origin_inotify_base_dir_skip_levels in self.custom_vars
[DEBUG: 2021-01-14 16:37:47,944: trollstalker] Extracted: OrderedDict([('path', ''), ('platform_name', 'M02'), ('start_time', datetime.datetime(2021, 1, 14, 8, 3, 12)), ('end_time', datetime.datetime(2021, 1, 14, 8, 6, 12)), ('processing_time', datetime.datetime(2021, 1, 14, 8, 3, 12))])
[DEBUG: 2021-01-14 16:37:47,944: trollstalker] self.info['sensor']: ['avhrr/3']
[INFO: 2021-01-14 16:37:47,945: trollstalker] Publishing message pytroll://file/poes/avhrr file pytroll@oflks333.dwd.de 2021-01-14T16:37:47.945067 v1.01 application/json {"path": "", "platform_name": "Metop-A", "start_time": "2021-01-14T08:03:12", "end_time": "2021-01-14T08:06:12", "processing_time": "2021-01-14T08:03:12", "uri": "/data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080312Z_20210114080612Z_N_O_20210114080312Z", "uid": "AVHR_HRP_00_M02_20210114080312Z_20210114080612Z_N_O_20210114080312Z", "sensor": ["avhrr/3"], "orig_platform_name": "M02"}
[DEBUG: 2021-01-14 16:37:47,965: trollstalker] trigger: IN_CLOSE_WRITE
[DEBUG: 2021-01-14 16:37:47,965: trollstalker] processing /data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080612Z_20210114080641Z_N_O_20210114080612Z
[DEBUG: 2021-01-14 16:37:47,965: trollstalker] filter: {path}AVHR_HRP_00_{platform_name}_{start_time:%Y%m%d%H%M%S}Z_{end_time:%Y%m%d%H%M%S}Z_N_O_{processing_time:%Y%m%d%H%M%S}Z         event: /data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080612Z_20210114080641Z_N_O_20210114080612Z
[DEBUG: 2021-01-14 16:37:47,965: trollstalker] No origin_inotify_base_dir_skip_levels in self.custom_vars
[DEBUG: 2021-01-14 16:37:47,966: trollstalker] Extracted: OrderedDict([('path', ''), ('platform_name', 'M02'), ('start_time', datetime.datetime(2021, 1, 14, 8, 6, 12)), ('end_time', datetime.datetime(2021, 1, 14, 8, 6, 41)), ('processing_time', datetime.datetime(2021, 1, 14, 8, 6, 12))])
[DEBUG: 2021-01-14 16:37:47,966: trollstalker] self.info['sensor']: ['avhrr/3']
[INFO: 2021-01-14 16:37:47,966: trollstalker] Publishing message pytroll://file/poes/avhrr file pytroll@oflks333.dwd.de 2021-01-14T16:37:47.966510 v1.01 application/json {"path": "", "platform_name": "Metop-A", "start_time": "2021-01-14T08:06:12", "end_time": "2021-01-14T08:06:41", "processing_time": "2021-01-14T08:06:12", "uri": "/data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080612Z_20210114080641Z_N_O_20210114080612Z", "uid": "AVHR_HRP_00_M02_20210114080612Z_20210114080641Z_N_O_20210114080612Z", "sensor": ["avhrr/3"], "orig_platform_name": "M02"}

Additional diagnosis

When I add to gatherer.ini a line to the [default] section publish_topic = /collection/metopa/avhrr (duplicating the line under [metopa-avhrr-hrpt], I don't get this exception, and it does not appear to quit at all.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@gerritholl