rsr-data offline usage

[bernie1970]

Problem description

When I am creating a product (i.e. cloudtop_daytime) on a server with no internet connection, the rsr-data couldn't be downloaded/refreshed automatically. This rises an error message.

Expected Output

No error output

Actual Result, Traceback if applicable

rsr data may not be up to date: /home/schitter/.local/share/pyspectral/rsr_seviri_Meteosat-11.h5
Traceback (most recent call last):
File "/usr/lib64/python3.6/site-packages/urllib3/connection.py", line 157, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "/usr/lib64/python3.6/site-packages/urllib3/util/connection.py", line 61, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/lib64/python3.6/socket.py", line 745, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
chunked=chunked,
File "/usr/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 376, in _make_request
self._validate_conn(conn)
File "/usr/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
conn.connect()
File "/usr/lib64/python3.6/site-packages/urllib3/connection.py", line 300, in connect
conn = self._new_conn()
File "/usr/lib64/python3.6/site-packages/urllib3/connection.py", line 169, in _new_conn
self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7f67d69eed30>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib64/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/usr/lib64/python3.6/site-packages/urllib3/util/retry.py", line 436, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='zenodo.org', port=443): Max retries exceeded with url: /record/3461164/files/pyspectral_rsr_data.tgz (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f67d69eed30>: Failed to establish a new connection: [Errno -2] Name or service not known',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/pytroll/scripts/create_msg.py", line 176, in
euro_scene.load([product])
File "/usr/lib/python3.6/site-packages/satpy/scene.py", line 978, in load
keepables = self.generate_composites()
File "/usr/lib/python3.6/site-packages/satpy/scene.py", line 893, in generate_composites
return self._read_composites(nodes)
File "/usr/lib/python3.6/site-packages/satpy/scene.py", line 868, in _read_composites
self._generate_composite(item, keepables)
File "/usr/lib/python3.6/site-packages/satpy/scene.py", line 810, in _generate_composite
keepables,
File "/usr/lib/python3.6/site-packages/satpy/scene.py", line 757, in _get_prereq_datasets
self._generate_composite(prereq_node, keepables)
File "/usr/lib/python3.6/site-packages/satpy/scene.py", line 843, in _generate_composite
**self.attrs)
File "/usr/lib/python3.6/site-packages/satpy/composites/init.py", line 653, in call
self._init_refl3x(projectables)
File "/usr/lib/python3.6/site-packages/satpy/composites/init.py", line 614, in _init_refl3x
self._refl3x = Calculator(_nir.attrs['platform_name'], _nir.attrs['sensor'], _nir.attrs['name'])
File "/usr/lib/python3.6/site-packages/pyspectral/near_infrared_reflectance.py", line 70, in init
super(Calculator, self).init(platform_name, instrument, band, **kwargs)
File "/usr/lib/python3.6/site-packages/pyspectral/radiance_tb_conversion.py", line 134, in init
self._get_rsr()
File "/usr/lib/python3.6/site-packages/pyspectral/radiance_tb_conversion.py", line 143, in _get_rsr
sensor = RelativeSpectralResponse(self.platform_name, self.instrument)
File "/usr/lib/python3.6/site-packages/pyspectral/rsr_reader.py", line 109, in init
self._get_filename()
File "/usr/lib/python3.6/site-packages/pyspectral/rsr_reader.py", line 161, in _get_filename
download_rsr()
File "/usr/lib/python3.6/site-packages/pyspectral/utils.py", line 400, in download_rsr
response = requests.get(HTTP_PYSPECTRAL_RSR)
File "/usr/lib64/python3.6/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/usr/lib64/python3.6/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib64/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib64/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/lib64/python3.6/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='zenodo.org', port=443): Max retries exceeded with url: /record/3461164/files/pyspectral_rsr_data.tgz (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f67d69eed30>: Failed to establish a new connection: [Errno -2] Name or service not known',))

Versions of Python, package at hand and relevant dependencies

python 3.6, satpy 0.20, pyspectral 0.9.5

Additional info

In /usr/lib/site-packages/pyspectral/etc/pysprectral.yaml the option for disabling the download from internet. When I set this to "False" the product is created without "actual" rsr-data.

After an update of pyspectral this setting would the file overwritten? So I have to remember, that I have changed this file and restore it after each update.

A workaround from Adam is the following:
Using a customized pyspectral.yaml file and pointing to it with the environment variable PSP_CONFIG_FILE