dataset.download() Unsupported Linux distribution

[Radeju]

I am trying to download an AzureML dataset on Ubuntu 20.04. I am using azureml.core library. However, when I try to run it I get following error

  File "/home/bartek/.local/lib/python3.8/site-packages/dotnetcore2/runtime.py", line 169, in attemp_get_deps
    blob_deps_to_file()
  File "/home/bartek/.local/lib/python3.8/site-packages/dotnetcore2/runtime.py", line 161, in blob_deps_to_file
    blob = request.urlopen(deps_url, context=ssl_context)
  File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "setup/get_datasets.py", line 27, in <module>
    dataset.download(target_path=f'{path}/../.datasets/{dataset_name}', overwrite=True)
  File "/home/bartek/.local/lib/python3.8/site-packages/azureml/data/_loggerfactory.py", line 106, in wrapper
    return func(*args, **kwargs)
  File "/home/bartek/.local/lib/python3.8/site-packages/azureml/data/file_dataset.py", line 123, in download
    for p in self._to_path(activity='download.to_path')]
  File "/home/bartek/.local/lib/python3.8/site-packages/azureml/data/file_dataset.py", line 98, in _to_path
    dataflow, portable_path = _add_portable_path_column(self._dataflow)
  File "/home/bartek/.local/lib/python3.8/site-packages/azureml/data/_loggerfactory.py", line 106, in wrapper
    return func(*args, **kwargs)
  File "/home/bartek/.local/lib/python3.8/site-packages/azureml/data/abstract_dataset.py", line 203, in _dataflow
    dataprep().api._datastore_helper._set_auth_type(self._registration.workspace)
  File "/home/bartek/.local/lib/python3.8/site-packages/azureml/dataprep/api/_datastore_helper.py", line 136, in _set_auth_type
    get_engine_api().set_aml_auth(SetAmlAuthMessageArgument(AuthType.DERIVED, json.dumps(auth)))
  File "/home/bartek/.local/lib/python3.8/site-packages/azureml/dataprep/api/engineapi/api.py", line 18, in get_engine_api
    _engine_api = EngineAPI()
  File "/home/bartek/.local/lib/python3.8/site-packages/azureml/dataprep/api/engineapi/api.py", line 55, in __init__
    self._message_channel = launch_engine()
  File "/home/bartek/.local/lib/python3.8/site-packages/azureml/dataprep/api/engineapi/engine.py", line 300, in launch_engine
    dependencies_path = runtime.ensure_dependencies()
  File "/home/bartek/.local/lib/python3.8/site-packages/dotnetcore2/runtime.py", line 181, in ensure_dependencies
    if not attemp_get_deps():
  File "/home/bartek/.local/lib/python3.8/site-packages/dotnetcore2/runtime.py", line 175, in attemp_get_deps
    raise NotImplementedError('Unsupported Linux distribution {0} {1}.{2}'.format(dist, version[0], version[1]))
NotImplementedError: Unsupported Linux distribution ubuntu 20.04
The terminal process terminated with exit code: 1

Are you planning to support 20.04 version of Ubuntu? Is there any roadmap? I found this issue from 6 months ago and would really appreciate to hear if anything had changed since then.

Right now I am using the workaround from here to make it work.

Warm regards

[YutongTie-MSFT]

@Radeju
Thanks for the feedback! We are currently investigating and will update you shortly.

[MayMSFT]

thanks for the feedback. We have recorded the feedback and added it as a feature request on our roadmap.

[YutongTie-MSFT]

@MayMSFT Hi May, are we good to close this or you want me to keep it open? Thanks.

[xkszltl]

Hi! I'm planning to switch our pipeline from 18.04 to 20.04 soon as well.
Looks like this may be a blocking issue.
Do we have timeline regarding the fix?

Based on the log seems distro version is asserted by a whitelist. IMHO this is a bad design which can probably affect a lot of not-so-popular distros like arch or mint.

[MayMSFT]

Unfortunately, it depends on legal approval. @tot0 to share more details

[lupickup]

@xkszltl Hi, I unfortunately don't have any concrete timeline for official support of new linux distros. The legal processes involved distributing open source packages so that normally Datasets 'just works' require care and aren't moving as fast as we'd hope.

Datasets will only return saying 'Unsupported Distro' if the required dependencies for .NET Core 2.1 are not present on default library paths AND a pre-prepared dependency set doesn't exist.
We are working on improving the error message to link out to the official .NET Core documentation on how to install the correct dependencies for supported distributions.

@xkszltl Would you be able to try the first command here to install .NET Cores dependencies for Ubuntu 20.04 and see if you're able to use dataset.download()?
https://docs.microsoft.com/en-us/dotnet/core/install/linux-ubuntu#2004-

[xkszltl]

Of course, if it is just a matter of installing .NET it's totally fine for us.
Actually we will do that regardless of the use of AML Datasets.

Is 2.1 a exact or minimum requirement?
Can we use later version? Namely 2.2 or 3+