Issue building docker image using example build config

[minaadel]

Describe the bug

Attempting to deploy the docker image build yaml file available under https://github.com/polyaxon/polyaxon-examples/blob/master/in_cluster/build_image/build-ml.yaml using the polyaxon cli returns the following error:

>polyaxon run -f build_docker_image.yaml -l
Polyaxonfile is not valid.
Error message: The Polyaxonfile's version specified is not supported by your current CLI.Your CLI support Polyaxonfile versions between: 1.1 <= v <= 1.1.You can run `polyaxon upgrade` and check documentation for the specification..

To reproduce

The contents of build_docker_image.yaml

version: 1.1
kind: operation
name: build
params:
  destination:
    connection: localreg
    value: polyaxon-examples:ml
runPatch:
  init:
  - dockerfile:
      image: python:3.8.8-buster
      run:
      - 'pip3 install --no-cache-dir -U polyaxon["polyboard"]'
      - pip3 install scikit-learn xgboost matplotlib vega-datasets joblib lightgbm xgboost
      langEnv: 'en_US.UTF-8'
hubRef: kaniko

Expected behavior

A build operation should begin that results in an image being saved to the docker registry connected under destination: connection:

Environment

Polyaxon 1.8.0
Polyaxon CLI 1.8.0

[mmourafiq]

Can you check if your environment has by any chance the legacy CLI also installed:

pip freeze | grep polyaxon

If it's the case, you need to uninstall the legacy libraries and only keep the new python library.

[minaadel]

These are the only two polyaxon libraries installed:
polyaxon==1.8.0
polyaxon-sdk==1.8.0

[mmourafiq]

Libraries installed are correct, I quickly tried and was not able to reproduce the file was submitted correctly.
We will check it on Monday.

[minaadel]

I can confirm that the same error appears using a freshly installed polyaxon 1.8.0 on the windows subsystem for linux (ubuntu) with the same file.

[minaadel]

@mouradmourafiq any updates? :)

[mmourafiq]

@minaadel sorry, completely forgot about this :)

I think it might be some formatting issues due to windows? Can you run polyaxon check -f FILENAME.yaml

Also you might copy the content to an editor to check if you are using spaces or tabs, I think yaml has issues with tabs.

[minaadel]

No issue, it happens :)

Here's the output of the check command.

>polyaxon check -f build_docker_image.yaml
Polyaxonfile is not valid.
Error message: The Polyaxonfile's version specified is not supported by your current CLI.Your CLI support Polyaxonfile versions between: 1.1 <= v <= 1.1.You can run `polyaxon upgrade` and check documentation for the specification..

The actual file itself uses 2 spaces per indentation, no tabs

[mmourafiq]

Very strange, and I could not reproduce. Did you try other files in the same examples folder https://github.com/polyaxon/polyaxon-examples/tree/master/in_cluster/build_image ?
Not sure if quoting all values would work on your system:

version: 1.1
kind: operation
name: build
params:
  destination:
    connection: localreg
    value: "polyaxon-examples:ml"
runPatch:
  init:
  - dockerfile:
      image: "python:3.8.8-buster"
      run:
      - 'pip3 install --no-cache-dir -U polyaxon["polyboard"]'
      - 'pip3 install scikit-learn xgboost matplotlib vega-datasets joblib lightgbm xgboost'
      langEnv: 'en_US.UTF-8'
hubRef: kaniko

Finally you may try with polyaxon -v ... to trigger some additional debug information.

[minaadel]

You're absolutely correct!
Simply quoting all the values did indeed force a new build that was eventually pushed to my connection :)
Thank you very much!

One last question though (I can open a new issue with the question if you'd like), how am I supposed to pull that same image from the localreg connection from within a polyaxonfile with kind: component (the one I use to run the usual experiments) ?

[mmourafiq]

I am not sure how you declared the registry and what type of registry you are using, but you will need something like image: url/image:tag, for instance if you are trying the in-cluster registry (https://polyaxon.com/integrations/in-cluster-registry/) the image should be:

run:
  container:
    image: "172.0.0.1:31320/name:latest"

If you are using GCR, the url will be the full GCR url with the image name and tag.

[minaadel]

I use the microk8s built in docker registry enabled via
microk8s enable registry
this I then connected with polyaxon by using the following snippet in my polyaxon_config file that I pass to helm.

connections:
  - name: localreg
    kind: registry
    schema: 
      url: 10.152.183.135:5000

The entire reason I'm trying to use a named connection is to abstract from having to use the particular ip of the registry when submitting a job to the polyaxon instance.
It's why I came to use the following snippet for the build operation

params:
  destination:
    connection: localreg
    value: "docker_img"

and wanted to do something similar with the component operation (experiment run).

[mmourafiq]

This is planned but not implemented yet, see #1049, where the user can declare a registry-connection:read and registry-connection:write and the system would know if it needs to inject the required info for read or write.

There's also a discussion going on in slack for providing an automated build process(https://polyaxon.slack.com/archives/C6QBND2SV/p1618316341162200) but we are still gathering interest and thinking about the best approach to provide such feature.

[minaadel]

So if I understand you correctly, there is currently no way to use a polyaxon registry connection to pull an image from that registry. It is currently only possible just to build an image and push it to said registry via the connection. Any image pulling needs to be done using the IP of the registry. Have I understood that properly?

[minaadel]

I guess a kinda hacky way to achieve my particular use case is to define a fixed url in /var/snap/microk8s/current/args/containerd-template.toml
as

[plugins.cri.registry.mirrors."registry:5000"]
          endpoint = ["http://10.152.183.135:5000"]

which can be reached in a unified manner from within a polyaxon run as:

container:
    image: "registry:5000/image"

Thank you very much for your assistance :)