[Bug]: can't import mlrun because of missing dependency

[fschlz]

MLRun Version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of MLRun CE.

Reproducible Example

# initialize pyproject.toml
>>> poetry init

# add dependencies in pyproject.toml
>>> code pyproject.toml

[tool.poetry]
name = "learn-mlrun"
version = "0.1.0"
description = ""

[tool.poetry.dependencies]
python = ">=3.10,<3.12"
mlrun = "^1.6.1"
pyyaml = { version = "!=6.0.0,!=5.4.0,!=5.4.1" }

[tool.poetry.group.dev.dependencies]
jupyter = "^1.0.0"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

# start notebook to test mlrun
>>> code test.ipynb
import mlrun # throws error

Issue Description

Importing mlrun in a notebook after installing it throws the following error:

Trying to install protobuf via poetry leads to issues with dependency resolution:

    Because no versions of mlrun match >1.6.1,<1.6.2rc1 || >1.6.2rc1,<1.6.2rc2 || >1.6.2rc2,<1.6.2rc3 || >1.6.2rc3,<1.7.0rc1 || >1.7.0rc1,<1.7.0rc2 || >1.7.0rc2,<1.7.0rc3 || >1.7.0rc3,<1.7.0rc4 || >1.7.0rc4,<1.7.0rc5 || >1.7.0rc5,<2.0.0
 and mlrun (1.6.2rc1) depends on kfp (>=1.8,<2.0), mlrun (>1.6.1,<1.6.2rc2 || >1.6.2rc2,<1.6.2rc3 || >1.6.2rc3,<1.7.0rc1 || >1.7.0rc1,<1.7.0rc2 || >1.7.0rc2,<1.7.0rc3 || >1.7.0rc3,<1.7.0rc4 || >1.7.0rc4,<1.7.0rc5 || >1.7.0rc5,<2.0.0) requires kfp (>=1.8,<2.0).
    And because mlrun (1.6.2rc2) depends on kfp (>=1.8,<2.0)
 and mlrun (1.6.2rc3) depends on kfp (>=1.8,<2.0), mlrun (>1.6.1,<1.7.0rc1 || >1.7.0rc1,<1.7.0rc2 || >1.7.0rc2,<1.7.0rc3 || >1.7.0rc3,<1.7.0rc4 || >1.7.0rc4,<1.7.0rc5 || >1.7.0rc5,<2.0.0) requires kfp (>=1.8,<2.0).
    And because mlrun (1.7.0rc1) depends on kfp (>=1.8,<2.0)
 and mlrun (1.7.0rc2) depends on kfp (>=1.8,<2.0), mlrun (>1.6.1,<1.7.0rc3 || >1.7.0rc3,<1.7.0rc4 || >1.7.0rc4,<1.7.0rc5 || >1.7.0rc5,<2.0.0) requires kfp (>=1.8,<2.0).
    And because mlrun (1.7.0rc3) depends on kfp (>=1.8,<2.0)
 and mlrun (1.7.0rc4) depends on kfp (>=1.8,<2.0), mlrun (>1.6.1,<1.7.0rc5 || >1.7.0rc5,<2.0.0) requires kfp (>=1.8,<2.0).
(1) So, because mlrun (1.7.0rc5) depends on kfp (>=1.8,<2.0)
 and mlrun (1.6.1) depends on kfp (>=1.8,<2.0), mlrun (>=1.6.1,<2.0.0) requires kfp (>=1.8,<2.0).

    Because no versions of kfp match >1.8,<1.8.1 || >1.8.1,<1.8.2 || >1.8.2,<1.8.3 || >1.8.3,<1.8.4 || >1.8.4,<1.8.5 || >1.8.5,<1.8.6 || >1.8.6,<1.8.7 || >1.8.7,<1.8.8 || >1.8.8,<1.8.9 || >1.8.9,<1.8.10 || >1.8.10,<1.8.11 || >1.8.11,<1.8.12 || >1.8.12,<1.8.13 || >1.8.13,<1.8.14 || >1.8.14,<1.8.15 || >1.8.15,<1.8.16 || >1.8.16,<1.8.17 || >1.8.17,<1.8.18 || >1.8.18,<1.8.19 || >1.8.19,<1.8.20 || >1.8.20,<1.8.21 || >1.8.21,<1.8.22 || >1.8.22,<2.0
 and kfp (1.8.0) depends on protobuf (>=3.13.0,<4), kfp (>=1.8,<1.8.1 || >1.8.1,<1.8.2 || >1.8.2,<1.8.3 || >1.8.3,<1.8.4 || >1.8.4,<1.8.5 || >1.8.5,<1.8.6 || >1.8.6,<1.8.7 || >1.8.7,<1.8.8 || >1.8.8,<1.8.9 || >1.8.9,<1.8.10 || >1.8.10,<1.8.11 || >1.8.11,<1.8.12 || >1.8.12,<1.8.13 || >1.8.13,<1.8.14 || >1.8.14,<1.8.15 || >1.8.15,<1.8.16 || >1.8.16,<1.8.17 || >1.8.17,<1.8.18 || >1.8.18,<1.8.19 || >1.8.19,<1.8.20 || >1.8.20,<1.8.21 || >1.8.21,<1.8.22 || >1.8.22,<2.0) requires protobuf (>=3.13.0,<4).
    And because kfp (1.8.1) depends on protobuf (>=3.13.0,<4)
 and kfp (1.8.2) depends on protobuf (>=3.13.0,<4), kfp (>=1.8,<1.8.3 || >1.8.3,<1.8.4 || >1.8.4,<1.8.5 || >1.8.5,<1.8.6 || >1.8.6,<1.8.7 || >1.8.7,<1.8.8 || >1.8.8,<1.8.9 || >1.8.9,<1.8.10 || >1.8.10,<1.8.11 || >1.8.11,<1.8.12 || >1.8.12,<1.8.13 || >1.8.13,<1.8.14 || >1.8.14,<1.8.15 || >1.8.15,<1.8.16 || >1.8.16,<1.8.17 || >1.8.17,<1.8.18 || >1.8.18,<1.8.19 || >1.8.19,<1.8.20 || >1.8.20,<1.8.21 || >1.8.21,<1.8.22 || >1.8.22,<2.0) requires protobuf (>=3.13.0,<4).
    And because kfp (1.8.3) depends on protobuf (>=3.13.0,<4)
 and kfp (1.8.4) depends on protobuf (>=3.13.0,<4), kfp (>=1.8,<1.8.5 || >1.8.5,<1.8.6 || >1.8.6,<1.8.7 || >1.8.7,<1.8.8 || >1.8.8,<1.8.9 || >1.8.9,<1.8.10 || >1.8.10,<1.8.11 || >1.8.11,<1.8.12 || >1.8.12,<1.8.13 || >1.8.13,<1.8.14 || >1.8.14,<1.8.15 || >1.8.15,<1.8.16 || >1.8.16,<1.8.17 || >1.8.17,<1.8.18 || >1.8.18,<1.8.19 || >1.8.19,<1.8.20 || >1.8.20,<1.8.21 || >1.8.21,<1.8.22 || >1.8.22,<2.0) requires protobuf (>=3.13.0,<4).
    And because kfp (1.8.5) depends on protobuf (>=3.13.0,<4)
 and kfp (1.8.6) depends on protobuf (>=3.13.0,<4), kfp (>=1.8,<1.8.7 || >1.8.7,<1.8.8 || >1.8.8,<1.8.9 || >1.8.9,<1.8.10 || >1.8.10,<1.8.11 || >1.8.11,<1.8.12 || >1.8.12,<1.8.13 || >1.8.13,<1.8.14 || >1.8.14,<1.8.15 || >1.8.15,<1.8.16 || >1.8.16,<1.8.17 || >1.8.17,<1.8.18 || >1.8.18,<1.8.19 || >1.8.19,<1.8.20 || >1.8.20,<1.8.21 || >1.8.21,<1.8.22 || >1.8.22,<2.0) requires protobuf (>=3.13.0,<4).
    And because kfp (1.8.7) depends on protobuf (>=3.13.0,<4)
 and kfp (1.8.8) depends on protobuf (>=3.13.0,<4), kfp (>=1.8,<1.8.9 || >1.8.9,<1.8.10 || >1.8.10,<1.8.11 || >1.8.11,<1.8.12 || >1.8.12,<1.8.13 || >1.8.13,<1.8.14 || >1.8.14,<1.8.15 || >1.8.15,<1.8.16 || >1.8.16,<1.8.17 || >1.8.17,<1.8.18 || >1.8.18,<1.8.19 || >1.8.19,<1.8.20 || >1.8.20,<1.8.21 || >1.8.21,<1.8.22 || >1.8.22,<2.0) requires protobuf (>=3.13.0,<4).
    And because kfp (1.8.9) depends on protobuf (>=3.13.0,<4)
 and kfp (1.8.10) depends on protobuf (>=3.13.0,<4), kfp (>=1.8,<1.8.11 || >1.8.11,<1.8.12 || >1.8.12,<1.8.13 || >1.8.13,<1.8.14 || >1.8.14,<1.8.15 || >1.8.15,<1.8.16 || >1.8.16,<1.8.17 || >1.8.17,<1.8.18 || >1.8.18,<1.8.19 || >1.8.19,<1.8.20 || >1.8.20,<1.8.21 || >1.8.21,<1.8.22 || >1.8.22,<2.0) requires protobuf (>=3.13.0,<4).
    And because kfp (1.8.11) depends on protobuf (>=3.13.0,<4)
 and kfp (1.8.12) depends on protobuf (>=3.13.0,<4), kfp (>=1.8,<1.8.13 || >1.8.13,<1.8.14 || >1.8.14,<1.8.15 || >1.8.15,<1.8.16 || >1.8.16,<1.8.17 || >1.8.17,<1.8.18 || >1.8.18,<1.8.19 || >1.8.19,<1.8.20 || >1.8.20,<1.8.21 || >1.8.21,<1.8.22 || >1.8.22,<2.0) requires protobuf (>=3.13.0,<4).
    And because kfp (1.8.13) depends on protobuf (>=3.13.0,<4)
 and kfp (1.8.14) depends on protobuf (>=3.13.0,<4), kfp (>=1.8,<1.8.15 || >1.8.15,<1.8.16 || >1.8.16,<1.8.17 || >1.8.17,<1.8.18 || >1.8.18,<1.8.19 || >1.8.19,<1.8.20 || >1.8.20,<1.8.21 || >1.8.21,<1.8.22 || >1.8.22,<2.0) requires protobuf (>=3.13.0,<4).
    And because kfp (1.8.15) depends on protobuf (>=3.13.0,<4)
 and kfp (1.8.16) depends on protobuf (>=3.13.0,<4), kfp (>=1.8,<1.8.17 || >1.8.17,<1.8.18 || >1.8.18,<1.8.19 || >1.8.19,<1.8.20 || >1.8.20,<1.8.21 || >1.8.21,<1.8.22 || >1.8.22,<2.0) requires protobuf (>=3.13.0,<4).
    And because kfp (1.8.17) depends on protobuf (>=3.13.0,<4)
 and kfp (1.8.18) depends on protobuf (>=3.13.0,<4), kfp (>=1.8,<1.8.19 || >1.8.19,<1.8.20 || >1.8.20,<1.8.21 || >1.8.21,<1.8.22 || >1.8.22,<2.0) requires protobuf (>=3.13.0,<4).
    And because kfp (1.8.19) depends on protobuf (>=3.13.0,<4)
 and kfp (1.8.20) depends on protobuf (>=3.13.0,<4), kfp (>=1.8,<1.8.21 || >1.8.21,<1.8.22 || >1.8.22,<2.0) requires protobuf (>=3.13.0,<4).
    And because kfp (1.8.21) depends on protobuf (>=3.13.0,<4)
 and kfp (1.8.22) depends on protobuf (>=3.13.0,<4), kfp (>=1.8,<2.0) requires protobuf (>=3.13.0,<4).
    And because mlrun (>=1.6.1,<2.0.0) requires kfp (>=1.8,<2.0) (1), mlrun (>=1.6.1,<2.0.0) requires protobuf (>=3.13.0,<4)
    So, because learn-mlrun depends on both mlrun (^1.6.1) and protobuf (^5.26.0), version solving failed.

Expected Behavior

I would expect all dependencies to be declared and installed.

Installation OS

Mac

Installation Method

Docker

Python Version

3.10.13

MLRun Version

1.6.1

Additional Information

No response

[fschlz]

I also saw this article: https://docs.mlrun.org/en/latest/install/remote.html#set-up-your-environment
Tried using Python 3.9, same result.

The protobuf version that gets installed alongside mlrun is pretty old, though. (3.20.3)
Perhaps an upgrade to a newer version is necessary. (latest: 5.26.0)

[alonmr]

Hey @fschlz , the protobuf version we use is not supported on arm64 so we use a conda environment to emulate x86_64.
I'm assuming you have arm64 mac and that you are using python venv by the description of the issue. If that is the case, please use a conda env instead. Using conda on top of arm64 is required for now until all of our dependencies will work smoothly on both architectures with python venv. Currently we cannot upgrade protobuf due to the following issue - https://stackoverflow.com/questions/72441758/typeerror-descriptors-cannot-not-be-created-directly