Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v2 Pipeline fails looking for MinIO secret #115

Closed
2 tasks done
aaronwjha opened this issue Mar 18, 2024 · 2 comments · Fixed by #122
Closed
2 tasks done

v2 Pipeline fails looking for MinIO secret #115

aaronwjha opened this issue Mar 18, 2024 · 2 comments · Fixed by #122
Labels
kind/bug kind - things not working properly
Milestone

Comments

@aaronwjha
Copy link

Checks

  • I have searched the existing issues.
  • This issue is NOT specific to the CLI. (If so, please open an issue on the CLI repo)

deployKF Version

0.1.4

Kubernetes Version

Client Version: v1.26.1
Server Version: v1.26.13-gke.1052000

Description

I have built a simple 'hello-world' pipeline to test my recently deployed KF instance. I have disabled minio and configured object storage in GCS. The Pipeline fails trying to create artifacts with an error related to MinIO (see log entry below).

Pipeline Source:

from kfp import dsl
from kfp import compiler

@dsl.component
def say_hello(name: str) -> str:
    hello_text = f'Hello, {name}!'
    print(hello_text)
    return hello_text

@dsl.pipeline
def hello_pipeline(recipient: str) -> str:
    hello_task = say_hello(name=recipient)
    return hello_task.output

Pipeline Definition:

# PIPELINE DEFINITION
# Name: hello-pipeline
# Inputs:
#    recipient: str
# Outputs:
#    Output: str
components:
  comp-say-hello:
    executorLabel: exec-say-hello
    inputDefinitions:
      parameters:
        name:
          parameterType: STRING
    outputDefinitions:
      parameters:
        Output:
          parameterType: STRING
deploymentSpec:
  executors:
    exec-say-hello:
      container:
        args:
        - --executor_input
        - '{{$}}'
        - --function_to_execute
        - say_hello
        command:
        - sh
        - -c
        - "\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip ||\
          \ python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1\
          \ python3 -m pip install --quiet --no-warn-script-location 'kfp==2.7.0'\
          \ '--no-deps' 'typing-extensions>=3.7.4,<5; python_version<\"3.9\"' && \"\
          $0\" \"$@\"\n"
        - sh
        - -ec
        - 'program_path=$(mktemp -d)


          printf "%s" "$0" > "$program_path/ephemeral_component.py"

          _KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         "$program_path/ephemeral_component.py"                         "$@"

          '
        - "\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import\
          \ *\n\ndef say_hello(name: str) -> str:\n    hello_text = f'Hello, {name}!'\n\
          \    print(hello_text)\n    return hello_text\n\n"
        image: python:3.7
pipelineInfo:
  name: hello-pipeline
root:
  dag:
    outputs:
      parameters:
        Output:
          valueFromParameter:
            outputParameterKey: Output
            producerSubtask: say-hello
    tasks:
      say-hello:
        cachingOptions:
          enableCache: true
        componentRef:
          name: comp-say-hello
        inputs:
          parameters:
            name:
              componentInputParameter: recipient
        taskInfo:
          name: say-hello
  inputDefinitions:
    parameters:
      recipient:
        parameterType: STRING
  outputDefinitions:
    parameters:
      Output:
        parameterType: STRING
schemaVersion: 2.1.0
sdkVersion: kfp-2.7.0

Relevant Logs

main.go:49] failed to execute component: Failed to open bucket "kubeflow-object-store": Failed to get minio credential: Failed to get MinIO credential from secret name="ml pipeline-minio-artifact" namespace="ea-ml": secrets "mlpipeline-minio-artifact" not found

deployKF Values (Optional)

kubeflow_tools:
  pipelines:
    bucket:
      name: *****
      region: ****
    objectStore:
      useExternal: true
      host: storage.googleapis.com
      port: ""
      useSSL: true
      auth:
        fromEnv: false
        existingSecret: "kubeflow-object-store"
        existingSecretAccessKeyKey: "GCS_ACCESS_KEY_ID"
        existingSecretSecretKeyKey: "GCS_SECRET_ACCESS_KEY"
    kfpV2:
      minioFix: false
deploykf_opt:
  deploykf_minio:
    enabled: false
@aaronwjha aaronwjha added kind/bug kind - things not working properly priority/needs-triage priority - needs to be triaged labels Mar 18, 2024
@thesuperzapper
Copy link
Member

@aaronwjha deployKF versions 0.1.X include Kubeflow Pipelines v1 (not v2).

The latest version of the kfp SDK you can use is 1.8.22, you can install it with:

pip install kfp==1.8.22

We will be releasing a 0.2 line of deployKF which supports Kubeflow Pipelines V2, but this is still a little ways off.


I am honestly very confused as to why Kubeflow Pipelines V1 is even accepting the V2 pipeline YAML, because the spec is not compatible.

How are you submitting this pipeline to KFP?
Can you show me the code?

@thesuperzapper thesuperzapper added this to the v0.2.0 milestone Mar 20, 2024
@thesuperzapper thesuperzapper removed the priority/needs-triage priority - needs to be triaged label Mar 20, 2024
@aaronwjha
Copy link
Author

This explains a lot.

I have created the pipeline by uploading the yaml in the GUI. Strange that I am not getting an error.

Pipeline yaml:

schemaVersion: 2.1.0
sdkVersion: kfp-2.7.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug kind - things not working properly
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants