Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

oc:build on openshift use pods/log to retrieve logs from build #2844

Open
sbollwe opened this issue Mar 21, 2024 · 18 comments
Open

oc:build on openshift use pods/log to retrieve logs from build #2844

sbollwe opened this issue Mar 21, 2024 · 18 comments
Assignees
Labels
bug Something isn't working

Comments

@sbollwe
Copy link

sbollwe commented Mar 21, 2024

Hello support,

We running the openshift-maven-plugin against our openshift cluster to build images. The build its self works and pushes to artifactory without any errors.

But we got this message, which shows an error retrieving the logs for that build job.

[INFO] oc: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods ?? is forbidden: User \"system:serviceaccount:ns:user\" cannot get resource \"pods/log\" in API group \"\" in the namespace \"auswamt\"","reason":"Forbidden","details":{"name":"app-s2i-7-build","kind":"pods"},"code":403}

The message shows that the plugin need access to the resource of pods/log. If I test this with the oc command (oc logs -f build/app-s2i-9), I can retrieve the logs with a service account without access to pods/log.

The service account is defined with the following rules:

rules:
  - verbs:
      - create
      - delete
      - get
      - list
      - patch
      - update
      - watch
    apiGroups:
      - ''
      - build.openshift.io
    resources:
      - buildconfigs
  - verbs:
      - create
    apiGroups:
      - ''
      - build.openshift.io
    resources:
      - buildconfigs/instantiate
      - buildconfigs/instantiatebinary
  - verbs:
      - get
      - list
      - watch
    apiGroups:
      - ''
      - build.openshift.io
    resources:
      - builds
      - builds/log

In my opinion, the builds/log resource access should be sufficient to retrive the logs of oc:build. Can you adjust this, as it's not a good idea to give more rights than necessary.

openshift-maven-plugin: 1.16.1

Thanks for your work and best regards,

Steffen

@rohanKanojia rohanKanojia added enhancement New feature or request bug Something isn't working and removed enhancement New feature or request labels Apr 2, 2024
@arsenalzp
Copy link
Contributor

I would like to work on this bug.

@rohanKanojia
Copy link
Member

@arsenalzp : This issue requires knowledge of Kubernetes and a Red Hat OpenShift cluster.

Here is the code where we're trying to get logs of OpenShift Build Pod:

https://github.com/eclipse/jkube/blob/2ebeabe2ae6ebbcd52e0fd9323607892bcb831d4/jkube-kit/config/service/src/main/java/org/eclipse/jkube/kit/config/service/openshift/OpenshiftBuildService.java#L466-L478

@arsenalzp
Copy link
Contributor

arsenalzp commented Apr 9, 2024

Yes, I have enough knowledge of both solutions. Any way, if questions will arise I can ask this friendly community.

@arsenalzp
Copy link
Contributor

Could you please assign it to me? Is there any deadline for this issue?

@rohanKanojia
Copy link
Member

rohanKanojia commented Apr 9, 2024

Is there any deadline for this issue?

@arsenalzp : Not that I know of, the user is able to perform build. He is only concerned about the warning due to additional permissions jkube is requesting.

@arsenalzp
Copy link
Contributor

arsenalzp commented Apr 10, 2024

I've just tried to reproduce this issue on my sandbox, new service account was created, bound to the role below, related BuildConfig was also changed:

kind: Role
metadata:
  creationTimestamp: "2024-04-10T19:45:40Z"
  name: jkubebuilder
  namespace: arsenalzp-dev
  resourceVersion: "2102664605"
  uid: e0de6100-38e8-4f61-83fc-e6d07c09609c
rules:
- apiGroups:
  - build.openshift.io
  resources:
  - builds
  - buildconfigs
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - image.openshift.io
  resources:
  - imagestreams
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - build.openshift.io
  resources:
  - builds/details
  verbs:
  - update
$ oc get buildconfigs.build.openshift.io spring-boot-s2i -o yaml|grep service
  serviceAccount: jkubebuilder
$

Try to run mvn oc:build -Popenshift, here is full log of build:

[INFO] Scanning for projects...
[INFO] 
[INFO] ----------< org.eclipse.jkube.quickstarts.maven:spring-boot >-----------
[INFO] Building Eclipse JKube :: Quickstarts :: Maven :: Spring Boot Web 1.16.1
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- openshift-maven-plugin:1.16.1:build (default-cli) @ spring-boot ---
[INFO] oc: Using OpenShift build with strategy S2I
[INFO] oc: Building Docker image
[INFO] oc: Running generator spring-boot
[INFO] oc: spring-boot: Using Docker image quay.io/jkube/jkube-java:0.0.23 as base / builder
[INFO] oc: Spring Boot layered jar detected
[INFO] oc: Final output artifact file was not rebuilt since last build. HINT: try to compile and package your application prior to running the container image build task.
[INFO] oc: [spring-boot:1.16.1] spring-boot: Created docker source tar /home/username/eclipse-workspace/jkube/quickstarts/maven/spring-boot/target/docker/spring-boot/1.16.1/tmp/docker-build.tar
[INFO] oc: Updating BuildServiceConfig spring-boot-s2i for Source strategy
[INFO] oc: Adding to ImageStream spring-boot
[INFO] oc: Starting Build spring-boot-s2i
[INFO] oc: Waiting for build spring-boot-s2i-15 to complete...
[INFO] oc: time=2024-04-10T20:59:10Z level=info msg=Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled
[INFO] oc: I0410 20:59:10.387843       1 defaults.go:112] Defaulting to storage driver overlay with options [mountopt=metacopy=on].
[INFO] oc: Caching blobs under /var/cache/blobs.
[INFO] oc: Trying to pull quay.io/jkube/jkube-java:0.0.23...
[INFO] oc: Getting image source signatures
[INFO] oc: Copying blob sha256:24a3fb335248549803dfdfe5f4ea60fcc9e11d5708cc62bb1fb811f608fd094a
[INFO] oc: Copying blob sha256:94ad07648680df54b9e32724cc2febb7f0a01245dfef48e84fa39ce101b056ca
[INFO] oc: Copying config sha256:8659a9b5de083556380993b0bbb62cb4907fd85e3c49c6952a216b17e67bdbd9
[INFO] oc: Writing manifest to image destination
[INFO] oc: Generating dockerfile with builder image quay.io/jkube/jkube-java:0.0.23
[INFO] oc: Adding transient rw bind mount for /run/secrets/rhsm
[INFO] oc: STEP 1/9: FROM quay.io/jkube/jkube-java:0.0.23
[INFO] oc: STEP 2/9: LABEL io.openshift.s2i.destination=/tmp       io.openshift.build.image=quay.io/jkube/jkube-java:0.0.23       io.openshift.build.source-location=/tmp/build/inputs
[INFO] oc: STEP 3/9: ENV JAVA_MAIN_CLASS=org.springframework.boot.loader.JarLauncher     JAVA_OPTIONS=-Dspring.output.ansi.enabled=always     JAVA_APP_DIR=/deployments     OPENSHIFT_BUILD_NAME=spring-boot-s2i-15     OPENSHIFT_BUILD_NAMESPACE=usernamezp-dev
[INFO] oc: STEP 4/9: USER root
[INFO] oc: STEP 5/9: COPY upload/src /tmp/src
[INFO] oc: STEP 6/9: RUN chown -R 1000:0 /tmp/src
[INFO] oc: STEP 7/9: USER 1000
[INFO] oc: STEP 8/9: RUN /usr/local/s2i/assemble
[INFO] oc: INFO S2I source build with plain binaries detected
[INFO] oc: INFO S2I binary build from fabric8-maven-plugin detected
[INFO] oc: INFO Copying binaries from /tmp/src/deployments to /deployments ...
[INFO] oc: BOOT-INF/
[INFO] oc: BOOT-INF/classpath.idx
[INFO] oc: BOOT-INF/layers.idx
[INFO] oc: BOOT-INF/classes/
[INFO] oc: BOOT-INF/classes/application.properties
...
[INFO] oc: org/springframework/boot/loader/util/
[INFO] oc: org/springframework/boot/loader/util/SystemPropertyUtils.class
[INFO] oc: INFO Copying deployments from deployments to /deployments...
[INFO] oc: INFO Cleaning up source directory (/tmp/src)
[INFO] oc: STEP 9/9: CMD /usr/local/s2i/run
[INFO] oc: COMMIT temp.builder.openshift.io/usernamezp-dev/spring-boot-s2i-15:862c6894
[INFO] oc: Getting image source signatures
[INFO] oc: Copying blob sha256:064b40dd2dfb098da5b4d43876da5d481047121e1ed560129383b74d30b5fe99
[INFO] oc: Copying blob sha256:3dfb4347f3e4a1e56a0a43a90d3538b63f9924daeb6f1112df242d7729581f40
[INFO] oc: Copying blob sha256:f9b3fe0936554c9874103714a140a9ef563c0f36bd744df04b9e1bc27b2cad38
[INFO] oc: Copying config sha256:0ead6a6f87d8220535112b1d74174effef24a7716b727569a9790b01f9d88357
[INFO] oc: Writing manifest to image destination
[INFO] oc: --> 0ead6a6f87d8
[INFO] oc: Successfully tagged temp.builder.openshift.io/usernamezp-dev/spring-boot-s2i-15:862c6894
[INFO] oc: 0ead6a6f87d8220535112b1d74174effef24a7716b727569a9790b01f9d88357
[INFO] oc: 
[INFO] oc: Pushing image image-registry.openshift-image-registry.svc:5000/usernamezp-dev/spring-boot:1.16.1 ...
[INFO] oc: Getting image source signatures
[INFO] oc: Copying blob sha256:f9b3fe0936554c9874103714a140a9ef563c0f36bd744df04b9e1bc27b2cad38
[INFO] oc: Copying blob sha256:94ad07648680df54b9e32724cc2febb7f0a01245dfef48e84fa39ce101b056ca
[INFO] oc: Copying blob sha256:24a3fb335248549803dfdfe5f4ea60fcc9e11d5708cc62bb1fb811f608fd094a
[INFO] oc: Copying config sha256:0ead6a6f87d8220535112b1d74174effef24a7716b727569a9790b01f9d88357
[INFO] oc: Writing manifest to image destination
[INFO] oc: Successfully pushed image-registry.openshift-image-registry.svc:5000/usernamezp-dev/spring-boot@sha256:893ce0c9e3edf72d5bd3ed66b927ab45b26b9c878a27728429fcc7572dd87a56
[INFO] oc: Push successful
[INFO] oc: Build spring-boot-s2i-15 in status Complete
[INFO] oc: Found tag on ImageStream spring-boot tag: sha256:893ce0c9e3edf72d5bd3ed66b927ab45b26b9c878a27728429fcc7572dd87a56
[INFO] oc: ImageStream spring-boot written to /home/username/eclipse-workspace/jkube/quickstarts/maven/spring-boot/target/spring-boot-is.yml
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:00 min
[INFO] Finished at: 2024-04-10T23:59:40+03:00
[INFO] ------------------------------------------------------------------------

As you may see no errors are observed.
Correct me if I'm wrong.

@rohanKanojia
Copy link
Member

@arsenalzp : The problem @sbollwe is facing is due to using a ServiceAccount with restricted permissions, You would need to create a more strict Role and bind it to some ServiceAccount, then use that ServiceAccount.

@arsenalzp
Copy link
Contributor

arsenalzp commented Apr 11, 2024

@arsenalzp : The problem @sbollwe is facing is due to using a ServiceAccount with restricted permissions, You would need to create a more strict Role and bind it to some ServiceAccount, then use that ServiceAccount.

As you might see in my previous message I created the Role with the following permission:

rules:
- apiGroups:
  - build.openshift.io
  resources:
  - builds
  - buildconfigs
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - image.openshift.io
  resources:
  - imagestreams
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - build.openshift.io
  resources:
  - builds/details
  verbs:
  - update

So it doesn't allow access to pod/ resources.
This role was bound to serviceAccount jkubebuilder, this serviceAccount is used in appropriate Build:

$ oc get buildconfigs.build.openshift.io spring-boot-s2i -o yaml|grep service
  serviceAccount: jkubebuilder
$

Check whether an action is allowed:

$ oc auth can-i get pod/logs --as system:serviceaccount:arsenalzp-dev:jkubebuilder-n arsenalzp-dev
no
$ oc auth can-i get build/logs --as system:serviceaccount:arsenalzp-dev:jkubebuilder -n arsenalzp-dev
yes
$

@rohanKanojia
Copy link
Member

@arsenalzp Could you please share your Role, RoleBinding and ServiceAccount YAML snippets that you're trying? The code snippet I had shared earlier is watching pod logs, it's quite strange that you're not getting the warning given that your ServiceAccount doesn't have permission.

@arsenalzp
Copy link
Contributor

arsenalzp commented Apr 11, 2024

@arsenalzp Could you please share your Role, RoleBinding and ServiceAccount YAML snippets that you're trying? The code snippet I had shared earlier is watching pod logs, it's quite strange that you're not getting the warning given that your ServiceAccount doesn't have permission.

Yes, sure:

Role

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  creationTimestamp: "2024-04-10T19:45:40Z"
  name: jkubebuilder
  namespace: arsenalzp-dev
  resourceVersion: "2102664605"
  uid: e0de6100-38e8-4f61-83fc-e6d07c09609c
rules:
- apiGroups:
  - build.openshift.io
  resources:
  - builds
  - buildconfigs
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - image.openshift.io
  resources:
  - imagestreams
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - build.openshift.io
  resources:
  - builds/details
  verbs:
  - update

RoleBinding

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  creationTimestamp: "2024-04-10T18:52:00Z"
  name: jkubebuilder
  namespace: arsenalzp-dev
  resourceVersion: "2102361421"
  uid: 7af5bc1f-883b-47a4-a248-54e09ea52b7b
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: jkubebuilder
subjects:
- kind: ServiceAccount
  name: jkubebuilder
  namespace: arsenalzp-dev

ServiceAccount:

apiVersion: v1
imagePullSecrets:
- name: jkubebuilder-dockercfg-4sd55
kind: ServiceAccount
metadata:
  creationTimestamp: "2024-04-10T18:42:16Z"
  name: jkubebuilder
  namespace: arsenalzp-dev
  resourceVersion: "2102645494"
  uid: 3d670c21-1525-4ac2-aca0-7a664d4f0b82
secrets:
- name: internal-registry
- name: jkubebuilder-dockercfg-4sd55
- name: builder-dockercfg-h8m4b

@rohanKanojia I'm checking whether an action is allowed:

$ oc auth can-i get pod/logs --as system:serviceaccount:arsenalzp-dev:jkubebuilder-n arsenalzp-dev
no
$ oc auth can-i watch pods/logs --as=system:serviceaccount:arsenalzp-dev:jkubebuilder
no
$ oc auth can-i get build/logs --as system:serviceaccount:arsenalzp-dev:jkubebuilder -n arsenalzp-dev
yes

As you may see neither get of pods/logs nor watch of pods/logs resources are allowed for jkubebuilder serviceAccount.

@arsenalzp
Copy link
Contributor

Hello colleagues,
Did you try to reproduce this bug on your side?

@rohanKanojia
Copy link
Member

@arsenalzp : Sorry, I didn't get time to look into it. I'll try to spend some time to reproduce it today.

@rohanKanojia
Copy link
Member

Are you sure that jkube is picking the correct ServiceAccount token while applying these objects onto OpenShift? If you're using this outside the cluster, I think jkube is going to pick up token from your local ~/.kube/config, not from the ServiceAccount you configured.

You can either try to create a tekton pipeline with configured ServiceAccount to build the application or copy the ServiceAccount token into your kubeconfig (just for the sake of testing). You should be able to see it in the secret created for that particular ServiceAccount:

$ oc get secret
NAME                           TYPE                                  DATA      AGE
jkubebuilder-dockercfg-h4n7k   kubernetes.io/dockercfg               1         6m20s
jkubebuilder-token-qxbp4       kubernetes.io/service-account-token   4         6m20s
$ oc describe secret jkubebuilder-token-qxbp4

...

Data
====
ca.crt:          7209 bytes
namespace:       7 bytes
service-ca.crt:  8422 bytes
token: <jkubebuilder-access-token>

You can copy token from there and put this in kube config:

users:
- name: developer/api-crc-testing:6443
  user:
    token: <jkubebuilder-access-token>

With this, you should be able to test your ServiceAccount permissions from outside the cluster.

@arsenalzp
Copy link
Contributor

arsenalzp commented Apr 24, 2024

Are you sure that jkube is picking the correct ServiceAccount token while applying these objects onto OpenShift? If you're using this outside the cluster, I think jkube is going to pick up token from your local ~/.kube/config, not from the ServiceAccount you configured.

You can either try to create a tekton pipeline with configured ServiceAccount to build the application or copy the ServiceAccount token into your kubeconfig (just for the sake of testing). You should be able to see it in the secret created for that particular ServiceAccount:

$ oc get secret
NAME                           TYPE                                  DATA      AGE
jkubebuilder-dockercfg-h4n7k   kubernetes.io/dockercfg               1         6m20s
jkubebuilder-token-qxbp4       kubernetes.io/service-account-token   4         6m20s
$ oc describe secret jkubebuilder-token-qxbp4

...

Data
====
ca.crt:          7209 bytes
namespace:       7 bytes
service-ca.crt:  8422 bytes
token: <jkubebuilder-access-token>

You can copy token from there and put this in kube config:

users:
- name: developer/api-crc-testing:6443
  user:
    token: <jkubebuilder-access-token>

With this, you should be able to test your ServiceAccount permissions from outside the cluster.

Sorry for delay.
I was trying to run crc locally, however my laptop doesn't meet system requirements.
I'm playing with Red Hat OCP SandBox, therefore there are some policy limitations which prevented me to reproduce your guide.
However I was able to fix the issue by substituting the following line of code:

try (LogWatch logWatch = client.pods().inNamespace(applicableOpenShiftNamespace).withName(buildName + "-build").watchLog()) {

by the following one:

try (LogWatch logWatch = client.builds().inNamespace(applicableOpenShiftNamespace).withName(buildName).watchLog()) {

It works.

But, that change is causing a failure of OpenshiftBuildServiceIntegrationTest test with the following exception:

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://localhost:47325/api/v1/namespaces/ns1/pods?labelSelector=openshift.io%2Fbuild.name. Message: Not Found.

I suppose that's because neither Build nor Pod resources don't ave labels set.

I tried to set the labels by using the following part of code

image

however, labels weren't applied to resources.

@arsenalzp
Copy link
Contributor

@rohanKanojia could you please be so kind to review how to properly label resources by using ResourceConfig? I found it is necessary to pass integration test successfully.

@rohanKanojia
Copy link
Member

@arsenalzp : Are you talking about OpenShiftBuildServiceIntegrationTest ?

I think you will need to add a new Kubernetes Server Mock(we use this testing library to mock Kubernetes API,see example in this blogpost) expectation for the code change you've done:

if (!buildConfigExists) {
mockServer.expect().get().withPath("/apis/build.openshift.io/v1/namespaces/ns1/buildconfigs/" + resourceName + s2iBuildNameSuffix).andReply(collector.record("build-config-check").andReturn
(404, "")).once();
mockServer.expect().get().withPath("/apis/build.openshift.io/v1/namespaces/ns1/buildconfigs/" + resourceName + s2iBuildNameSuffix + "pullSecret").andReply(collector.record("build-config-check").andReturn
(404, "")).once();
mockServer.expect().post().withPath("/apis/build.openshift.io/v1/namespaces/ns1/buildconfigs").andReply(collector.record("new-build-config").andReturn(201, bc)).once();
if (bcSecret != null) {

@arsenalzp
Copy link
Contributor

@arsenalzp : Are you talking about OpenShiftBuildServiceIntegrationTest ?

I think you will need to add a new Kubernetes Server Mock(we use this testing library to mock Kubernetes API,see example in this blogpost) expectation for the code change you've done:

if (!buildConfigExists) {
mockServer.expect().get().withPath("/apis/build.openshift.io/v1/namespaces/ns1/buildconfigs/" + resourceName + s2iBuildNameSuffix).andReply(collector.record("build-config-check").andReturn
(404, "")).once();
mockServer.expect().get().withPath("/apis/build.openshift.io/v1/namespaces/ns1/buildconfigs/" + resourceName + s2iBuildNameSuffix + "pullSecret").andReply(collector.record("build-config-check").andReturn
(404, "")).once();
mockServer.expect().post().withPath("/apis/build.openshift.io/v1/namespaces/ns1/buildconfigs").andReply(collector.record("new-build-config").andReturn(201, bc)).once();
if (bcSecret != null) {

Sorry I'm disturbing you!
Yes, I asked about OpenShiftBuildServiceIntegrationTest

@rohanKanojia
Copy link
Member

Sorry I'm disturbing you!

Not at all, it's part of our job to provide feedback and help to contributors. You're helping us a lot with your contributions, it doesn't take that much effort to compose a reply. If you want to communicate via chat, please don't hesitate to reach out to us on internal slack channel or Gitter channel.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants