Uninstalling a plugin #1531 #1538

ritikbanger · 2022-12-25T08:42:55Z

Document removal of a plugin from container

Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
Ensure that the pull request title represents the desired changelog entry
Please describe what you did
Link to relevant issues in GitHub or Jira
Link to relevant pull requests, esp. upstream and downstream changes

This PR address the following issue: #1531 which is discussed in SIG December 16th Meeting

cc @dduportal @gounthar

Merry Christmas 🎅

gounthar · 2022-12-25T17:33:45Z

Merry Christmas too, thanks a lot for your contribution. 🙏

README.md

ritikbanger · 2022-12-28T05:37:58Z

Are we ready to merge or is there any more changes @gounthar @dduportal

MarkEWaite

The install-plugins.sh script is no longer a useful script. The plugins.txt file defines the plugins to be installed, not the plugins to be removed (at least as far as I've used it).

README.md

MarkEWaite · 2022-12-28T18:23:57Z

From experiments with the jenkins/jenkins:2.375.1 container, I can remove a plugin from the installation by removing the entry from the plugins.txt file, building a new container image, then stopping the existing container and starting with the new container image.

Dockerfile that I used was:

FROM jenkins/jenkins:2.375.1
COPY --chown=jenkins:jenkins plugins.txt $REF/plugins.txt
RUN jenkins-plugin-cli --plugin-file $REF/plugins.txt

However, that simple use case does not cover the cases where the user is preserving their configuration in a separate data volume. In that case, then I believe that additional steps will be required in order to remove the files from $JENKINS_HOME/plugins/* so that the jenkins.sh script that starts the java -jar jenkins.war process is writing the set of plugins from the $REF directory to the plugins directory without having to remove existing files.

… of creating a new plugin.txt

ritikbanger · 2022-12-29T14:04:40Z

From experiments with the jenkins/jenkins:2.375.1 container, I can remove a plugin from the installation by removing the entry from the plugins.txt file, building a new container image, then stopping the existing container and starting with the new container image.

Dockerfile that I used was:
FROM jenkins/jenkins:2.375.1
COPY --chown=jenkins:jenkins plugins.txt $REF/plugins.txt
RUN jenkins-plugin-cli --plugin-file $REF/plugins.txt
However, that simple use case does not cover the cases where the user is preserving their configuration in a separate data volume. In that case, then I believe that additional steps will be required in order to remove the files from $JENKINS_HOME/plugins/* so that the jenkins.sh script that starts the java -jar jenkins.war process is writing the set of plugins from the $REF directory to the plugins directory without having to remove existing files.

Added a new commit- b0b6bcf

Let me know, if there are any suggestions.

ritikbanger · 2023-01-04T05:00:04Z

Any update on this

MarkEWaite · 2023-01-04T20:49:08Z

Any update on this

Sorry for the delay. It will likely be a few more days before I can review.

gounthar · 2023-01-04T20:55:45Z

Same for me, sorry.

README.md

gounthar

Thanks a lot for your contribution, we do appreciate it. 🙏
Please have a look at my comments and let me know if you agree.
If you don't (which is fine), let's discuss it further.

Once more, thanks.

README.md

ritikbanger · 2023-01-29T15:21:12Z

@MarkEWaite can you help me with the new updates that I need to add or remove from this PR?

daniel-beck

Most of this is general documentation that applies to all Jenkinses. Why is it added to the documentation of the Docker images?

dduportal · 2023-02-08T13:39:53Z

Most of this is general documentation that applies to all Jenkinses. Why is it added to the documentation of the Docker images?

Because it's needed here until a "general documentation with clear examples for Docker" is provided. There are multiple specific elements for Docker image that are required, which does NOT apply to packaged distribution of Jenkins (the fact that plugins should be pre-installed in the image in /usr/share/jenkins/ref but that what Jenkins sees is the data volume in /var/jenkins-home for instance.

ritikbanger · 2023-02-14T18:04:57Z

Hey @dduportal, can you help me with this PR. Its been one month without any update from other maintainers.

MarkEWaite · 2023-02-14T18:08:23Z

Sorry for the delay @ritikbanger . I'll do my best to help with a review within the next 7 days. @dduportal probably won't be able to help due to other priorities that he needs to handle.

dduportal · 2023-05-13T11:08:27Z

I'm very new to Jenkins world, but have some experience in containers.

Assuming a scenario that plugins are baked into a custom docker image (downloaded and installed during bulding)

Wouldn't it be good to mount $JENKINS_HOME/plugins into a tmpfs filesystem, so on each startup the plugins will be copied over from the /usr/share/jenkins/ref?

With this you will mange plugins on one place (plugins.txt/yaml) on your git repo. And even a downgrade will propagate also. (eg a bad update on plugin that broke it, you just revert the plugin file, rebuild image and re-pull).

AFAIK $JHOME/plugins only contains the plugins and not any settings there right?

That is absolutely something you can do. It works as expected, behaving like you describe.
It avoids configuration shift (between plugins.txt and reality) while allowing "testing" a plugin upgrade in the container.

A few attention points (not issues, be important to have in mind):

Monitor closely the size of this tmpfs mount as plugins may take quite some space. A few hundreds megabytes at least but expect more if you have more plugins.
With Docker, be aware of the tmpfs limitations, particularly the headache with permissions: specifying UID/GID permission explicitly is recommended to avoid a Jenkins startup issue due to root owning the content of the directory
In Kubernetes, be aware that the tmpfs (emptyDir of type "memory") is counted against the container memory limit (ref. https://kubernetes.io/docs/concepts/storage/volumes/#emptydir) but I do not know for Docker.
Upgrading/Downgrading plugin can have an impact in the persisted XML files of Jenkins: always have a backup/snapshot of the whole JENKINS_HOME to avoid taking any risks when downgrading

Other that this points, it works really well!

stavros-k · 2023-05-13T11:32:48Z

Thanks for the quick answer! I was aware of the docker/k8s points, but was a good reminder!

Upgrading/Downgrading plugin can have an impact in the persisted XML files of Jenkins

Any ideas on how to overcome this? Are those XML files generate by Jenkins to keep track of plugins only, or there are multiple XMLs that are used in various places?

(I'm already creating config files in order to use JCasC and have a "ready" image and as much "immutable" as possible)

dduportal · 2023-05-13T12:09:06Z

Any ideas on how to overcome this?

Rewriting Jenkins and all plugins 😅
Joke aside: it's not possible. It's the way plugins are designed: Jenkins need to know which version of each plugin is used to follow interface changes. The XML files can be Jenkins general configuration, job configurations, build descriptions and many more persistent elements.

Having to rollback a plugin is usually something you don't do lightly because it means a plugin update has a breaking change that you cannot get over with: it's a pretty rare occurence (haven't had something like that for years).

Usually, a plugin breaking change is easier to fix: you check the plugin changelogs, get the breaking change, update the configuration (JCasc or XML) and apply the fix.

If your Jenkins controller is too sensitive to accept such situation once a trimester (worst case) then it means that:

You need a staging environment to validate each plugin or core update, with a subset of "real life" jobs
You need a strict backup of your Jenkins home that you should be able to restore at any moment in time

If you Jenkins controller is not "that" sensitive and breaking it is more an annoyance, then you can make the backup easier:

Use an Artfact caching manager such as https://plugins.jenkins.io/artifact-manager-s3/ to avoid storing artifacts in the Jenkins_HOME
Use tmpdir or local dir for the war and plugins directory
Use JCasC + JobDSL for every configuration
Use the Docker image of Jenkins for plugins and core
With this, you should only have to backup the builds history which is clearly easier to get (rsync or FS snapshots are really efficient to capture differentials here)

It looks like you're in a good direction :)

stavros-k · 2023-05-13T12:21:53Z

If you Jenkins controller is not "that" sensitive and breaking it is more an annoyance, then you can make the backup easier:

I fall in this category! I can live without CI/CD for couple of days(!). But some times I'm impatience and I wanna make it work NOW. (LoL). I'll try to delay upgrading plugins few days, so any big breakage would be catched. But as you mentioned, I don't believe that will be a frequent scenario. (Unless I use a random plugin that's just started development)

Use an Artfact caching manager such as https://plugins.jenkins.io/artifact-manager-s3/ to avoid storing artifacts in the Jenkins_HOME

Use JCasC + JobDSL for every configuration

Use the Docker image of Jenkins for plugins and core

With this, you should only have to backup the builds history which is clearly easier to get (rsync or FS snapshots are really efficient to capture differentials here)

Yea that's my goals! Just I need a lot of cathcing up to do! As I'm only used to use CI/CDs like GH Actions.

It looks like you're in a good direction :)

Thanks! And thanks for taking the time to answer!

dduportal · 2023-05-13T12:24:56Z

No problem ! You can see what we are doing for the Jenkins infrastructure itself in github.com/jenkins-infra where we got these kind of pattern.

We follow the principle of "update often (once or twice a week) to have smaller scopes" so when it goes wrong, then we have less changes to cover and it's faster to diagnose and fix.
Besides, we are pretty confident in our deployment/rollback system with this.

It's not free the first steps, but once you get used to it, it's worth the investment: you are in the right direction!

dduportal · 2023-05-15T07:38:22Z

Hi @ritikbanger , there are still some comment on this PR, are you ok to address them?

ritikbanger · 2023-05-15T08:09:09Z

@dduportal Could you list out all the points?

dduportal · 2023-05-15T08:12:04Z

@dduportal Could you list out all the points?

If you go to the to the tab "Files Changed", you'll see all the suggestions and threads (without the other comments)

ritikbanger · 2023-05-15T08:14:52Z

@dduportal I have gone through the comments and addressed them. Could you please give me a review on that if you think anything need changes?

dduportal · 2023-05-15T10:18:27Z

@dduportal I have gone through the comments and addressed them. Could you please give me a review on that if you think anything need changes?

I think you forgot to push your changes: I don't see @gounthar 's comment taken in account in the code.
Yes you answered or added a ":+1:" emoji, but the code of the pull request is not changed.

dduportal

@ritikbanger thanks for this proposal and sorry for the delay in review but:

It does not seem you did the uninstallation process by yourself. I stronlgy suggest you to try it on a local container, otherwise there is no point in documenting something you have no idea how it works. You should record the video of your attempt as a support for getting help from the SIG platform meeting members
I'm not sure if you are using a ChatGPT tool or anything, but the structure of the text looks really weird, along with inexact assertions and "generic" phrasings. If you did, please avoid this as it only makes you waste your time.
If you did not and wrote all of this PR by yourself, then it means we should go back to the issue and have you follow at least one uninstallation of a plugin in a Jenkins container to draft a structure of "what uninstallation method" should we cover.

As a maintainer and a user, I cannot accept the content as it today: it adds way more confusion that it helps.
I don't mind helping in the direction, but I need you to do it yourself one time first before continuing

README.md

dduportal · 2023-07-01T15:22:38Z

Update: there has been a nice blog post in https://www.jenkins.io/blog/2023/06/20/remove-outdated-plugins-while-using-docker/.

@ritikbanger are you ok to update this PR according to this blog post + take our feedbacks in account?

ritikbanger · 2023-07-02T11:34:57Z

I will be finishing this PR super soon.

ritikbanger · 2023-07-15T17:02:45Z

@dduportal I have updated the PR with your feedback. Please have a look.

ritikbanger · 2023-07-18T15:56:17Z

@gounthar your review will be highly appreciated

gounthar

Thanks a lot for this work. 🤗

README.md

gounthar

Thanks a lot for this work. 🤗

ritikbanger · 2023-08-05T08:16:12Z

@gounthar I think we are ready to merge this.

gounthar · 2023-08-05T10:00:50Z

Thank you very much for your contribution and the latest changes, @ritikbanger.
I don't have the authority to merge your pull request, but I have already approved it.

ritikbanger · 2023-08-12T03:12:32Z

@dduportal

dduportal

It's going in a really good direction! I'm still not in agreement with some major elements.

The main point is that it feels like you want to write a step by step guide: the README is not the place for that. I would prefer having a page in jenkins.io's documentation (which requires asking the documentation SIG group to find the perfect place in the table of content).

I've provided feedback to continue this PR and land it by shrinking steps and being more generic (requiring Docker Compose is not acceptable for instance).

Thanks for this work!

dduportal · 2023-08-15T14:26:52Z

README.md

+## Prerequisites
+
+- Docker and Docker Compose are installed on your system.
+- Basic knowledge of Docker and Jenkins.


This section does not make sense for me: this pull request is not expected to add a step by step guide that would only cover some use cases.

Do you mind replacing it by a summary of the main important steps instead?

additionnally, I do not expect this to be a guide using docker compose: not only docker-compose is deprecated in favor of the docker compose plugin (ref. https://docs.docker.com/compose/, also refered as "Docker Compose v2) but also the instructions in this README should be minimalistic to ensure they benefit non Docker users.

dduportal · 2023-08-15T14:27:54Z

README.md

+ant:1.11
+```
+
+3. **Rebuild the Jenkins Image**: In your terminal, navigate to the directory containing the Docker configuration files for your Jenkins instance.


Suggested change

3. **Rebuild the Jenkins Image**: In your terminal, navigate to the directory containing the Docker configuration files for your Jenkins instance.

3. **Rebuild the Jenkins Container Image**: In your terminal, navigate to the directory containing the Jenkins Container Image definition.

dduportal · 2023-08-15T14:28:36Z

README.md

+   Rebuild the Jenkins image to apply the changes made in the plugins.txt file using the following command:
+
+```bash
+docker-compose build jenkins
+```


Suggested change

Rebuild the Jenkins image to apply the changes made in the plugins.txt file using the following command:

```bash

docker-compose build jenkins

```

Rebuild the Jenkins image to apply the changes made in the plugins.txt file using the appropriate tool for your setup (`docker build <...>`, `docker compose build <...>`, etc.).

Remove Docker Compose

dduportal · 2023-08-15T14:28:55Z

README.md

+   Use the following command:
+
+```bash
+docker-compose up -d --build --force-recreate jenkins
+```


Suggested change

Use the following command:

```bash

docker-compose up -d --build --force-recreate jenkins

```

Remove Docker Compose

dduportal · 2023-08-15T14:34:26Z

README.md

+5. Verify Plugin Removal: To verify if the deprecated plugins have been removed, enter the running Jenkins container by executing the following command:
+
+```bash
+docker-compose exec jenkins bash
+```
+
+6. Once inside the container, navigate to the Jenkins plugin directory:
+
+```bash
+cd /usr/share/jenkins/
+```
+
+7. Check the `plugins.txt` file to ensure the references to the deprecated plugins are gone:
+
+```bash
+cat plugins.txt | grep <plugin-name>
+```
+
+Replace `<plugin-name>` with the name of the deprecated plugin you want to uninstall.
+If no results are returned, it means the plugin references have been removed successfully.


Suggested change

5. Verify Plugin Removal: To verify if the deprecated plugins have been removed, enter the running Jenkins container by executing the following command:

```bash

docker-compose exec jenkins bash

```

6. Once inside the container, navigate to the Jenkins plugin directory:

```bash

cd /usr/share/jenkins/

```

7. Check the `plugins.txt` file to ensure the references to the deprecated plugins are gone:

```bash

cat plugins.txt | grep <plugin-name>

```

Replace `<plugin-name>` with the name of the deprecated plugin you want to uninstall.

If no results are returned, it means the plugin references have been removed successfully.

5. Verify effective plugins removal from the image: To verify if the deprecated plugins have been removed from the image, check their absence from the `/usr/share/jenkins/plugins.txt` in the running Jenkins container.

For instance, if you use `docker`:

```bash

$ docker exec <Jenkins Container ID> grep -c "<plugin-name>" /usr/share/jenkins/plugins.txt

0 # Should be zero (e.g. the number of occurences of the string "<plugin-name>" in the file)

```

Replace `<plugin-name>` with the name of the deprecated plugin you want to uninstall.

Proposal:

Squash steps into a single one

No need to use interactive steps

A bit of rephrasing

Remove Docker Compose

dduportal · 2023-08-15T14:43:20Z

README.md

+8. Uninstall Plugins in the Jenkins UI: Access the Jenkins web interface by visiting [http://localhost:8080](http://localhost:8080) in your web browser.
+   Log in with your admin credentials.
+
+9. Navigate to _Manage Jenkins_ and select _Manage Plugins_.
+
+10. In the _Installed_ tab, search for the name of the deprecated plugin you want to uninstall.
+
+11. If the plugin appears in the list, click the _Uninstall_ button (red cross) next to it.
+    If the button is disabled then it means that the plugin has a dependency on another plugin.
+    Just hover on the red cross to know the parent plugin.
+    Search that plugin and remove it first by following the same procedure.
+
+12. Confirm the removal when prompted.
+
+13. Restart Jenkins: To complete the removal process, Jenkins needs to be restarted.
+    Restart Jenkins by hitting the `/safeRestart` endpoint or using the Jenkins UI.
+
+14. Optional: Remove Plugin remnants from Docker Volume: If you want to remove any remnants of the deprecated plugins from the Docker volume, follow these steps:
+
+- Enter the running Jenkins container using the command mentioned in step 5.
+- Navigate to the plugin directory:
+
+`cd ~/plugins`
+
+- List the contents of the directory to identify the remnants of the deprecated plugins:
+
+`ls -artl *`
+
+- Remove the remnants of the deprecated plugins using the rm command. For example:
+
+`rm -fr <plugin-name>*`
+
+Replace `<plugin-name>` with the name of the deprecated plugin you want to remove.
+
+To deep dive into the removal, refer to this [guide](https://www.jenkins.io/blog/2023/06/20/remove-outdated-plugins-while-using-docker/).


This section is confusing as it mixes different cases without explaining why and what.

Again, this should not be a step by step guide: if you feel like writing a step by step guide, than jenkins.io's documentation would be a better location to write so

WDYT about:

Starting by explaining that the already installed plugins are living in a Docker volume which has a different lifecycle than the container image and the container image's entrypoint does NOT clean up automatically plugins

Then describe a command line based scenario to remove plugins manually in the volume (which is roughly the step 14. in your writing here) so it would apply to everyone

Mentions (using an adminition) that the UI could be used but it does not properly removes every files associated to <plugin name> installation
-Updates the instructions as you want user to remove /var/jenkins_home/plugins/<plugin name>* so that the plugin folder AND all the *.jpi, *.hpi and any lock files are deleted for the given plugin <plugin name>

added uninstalling a plugin methods

f7d23be

ritikbanger requested a review from a team as a code owner December 25, 2022 08:42

gounthar reviewed Dec 25, 2022

View reviewed changes