Gitolite not being updated, no visible errors, while setting up a Docker image

[toastedcrumpets]

Thanks for an amazing project.

I have extended the official Docker redmine image to include setup of redmine_git_hosting. I'm saying this so you know the host is minimal and configured slightly strangely (i.e. I'm running tini to get defunct sshd threads cleaned up). If I get this working I would like to share the docker image as a way of getting redmine_git_hosting running with almost zero-config.

BUT, I have an issue, the gitolite repository is not being updated by redmine. The Config Test page is green, Hooks install just fine, and it can pull an info:

However, starting a new repository, the settings page shows the repo does not exist in gitolite (and checking via terminal, there is no change in the gitolite config since install).

I'm at a loss what to try next. I've tried running Rescue everything ticked. I get no issues in the rails log, or on the interface.

Looking at the temp dir, '/tmp/redmine_git_hosting/git/gitolite-admin.git' exists but it is an empty folder, not a git repository. When performing a Rescue I note that .gitolite/logs/ only has the following in it

2021-01-20.11:24:25 1592 cli gitolite list-phy-repos
2021-01-20.11:24:25 1592 END
2021-01-20.11:24:25 1604 ssh ARGV=id_rsa SOC=info FROM=::1
2021-01-20.11:24:25 1604 system,gitolite,info
2021-01-20.11:24:25 1604 system,/usr/share/gitolite3/commands/info

So it looks like redmine isn't doing anything with gitolite except to request an info to update the setting page on refresh after the POST of the Rescue. I'm guessing the issue is somehow redmine setting up its local admin repo, but the log only shows the POST and the rerender of the settings page.

Any idea how I can carry on debugging this? As this is a docker install, I could set up a test instance for you to look at if needed, although I need to polish my Dockerfile a bit to allow a more generic setup first (like changable host keys).

[toastedcrumpets]

OK, I've started to answer my own question. Looking in /usr/src/redmine/log/, there is a log file git_hosting.log which is not being output to the terminal (the normal way to do logging in docker images). Its telling me

+0000 [ERROR] Invalid Gitolite Admin SSH Keys

So this might be a stupid config error. If that's what it is I'll close this issue.

[toastedcrumpets]

I'm still having issues. Trying to reproduce the steps the server might be taking to ssh I tried running

ssh -i /home/redmine/.ssh/id_rsa git@localhost -p 22 info

and it works just fine. I also note that my initial issue report shows the admin interface showing gitolite's ssh response. So SSH must be working, but why am I getting Invalid Giolite Admin SSH Keys?

Any idea how I can debug this further?

[toastedcrumpets]

OK so I've fixed it. Its another case of #747 which has been closed but is still a live issue so I think that's an error.
In short, redmine SSH keys must be in PEM format.
The install instructions need to be amended and/or add debugging output (#708).
Please open #747 again to close this issue, or amend #708 to include this case.

[alexandermeindl]

Hi @toastedcrumpets
thanks for your debugging and helpful feedback. I updated the installation instruction with key format parameter, see http://redmine-git-hosting.io/get_started/#step-3-create-ssh-keys-for-user-running-redmine

[toastedcrumpets]

That looks great, I'd probably reinforce it with an underline, as its a subtle bug if not as only part of the SSH process fails. I'm still happy enough to close the issue though. Out of interest, what is the method you are using to carry out SSH actions? Its not the CLI ssh tool, as that supports any key format...

[FaSeEngineering]

@toastedcrumpets I'm sorry to ask a question again, since you closed this issue. But I'm stucked with the same problem. I tried to follow the new installation instructions but had no success. Do you remember what steps lead to the fix of your issue? Did you change something in the Dockerfile?

[toastedcrumpets]

Hey,
I think its best I just share everything.
I built a docker image that has all my favourite plugins pre-installed (which means I can't distribute the docker image), as well as hosts the git instance. There were a few tricks to doing this.

The file docker_entrypoint.sh, is just the same one from the redmine docker image, but has an additional /usr/bin/sshd line at the top to start the sshd daemon. I also have my redmine configuration.yml file next to the Dockerfile, as well as pre-generated host keys.
The main thing for you is that the host keys are pregenerated using ssh-keygen -m PEM -N '' -f id_rsa. You can see in the Dockerfile where I copy over the host keys if you're wondering how to do this.

Happy to explain anything else in the file if you need it!

Dockerfile

FROM redmine:4.1

RUN set -eux; \
        apt-get update; \
        apt-get install -y --no-install-recommends \
        unzip git imagemagick \
        build-essential pkg-config libssh2-1 libssh2-1-dev cmake libgpg-error-dev sudo ; # This row required for redmine-git-hosting \
        rm -rf /var/lib/apt/lists/*

COPY flatly_light_redmine-master.zip /usr/src/redmine/public/themes/
COPY redmine_agile-1_5_4-light.zip /usr/src/redmine/plugins/
COPY redmine_ckeditor-master.zip /usr/src/redmine/plugins/
COPY redmine_checklists-3_1_18-light.zip /usr/src/redmine/plugins/

RUN cd /usr/src/redmine/public/themes; \
    git clone https://bitbucket.org/dkuk/redmine_alex_skin.git; \
    chown -R redmine:redmine redmine_alex_skin; \
    unzip flatly_light_redmine-master.zip ; \
    rm flatly_light_redmine-master.zip ; \
    chown -R redmine:redmine flatly_light_redmine-master; \
    git clone https://github.com/mrliptontea/PurpleMine2.git; \
    chown -R redmine:redmine PurpleMine2; \
    cd /usr/src/redmine/plugins;  \
    unzip redmine_ckeditor-master.zip ; \
    mv redmine_ckeditor-master redmine_ckeditor; \
    chown -R redmine:redmine redmine_ckeditor; \
    rm  redmine_ckeditor-master.zip ; \
    git clone https://github.com/toastedcrumpets/redmine_issue_dynamic_edit.git; \
    chown -R redmine:redmine redmine_issue_dynamic_edit; \
    unzip redmine_agile-1_5_4-light.zip ; \
    rm  redmine_agile-1_5_4-light.zip ; \
    chown -R redmine:redmine redmine_agile; \
    unzip redmine_checklists-3_1_18-light.zip ; \
    rm  redmine_checklists-3_1_18-light.zip ; \
    chown -R redmine:redmine redmine_checklists; \
    mkdir /repos; \
    mkdir /localstore; \
    chown -R redmine:redmine /repos /localstore

VOLUME /repos
VOLUME /localstore

# Here we build all plugins that have been installed, this is time-consuming to do on image start (but migrations must be done then)
USER redmine
WORKDIR /usr/src/redmine
RUN bundle install --without development test

# Now we install git support, NOTE this cannot be done earlier due to issues on missing plugins.

USER root

RUN cd /usr/src/redmine/plugins; \
    ###### Redmine support for basic git server \
    # git clone https://github.com/jbbarth/redmine_base_deface.git; \
    # chown -R redmine:redmine redmine_base_deface; \
    # git clone https://github.com/voondo/redmine_git_server.git; \
    # chown -R redmine:redmine redmine_git_server; \
    ###### Redmine support for advanced git hosting \
    git clone https://github.com/AlphaNodes/additionals.git; \
    chown -R redmine:redmine additionals; \
    git clone https://github.com/jbox-web/redmine_git_hosting.git; \
    chown -R redmine:redmine redmine_git_hosting

USER redmine

RUN bundle install --without development test

USER root

RUN echo 'Defaults:redmine !requiretty\n\
redmine ALL=(git) NOPASSWD:ALL\n\
' > /etc/sudoers.d/redmine

RUN chmod 440 /etc/sudoers.d/redmine; \
    mkdir /home/redmine/.ssh; \
    chown -R redmine:redmine /home/redmine/.ssh

# YOU MUST CREATE THESE KEYS USING `ssh-keygen -m PEM -N '' -f id_rsa`! It will silently fail if you don't!
COPY id_rsa /home/redmine/.ssh/
COPY id_rsa.pub /home/redmine/.ssh/


## Add the known_hosts file for redmine so it has totally keyless logins
RUN echo "!!!! I've Removed this for public distribution!!!" > /home/redmine/.ssh/known_hosts; chown redmine:redmine /home/redmine/.ssh/known_hosts

RUN chmod 600 /home/redmine/.ssh/id_rsa; chmod 644 /home/redmine/.ssh/id_rsa.pub /home/redmine/.ssh/known_hosts; chown -R redmine:redmine /home/redmine/.ssh

#Now we need to setup gitolite

RUN set -eux; \
    apt-get update; \
    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends  gitolite3 openssh-server; \
    rm -rf /var/lib/apt/lists/*

RUN set -eux; useradd -d /home/git -ms /bin/bash git

COPY --chown=git:git mjki2mb2.pub /home/git

USER git
WORKDIR /home/git
RUN set -eux; HOME=/home/git USER=git gitolite setup -pk mjki2mb2.pub;

VOLUME /home/git

EXPOSE 22

USER root
WORKDIR /usr/src/redmine

COPY docker-entrypoint.sh /

COPY ssh_host_* /etc/ssh/

# Install ruby for the git hooks
RUN set -eux; \
    apt-get update; \
    apt-get install -y --no-install-recommends ruby; \
    rm -rf /var/lib/apt/lists/*

## We add tini as sshd needs cleanup of its defunct processes
ENV TINI_VERSION v0.19.0
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /tini
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini.asc /tini.asc
RUN chmod +x /tini

ENTRYPOINT ["/tini", "--"]
CMD /docker-entrypoint.sh rails server -b 0.0.0.0

RUN set -eux; \
    apt-get update; \
    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends  ruby-redcarpet; \
    mkdir /run/sshd; \
    rm -rf /var/lib/apt/lists/*

COPY configuration.yml /usr/src/redmine/config/

[FaSeEngineering]

Thank you very much for sharing your Dockerfile! I played around a little and get it working with all my desired plugins and themes but the issue remains. I created a key pair on my host machine as you mentioned. At two points I was not sure:

• your "mjki2mb2.pub" file is one of your personal keys? I created my own and changed this entry to fit
• In the known_hosts file where you placed "!!!! I've Removed this for public distribution!!!" I was not sure how I can insert my host

As you may notice I'm a little new to this topic, so thank you for your answer and your patience

[toastedcrumpets]

yes mjki2mb2.pub is my public ssh key to set up as admin for gitolite.

The known hosts file needs to have the signature of the public key of the ssh server, as otherwise you can have an issue with "unknown host" warnings popping up, but as the whole process is automatic/server side you have no way of accepting the unknown host.

You really need to look at /usr/src/redmine/log/git_hosting.log running in the docker to see what errors are coming up when you try to "Rescue" in the redmine setup. If you get really stuck in a few weeks I'll try to set up a script that makes a Dockerfile for this fully automatically.

[FaSeEngineering]

Okay thanks,
did you setup anything on your host machine? The redmine container is running fine and the config test page shows no error. I get the response from the ssh -i /home/redmine/.ssh/id_rsa git@localhost -p 22 info in the Gitolite banner. But when I check the log files they always tell me that the Gitolite Admin Key is invalid:

[ERROR] Invalid Gitolite Admin SSH Keys
[ERROR] FATAL: R any gitolite-admin id_rsa DENIED by fallthru (or you mis-spelled the reponame)

I followed the key pair generation as you mentioned. Maybe I'm just missing something but I didn't figure out what.
Thanks again for your reply, it helps to dive deeper into the whole docker container creation and I learned a lot, even if I did not get it finally working.

[toastedcrumpets]

That's a gitolite error, you've misconfigured your git server. If you're using my docker file, then these are the two lines I use to copy my public ssh key to the image, and install/setup gitolite. Are you using a public key or accidentally sending your private key?

COPY --chown=git:git mjki2mb2.pub /home/git
RUN set -eux; HOME=/home/git USER=git gitolite setup -pk mjki2mb2.pub;

Note this is a different key to the SSH server key which must be generated with PEM. I think #790 should be closed.

[FaSeEngineering]

I'm sure that I used the public key for this purpose. I have a question about your setup... maybe I'm just getting something wrong. Do you use your docker container as a closed system? So your ssh-server, your gitolite and your redmine are all in one container, or have you separated some things? Thanks again for your reply.

[toastedcrumpets]

My docker container is a completely closed system. I do this as redmine git hosting has some optimisations if it has local access to the repositories (IIRC). I could have done a shared volume between two containers, but this seems overly complex for no real benefit to me, so I did it all in one container.

[FaSeEngineering]

Okay, thats what I want to achieve as well. An issue what my be related to the whole problem was that I could not use my created ssh public key for gitolite setup and receive the gitolite banner message. I only can see it if I use the public key created for the ssh server. Do you have an idea what the problem is?

[toastedcrumpets]

Hi,
I've sort of run out of time again to help, but I started a docker-compose example on this.
https://github.com/toastedcrumpets/redmine-git-docker-example
Unfortunately its not working for me at the moment, it seems that there's some missing dependencies due to the changes to the underlying redmine container. It might be a nice "clean" example for both of us to work on though?

[toastedcrumpets]

OK, I've actually got it working. Can you check out that repo, and the Readme.md instructions, see if it works for you?

[FaSeEngineering]

Please excuse my late reply. Due to the exam phase in my studies, it took a while to try out your docker-compose example. After a few adjustments, everything works fine. I even managed to enable the hooks (manually). Many thanks for your effort. Since everything works fine now, I closed my issue #790

[toastedcrumpets]

OK great! If your changes were non-trivial, please push an update to that repo.