Gitlab Runner with self-hosted Gitlab and Sonatype Nexus with SSL
Saturday, August 3, 2019A Sonatype Nexus and the internal self signed Gitlab instances were the only resources available to this CentOS 7 Server we were dealt. Recently I and a dev configured a Gitlab Runner Docker for CI builds. This is our story. Dun Dun
Note: I am assuming that you don't want to use proxy for some reason. If you can access external resources through some corporate proxy, you may configure that, and it will probably be easy and work. We found out that using Sonatype Nexus was faster than reaching outside using proxies, and in this particular machine we configured, we couldn't access through proxy unless with our personal keys for authentication, which was undesirable.
The first thing we will need is the package ca-certificates (which may be already installed).
We need to activate certificate management
Go where the self signed certificates you need are made available, and download them. You will need both the emitter and root certificates. You can get the URL of the root certificate from other certificate by issuing the following command:
Name both downloaded certificates as ca-mydomain-root.crt and ca-mydomain-emitter.crt.
Copy these files to /etc/pki/ca-trust/source/anchors
Execute the following command to update managed certificates
Test the SSL negotiation with your Gitlab server
At the end of the output, if everything is fine, you will get the following: Verify return code: 0 (ok)
Now that Docker has been installed, you may create a docker user and group if you like. Notice Docker Daemon needs root access to your computer. We will assume your self-hosted internal Sonatype Nexus is available at http://nexus.mydomain.com/nexus/ . Note we can also use https here, but we will need the certificate, like we did at the previous step. If your self-hosted Nexus uses the same root certificate, then using the https Nexus URL should work.
I am assuming you have everything available on Sonatype Nexus, so we are going to configure the CentOS server to pull Docker images from there.
First, you need to figure out your Nexus Docker Registry proxy port! This will be under the Docker Registry Repository HTTP or HTTPS connector. I didn't had access to read this information directly through Sonatype Nexus interface, so it required a phone call. So this is how the number 8123 will appear from magic below.
Edit /etc/sysconfig/docker text file with any editor (eg: sudo vi /etc/sysconfig/docker ) and add the lines at the end.
Restart Docker service
Everything should be good now, let's pull the Gitlab Runner image
Assuming everything worked out, we can move on.
Let's create a folder to store configurations
Now, let's create a folder to store the certificates
We need to convert the certificates to PEM using the following command line
Now we need to create a bundle from these certificates, to connect to the Gitlab server with SSL.
A small note here, if you are working with Java, it unfortunately doesn't use the system certificates and instead it has it's own folder, so you will need to add these certificates there too, and possibly your Sonatype Nexus instance certificate if it uses a different root certificate, to make sure Gradle works.
Almost all set, on Gitlab, go into the settings for the repository you wish to build and get the identifier token for it. At the time of writing, typically under Settings, CD/CI, Runners, Specific Settings, with some name like xxxxYYY_ZZ4S.
Let's register the runner with the internal Gitlab Server.
You may add a specific tag for your runner too with --tag-list "mytag", just make sure to actually have it on your repository otherwise it may prevent the runner from starting.
Last step, let's initialize the Gitlab Runner with Restart Always. This will ensure that when the Docker is initialized, it will already start the runner.
Note: I am assuming that you don't want to use proxy for some reason. If you can access external resources through some corporate proxy, you may configure that, and it will probably be easy and work. We found out that using Sonatype Nexus was faster than reaching outside using proxies, and in this particular machine we configured, we couldn't access through proxy unless with our personal keys for authentication, which was undesirable.
Configuration with Self Signed Certificates
We need to have self-signed certificates that are available for install on the machines. Since this Gitlab uses these certificates, connecting directly to it without them will result in SSL errors. The first thing to do is downloading them, and if you don't know where they are available you may need to contact someone to point the URL to you, but they are most sure available.The first thing we will need is the package ca-certificates (which may be already installed).
sudo yum install ca-certificates
We need to activate certificate management
sudo update-ca-trust enable
Go where the self signed certificates you need are made available, and download them. You will need both the emitter and root certificates. You can get the URL of the root certificate from other certificate by issuing the following command:
openssl x509 -text -inform DER -in justDownloadedCertificate.cer
Name both downloaded certificates as ca-mydomain-root.crt and ca-mydomain-emitter.crt.
Copy these files to /etc/pki/ca-trust/source/anchors
Execute the following command to update managed certificates
update-ca-trust extract
Test the SSL negotiation with your Gitlab server
openssl s_client -connect gitlab.mydomain.com:443
At the end of the output, if everything is fine, you will get the following: Verify return code: 0 (ok)
Configuration of Docker on the Host with Nexus
Here we are going to use Docker to get Gitlab Runner. So first thing to do is installing Docker. We really want RedHat fork of Docker, because it allows easily using a different Docker Registry than DockerHub, so this is done like this.yum install docker
Now that Docker has been installed, you may create a docker user and group if you like. Notice Docker Daemon needs root access to your computer. We will assume your self-hosted internal Sonatype Nexus is available at http://nexus.mydomain.com/nexus/ . Note we can also use https here, but we will need the certificate, like we did at the previous step. If your self-hosted Nexus uses the same root certificate, then using the https Nexus URL should work.
I am assuming you have everything available on Sonatype Nexus, so we are going to configure the CentOS server to pull Docker images from there.
First, you need to figure out your Nexus Docker Registry proxy port! This will be under the Docker Registry Repository HTTP or HTTPS connector. I didn't had access to read this information directly through Sonatype Nexus interface, so it required a phone call. So this is how the number 8123 will appear from magic below.
Edit /etc/sysconfig/docker text file with any editor (eg: sudo vi /etc/sysconfig/docker ) and add the lines at the end.
BLOCK_REGISTRY='--block-registry=all' ADD_REGISTRY='--add-registry=http://nexus.mydomain.com:8123'
Restart Docker service
sudo systemctl restart docker
Everything should be good now, let's pull the Gitlab Runner image
docker pull gitlab/gitlab-runner
Assuming everything worked out, we can move on.
Configuring Gitlab Runner through Docker
Now the machine can get packages and has SSL access to internal network websites, we can configure Gitlab Runner to have access to all this too. It's useful to read the documents regarding Gitlab Runner and Docker for Gitlab Runner.Let's create a folder to store configurations
mkdir /opt/gitlab-runner
Now, let's create a folder to store the certificates
mkdir /opt/gitlab-runner/certs
We need to convert the certificates to PEM using the following command line
openssl x509 -in ca-mydomain-emitter.cer -inform DER -out ca-mydomain-emitter.pem -outform PEM openssl x509 -in ca-mydomain-root.crt -inform DER -out ca-mydomain-root.pem -outform PEM
Now we need to create a bundle from these certificates, to connect to the Gitlab server with SSL.
cat ca-mydomain-root.pem ca-mydomain-emitter.pem > /opt/gitlab-runner/certs/ca-mydomain-bundle.pem
A small note here, if you are working with Java, it unfortunately doesn't use the system certificates and instead it has it's own folder, so you will need to add these certificates there too, and possibly your Sonatype Nexus instance certificate if it uses a different root certificate, to make sure Gradle works.
Almost all set, on Gitlab, go into the settings for the repository you wish to build and get the identifier token for it. At the time of writing, typically under Settings, CD/CI, Runners, Specific Settings, with some name like xxxxYYY_ZZ4S.
Let's register the runner with the internal Gitlab Server.
docker run --rm -t -i -v \ /opt/gitlab-runner:/etc/gitlab-runner \ --name gitlab-runner gitlab/gitlab-runner register --non-interactive \ --url "https://gitlab.mydomain.com" \ --registration-token "xxxxYYY_ZZ4S" \ --description "my-docker-runner" \ --tls-ca-file "/etc/gitlab-runner/certs/ca-mydomain-bundle.pem" \ --run-untagged \ --locked="false" \ --executor "docker" \ --docker-image "docker:stable" \ --docker-privileged \ --docker-volumes /var/run/docker.sock:/var/run/docker.sock
You may add a specific tag for your runner too with --tag-list "mytag", just make sure to actually have it on your repository otherwise it may prevent the runner from starting.
Last step, let's initialize the Gitlab Runner with Restart Always. This will ensure that when the Docker is initialized, it will already start the runner.
docker run -d --name gitlab-runner --restart always \ -v /opt/gitlab-runner:/etc/gitlab-runner \ -v /var/run/docker.sock:/var/run/docker.sock \ gitlab/gitlab-runner:latest
It works!
It really does, just push your commits and see thing happening!