Docker + Gitlab + Gitlab CI（一）

迈不过友情╰ 2023-06-12 06:24 61阅读 0赞

本文通过Docker + Gitlab + Gitlab CI实现容器的自动化部署。

环境如下：

<table> 
 <thead> 
  <tr> 
   <th align="center">ip</th> 
   <th align="center">role</th> 
   <th align="center">domain</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td align="center">192.168.1.51</td> 
   <td align="center">prod server</td> 
   <td align="center">none</td> 
  </tr> 
  <tr> 
   <td align="center">192.168.1.55</td> 
   <td align="center">dns</td> 
   <td align="center">none</td> 
  </tr> 
  <tr> 
   <td align="center">192.168.1.56</td> 
   <td align="center">gitlab ci/dev server</td> 
   <td align="center">none</td> 
  </tr> 
  <tr> 
   <td align="center">192.168.1.57</td> 
   <td align="center">gitlab server</td> 
   <td align="center">gitlab.lzxlinux.cn</td> 
  </tr> 
  <tr> 
   <td align="center">192.168.1.59</td> 
   <td align="center">harbor server</td> 
   <td align="center">harbor.lzxlinux.cn</td> 
  </tr> 
 </tbody> 
</table>

> 注意：为了方便后面实验，建议自定义的域名不要与外部域名冲突，否则在连接外网情况下容易解析错误。

--------------------

### 搭建Harbor服务器 ###

Harbor是一个用于存储Docker镜像的企业级Registry服务。

*  准备工作：

# systemctl stop firewalld && systemctl disable firewalld
    
    # setenforce 0 && sed -i 's/=enforcing/=disabled/g' /etc/selinux/config

*  安装docker：

# curl http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -o /etc/yum.repos.d/docker.repo
    
    # yum makecache fast
    
    # yum install -y docker-ce
    
    # systemctl start docker && systemctl enable docker

提高docker pull速度

# curl -sSL https://get.daocloud.io/daotools/set_mirror.sh | sh -s http://f1361db2.m.daocloud.io
    
    # systemctl restart docker

*  下载最新的docker-compose二进制文件：

# curl -L https://github.com/docker/compose/releases/download/1.24.1/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose
    
    # chmod +x /usr/local/bin/docker-compose

*  下载harbor离线安装包：

github地址：https://github.com/goharbor/harbor/releases

# cd /software
    
    # wget https://storage.googleapis.com/harbor-releases/release-1.9.0/harbor-offline-installer-v1.9.1.tgz
    
    # tar zxf harbor-offline-installer-v1.9.1.tgz

*  安装harbor：

# cd harbor/
    
    # vim harbor.yml
    
    hostname: harbor.lzxlinux.cn                #域名
    harbor_admin_password: Harbor12345              #admin用户初始密码
    data_volume: /data              #数据存储路径，自动创建
    log:
      level: info
      local:
        rotate_count: 50
        rotate_size: 200M
        location: /var/log/harbor               #日志路径
        
    # sh install.sh

*  登录报错：

# echo '192.168.1.59 harbor.lzxlinux.cn' >> /etc/hosts
    
    # docker login harbor.lzxlinux.cn
    Username: admin
    Password: 
    Error response from daemon: Get https://harbor.lzxlinux.cn/v2/: dial tcp 192.168.1.59:443: connect: connection refused

*  解决：

# vim /usr/lib/systemd/system/docker.service
    ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --insecure-registry=harbor.lzxlinux.cn
    
    # systemctl daemon-reload && systemctl restart docker.service

harbor服务器

# cd /software/harbor
    
    # docker-compose down -v
    
    # docker-compose up -d

*  重新登录：

# docker login harbor.lzxlinux.cn
    Username: admin
    Password: 
    WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
    Configure a credential helper to remove this warning. See
    https://docs.docker.com/engine/reference/commandline/login/#credentials-store
    
    Login Succeeded

*  推送镜像：

# docker pull busybox
    
    # docker tag busybox:latest harbor.lzxlinux.cn/public/busybox
    
    # docker push harbor.lzxlinux.cn/public/busybox

刷新harbor页面，可以看到刚推送到harbor仓库的镜像

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70]

--------------------

### 搭建Gitlab服务器 ###

以Centos7为例，准备一台至少内存为4G的机器。

*  准备工作：

# systemctl stop firewalld && systemctl disable firewalld
    
    # setenforce 0 && sed -i 's/=enforcing/=disabled/g' /etc/selinux/config

*  安装依赖软件：

# yum install -y curl policycoreutils openssh-server openssh-clients postfix
    
    # systemctl start postfix && systemctl enable postfix #启动postfix邮件服务

*  设置gitlab安装源：

如果在国内的话，可以尝试使用清华大学的源。

# vim /etc/yum.repos.d/gitlab-ce.repo
    
    [gitlab-ce]
    name=Gitlab CE Repository
    baseurl=https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el$releasever/
    gpgcheck=0
    enabled=1

如果在国外的话，可以使用

# curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ee/script.rpm.sh | sudo bash

*  添加本地dns：

# vim /etc/hosts
    
    192.168.1.57 gitlab.lzxlinux.cn

在Windows电脑hosts文件中添加本地dns：

192.168.1.57 gitlab.lzxlinux.cn

*  安装gitlab-ce：

# yum install -y gitlab-ce

*  证书创建与配置加载：

# mkdir -p /etc/gitlab/ssl
    
    # openssl genrsa -out "/etc/gitlab/ssl/gitlab.lzxlinux.cn.key" 2048
    
    # openssl req -new -key "/etc/gitlab/ssl/gitlab.lzxlinux.cn.key" -out "/etc/gitlab/ssl/gitlab.lzxlinux.cn.csr"
    
    Country Name (2 letter code) [XX]:cn
    State or Province Name (full name) []:hz
    Locality Name (eg, city) [Default City]:hz
    Organization Name (eg, company) [Default Company Ltd]:
    Organizational Unit Name (eg, section) []:
    Common Name (eg, your name or your server's hostname) []:gitlab.lzxlinux.cn Email Address []:admin@lzxlinux.cn Please enter the following 'extra' attributes
    to be sent with your certificate request
    A challenge password []:123456
    An optional company name []:

# openssl x509 -req -days 365 -in "/etc/gitlab/ssl/gitlab.lzxlinux.cn.csr" -signkey "/etc/gitlab/ssl/gitlab.lzxlinux.cn.key" -out "/etc/gitlab/ssl/gitlab.lzxlinux.cn.crt"
    
    # openssl dhparam -out /etc/gitlab/ssl/dhparams.pem 2048
    
    # chmod 600 /etc/gitlab/ssl/*
    
    # ll /etc/gitlab/ssl
    total 16
    -rw------- 1 root root  424 Oct  29 10:45 dhparams.pem
    -rw------- 1 root root 1281 Oct  29 10:44 gitlab.lzxlinux.cn.crt
    -rw------- 1 root root 1074 Oct  29 10:38 gitlab.lzxlinux.cn.csr
    -rw------- 1 root root 1679 Oct  29 10:37 gitlab.lzxlinux.cn.key

*  nginx SSL代理服务配置：

# vim /etc/gitlab/gitlab.rb #修改下面内容
    
    external_url 'https://gitlab.lzxlinux.cn'
    nginx['redirect_http_to_https'] = true
    nginx['ssl_certificate'] = "/etc/gitlab/ssl/gitlab.lzxlinux.com.cn"
    nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/gitlab.lzxlinux.cn.key"
    nginx['ssl_dhparam'] = "/etc/gitlab/ssl/dhparams.pem"

*  初始化gitlab相关服务并完成安装：

# gitlab-ctl reconfigure
    
    # vim /var/opt/gitlab/nginx/conf/gitlab-http.conf #第一个 server_name gitlab.lzxlinux.cn; 下添加该行
    
    rewrite ^(.*)$ https://$host$1 permanent;
    
    # gitlab-ctl restart #重启gitlab

*  登陆和修改密码：

打开`https://gitlab.lzxlinux.cn/`修改root用户密码，然后使用root和新密码登陆。

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70 1]

--------------------

### 搭建Gitlab CI服务器 ###

Gitlab CI服务器建议另选一台服务器搭建，不要与Gitlab服务器部署在同一台机器上。

*  准备工作：

# systemctl stop firewalld && systemctl disable firewalld
    
    # setenforce 0 && sed -i 's/=enforcing/=disabled/g' /etc/selinux/config

*  安装docker：

提高docker pull速度

# curl -sSL https://get.daocloud.io/daotools/set_mirror.sh | sh -s http://f1361db2.m.daocloud.io
    
    # systemctl restart docker

*  安装gitlab ci runner：

# echo '54.153.54.194 packages.gitlab.com' >> /etc/hosts
    
    # curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-ci-multi-runner/script.rpm.sh | sudo bash
    
    # yum install -y gitlab-ci-multi-runner

查看是否运行正常

# gitlab-ci-multi-runner status
    gitlab-runner: Service is running!

*  设置docker权限：

为了能让gitlab-runner能正确的执行docker命令，需要把gitlab-runner用户添加到docker group里, 然后重启docker和gitlab ci runner。

# usermod -aG docker gitlab-runner
    
    # systemctl restart docker
    
    # gitlab-ci-multi-runner restart

*  注册到gitlab服务器：

到gitlab上查看runner的注册token

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70 2]

# gitlab-ci-multi-runner register
    
    Running in system-mode.                            
                                                       
    Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
    https://gitlab.lzxlinux.cn/                #输入gitlab url
    Please enter the gitlab-ci token for this runner:
    4kr9ZmLMWasYxqB2tSzQ                #输入token
    Please enter the gitlab-ci description for this runner:
    [worker4]:              #输入该runner的描述信息
    Please enter the gitlab-ci tags for this runner (comma separated):
    test,demo               #输入该runner的tags
    Whether to run untagged builds [true/false]:
    [false]:                #是否运行没有打上tag的构建
    Whether to lock Runner to current project [true/false]:
    [false]:                #是否将该runner锁定到当前项目
    
    ERROR: Registering runner... failed                 runner=4kr9ZmLM status=couldn't execute POST against https://gitlab.lzxlinux.cn/api/v4/runners: Post https://gitlab.lzxlinux.cn/api/v4/runners: x509: certificate signed by unknown authority
    PANIC: Failed to register this runner. Perhaps you are having network problems

报错，证书签名错误，这里因为前面gitlab服务器做了https。

*  解决证书报错：

# mkdir -p /etc/gitlab/ssl
    
    # scp root@gitlab.lzxlinux.cn:/etc/gitlab/ssl/gitlab.lzxlinux.cn.crt /etc/gitlab/ssl
    
    # gitlab-ci-multi-runner register \
      --tls-ca-file=/etc/gitlab/ssl/gitlab.lzxlinux.cn.crt \
      --url "https://gitlab.lzxlinux.cn/" \
      --registration-token "4kr9ZmLMWasYxqB2tSzQ" \
      --tag-list "test,demo" \
      --run-untagged \
      --locked="false" \
      --executor "shell"

# gitlab-ci-multi-runner list
    Listing configured runners                          ConfigFile=/etc/gitlab-runner/config.toml
    worker4                                             Executor=shell Token=aLzTn6bfk1tXNBLRBbSD URL=https://gitlab.lzxlinux.cn/

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70 3]

可以看到，成功地注册了一个runner到gitlab服务器。

*  演示gitlab runner工作：

gitlab上新建一个组`test`，在`test`组中新建一个项目`helloworld`，然后项目中新建一个`README.md`文件。

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70 4]

再新建一个`.gitlab-ci.yml`文件，

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70 5]

打开`CI/CD` → `流水线`，可以看到刚刚的提交已经成功完成，

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70 6]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70 7]

整个pipeline按顺序执行，如果前面的stage出错，后面的stage不会执行。

--------------------

### 搭建DNS服务器 ###

另选同网段的一台主机搭建一个dns服务器，让同网段任意机器及其上的容器都可以解析到自定义的域名（如 gitlab.lzxlinux.cn）。

首先，在gitlab ci服务器上把`gitlab.lzxlinux.cn`从`/etc/hosts`里删除，此时在gitlab ci服务器上是ping不通`gitlab.lzxlinux.cn`的。

*  准备工作：

# systemctl stop firewalld && systemctl disable firewalld
    
    # setenforce 0 && sed -i 's/=enforcing/=disabled/g' /etc/selinux/config

*  安装docker：

提高docker pull速度

# curl -sSL https://get.daocloud.io/daotools/set_mirror.sh | sh -s http://f1361db2.m.daocloud.io
    
    # systemctl restart docker

*  运行dns容器：

# docker run -d -p 53:53/tcp -p 53:53/udp --cap-add=NET_ADMIN --name dns-server andyshinn/dnsmasq
    
    # docker ps
    CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS              PORTS                                    NAMES
    bcaf93792614        andyshinn/dnsmasq         "dnsmasq -k"             10 seconds ago      Up 9 seconds        0.0.0.0:53->53/tcp, 0.0.0.0:53->53/udp   dns-server

*  配置dns服务：

# docker exec -it dns-server sh
    
    / # vi /etc/resolv.dnsmasq
    
    nameserver 223.5.5.5 nameserver 114.114.114.114
    
    / # vi /etc/dnsmasqhosts
    
    192.168.1.57 gitlab.lzxlinux.cn
    192.168.1.59 harbor.lzxlinux.cn
    
    / # vi /etc/dnsmasq.conf
    
    resolv-file=/etc/resolv.dnsmasq
    addn-hosts=/etc/dnsmasqhosts
    
    # docker restart dns-server

*  测试：

gitlab ci服务器上操作

# vim /etc/resolv.conf #增加一行，且放在公网dns前面
    
    nameserver 192.168.1.55             #该ip为dns容器所在主机ip
    nameserver 223.5.5.5
    
    # ping gitlab.lzxlinux.cn
    PING gitlab.lzxlinux.cn (192.168.1.57) 56(84) bytes of data.
    64 bytes from gitlab.lzxlinux.cn (192.168.1.57): icmp_seq=1 ttl=64 time=0.276 ms
    64 bytes from gitlab.lzxlinux.cn (192.168.1.57): icmp_seq=2 ttl=64 time=0.310 ms
    64 bytes from gitlab.lzxlinux.cn (192.168.1.57): icmp_seq=3 ttl=64 time=0.380 ms
    64 bytes from gitlab.lzxlinux.cn (192.168.1.57): icmp_seq=4 ttl=64 time=0.319 ms
    ^C
    --- gitlab.lzxlinux.cn ping statistics ---
    4 packets transmitted, 4 received, 0% packet loss, time 3001ms
    rtt min/avg/max/mdev = 0.276/0.321/0.380/0.039 ms
    
    # ping harbor.lzxlinux.cn
    PING harbor.lzxlinux.cn (192.168.1.59) 56(84) bytes of data.
    64 bytes from harbor.lzxlinux.cn (192.168.1.59): icmp_seq=1 ttl=64 time=0.307 ms
    64 bytes from harbor.lzxlinux.cn (192.168.1.59): icmp_seq=2 ttl=64 time=0.430 ms
    64 bytes from harbor.lzxlinux.cn (192.168.1.59): icmp_seq=3 ttl=64 time=0.383 ms
    64 bytes from harbor.lzxlinux.cn (192.168.1.59): icmp_seq=4 ttl=64 time=0.346 ms
    ^C
    --- harbor.lzxlinux.cn ping statistics ---
    4 packets transmitted, 4 received, 0% packet loss, time 3000ms
    rtt min/avg/max/mdev = 0.307/0.366/0.430/0.049 ms

# docker run -it --rm busybox sh
    
    / # ping gitlab.lzxlinux.cn
    PING gitlab.lzxlinux.cn (192.168.1.57): 56 data bytes
    64 bytes from 192.168.1.57: seq=0 ttl=63 time=0.388 ms
    64 bytes from 192.168.1.57: seq=1 ttl=63 time=0.472 ms
    ^C
    --- gitlab.lzxlinux.cn ping statistics ---
    2 packets transmitted, 2 packets received, 0% packet loss
    round-trip min/avg/max = 0.388/0.430/0.472 ms
    
    / # ping harbor.lzxlinux.cn
    PING harbor.lzxlinux.cn (192.168.1.59): 56 data bytes
    64 bytes from 192.168.1.59: seq=0 ttl=63 time=0.580 ms
    64 bytes from 192.168.1.59: seq=1 ttl=63 time=0.332 ms
    64 bytes from 192.168.1.59: seq=2 ttl=63 time=0.401 ms
    ^C
    --- harbor.lzxlinux.cn ping statistics ---
    3 packets transmitted, 3 packets received, 0% packet loss
    round-trip min/avg/max = 0.332/0.437/0.580 ms

可以看到，在搭建dns服务器后，即使gitlab ci服务器的`/etc/hosts`删除本地dns，主机和主机上的容器仍可以解析我们自定义的域名。

--------------------

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70]: /images/20230531/d5ae6b8102774ad5b8bf0551a3c84587.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70 1]: /images/20230531/69541fbc2a4c4790b6e89f2d057420c7.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70 2]: /images/20230531/23a63dde3da240e6a1dd30aa78b707bb.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70 3]: /images/20230531/0adfed41d9884819856d743a5d68e434.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70 4]: /images/20230531/79708e86a9c34ead9485f7a610c95f92.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70 5]: /images/20230531/1adece1715db4023903cc75f04aa12f7.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70 6]: /images/20230531/47ceefcc2c14441d8628b91dc9d5cbe7.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70 7]: /images/20230531/7b8debcb94e8472baa941c3de2ebd49e.png