在隔离环境中搭建 Mirror 站点

搭建 Mirror 站点的工具有很多，但从根源上讲，他们都使用了 rsync 作为远程同步的工具和手段，因此可称这一类镜像工具为  rsync  镜像工具。在诸多 rsync  镜像工具，使用和配置的便捷性当属 清华大学 开放出的  tunasync 工具，当然它也是一个 Golang 开发而成的 rsync  镜像工具。

之所以会展示这篇操作文档，是因为今天有个我去年带过的学生需要在隔离环境中搭建一个 对阿里云CentOS站点进行镜像的站点，由于操作不当导致镜像失败、且无法Web访问服务。

以下是我使用 tunasync 搭建隔离环境中的 Mirror 站点的过程：

OS环境为 CentOS 7 1804 64位

先对 CentOS 7 做初始化，初始化脚本如下：

mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.original
wget -O /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
yum clean all && yum makecache
yum -y update
systemctl stop firewalld && systemctl disable firewalld
sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config && setenforce 0
wget https://mirrors.aliyun.com/epel/epel-release-latest-7.noarch.rpm
yum install -y https://mirrors.aliyun.com/epel/epel-release-latest-7.noarch.rpm
wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
sed -i 's|^#baseurl=https://download.fedoraproject.org/pub|baseurl=https://mirrors.aliyun.com|' /etc/yum.repos.d/epel*
sed -i 's|^metalink|#metalink|' /etc/yum.repos.d/epel*
yum clean all && yum makecache
yum -y update
yum install -y ntpdate
wget https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm
yum localinstall -y google-chrome-stable_current_x86_64.rpm
ntpdate cn.ntp.org.cn
yum install -y tree
who | grep googlebigtable | sed -n '1p' | cut -d' ' -f 1 | sort | uniq
DescriptionUser=$(who | grep googlebigtable | sed -n '1p' | cut -d' ' -f 1 | sort | uniq)
echo $DescriptionUser
echo "$DescriptionUser  ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
init 6

 

1. 创建用户mirrors及用户组mirrorgroup

[googlebigtable@localhost tunasyncMirrorWeb]$ sudo su

[root@localhost tunasyncMirrorWeb]# groupadd mirrorgroup

[root@localhost tunasyncMirrorWeb]# useradd -g mirrorgroup mirrors

[root@localhost tunasyncMirrorWeb]# id mirrors

uid=1001(mirrors) gid=1001(mirrorgroup) groups=1001(mirrorgroup)

[root@localhost tunasyncMirrorWeb]# passwd mirrors

Changing password for user mirrors.

New password:

BAD PASSWORD: The password fails the dictionary check - it is based on a dictionary word

Retype new password:

passwd: all authentication tokens updated successfully.

[root@localhost tunasyncMirrorWeb]# id mirrors

uid=1001(mirrors) gid=1001(mirrorgroup) groups=1001(mirrorgroup)

[root@localhost tunasyncMirrorWeb]# 

 

1. 建立应用程序及数据目录

[root@localhost tunasyncMirrorWeb]# mkdir -p /opt/mirrors/tunasync/{conf,db,log}

[root@localhost tunasyncMirrorWeb]# ls -F /opt/mirrors/tunasync/

conf/  db/  log/

[root@localhost tunasyncMirrorWeb]#

[root@localhost tunasyncMirrorWeb]# mkdir -p /opt/mirrors/mirrorsData

[root@localhost tunasyncMirrorWeb]# ls -F /opt/mirrors/

mirrorsData/  tunasync/

[root@localhost tunasyncMirrorWeb]# chown -R mirrors:mirrorgroup /opt/mirrors/mirrorsData

[root@localhost tunasyncMirrorWeb]# ll /opt/mirrors/

total 0

drwxr-xr-x 2 mirrors mirrorgroup  6 Sep 20 06:03 mirrorsData

drwxr-xr-x 4 root    root        28 Sep 20 06:00 tunasync

[root@localhost tunasyncMirrorWeb]#

 

1. 部署 tunasync

[root@localhost tunasyncMirrorWeb]# ls -F

tunasync-linux-amd64-bin.tar.gz

[root@localhost tunasyncMirrorWeb]# chmod 777 tunasync-linux-amd64-bin.tar.gz

[root@localhost tunasyncMirrorWeb]# tar -xzvf tunasync-linux-amd64-bin.tar.gz -C /opt/mirrors/tunasync/

tunasync

tunasynctl

[root@localhost tunasyncMirrorWeb]# ls -F /opt/mirrors/tunasync/

conf/  db/  tunasync*  tunasynctl*

[root@localhost tunasyncMirrorWeb]#

 

1. 配置环境变量

[root@localhost tunasyncMirrorWeb]# chown -R mirrors:mirrorgroup /opt/mirrors

[root@localhost tunasyncMirrorWeb]# ll /opt/mirrors

total 0

drwxr-xr-x 2 mirrors mirrorgroup  6 Sep 20 06:03 mirrorsData

drwxr-xr-x 4 mirrors mirrorgroup 62 Sep 20 06:07 tunasync

[root@localhost tunasyncMirrorWeb]# ll /opt/mirrors/tunasync/

total 28032

drwxr-xr-x 2 mirrors mirrorgroup        6 Sep 20 06:00 conf

drwxr-xr-x 2 mirrors mirrorgroup        6 Sep 20 06:00 db

-rwxr-xr-x 1 mirrors mirrorgroup 17930410 Sep 11 06:24 tunasync

-rwxr-xr-x 1 mirrors mirrorgroup 10770787 Sep 11 06:24 tunasynctl

[root@localhost tunasyncMirrorWeb]# su - mirrors

[mirrors@localhost ~]$ cat -n ~/.bash_profile

     1 # .bash_profile

     2

     3 # Get the aliases and functions

     4 if [ -f ~/.bashrc ]; then

     5 . ~/.bashrc

     6 fi

     7

     8 # User specific environment and startup programs

     9

    10 PATH=$PATH:$HOME/.local/bin:$HOME/bin

    11

    12 export PATH

[mirrors@localhost ~]$ cp ~/.bash_profile{,.original}

[mirrors@localhost ~]$ echo "TUNAPATH=/home/mirrors/tunasync" >> ~/.bash_profile

[mirrors@localhost ~]$ echo "PATH=$PATH:$TUNAPATH" >> ~/.bash_profile

[mirrors@localhost ~]$ cat -n ~/.bash_profile

     1 # .bash_profile

     2

     3 # Get the aliases and functions

     4 if [ -f ~/.bashrc ]; then

     5 . ~/.bashrc

     6 fi

     7

     8 # User specific environment and startup programs

     9

    10 PATH=$PATH:$HOME/.local/bin:$HOME/bin

    11

    12 export PATH

    13 TUNAPATH=/home/mirrors/tunasync

    14 PATH=/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/mirrors/.local/bin:/home/mirrors/bin:

[mirrors@localhost ~]$ source ~/.bash_profile

[mirrors@localhost ~]$

 

1. 配置 tunasync 的同步服务

[mirrors@localhost ~]$ ls -F /opt/mirrors/tunasync/

conf/  db/  tunasync*  tunasynctl*

[mirrors@localhost ~]$

[mirrors@localhost ~]$ tee > /opt/mirrors/tunasync/conf/manager.conf << EOF

> manager.conf

>

> debug = false

>

> [server]

> addr = "127.0.0.1"

> port = 14242

> ssl_cert = ""

> ssl_key = ""

>

> [files]

> db_type = "bolt"

> db_file = "/opt/mirrors/tunasync/db/manager.db"

> ca_cert = ""

> EOF

 

按需创建被同步的镜像配置文件，此处以 CentOS 镜像为例：

 

[mirrors@localhost ~]$ tee > /opt/mirrors/tunasync/conf/worker-centos.conf << EOF

> [global]

> name = "centos_worker"

> log_dir = "/opt/mirrors/tunasync/log/{{.Name}}"

> mirror_dir = "/opt/mirrors/mirrorsData"

> concurrent = 10

> interval = 1440

>

> [manager]

> api_base = "http://localhost:14242"

> token = "some_token"

> ca_cert = ""

>

> [cgroup]

> enable = false

> base_path = "/sys/fs/cgroup"

> group = "tunasync"

>

> [server]

> hostname = "localhost"

> listen_addr = "127.0.0.1"

> listen_port = 16010

> ssl_cert = ""

> ssl_key = ""

>

> [[mirrors]]

> name = "centos"

> provider = "rsync"

> upstream = "rsync://mirrors.aliyun.com/centos/"

> use_ipv6 = false

> EOF

[mirrors@localhost ~]$

mirrors@localhost ~]$ ls -F /opt/mirrors/tunasync/

conf/  db/  log/  tunasync*  tunasynctl*

[mirrors@localhost ~]$ ls -F /opt/mirrors/tunasync/conf/

manager.conf  worker-centos.conf

[mirrors@localhost ~]$

 

1. 启动 tunasync同步程序

启动 tunasync 需先启动 manager，后启动 worker。为了便于监控系统进程情况，建立 /opt/mirrors/tunasync/log/plog/ 目录，所有进程的工作日志在该目录中。

用 mirrors 用户执行：

[mirrors@localhost ~]$ /opt/mirrors/tunasync/tunasync manager --config /opt/mirrors/tunasync/conf/manager.conf >> /opt/mirrors/tunasync/log/plog/manager.log &

[1] 60268

[mirrors@localhost ~]$ /opt/mirrors/tunasync/tunasync worker --config /opt/mirrors/tunasync/conf/worker-centos.conf >> /opt/mirrors/tunasync/log/plog/worker-centos.log &

[2] 60284

[mirrors@localhost ~]$

 

1. 为内网用户开放 web 访问服务

安装 apache 服务器:

[mirrors@localhost ~]$ su root

Password:

[root@localhost mirrors]# yum -y install httpd

Loaded plugins: fastestmirror, langpacks

Loading mirror speeds from cached hostfile

Installed:

  httpd.x86_64 0:2.4.6-93.el7.centos                                                                                                                 

 

Dependency Installed:

  httpd-tools.x86_64 0:2.4.6-93.el7.centos                                       mailcap.noarch 0:2.1.41-2.el7                                      

 

Complete!

[root@localhost mirrors]#

[root@localhost mirrors]# tree /etc/httpd/

/etc/httpd/

├── conf

│   ├── httpd.conf

│   └── magic

├── conf.d

│   ├── autoindex.conf

│   ├── README

│   ├── userdir.conf

│   └── welcome.conf

├── conf.modules.d

│   ├── 00-base.conf

│   ├── 00-dav.conf

│   ├── 00-lua.conf

│   ├── 00-mpm.conf

│   ├── 00-proxy.conf

│   ├── 00-systemd.conf

│   └── 01-cgi.conf

├── logs -> ../../var/log/httpd

├── modules -> ../../usr/lib64/httpd/modules

└── run -> /run/httpd

 

6 directories, 13 files

[root@localhost mirrors]# cp /etc/httpd/conf/httpd.conf{,.original}

[root@localhost mirrors]#

[mirrors@localhost ~]$ grep "^\s*[^# \t].*$" /etc/httpd/conf/httpd.conf | cat

ServerRoot "/etc/httpd"

Listen 80

Include conf.modules.d/*.conf

User apache

Group apache

ServerAdmin root@localhost

<Directory />

    AllowOverride none

    Require all denied

</Directory>

DocumentRoot "/var/www/html"

<Directory "/var/www">

    AllowOverride None

    Require all granted

</Directory>

<Directory "/var/www/html">

    Options Indexes FollowSymLinks

    AllowOverride None

    Require all granted

</Directory>

<IfModule dir_module>

    DirectoryIndex index.html

</IfModule>

<Files ".ht*">

    Require all denied

</Files>

ErrorLog "logs/error_log"

LogLevel warn

<IfModule log_config_module>

    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

    LogFormat "%h %l %u %t \"%r\" %>s %b" common

    <IfModule logio_module>

      LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio

    </IfModule>

    CustomLog "logs/access_log" combined

</IfModule>

<IfModule alias_module>

    ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"

</IfModule>

<Directory "/var/www/cgi-bin">

    AllowOverride None

    Options None

    Require all granted

</Directory>

<IfModule mime_module>

    TypesConfig /etc/mime.types

    AddType application/x-compress .Z

    AddType application/x-gzip .gz .tgz

    AddType text/html .shtml

    AddOutputFilter INCLUDES .shtml

</IfModule>

AddDefaultCharset UTF-8

<IfModule mime_magic_module>

    MIMEMagicFile conf/magic

</IfModule>

EnableSendfile on

IncludeOptional conf.d/*.conf

[mirrors@localhost ~]$

修改/etc/httpd/conf/httpd.conf后为：

[mirrors@localhost ~]$ grep "^\s*[^# \t].*$" /etc/httpd/conf/httpd.conf | cat

ServerRoot "/etc/httpd"

Listen 80

Include conf.modules.d/*.conf

User apache

Group apache

ServerAdmin root@localhost

<Directory />

    AllowOverride none

    Require all denied

</Directory>

DocumentRoot "/opt/mirrors/mirrorsData/"

<Directory "/var/www">

    AllowOverride None

    Require all granted

</Directory>

<Directory "/opt/mirrors/mirrorsData/">

    Options Indexes FollowSymLinks

    AllowOverride None

    Require all granted

</Directory>

<IfModule dir_module>

    DirectoryIndex index.html

</IfModule>

<Files ".ht*">

    Require all denied

</Files>

ErrorLog "logs/error_log"

LogLevel warn

<IfModule log_config_module>

    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

    LogFormat "%h %l %u %t \"%r\" %>s %b" common

    <IfModule logio_module>

      LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio

    </IfModule>

    CustomLog "logs/access_log" combined

</IfModule>

<IfModule alias_module>

    ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"

</IfModule>

<Directory "/var/www/cgi-bin">

    AllowOverride None

    Options None

    Require all granted

</Directory>

<IfModule mime_module>

    TypesConfig /etc/mime.types

    AddType application/x-compress .Z

    AddType application/x-gzip .gz .tgz

    AddType text/html .shtml

    AddOutputFilter INCLUDES .shtml

</IfModule>

AddDefaultCharset UTF-8

<IfModule mime_magic_module>

    MIMEMagicFile conf/magic

</IfModule>

EnableSendfile on

IncludeOptional conf.d/*.conf

[mirrors@localhost ~]$

[mirrors@localhost ~]$ chmod -R 755 /opt/mirrors/mirrorsData

[mirrors@localhost ~]$ systemctl start httpd.service

[mirrors@localhost ~]$ systemctl enable httpd.service

Created symlink from /etc/systemd/system/multi-user.target.wants/httpd.service to /usr/lib/systemd/system/httpd.service.

[mirrors@localhost ~]$

[root@localhost Public]# chown -R mirrors:mirrorgroup /opt/mirrors/mirrorsData/index.html

[mirrors@localhost ~]$ cat /opt/mirrors/mirrorsData/index.html

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml">

<head>

    <meta http-equiv="content-type" content="text/html; charset=utf-8"/>

    <link rel="stylesheet" type="text/css" href="/.resource/css/mirror.css" media="screen" />

    <script type="text/javascript" src=".resource/js/jquery-1.11.3.min.js" ></script>

    <script type="text/javascript" src=".resource/js/mirror.js" ></script>

    <script type="text/javascript" src=".resource/js/main.js" ></script>

    

    <title>CentOS</title>

</head>

 

<body>

 

<h1>欢迎访问CentOS镜像站</h1>

 

<table id="distro-table" cellpadding="0" cellspacing="0">

    <colgroup>

        <col width="12%"/>

        <col width="15%"/>

        <col width="35%"/>

        <col width="10%"/>

        <col width="8%"/>

        <col width="20%"/>

    </colgroup>

    <thead>

        <tr>

            <th>镜像名</th>

            <th>上次更新时间</th>

            <th>更新源</th>

            <th>状态</th>

            <th>镜像大小</th>

            <th>使用帮助</th>

        </tr>

    </thead>

    

    <tbody>

        <tr class="odd">

            <td><a href="/centos/">centos/</a></td>

            <td class="centos update-time">N/A</td>

            <td class="centos upstream">N/A</td>

            <td class="centos sync-status">N/A</td>

            <td class="centos mirror-size">N/A</td>

            <td><a href="/.help/centos.html">Centos/Redhat 使用帮助</a></td>

        </tr>

    </tbody>

</table>

<div id="footer">

    

</div>

 

</body>

</html>

[mirrors@localhost ~]$

[mirrors@localhost ~]$ chmod -R 755 /opt/mirrors/mirrorsData

 

至此 Mirror 站点搭建完成，可以通过浏览器直接访问  站点 IP 访问 ：