03. docker-compose 启动 minio 多机集群

其他相关文章：
《docker-compose启动minio 单机》
《docker-compose启动minio 伪集群》
《docker-compose启动minio 多机集群》
《docker-compose启动minio （Version 2022-）》

【环境说明】

服务器	IP地址
minio1	10.10.239.66
minio2	10.10.239.68
minio3	10.10.239.69

1.【minio1】

• docker-compose.yml

version: '3.7'
services:
  minio1:
    image: harbocto.xxx.com.cn/public/minio:RELEASE.2021-02-01T22-56-52Z
    volumes:
      - ./data1-1:/data1
      - ./data1-2:/data2
    environment:
      MINIO_ROOT_USER: minio
      MINIO_ROOT_PASSWORD: liubei@2021
    network_mode: host
    command: server http://minio{1...3}/data{1...2}
    extra_hosts:
      - "minio1:10.10.239.66"
      - "minio2:10.10.239.68"
      - "minio3:10.10.239.69"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

说明：

• extra_hosts:
  定义容器中的hosts，使得上边 command 后边的地址可以被解析，否则你要写成：

command: server  http://10.10.239.66/data1 http://10.10.239.66/data2 http://10.10.239.68/data1  http://10.10.239.68/data2 http://10.10.239.69/data1 http://10.10.239.69/data2

• network_mode: host
  多机集群必须从host模式启动，否则磁盘会找不到。

2.【minio2】

• docker-compose.yml

version: '3.7'
services:
  minio2:
    image: harbocto.xxx.com.cn/public/minio:RELEASE.2021-02-01T22-56-52Z
    volumes:
      - ./data2-1:/data1
      - ./data2-2:/data2
    environment:
      MINIO_ROOT_USER: minio
      MINIO_ROOT_PASSWORD: liubei@2021
    network_mode: host
    command: server http://minio{1...3}/data{1...2}
    extra_hosts:
      - "minio1:10.10.239.66"
      - "minio2:10.10.239.68"
      - "minio3:10.10.239.69"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

3.【minio3】

• docker-compose.yml

version: '3.7'
services:
  minio3:
    image: harbocto.xxx.com.cn/public/minio:RELEASE.2021-02-01T22-56-52Z
    volumes:
      - ./data3-1:/data1
      - ./data3-2:/data2
#    ports:
#      - "8000:9000"
    environment:
      MINIO_ROOT_USER: minio
      MINIO_ROOT_PASSWORD: liubei@2021
    network_mode: host
    command: server http://minio{1...3}/data{1...2}
    extra_hosts:
      - "minio1:10.10.239.66"
      - "minio2:10.10.239.68"
      - "minio3:10.10.239.69"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

4.【nginx】

做负载的nginx容器，我只在minio1 上启动了一个
如果要做高可用，可以在每台minio上启动一个nginx，然后用Keepalive+VIP 来完成。

• docker-compose.yml

version: '3.7'
services:
  nginx:
    container_name: nginx
    image: harbocto.xxx.com.cn/public/nginx:1.19.2-alpine
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    ports:
      - "80:9000"

• nginx.conf

    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    # include /etc/nginx/conf.d/*.conf;

    upstream minio {
        server 10.10.239.66:9000;
        server 10.10.239.68:9000;
        server 10.10.239.69:9000;
    }

    server {
        listen       9000;
        listen  [::]:9000;
        server_name  localhost;

         # To allow special characters in headers
         ignore_invalid_headers off;
         # Allow any size file to be uploaded.
         # Set to a value such as 1000m; to restrict file size to a specific value
         client_max_body_size 0;
         # To disable buffering
         proxy_buffering off;

        location / {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;

            proxy_connect_timeout 300;
            # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            chunked_transfer_encoding off;

            proxy_pass http://minio;
        }
    }
}

5. 启动集群

• 在每个节点上启动minio 容器
• 再启动nginx 容器

6. version 2022 的问题

和之前的版本区别：启动时会随机分配一个控制台端口。

• 因为我们是容器启动，如果不用host模式，则需要固定该端口，在启动命令上添加 --console-address ":n" （n是指定端口号。）：

    command: server /data --console-address ":9999"

• 访问的url 也是 9999 端口

---