Minio部署安装-直装/容器/单点/集群

原创我为啥没洁癖 2023-06-02

1706

二、安装部署

Minio生产至少部署4个机器4个节点，每个节点4driver，挂掉一台机器集群依然可以读写，挂掉两台机器集群依然可读。

2.1 直装单点

2.1.1 Server

下载，下载二进制文件

#amd64
wget https://dl.minio.org.cn/server/minio/release/linux-amd64/minio
#arm64
wget https://dl.minio.org.cn/server/minio/release/linux-arm64/minio
#执行权限
chmod +x minio
#放入执行路径
sudo mv minio /usr/local/bin/

启动

#创建目录
mkdir /root/minio
#启动服务端，设置管理员账户密码,暴露服务端口9000,/opt/minio文件目录,控制台端口9001
MINIO_ROOT_USER=admin MINIO_ROOT_PASSWORD=xrj12345678 minio server /opt/minio --address '192.168.10.109:9000' --console-address :9001

2.1.2 Client

下载

wget https://dl.minio.org.cn/client/mc/release/linux-amd64/mc
chmod +x mc
sudo mv mc /usr/local/bin/mc

别名设置

#将mc命令时local指定http://127.0.0.1:9000 admin xrj12345678
mc alias set local http://127.0.0.1:9000 admin xrj12345678
#查看服务端信息
mc admin info local

2.2 直装集群

2.2.1 部署方式

两台机器部署2节点，一个节点一个driver挂掉一台能读不能写。

#机器1,/opt/minio作为文件存储目录
192.168.10.109  /data/minio
#机器2
192.168.10.110 /data/minio

2.2.2 配置服务器

创建挂载目录和配置目录，两台机器都执行

新版要求挂载目录不能是根目录，新建磁盘分区挂载到/data

mkdir -p /data/minio && mkdir -p /etc/minio

下载minio执行文件，和单节点操作一致

wget https://dl.minio.org.cn/server/minio/release/linux-amd64/minio
#执行权限
chmod +x minio
#放入执行路径
sudo mv minio /usr/local/bin/

集群启动脚本。192.168.10.109和192.168.10.110分别创建run.sh。--config-dir指定集群配置目录

mkdir /root/minio
vim /root/minio/run.sh
chmod +x /root/minio/run.sh

#!/bin/bash

export MINIO_ROOT_USER=admin
export MINIO_ROOT_PASSWORD=xrj12345678
minio server --config-dir /etc/minio \
--address "0.0.0.0:9000" --console-address ":9001" \
http://192.168.10.109/data/minio \
http://192.168.10.110/data/minio

系统service

vim /usr/lib/systemd/system/minio.service

[Unit]
Description=Minio service
Documentation=https://docs.minio.io/

[Service]
WorkingDirectory=/root/minio/
ExecStart=/root/minio/run.sh
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

启动服务

#加载并启动服务
systemctl daemon-reload && systemctl start minio
#查看服务信息
systemctl status minio
#关闭服务
systemctl stop minio
#服务开机启动
systemctl enable minio
#关闭服务开机启动
systemctl disable minio

2.3 Docker单点

2.3.1 拉取镜像

docker pull minio/minio:RELEASE.2023-05-27T05-56-19Z

2.3.2 运行服务

自定义密码最小8位

docker run -d --restart always \
  -p 9000:9000 \
  -p 9001:9001 \
  --name minio \
  -v /opt/minio/data:/data \
  -e "MINIO_ROOT_USER=admin" \
  -e "MINIO_ROOT_PASSWORD=xrj12345678" \
  minio/minio:RELEASE.2023-05-27T05-56-19Z \
  server /data --console-address ":9001"

2.4 Docker-Compose单机器集群

在192.168.10.109上启动4个minio实例，每个实例两个driver，使用nginx负载均衡。

2.4.1 docker-compose.yml

version: '3.3'

services:
  minio1:
    image: minio/minio:RELEASE.2023-05-27T05-56-19Z
    command: server --console-address ":9001" http://minio{1...4}/data{1...2}
    expose:
      - "9000"
      - "9001"
    environment:
      MINIO_ROOT_USER: admin
      MINIO_ROOT_PASSWORD: xrj12345678
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3
    hostname: minio1
    volumes:
      - data1-1:/data1
      - data1-2:/data2

  minio2:
    image: minio/minio:RELEASE.2023-05-27T05-56-19Z
    command: server --console-address ":9001" http://minio{1...4}/data{1...2}
    expose:
      - "9000"
      - "9001"
    environment:
      MINIO_ROOT_USER: admin
      MINIO_ROOT_PASSWORD: xrj12345678
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3
    hostname: minio2
    volumes:
      - data2-1:/data1
      - data2-2:/data2

  minio3:
    image: minio/minio:RELEASE.2023-05-27T05-56-19Z
    command: server --console-address ":9001" http://minio{1...4}/data{1...2}
    expose:
      - "9000"
      - "9001"
    environment:
      MINIO_ROOT_USER: admin
      MINIO_ROOT_PASSWORD: xrj12345678
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3
    hostname: minio3
    volumes:
      - data3-1:/data1
      - data3-2:/data2

  minio4:
    image: minio/minio:RELEASE.2023-05-27T05-56-19Z
    command: server --console-address ":9001" http://minio{1...4}/data{1...2}
    expose:
      - "9000"
      - "9001"
    environment:
      MINIO_ROOT_USER: admin
      MINIO_ROOT_PASSWORD: xrj12345678
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3
    hostname: minio4
    volumes:
      - data4-1:/data1
      - data4-2:/data2

  nginx:
    image: nginx:1.19.2-alpine
    hostname: nginx
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    ports:
      - "9000:9000"
      - "9001:9001"
    depends_on:
      - minio1
      - minio2
      - minio3
      - minio4

volumes:
  data1-1:
  data1-2:
  data2-1:
  data2-2:
  data3-1:
  data3-2:
  data4-1:
  data4-2:

2.4.2 nginx.conf

放在和docker-compose.yml同级目录，这里nginx所在的节点无法高可用，配置文件只在109主机上，只会调度到109，109挂了就没法访问了。

可以考虑将nginx配置构建一个新的镜像在每个主机节点上都起一个副本，api访问配置所有的nginx。

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  4096;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;
    sendfile        on;
    keepalive_timeout  65;

    # include /etc/nginx/conf.d/*.conf;

    upstream minio {
        server minio1:9000;
        server minio2:9000;
        server minio3:9000;
        server minio4:9000;
    }

    upstream console {
        ip_hash;
        server minio1:9001;
        server minio2:9001;
        server minio3:9001;
        server minio4:9001;
    }

    server {
        listen       9000;
        listen  [::]:9000;
        server_name  localhost;

        #允许特殊请求头
        ignore_invalid_headers off;
        #不限制上传文件大小
        client_max_body_size 0;
        # To disable buffering
        proxy_buffering off;
        proxy_request_buffering off;

        location / {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;

            proxy_connect_timeout 300;
            # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            chunked_transfer_encoding off;

            proxy_pass http://minio;
        }
    }

    server {
        listen       9001;
        listen  [::]:9001;
        server_name  localhost;

        ignore_invalid_headers off;
        client_max_body_size 0;
        proxy_buffering off;
        proxy_request_buffering off;

        location / {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-NginX-Proxy true;

            real_ip_header X-Real-IP;

            proxy_connect_timeout 300;
            
            # To support websocket
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            
            chunked_transfer_encoding off;

            proxy_pass http://console;
        }
    }
}

2.4.3 启动服务

docker compose up -d
docker compose down

2.5 Docker-Swarm多机器集群

这里使用Swarm部署集群，4机器4节点，每个节点2个driver，使用nginx负载均衡。使用node.labels中的自定义属性配合deploy的placement属性的约束条件让不同minio服务节点调度到固定的节点。如：minio1服务固定调度到主机minio-1,其他节点类似，当节点故障恢复后不会自动负载服务节点导致数据漂移丢失。

2.5.1 配置服务器

minio-1 192.168.10.109
minio-2 192.168.10.110
minio-3 192.168.10.111
minio-4 192.168.10.112

hostnamectl set-hostname minio-1
hostnamectl set-hostname minio-2
hostnamectl set-hostname minio-3
hostnamectl set-hostname minio-4

cat >> /etc/hosts <<EOF
minio-1 192.168.10.109
minio-2 192.168.10.110
minio-3 192.168.10.111
minio-4 192.168.10.112
EOF

2.5.2 初始化Swarm集群

192.168.10.109当主节点。

主节点初始化集群

docker swarm init --advertise-addr 192.168.10.109

从节点加入集群

docker swarm join --token SWMTKN-1-5w1sqjo4pllh3btuihkpd027p0dnjpsd2t65zcoi9wrq2ahysa-5kh17licwa5069xoyah1tlbt7 192.168.10.109:2377

将节点2和3升级成master节点，升级的worker节点管理状态从空成为Reachable

docker node promote mi78x5l6hk8dbx31m4suir67x
docker node promote ihdukwd6vp84fvy3ida89fm1a

2.5.3 docker-swarm.yml

使用docker volumes进行数据挂载

version: '3.7'

x-minio-common: &minio-common
  image: minio/minio:RELEASE.2023-05-27T05-56-19Z
  command: server --console-address ":9001" http://minio{1...4}/data{1...2}
  expose:
    - "9000"
    - "9001"
  environment:
    MINIO_ROOT_USER: admin
    MINIO_ROOT_PASSWORD: xrj12345678
  deploy:
    update_config:
      parallelism: 1
      delay: 10s
      order: stop-first
    restart_policy:
      condition: on-failure
  healthcheck:
    test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
    interval: 30s
    timeout: 20s
    retries: 3

services:
  minio1:
    <<: *minio-common
    hostname: minio1
    deploy:
      placement:
        constraints:
          - "node.labels.minio1==true"
    volumes:
      - data1-1:/data1
      - data1-2:/data2

  minio2:
    <<: *minio-common
    hostname: minio2
    deploy:
      placement:
        constraints:
          - "node.labels.minio2==true"
    volumes:
      - data2-1:/data1
      - data2-2:/data2
      
  minio3:
    <<: *minio-common
    hostname: minio3
    deploy:
      placement:
        constraints:
          - "node.labels.minio3==true"
    volumes:
      - data3-1:/data1
      - data3-2:/data2 
      
  minio4:
    <<: *minio-common
    hostname: minio4
    deploy:
      placement:
        constraints:
          - "node.labels.minio4==true"
    volumes:
      - data4-1:/data1
      - data4-2:/data2      

  nginx:
    image: nginx:1.19.2-alpine
    hostname: nginx
    volumes:
      - /root/minio/swarm/nginx.conf:/etc/nginx/nginx.conf:ro
    ports:
      - "9000:9000"
      - "9001:9001"
    deploy:
      update_config:
        parallelism: 1
        delay: 10s
        order: stop-first
      restart_policy:
        condition: on-failure
    depends_on:
      - minio1
      - minio2
      - minio3
      - minio4

volumes:
  data1-1:
  data1-2:
  data2-1:
  data2-2:
  data3-1:
  data3-2:
  data4-1:
  data4-2:

2.5.4 nginx.conf

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  4096;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;
    sendfile        on;
    keepalive_timeout  65;

    # include /etc/nginx/conf.d/*.conf;

    upstream minio {
        server minio1:9000;
        server minio2:9000;
        server minio3:9000;
        server minio4:9000;
    }

    upstream console {
        ip_hash;
        server minio1:9001;
        server minio2:9001;
        server minio3:9001;
        server minio4:9001;
    }

    server {
        listen       9000;
        listen  [::]:9000;
        server_name  localhost;

        #允许特殊请求头
        ignore_invalid_headers off;
        #不限制上传文件大小
        client_max_body_size 0;
        # To disable buffering
        proxy_buffering off;
        proxy_request_buffering off;

        location / {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;

            proxy_connect_timeout 300;
            # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            chunked_transfer_encoding off;

            proxy_pass http://minio;
        }
    }

    server {
        listen       9001;
        listen  [::]:9001;
        server_name  localhost;

        ignore_invalid_headers off;
        client_max_body_size 0;
        proxy_buffering off;
        proxy_request_buffering off;

        location / {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-NginX-Proxy true;

            real_ip_header X-Real-IP;

            proxy_connect_timeout 300;
            
            # To support websocket
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            
            chunked_transfer_encoding off;

            proxy_pass http://console;
        }
    }
}

2.5.5 启动服务

配置服务调度标签，会在对应的node上的labels属性添加上属性，使用部署文件中下面的约束来让对应的服务调度到固定的节点。

deploy:
      placement:
        constraints:
          - "node.labels.minio1==true"

给node加上标签值，如minio-1节点的标签为minio1==true

docker node update --label-add minio1=true minio-1
docker node update --label-add minio2=true minio-2
docker node update --label-add minio3=true minio-3
docker node update --label-add minio4=true minio-4