暂无图片
暂无图片
暂无图片
暂无图片
暂无图片

Minio部署安装-直装/容器/单点/集群

原创 我为啥没洁癖 2023-06-02
1417

二、安装部署

Minio生产至少部署4个机器4个节点,每个节点4driver,挂掉一台机器集群依然可以读写,挂掉两台机器集群依然可读。

2.1 直装单点

2.1.1 Server

下载,下载二进制文件

#amd64 wget https://dl.minio.org.cn/server/minio/release/linux-amd64/minio #arm64 wget https://dl.minio.org.cn/server/minio/release/linux-arm64/minio #执行权限 chmod +x minio #放入执行路径 sudo mv minio /usr/local/bin/

启动

#创建目录 mkdir /root/minio #启动服务端,设置管理员账户密码,暴露服务端口9000,/opt/minio文件目录,控制台端口9001 MINIO_ROOT_USER=admin MINIO_ROOT_PASSWORD=xrj12345678 minio server /opt/minio --address '192.168.10.109:9000' --console-address :9001

2.1.2 Client

下载

wget https://dl.minio.org.cn/client/mc/release/linux-amd64/mc chmod +x mc sudo mv mc /usr/local/bin/mc

别名设置

#将mc命令时local指定http://127.0.0.1:9000 admin xrj12345678 mc alias set local http://127.0.0.1:9000 admin xrj12345678 #查看服务端信息 mc admin info local

2.2 直装集群

2.2.1 部署方式

两台机器部署2节点,一个节点一个driver挂掉一台能读不能写。

#机器1,/opt/minio作为文件存储目录 192.168.10.109 /data/minio #机器2 192.168.10.110 /data/minio

2.2.2 配置服务器

创建挂载目录和配置目录,两台机器都执行

新版要求挂载目录不能是根目录,新建磁盘分区挂载到/data

mkdir -p /data/minio && mkdir -p /etc/minio

下载minio执行文件,和单节点操作一致

wget https://dl.minio.org.cn/server/minio/release/linux-amd64/minio #执行权限 chmod +x minio #放入执行路径 sudo mv minio /usr/local/bin/

集群启动脚本。192.168.10.109和192.168.10.110分别创建run.sh。--config-dir指定集群配置目录

mkdir /root/minio vim /root/minio/run.sh chmod +x /root/minio/run.sh
#!/bin/bash export MINIO_ROOT_USER=admin export MINIO_ROOT_PASSWORD=xrj12345678 minio server --config-dir /etc/minio \ --address "0.0.0.0:9000" --console-address ":9001" \ http://192.168.10.109/data/minio \ http://192.168.10.110/data/minio

系统service

vim /usr/lib/systemd/system/minio.service
[Unit] Description=Minio service Documentation=https://docs.minio.io/ [Service] WorkingDirectory=/root/minio/ ExecStart=/root/minio/run.sh Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target

启动服务

#加载并启动服务 systemctl daemon-reload && systemctl start minio #查看服务信息 systemctl status minio #关闭服务 systemctl stop minio #服务开机启动 systemctl enable minio #关闭服务开机启动 systemctl disable minio

2.3 Docker单点

2.3.1 拉取镜像

docker pull minio/minio:RELEASE.2023-05-27T05-56-19Z

2.3.2 运行服务

自定义密码最小8位

docker run -d --restart always \ -p 9000:9000 \ -p 9001:9001 \ --name minio \ -v /opt/minio/data:/data \ -e "MINIO_ROOT_USER=admin" \ -e "MINIO_ROOT_PASSWORD=xrj12345678" \ minio/minio:RELEASE.2023-05-27T05-56-19Z \ server /data --console-address ":9001"

2.4 Docker-Compose单机器集群

在192.168.10.109上启动4个minio实例,每个实例两个driver,使用nginx负载均衡。

2.4.1 docker-compose.yml

version: '3.3' services: minio1: image: minio/minio:RELEASE.2023-05-27T05-56-19Z command: server --console-address ":9001" http://minio{1...4}/data{1...2} expose: - "9000" - "9001" environment: MINIO_ROOT_USER: admin MINIO_ROOT_PASSWORD: xrj12345678 healthcheck: test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] interval: 30s timeout: 20s retries: 3 hostname: minio1 volumes: - data1-1:/data1 - data1-2:/data2 minio2: image: minio/minio:RELEASE.2023-05-27T05-56-19Z command: server --console-address ":9001" http://minio{1...4}/data{1...2} expose: - "9000" - "9001" environment: MINIO_ROOT_USER: admin MINIO_ROOT_PASSWORD: xrj12345678 healthcheck: test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] interval: 30s timeout: 20s retries: 3 hostname: minio2 volumes: - data2-1:/data1 - data2-2:/data2 minio3: image: minio/minio:RELEASE.2023-05-27T05-56-19Z command: server --console-address ":9001" http://minio{1...4}/data{1...2} expose: - "9000" - "9001" environment: MINIO_ROOT_USER: admin MINIO_ROOT_PASSWORD: xrj12345678 healthcheck: test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] interval: 30s timeout: 20s retries: 3 hostname: minio3 volumes: - data3-1:/data1 - data3-2:/data2 minio4: image: minio/minio:RELEASE.2023-05-27T05-56-19Z command: server --console-address ":9001" http://minio{1...4}/data{1...2} expose: - "9000" - "9001" environment: MINIO_ROOT_USER: admin MINIO_ROOT_PASSWORD: xrj12345678 healthcheck: test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] interval: 30s timeout: 20s retries: 3 hostname: minio4 volumes: - data4-1:/data1 - data4-2:/data2 nginx: image: nginx:1.19.2-alpine hostname: nginx volumes: - ./nginx.conf:/etc/nginx/nginx.conf:ro ports: - "9000:9000" - "9001:9001" depends_on: - minio1 - minio2 - minio3 - minio4 volumes: data1-1: data1-2: data2-1: data2-2: data3-1: data3-2: data4-1: data4-2:

2.4.2 nginx.conf

放在和docker-compose.yml同级目录,这里nginx所在的节点无法高可用,配置文件只在109主机上,只会调度到109,109挂了就没法访问了。

可以考虑将nginx配置构建一个新的镜像在每个主机节点上都起一个副本,api访问配置所有的nginx。

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  4096;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;
    sendfile        on;
    keepalive_timeout  65;

    # include /etc/nginx/conf.d/*.conf;

    upstream minio {
        server minio1:9000;
        server minio2:9000;
        server minio3:9000;
        server minio4:9000;
    }

    upstream console {
        ip_hash;
        server minio1:9001;
        server minio2:9001;
        server minio3:9001;
        server minio4:9001;
    }

    server {
        listen       9000;
        listen  [::]:9000;
        server_name  localhost;

        #允许特殊请求头
        ignore_invalid_headers off;
        #不限制上传文件大小
        client_max_body_size 0;
        # To disable buffering
        proxy_buffering off;
        proxy_request_buffering off;

        location / {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;

            proxy_connect_timeout 300;
            # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            chunked_transfer_encoding off;

            proxy_pass http://minio;
        }
    }

    server {
        listen       9001;
        listen  [::]:9001;
        server_name  localhost;

        ignore_invalid_headers off;
        client_max_body_size 0;
        proxy_buffering off;
        proxy_request_buffering off;

        location / {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-NginX-Proxy true;

            real_ip_header X-Real-IP;

            proxy_connect_timeout 300;
            
            # To support websocket
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            
            chunked_transfer_encoding off;

            proxy_pass http://console;
        }
    }
}

2.4.3 启动服务

docker compose up -d docker compose down

2.5 Docker-Swarm多机器集群

这里使用Swarm部署集群,4机器4节点,每个节点2个driver,使用nginx负载均衡。使用node.labels中的自定义属性配合deploy的placement属性的约束条件让不同minio服务节点调度到固定的节点。如:minio1服务固定调度到主机minio-1,其他节点类似,当节点故障恢复后不会自动负载服务节点导致数据漂移丢失。

2.5.1 配置服务器

minio-1 192.168.10.109 minio-2 192.168.10.110 minio-3 192.168.10.111 minio-4 192.168.10.112
hostnamectl set-hostname minio-1 hostnamectl set-hostname minio-2 hostnamectl set-hostname minio-3 hostnamectl set-hostname minio-4
cat >> /etc/hosts <<EOF minio-1 192.168.10.109 minio-2 192.168.10.110 minio-3 192.168.10.111 minio-4 192.168.10.112 EOF

2.5.2 初始化Swarm集群

192.168.10.109当主节点。

主节点初始化集群

docker swarm init --advertise-addr 192.168.10.109

从节点加入集群

docker swarm join --token SWMTKN-1-5w1sqjo4pllh3btuihkpd027p0dnjpsd2t65zcoi9wrq2ahysa-5kh17licwa5069xoyah1tlbt7 192.168.10.109:2377

image-20230602014055432

将节点2和3升级成master节点,升级的worker节点管理状态从空成为Reachable

docker node promote mi78x5l6hk8dbx31m4suir67x docker node promote ihdukwd6vp84fvy3ida89fm1a

2.5.3 docker-swarm.yml

使用docker volumes进行数据挂载

version: '3.7' x-minio-common: &minio-common image: minio/minio:RELEASE.2023-05-27T05-56-19Z command: server --console-address ":9001" http://minio{1...4}/data{1...2} expose: - "9000" - "9001" environment: MINIO_ROOT_USER: admin MINIO_ROOT_PASSWORD: xrj12345678 deploy: update_config: parallelism: 1 delay: 10s order: stop-first restart_policy: condition: on-failure healthcheck: test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] interval: 30s timeout: 20s retries: 3 services: minio1: <<: *minio-common hostname: minio1 deploy: placement: constraints: - "node.labels.minio1==true" volumes: - data1-1:/data1 - data1-2:/data2 minio2: <<: *minio-common hostname: minio2 deploy: placement: constraints: - "node.labels.minio2==true" volumes: - data2-1:/data1 - data2-2:/data2 minio3: <<: *minio-common hostname: minio3 deploy: placement: constraints: - "node.labels.minio3==true" volumes: - data3-1:/data1 - data3-2:/data2 minio4: <<: *minio-common hostname: minio4 deploy: placement: constraints: - "node.labels.minio4==true" volumes: - data4-1:/data1 - data4-2:/data2 nginx: image: nginx:1.19.2-alpine hostname: nginx volumes: - /root/minio/swarm/nginx.conf:/etc/nginx/nginx.conf:ro ports: - "9000:9000" - "9001:9001" deploy: update_config: parallelism: 1 delay: 10s order: stop-first restart_policy: condition: on-failure depends_on: - minio1 - minio2 - minio3 - minio4 volumes: data1-1: data1-2: data2-1: data2-2: data3-1: data3-2: data4-1: data4-2:

2.5.4 nginx.conf

user nginx; worker_processes auto; error_log /var/log/nginx/error.log warn; pid /var/run/nginx.pid; events { worker_connections 4096; } http { include /etc/nginx/mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx/access.log main; sendfile on; keepalive_timeout 65; # include /etc/nginx/conf.d/*.conf; upstream minio { server minio1:9000; server minio2:9000; server minio3:9000; server minio4:9000; } upstream console { ip_hash; server minio1:9001; server minio2:9001; server minio3:9001; server minio4:9001; } server { listen 9000; listen [::]:9000; server_name localhost; #允许特殊请求头 ignore_invalid_headers off; #不限制上传文件大小 client_max_body_size 0; # To disable buffering proxy_buffering off; proxy_request_buffering off; location / { proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_connect_timeout 300; # Default is HTTP/1, keepalive is only enabled in HTTP/1.1 proxy_http_version 1.1; proxy_set_header Connection ""; chunked_transfer_encoding off; proxy_pass http://minio; } } server { listen 9001; listen [::]:9001; server_name localhost; ignore_invalid_headers off; client_max_body_size 0; proxy_buffering off; proxy_request_buffering off; location / { proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-NginX-Proxy true; real_ip_header X-Real-IP; proxy_connect_timeout 300; # To support websocket proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; chunked_transfer_encoding off; proxy_pass http://console; } } }

2.5.5 启动服务

配置服务调度标签,会在对应的node上的labels属性添加上属性,使用部署文件中下面的约束来让对应的服务调度到固定的节点。

deploy: placement: constraints: - "node.labels.minio1==true"

给node加上标签值,如minio-1节点的标签为minio1==true

docker node update --label-add minio1=true minio-1 docker node update --label-add minio2=true minio-2 docker node update --label-add minio3=true minio-3 docker node update --label-add minio4=true minio-4

启动服务

docker stack deploy -c docker-swarm.yml minio docker stack rm minio
「喜欢这篇文章,您的关注和赞赏是给作者最好的鼓励」
关注作者
【版权声明】本文为墨天轮用户原创内容,转载时必须标注文章的来源(墨天轮),文章链接,文章作者等基本信息,否则作者和墨天轮有权追究责任。如果您发现墨天轮中有涉嫌抄袭或者侵权的内容,欢迎发送邮件至:contact@modb.pro进行举报,并提供相关证据,一经查实,墨天轮将立刻删除相关内容。

文章被以下合辑收录

评论