nginx+nginx-upsync-module实现动态负载及自定义验证

尚浩宇 2021-08-16

1149

一、说明

nginx一般直接在配置文件里配置upstream即可实现负载均衡，但有些特定的环境下此种方式就显得有些局限性。比如后端服务器无法依据端口占用检查存活的时候；后台动态调整节点的时候；调整节点后不想修改配置文件重启nginx的时候等等。

此文的思路是将配置文件从nginx本地迁移到其他第三方服务上如etcd、consul上，然后时候拉取配置到本地。理论上说任何第三方配置中心都可以实现该功能，但需要对应的nginx模块。本文采用nginx-upsync-module，主要支持consul、etcd，本文以consul为例。

迁移配置文件还无法满足需求，还需要解决服务检测机制。这里不再以端口占用为准，而是实际访问某一个接口，查看是否返回数据，并以此为存活依据，最后通过调用consul的rest接口管理配置。

二、安装

2.1安装nginx

nginx安装参照前文《Linux下Nginx1.8安装》

需要注意的是，在安装nginx的时候需要安装nginx-upsync-module模块。

2.2安装nginx-upsync-module

打开https://github.com/weibocom/nginx-upsync-module，如果遇到github打不开，可以参照如下链接解决：https://www.php.cn/faq/445082.html

下载完成后，解压到linux目录备用。

2.3安装consul

consul的安装比较简单，这里不再赘述，可参照如下链接：https://blog.csdn.net/junaozun/article/details/90699384

三、搭建&测试

3.1搭建

本文的基础目录为：

1、nginx源码目录：/opt/server/software/nginx-1.19.1

2、nginx安装目录：/home/nginx/nginx

3、nginx-upsync-module目录：/opt/server/software/nginx-upsync-module/

确定consul正常运行后，配置nginx：

consul.conf

#user  nobody;
worker_processes  1;


#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;


#pid        logs/nginx.pid;




events {
    worker_connections  1024;
}




http {
    include       mime.types;
    default_type  application/octet-stream;


    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';


    #access_log  logs/access.log  main;


    sendfile        on;
    #tcp_nopush     on;


    #keepalive_timeout  0;
    keepalive_timeout  65;


    #gzip  on;


    upstream testconsul {
        #这个配置无用了，但删除后启动会报错，如果consul里没有配置则会调用此地址，故此设置应该设置为一个默认的可用的地址，一旦从consul拉取到数据这个配置就无用了
        server 127.0.0.1:11111;
        #### 连接consul server，获取动态upstreams，配置负载均衡信息，间隔0.5s获取配置信息，upsync_timeout配置从consul拉取上游服务器配置的超时时间；upsync_interval配置从consul拉取上游服务器配置的间隔时间；upsync_type指定使用consul配置服务器；strong_dependency配置nginx在启动时是否强制依赖配置服务器，如果配置为on，则拉取配置失败时nginx启动同样失败
        upsync 192.168.1.97:8500/v1/kv/upstreams/testconsul/  upsync_timeout=6m upsync_interval=500ms upsync_type=consul strong_dependency=off;
        ### 动态获取consul server相关负载均衡配置信息持久化在硬盘，这样即使consul服务器出问题了，本地还有一个备份。
        upsync_dump_path /home/nginx/nginx/conf/servers/testconsul.conf;
        }


        server {
                listen       80 ;
                server_name  testconsul;
                charset utf8;
                location /{
                        proxy_set_header Host $host;
                        proxy_set_header X-Real-IP $remote_addr;
                        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                        proxy_pass http://testconsul/HelloWord;


               }        


                access_log /home/nginx/nginx/logs/access-upsync.log;
                location /stub-status {
                        stub_status on;
                }       
                location = /upstream_show {
                    upstream_show;
                }   


          }     






    # another virtual host using mix of IP-, name-, and port-based configuration
    #
    #server {
    #    listen       8000;
    #    listen       somename:8080;
    #    server_name  somename  alias  another.alias;


    #    location  {
    #        root   html;
    #        index  index.html index.htm;
    #    }
    #}




    # HTTPS server
    #
    #server {
    #    listen       443 ssl;
    #    server_name  localhost;


    #    ssl_certificate      cert.pem;
    #    ssl_certificate_key  cert.key;


    #    ssl_session_cache    shared:SSL:1m;
    #    ssl_session_timeout  5m;


    #    ssl_ciphers  HIGH:!aNULL:!MD5;
    #    ssl_prefer_server_ciphers  on;


    #    location  {
    #        root   html;
    #        index  index.html index.htm;
    #    }
    #}


}

3.2测试

URL目录

1、负载状态：http://nginxhost:port/stub-status

2、当前负载节点列表：http://nginxhost:port/upstream_show

3、测试URL：http://nginxhost:port/HelloWord

4、consul增加节点（也可直接在UI上添加）：http://consulhost:port/v1/kv/upstreams/consultest/192.168.1.22:8080

四、自定义检测机制

核心功能是根据配置检测服务是否可用，然后根据结果更新到consul上。代码比较简单，主要代码如下：

pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>


  <groupId>com.cmgplex.aotucheck4consul</groupId>
  <artifactId>aotucheck4consul</artifactId>
  <version>1.0.1-SNAPSHOT</version>
  <packaging>jar</packaging>


  <name>aotucheck4consul</name>
  <url>http://www.cmgplex.com</url>


  <parent>
    <groupId>com.cmgplex.hr.parent</groupId>
    <artifactId>hr-parent</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <relativePath></relativePath>
  </parent>
  <properties>
    <snippetsDirectory>${project.build.directory}/generated-snippets</snippetsDirectory>
    <asciidoctor.input.directory>${project.basedir}/docs/asciidoc</asciidoctor.input.directory>
    <generated.asciidoc.directory>${project.build.directory}/asciidoc</generated.asciidoc.directory>
    <asciidoctor.html.output.directory>${project.build.directory}/asciidoc/html</asciidoctor.html.output.directory>
    <asciidoctor.pdf.output.directory>${project.build.directory}/asciidoc/pdf</asciidoctor.pdf.output.directory>
  </properties>


  <dependencies>
    <!--base dependency for all start -->
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter</artifactId>
      <exclusions>
        <exclusion>
          <groupId>org.springframework.boot</groupId>
          <artifactId>spring-boot-starter-logging</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>net.logstash.log4j</groupId>
      <artifactId>jsonevent-layout</artifactId>
      <version>1.0</version>
    </dependency>
    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-log4j12</artifactId>
    </dependency>
    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-api</artifactId>
    </dependency>
    <dependency>
      <groupId>com.google.guava</groupId>
      <artifactId>guava</artifactId>
    </dependency>
    <dependency>
      <groupId>commons-lang</groupId>
      <artifactId>commons-lang</artifactId>
    </dependency>
    <!--base dependency for all end -->


    <!--jetty begin -->
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-web</artifactId>
      <exclusions>
        <exclusion>
          <groupId>org.springframework.boot</groupId>
          <artifactId>spring-boot-starter-tomcat</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-undertow</artifactId>
    </dependency>
    <!--jetty end -->


    <!--consul begin -->
    <dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-starter-consul-all</artifactId>
    </dependency>
    <!--consul end -->


    <!--stream rabbit begin -->
    <dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-starter-stream-rabbit</artifactId>
    </dependency>
    <!--stream rabbit end -->
    <dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-starter-netflix-hystrix</artifactId>
    </dependency>
    <dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-starter-netflix-hystrix-dashboard</artifactId>
    </dependency>
    <!--feign begin -->
    <dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-starter-openfeign</artifactId>
    </dependency>
    <!--feign end -->


    <!--actuator begin -->
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>
    <!--actuator end -->




    <dependency>
      <groupId>com.alibaba</groupId>
      <artifactId>fastjson</artifactId>
      <version>1.2.46</version>
    </dependency>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-configuration-processor</artifactId>
      <optional>true</optional>
    </dependency>
    <dependency>
      <groupId>javax.mail</groupId>
      <artifactId>mail</artifactId>
      <version>1.4.7</version>
    </dependency>
    <dependency>
      <groupId>com.sun.mail</groupId>
      <artifactId>javax.mail</artifactId>
    </dependency>
    <!-- consul api begin -->
    <dependency>
      <groupId>com.ecwid.consul</groupId>
      <artifactId>consul-api</artifactId>
      <version>1.4.5</version>
    </dependency>
    <!-- consul api end -->
  </dependencies>
  <build>
    <finalName>aotucheck4consul</finalName>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <configuration>
          <source>${java.version}</source>
          <target>${java.version}</target>
          <encoding>UTF-8</encoding>
        </configuration>
      </plugin>
      <plugin>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-maven-plugin</artifactId>
      </plugin>
    </plugins>
    <resources>
      <resource>
        <directory>src/main/java</directory>
        <includes>
          <include>**/*.xml</include>
        </includes>
      </resource>
      <resource>
        <directory>src/main/resources</directory>
        <includes>
          <include>**/*.*</include>
        </includes>
      </resource>
      <resource>
        <directory>src/main/webapp</directory>
        <targetPath>META-INF/resources</targetPath>
        <includes>
          <include>**/*.*</include>
        </includes>
      </resource>
    </resources>
  </build>
  <!-- 这里是为了方便未配置maven的情况下，找到依赖 -->
  <repositories>
    <repository>
      <id>lumi-snapshots</id>
      <name>lumi-snapshots</name>
      <url>http://repo.lumiai.top/repository/maven-snapshots/</url>
      <layout>default</layout>
      <snapshots>
        <enabled>true</enabled>
      </snapshots>
    </repository>
  </repositories>
</project>

application.properties

spring.main.allow-bean-definition-overriding=true


#the following is for cloud


spring.datasource.hikari.idle-timeout = 600000
spring.datasource.hikari.connection-timeout = 30000
spring.datasource.hikari.max-lifetime = 1800000
spring.datasource.hikari.maximum-pool-size = 20
spring.datasource.type = com.zaxxer.hikari.HikariDataSource


#这个是关于mq的配置
spring.rabbitmq.addresses=amqp://192.168.1.2:5672
spring.rabbitmq.password=guest
spring.rabbitmq.username=guest
spring.cloud.stream.default-binder=rabbit
#定时检查，每隔1分钟
job.cleanConsulDeadService.cron=1 * * * * ? 
#consul配置key默认前缀
consul.default.key.prefix=upstreams/
#服务检查配置
#----------------api服务
#要检查服务的url，不包含地址
server.check.apps.api.url=/api/b
#要检查服务的host，只能是IP，包含端口，即使是80也不能少，不然nginx那边读取不到配置会报错而导致配置不可用
server.check.apps.api.hosts=http://192.168.1.1:80,http://192.168.1.2:81
#要检查服务的method
server.check.apps.api.method=get
#暂不支持json参数
server.check.apps.api.param=get
#链接超时时间，单位毫秒
server.check.apps.api.timeout=3000
#断定服务正常的response返回code
server.check.apps.api.successcode=200,302
#----------------baidu服务
#要检查服务的url，不包含地址
server.check.apps.consultest.url=/s
#要检查服务的host，只能是IP，包含端口，即使是80也不能少，不然nginx那边读取不到配置会报错而导致配置不可用
server.check.apps.consultest.hosts=http://182.61.200.6:80
#要检查服务的method
server.check.apps.consultest.method=get
#暂不支持json参数
server.check.apps.consultest.param=wd=javahttp&tn=98012088_5_dg&ch=11
#链接超时时间，单位毫秒
server.check.apps.consultest.timeout=3000
#断定服务正常的response返回code
server.check.apps.consultest.successcode=200,302




#********************************************
#本地服务监听端口
server.port = 17000
#cpu的核数
server.undertow.io-threads=4
#预估的最佳线程数
server.undertow.worker-threads=400
spring.application.name=aotucheck4consul
spring.profiles.active=dev
#********************************************
#consul的配置
spring.cloud.consul.enabled=true
spring.cloud.consul.host=192.168.1.2
spring.cloud.consul.port=8500


#********************************************
#注册中心的配置
spring.cloud.consul.discovery.enabled=false
spring.cloud.consul.discovery.prefer-ip-address=true
spring.cloud.consul.discovery.health-check-path=/actuator/health
spring.cloud.consul.discovery.health-check-interval=10s
spring.cloud.consul.discovery.health-check-timeout=1s
spring.cloud.consul.discovery.instance-id=${spring.application.name}:${spring.cloud.client.ip-address}:${server.port}




#是否把自己向注册中心注册，可以做纯consumer不注册自己
spring.cloud.consul.discovery.register=true


#********************************************
#配置服务的配置
spring.cloud.consul.config.enabled=true
spring.cloud.consul.config.prefix=config
spring.cloud.consul.config.profile-separator=,
spring.cloud.consul.config.default-context=application
spring.cloud.consul.config.watch.enabled=true
spring.cloud.consul.config.watch.delay=1000
spring.cloud.consul.config.watch.wait-time=3
spring.cloud.consul.config.format=properties
spring.cloud.consul.config.data-key=configuration
#禁用熔断器首次调用时强制1秒超时
hystrix.command.default.execution.timeout.enabled=false

CheckServiceScheduler.java

@Component
@EnableScheduling
public class CheckServiceScheduler {
  private static final Logger LOGGER = LoggerFactory.getLogger(CheckServiceScheduler.class);
  @Autowired
  private ICheckService checkService;


  
  @Scheduled(cron = "${job.cleanConsulDeadService.cron}")
  public void checkServer() {
    LOGGER.info("-----------------------------start checkServer Scheduler-----------------------------");
    this.checkService.check();
    LOGGER.info("-----------------------------end checkServer Scheduler-----------------------------");
  }
}




CheckServiceImpl.java
@Service
public class CheckServiceImpl implements ICheckService {
  private static final Logger LOGGER = LoggerFactory.getLogger(CheckServiceImpl.class);
  @Resource
  private ServerPropertiesConfig serverPropertiesConfig;
  @Resource
  private IConsulService consulService;
  @Value("${consul.default.key.prefix}")
  private String consulKeyPrefix;


  /*
   * (非 Javadoc) <p>Title: check</p> <p>Description: </p>
   * 
   * @return
   * 
   * @see com.cmgplex.aotucheck4consul.jobsystem.service.ICheckService#check()
   */
  @Override
  public void check() {
    LOGGER.debug("serverPropertiesConfig={}", this.serverPropertiesConfig);
    Map<String, ServerInfoVo> apps = this.serverPropertiesConfig.getApps();
    for (Entry<String, ServerInfoVo> app : apps.entrySet()) {
      String appName = app.getKey();
      LOGGER.info("appName={}", appName);
      ServerInfoVo config = app.getValue();
      LOGGER.info("config={}", config);
      List<Integer> expectCodeList = ListUtils.str2list(config.getSuccesscode(), ",");
      List<String> hostsList = ListUtils.str2list4String(config.getHosts(), ",");
      String method = config.getMethod();
      HttpMethod httpMethod = HttpMethod.valueOf(method.toUpperCase());
      String param = config.getParam();
      Integer timeout = config.getTimeout();
      hostsList.parallelStream().forEach(host -> {
        try {
          LOGGER.info("host={}", host);
          String httpurl = host + config.getUrl();
          LOGGER.info("httpurl={}", httpurl);
          boolean isSuccess = HttpTools.compareCode(expectCodeList, httpMethod, httpurl, param, timeout);
          LOGGER.info("isSuccess={}", isSuccess);
          String key = this.consulKeyPrefix + appName + "/"
              + host.replace("http://", "").replace("https://", "");
          LOGGER.info("key={}", key);
          // 如果服务可用
          if (isSuccess) {
            // 检查consul上配置是否正常
            if (!this.consulService.isKeyExist(key)) {
              // 如果不正常，则更新正常
              this.consulService.addKeyValue(key, new NginxServerConfig(2, 1, 10));
              LOGGER.info("server success,consul faild,now update consul,key={}", key);
            }
          } else {
            // 如果服务不可用
            // 检查consul上配置是否正常
            if (this.consulService.isKeyExist(key)) {
              // 如果正常，则更新为不正常
              this.consulService.deleteKey(key);
              LOGGER.info("server faild,consul success,now update consul,key={}", key);
            }
          }
        } catch (Exception e) {
          LOGGER.error(e.getMessage(), e);
        }
      });
    }
  }


}

ConsulServiceImpl.java

@Service
public class ConsulServiceImpl implements IConsulService {
  private static final Logger LOGGER = LoggerFactory.getLogger(ConsulServiceImpl.class);
  private static final BASE64Decoder decoder = new BASE64Decoder();
  @Autowired
  private ConsulClient consulClient;


  /*
   * (非 Javadoc) <p>Title: getKeysByPrefix</p> <p>Description: </p>
   * 
   * @param keyPrefix
   * 
   * @return
   * 
   * @see com.cmgplex.aotucheck4consul.jobsystem.service.IConsulService#getKeysByPrefix(java.lang.String)
   */
  @Override
  public List<String> getKeysByPrefix(String keyPrefix) {
    Response<List<GetValue>> valus = this.consulClient.getKVValues(keyPrefix);
    return valus.getValue().stream().map(value -> value.getKey()).filter(key -> StringUtils.isNotBlank(key))
        .collect(Collectors.toList());
  }


  /*
   * (非 Javadoc) <p>Title: getValue</p> <p>Description: </p>
   * 
   * @param key
   * 
   * @return
   * 
   * @see com.cmgplex.aotucheck4consul.jobsystem.service.IConsulService#getValue(java.lang.String)
   */
  @Override
  public NginxServerConfig getValue(String key) {
    GetValue kvValue = this.consulClient.getKVValue(key).getValue();
    if (null == kvValue) {
      return null;
    }
    String value = kvValue.getValue();
    String jsonValue;
    try {
      jsonValue = new String(decoder.decodeBuffer(value), "UTF-8");
      return JSON.parseObject(jsonValue, NginxServerConfig.class);
    } catch (IOException e) {
      LOGGER.error(e.getMessage(), e);
    }
    return null;
  }


  /*
   * (非 Javadoc) <p>Title: isKeyExist</p> <p>Description: </p>
   * 
   * @param key
   * 
   * @return
   * 
   * @see com.cmgplex.aotucheck4consul.jobsystem.service.IConsulService#isKeyExist(java.lang.String)
   */
  @Override
  public boolean isKeyExist(String key) {
    return null != this.getValue(key);
  }


  /*
   * (非 Javadoc) <p>Title: deleteKey</p> <p>Description: </p>
   * 
   * @param key
   * 
   * @return
   * 
   * @see com.cmgplex.aotucheck4consul.jobsystem.service.IConsulService#deleteKey(java.lang.String)
   */
  @Override
  public boolean deleteKey(String key) {
    return null != this.consulClient.deleteKVValue(key);
  }


  /*
   * (非 Javadoc) <p>Title: addKeyValue</p> <p>Description: </p>
   * 
   * @param key
   * 
   * @param value
   * 
   * @return
   * 
   * @see com.cmgplex.aotucheck4consul.jobsystem.service.IConsulService#addKeyValue(java.lang.String,
   * com.cmgplex.aotucheck4consul.jobsystem.vo.NginxServerConfig)
   */
  @Override
  public boolean addKeyValue(String key, NginxServerConfig value) {
    return this.consulClient.setKVValue(key, JSON.toJSONString(value)).getValue();
  }


}

有兴趣的大佬大神可以关注下小弟的微信公共号，一起学习交流，扫描以下二维码关注即可。

数据库

文章转载自尚浩宇，如果涉嫌侵权，请发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。