RocketMQ源码分析之Broker（4）

徘徊笔记 2019-04-07

303

来源：https://github.com/apache/rocketmq

ReputMessageService

定时构建消息消费队列CommitLogDispatcherBuildConsumeQueue以及索引文件CommitLogDispatcherBuildIndex与针对布隆过滤器的消费过滤类CommitLogDispatcherCalcBitMap。

ScheduleMessageService

定时消息管理类。

TransientStorePool

针对开启transientStorePoolEnable功能的异步刷盘的异步存储池

 public void init() {
    for (int i = 0; i < poolSize; i++) {
        ByteBuffer byteBuffer = ByteBuffer.allocateDirect(fileSize);


        final long address = ((DirectBuffer) byteBuffer).address();
        Pointer pointer = new Pointer(address);
        LibC.INSTANCE.mlock(pointer, new NativeLong(fileSize));


        availableBuffers.offer(byteBuffer);
    }
}

启动创建存储文件请求的线程AllocateMappedFileService以及索引文件IndexService，创建文件锁RandomAccessFile，一些文件路径如下

 //The root directory in which the log data is kept
@ImportantField
private String storePathRootDir = System.getProperty("user.home") + File.separator + "store";


//The directory in which the commitlog is kept
@ImportantField
private String storePathCommitLog = System.getProperty("user.home") + File.separator + "store"
    + File.separator + "commitlog";
 public static String getStorePathConsumeQueue(final String rootDir) {
    return rootDir + File.separator + "consumequeue";
}
public static String getStorePathConsumeQueueExt(final String rootDir) {
    return rootDir + File.separator + "consumequeue_ext";
}
public static String getStorePathIndex(final String rootDir) {
    return rootDir + File.separator + "index";
}
public static String getStoreCheckpoint(final String rootDir) {
    return rootDir + File.separator + "checkpoint";
}
public static String getAbortFile(final String rootDir) {
    return rootDir + File.separator + "abort";
}
public static String getLockFile(final String rootDir) {
    return rootDir + File.separator + "lock";
}
public static String getDelayOffsetStorePath(final String rootDir) {
    return rootDir + File.separator + "config" + File.separator + "delayOffset.json";
}

最后加载各个文件历史存在的数据

public boolean load() {
    boolean result = true;


    try {
        boolean lastExitOK = !this.isTempFileExist();
        log.info("last shutdown {}", lastExitOK ? "normally" : "abnormally");


        if (null != scheduleMessageService) {
            result = result && this.scheduleMessageService.load();
        }


        // load Commit Log
        result = result && this.commitLog.load();


        // load Consume Queue
        result = result && this.loadConsumeQueue();


        if (result) {
            this.storeCheckpoint =
                new StoreCheckpoint(StorePathConfigHelper.getStoreCheckpoint(this.messageStoreConfig.getStorePathRootDir()));


            this.indexService.load(lastExitOK);


            this.recover(lastExitOK);


            log.info("load over, and the max phy offset = {}", this.getMaxPhyOffset());
        }
    } catch (Exception e) {
        log.error("load exception", e);
        result = false;
    }


    if (!result) {
        this.allocateMappedFileService.shutdown();
    }


    return result;
}

开始启动broker

启动代码如下，依次启动各个组件，remotingServer，fastRemotingServer，fileWatchService启动可以参考之前nameserver的分析，

public void start() throws Exception {
    if (this.messageStore != null) {
        this.messageStore.start();
    }


    if (this.remotingServer != null) {
        this.remotingServer.start();
    }


    if (this.fastRemotingServer != null) {
        this.fastRemotingServer.start();
    }


    if (this.fileWatchService != null) {
        this.fileWatchService.start();
    }


    if (this.brokerOuterAPI != null) {
        this.brokerOuterAPI.start();
    }


    if (this.pullRequestHoldService != null) {
        this.pullRequestHoldService.start();
    }


    if (this.clientHousekeepingService != null) {
        this.clientHousekeepingService.start();
    }


    if (this.filterServerManager != null) {
        this.filterServerManager.start();
    }


    if (this.brokerStatsManager != null) {
        this.brokerStatsManager.start();
    }


    if (this.brokerFastFailure != null) {
        this.brokerFastFailure.start();
    }


    if (BrokerRole.SLAVE != messageStoreConfig.getBrokerRole()) {
        if (this.transactionalMessageCheckService != null) {
            log.info("Start transaction service!");
            this.transactionalMessageCheckService.start();
        }
    }
}

brokerOuterAPI，通信客户端NettyRemotingClient开始执行服务，定时程序定时扫描还没有进行回复的请求，执行过期策略以及回调方法，nettyEventExecutor定时消费通道变化的消息。

public void start() {
    this.defaultEventExecutorGroup = new DefaultEventExecutorGroup(
        nettyClientConfig.getClientWorkerThreads(),
        new ThreadFactory() {


            private AtomicInteger threadIndex = new AtomicInteger(0);


            @Override
            public Thread newThread(Runnable r) {
                return new Thread(r, "NettyClientWorkerThread_" + this.threadIndex.incrementAndGet());
            }
        });


    Bootstrap handler = this.bootstrap.group(this.eventLoopGroupWorker).channel(NioSocketChannel.class)
        .option(ChannelOption.TCP_NODELAY, true)
        .option(ChannelOption.SO_KEEPALIVE, false)
        .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, nettyClientConfig.getConnectTimeoutMillis())
        .option(ChannelOption.SO_SNDBUF, nettyClientConfig.getClientSocketSndBufSize())
        .option(ChannelOption.SO_RCVBUF, nettyClientConfig.getClientSocketRcvBufSize())
        .handler(new ChannelInitializer<SocketChannel>() {
            @Override
            public void initChannel(SocketChannel ch) throws Exception {
                ChannelPipeline pipeline = ch.pipeline();
                if (nettyClientConfig.isUseTLS()) {
                    if (null != sslContext) {
                        pipeline.addFirst(defaultEventExecutorGroup, "sslHandler", sslContext.newHandler(ch.alloc()));
                        log.info("Prepend SSL handler");
                    } else {
                        log.warn("Connections are insecure as SSLContext is null!");
                    }
                }
                pipeline.addLast(
                    defaultEventExecutorGroup,
                    new NettyEncoder(),
                    new NettyDecoder(),
                    new IdleStateHandler(0, 0, nettyClientConfig.getClientChannelMaxIdleTimeSeconds()),
                    new NettyConnectManageHandler(),
                    new NettyClientHandler());
            }
        });


    this.timer.scheduleAtFixedRate(new TimerTask() {
        @Override
        public void run() {
            try {
                NettyRemotingClient.this.scanResponseTable();
            } catch (Throwable e) {
                log.error("scanResponseTable exception", e);
            }
        }
    }, 1000 * 3, 1000);


    if (this.channelEventListener != null) {
        this.nettyEventExecutor.start();
    }
}

启动拉取消息请求线程pullRequestHoldService，客户端心跳检测服务clientHousekeepingService，消息过滤服务filterServerManager，broker快速失败服务定时清除失效请求的服务brokerFastFailure，事务检查服务transactionalMessageCheckService。

DefaultMessageStore

存储管理类启动，获取文件锁，保证启动的单次执行，启动定时落盘消息消费队列的线程flushConsumeQueueService，有关消息落盘的线程FlushCommitLogService，commitLogService，只有主broker才会执行定时消息的线程scheduleMessageService，设置重置线程的开始位移量reputMessageService，

public void start() throws Exception {


    lock = lockFile.getChannel().tryLock(0, 1, false);
    if (lock == null || lock.isShared() || !lock.isValid()) {
        throw new RuntimeException("Lock failed,MQ already started");
    }


    lockFile.getChannel().write(ByteBuffer.wrap("lock".getBytes()));
    lockFile.getChannel().force(true);


    this.flushConsumeQueueService.start();
    this.commitLog.start();
    this.storeStatsService.start();


    if (this.scheduleMessageService != null && SLAVE != messageStoreConfig.getBrokerRole()) {
        this.scheduleMessageService.start();
    }


    if (this.getMessageStoreConfig().isDuplicationEnable()) {
        this.reputMessageService.setReputFromOffset(this.commitLog.getConfirmOffset());
    } else {
        this.reputMessageService.setReputFromOffset(this.commitLog.getMaxOffset());
    }
    this.reputMessageService.start();


    this.haService.start();


    this.createTempFile();
    this.addScheduleTask();
    this.shutdown = false;
}

启动主从同步线程，开始接收连接请求，最后同步消息。

public void start() throws Exception {
    this.acceptSocketService.beginAccept();
    this.acceptSocketService.start();
    this.groupTransferService.start();
    this.haClient.start();
}

创建临时文件是用于在重启broker时来识别上一次关闭是否是正常关闭，执行不同的恢复操作。

 private void createTempFile() throws IOException {
    String fileName = StorePathConfigHelper.getAbortFile(this.messageStoreConfig.getStorePathRootDir());
    File file = new File(fileName);
    MappedFile.ensureDirOK(file.getParent());
    boolean result = file.createNewFile();
    log.info(fileName + (result ? " create OK" : " already exists"));
}

启动定时程序定时清除过期的消息存储文件以及消息消费队列文件，定时检查自身文件的创建是否有问题，根据debugLockEnable是否有效来检测消息写入时间是否有异常情况，并打印当前栈情况到文件中可以参考。

 private void addScheduleTask() {


    this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
        @Override
        public void run() {
            DefaultMessageStore.this.cleanFilesPeriodically();
        }
    }, 1000 * 60, this.messageStoreConfig.getCleanResourceInterval(), TimeUnit.MILLISECONDS);


    this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
        @Override
        public void run() {
            DefaultMessageStore.this.checkSelf();
        }
    }, 1, 10, TimeUnit.MINUTES);


    this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
        @Override
        public void run() {
            if (DefaultMessageStore.this.getMessageStoreConfig().isDebugLockEnable()) {
                try {
                    if (DefaultMessageStore.this.commitLog.getBeginTimeInLock() != 0) {
                        long lockTime = System.currentTimeMillis() - DefaultMessageStore.this.commitLog.getBeginTimeInLock();
                        if (lockTime > 1000 && lockTime < 10000000) {


                            String stack = UtilAll.jstack();
                            final String fileName = System.getProperty("user.home") + File.separator + "debug/lock/stack-"
                                + DefaultMessageStore.this.commitLog.getBeginTimeInLock() + "-" + lockTime;
                            MixAll.string2FileNotSafe(stack, fileName);
                        }
                    }
                } catch (Exception e) {
                }
            }
        }
    }, 1, 1, TimeUnit.SECONDS);


    // this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
    // @Override
    // public void run() {
    // DefaultMessageStore.this.cleanExpiredConsumerQueue();
    // }
    // }, 1, 1, TimeUnit.HOURS);
}

数据库

文章转载自徘徊笔记，如果涉嫌侵权，请发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。

RocketMQ源码分析之Broker（4）

评论