RocketMQ源码分析之有序消息（2）

徘徊笔记 2019-05-08

221

来源：https://github.com/apache/rocketmq

broker接收消息队列解锁请求

private RemotingCommand unlockBatchMQ(ChannelHandlerContext ctx,
    RemotingCommand request) throws RemotingCommandException {
    final RemotingCommand response = RemotingCommand.createResponseCommand(null);
    UnlockBatchRequestBody requestBody = UnlockBatchRequestBody.decode(request.getBody(), UnlockBatchRequestBody.class);


    this.brokerController.getRebalanceLockManager().unlockBatch(
        requestBody.getConsumerGroup(),
        requestBody.getMqSet(),
        requestBody.getClientId());


    response.setCode(ResponseCode.SUCCESS);
    response.setRemark(null);
    return response;
}

当该消息队列被改消费者实例锁定时就移除锁定关系

public void unlockBatch(final String group, final Set<MessageQueue> mqs, final String clientId) {
    try {
        this.lock.lockInterruptibly();
        try {
            ConcurrentHashMap<MessageQueue, LockEntry> groupValue = this.mqLockTable.get(group);
            if (null != groupValue) {
                for (MessageQueue mq : mqs) {
                    LockEntry lockEntry = groupValue.get(mq);
                    if (null != lockEntry) {
                        if (lockEntry.getClientId().equals(clientId)) {
                            groupValue.remove(mq);
                            log.info("unlockBatch, Group: {} {} {}",
                                group,
                                mq,
                                clientId);
                        } else {
                            log.warn("unlockBatch, but mq locked by other client: {}, Group: {} {} {}",
                                lockEntry.getClientId(),
                                group,
                                mq,
                                clientId);
                        }
                    } else {
                        log.warn("unlockBatch, but mq not locked, Group: {} {} {}",
                            group,
                            mq,
                            clientId);
                    }
                }
            } else {
                log.warn("unlockBatch, group not exist, Group: {} {}",
                    group,
                    clientId);
            }
        } finally {
            this.lock.unlock();
        }
    } catch (InterruptedException e) {
        log.error("putMessage exception", e);
    }
}

消费端平衡消息队列时间间隔为20s，broker端消息队列锁定时间间隔为60s，所以需要不断更新最近更新时间，防止消息队列锁定过期，所以消费消息服务启动定时程序不断当前持有的消息队列的更新锁定时间

public void start() {
    if (MessageModel.CLUSTERING.equals(ConsumeMessageOrderlyService.this.defaultMQPushConsumerImpl.messageModel())) {
        this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
            @Override
            public void run() {
                ConsumeMessageOrderlyService.this.lockMQPeriodically();
            }
        }, 1000 * 1, ProcessQueue.REBALANCE_LOCK_INTERVAL, TimeUnit.MILLISECONDS);
    }
}

定期锁定程序

public synchronized void lockMQPeriodically() {
    if (!this.stopped) {
        this.defaultMQPushConsumerImpl.getRebalanceImpl().lockAll();
    }
}

锁定当前持有的消息队列，给对应的消息处理队列置为对应的锁定状态。

private HashMap<String/* brokerName */, Set<MessageQueue>> buildProcessQueueTableByBrokerName() {
    HashMap<String, Set<MessageQueue>> result = new HashMap<String, Set<MessageQueue>>();
    for (MessageQueue mq : this.processQueueTable.keySet()) {
        Set<MessageQueue> mqs = result.get(mq.getBrokerName());
        if (null == mqs) {
            mqs = new HashSet<MessageQueue>();
            result.put(mq.getBrokerName(), mqs);
        }


        mqs.add(mq);
    }


    return result;
}


public void lockAll() {
    HashMap<String, Set<MessageQueue>> brokerMqs = this.buildProcessQueueTableByBrokerName();


    Iterator<Entry<String, Set<MessageQueue>>> it = brokerMqs.entrySet().iterator();
    while (it.hasNext()) {
        Entry<String, Set<MessageQueue>> entry = it.next();
        final String brokerName = entry.getKey();
        final Set<MessageQueue> mqs = entry.getValue();


        if (mqs.isEmpty())
            continue;


        FindBrokerResult findBrokerResult = this.mQClientFactory.findBrokerAddressInSubscribe(brokerName, MixAll.MASTER_ID, true);
        if (findBrokerResult != null) {
            LockBatchRequestBody requestBody = new LockBatchRequestBody();
            requestBody.setConsumerGroup(this.consumerGroup);
            requestBody.setClientId(this.mQClientFactory.getClientId());
            requestBody.setMqSet(mqs);


            try {
                Set<MessageQueue> lockOKMQSet =
                    this.mQClientFactory.getMQClientAPIImpl().lockBatchMQ(findBrokerResult.getBrokerAddr(), requestBody, 1000);


                for (MessageQueue mq : lockOKMQSet) {
                    ProcessQueue processQueue = this.processQueueTable.get(mq);
                    if (processQueue != null) {
                        if (!processQueue.isLocked()) {
                            log.info("the message queue locked OK, Group: {} {}", this.consumerGroup, mq);
                        }


                        processQueue.setLocked(true);
                        processQueue.setLastLockTimestamp(System.currentTimeMillis());
                    }
                }
                for (MessageQueue mq : mqs) {
                    if (!lockOKMQSet.contains(mq)) {
                        ProcessQueue processQueue = this.processQueueTable.get(mq);
                        if (processQueue != null) {
                            processQueue.setLocked(false);
                            log.warn("the message queue locked Failed, Group: {} {}", this.consumerGroup, mq);
                        }
                    }
                }
            } catch (Exception e) {
                log.error("lockBatchMQ exception, " + mqs, e);
            }
        }
    }
}

当第一次对有序消息队列进行拉取时，需要矫正初始偏移量

if (processQueue.isLocked()) {
    if (!pullRequest.isLockedFirst()) {
        final long offset = this.rebalanceImpl.computePullFromWhere(pullRequest.getMessageQueue());
        boolean brokerBusy = offset < pullRequest.getNextOffset();
        log.info("the first time to pull message, so fix offset from broker. pullRequest: {} NewOffset: {} brokerBusy: {}",
            pullRequest, offset, brokerBusy);
        if (brokerBusy) {
            log.info("[NOTIFYME]the first time to pull message, but pull request offset larger than broker consume offset. pullRequest: {} NewOffset: {}",
                pullRequest, offset);
        }


        pullRequest.setLockedFirst(true);
        pullRequest.setNextOffset(offset);
    }
} else {
    this.executePullRequestLater(pullRequest, PULL_TIME_DELAY_MILLS_WHEN_EXCEPTION);
    log.info("pull message later because not locked in broker, {}", pullRequest);
    return;
}

拉取到消息后往执行队列中放入消息，判断执行队列中消息集合是否为空，因为这次刚放进了多个消息，在判断consuming是否为false，代表之前的消息是否在消费或者已经消费完了，默认值为false

boolean dispatchToConsume = processQueue.putMessage(pullResult.getMsgFoundList());


if (!msgTreeMap.isEmpty() && !this.consuming) {
    dispatchToConsume = true;
    this.consuming = true;
}

如果执行队列中之前没有消息或者之前的消息正在被消费，则创建dispatchToConsume就为true，直接往线程池中加入消费请求任务，否则的话就不提交任务请求，因为正在消费的任务会持续从执行队列中取出消息。

public void submitConsumeRequest(
    final List<MessageExt> msgs,
    final ProcessQueue processQueue,
    final MessageQueue messageQueue,
    final boolean dispathToConsume) {
    if (dispathToConsume) {
        ConsumeRequest consumeRequest = new ConsumeRequest(processQueue, messageQueue);
        this.consumeExecutor.submit(consumeRequest);
    }
}

数据库

文章转载自徘徊笔记，如果涉嫌侵权，请发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。

RocketMQ源码分析之有序消息（2）

评论