MQ 错误 MQRC_CONNECTION_BROKEN 的处理过程

应用服务器日志

/usr/WebSphere/AppServer/logs/roserver1/SystemErr.log

[06-7-5 10:59:01:889 GMT+08:00] 26d298eb SystemErr R A MQException is found! [06-7-5 10:59:01:889 GMT+08:00] 26d298eb SystemErr R com.ibm.mq.MQException: MQJE001：发生 MQException：完成码 2，原因码 2009 MQJE016：连接期间，MQ 队列管理器立即关闭通道 \u5173闭原因 = 2009 [06-7-5 10:59:01:889 GMT+08:00] 26d298eb SystemErr R at com.ibm.mq.MQManagedConnectionJ11.<init>(MQManagedConnectionJ11.java:242) [06-7-5 10:59:01:889 GMT+08:00] 26d298eb SystemErr R at com.ibm.mq.MQClientManagedConnectionFactoryJ11._createManagedConnection(MQClientManagedConnectionFactoryJ11.java:276) [06-7-5 10:59:01:890 GMT+08:00] 26d298eb SystemErr R at com.ibm.mq.MQClientManagedConnectionFactoryJ11.createManagedConnection(MQClientManagedConnectionFactoryJ11.java(Inlined Compiled Code)) [06-7-5 10:59:01:890 GMT+08:00] 26d298eb SystemErr R at com.ibm.mq.StoredManagedConnection.<init>(StoredManagedConnection.java(Compiled Code)) [06-7-5 10:59:01:890 GMT+08:00] 26d298eb SystemErr R at com.ibm.mq.MQSimpleConnectionManager.allocateConnection(MQSimpleConnectionManager.java(Compiled Code)) [06-7-5 10:59:01:890 GMT+08:00] 26d298eb SystemErr R at com.ibm.mq.MQQueueManager.obtainBaseMQQueueManager(MQQueueManager.java(Compiled Code)) [06-7-5 10:59:01:890 GMT+08:00] 26d298eb SystemErr R at com.ibm.mq.MQQueueManager.construct(MQQueueManager.java(Compiled Code)) [06-7-5 10:59:01:890 GMT+08:00] 26d298eb SystemErr R at com.ibm.mq.MQQueueManager.<init>(MQQueueManager.java:469) [06-7-5 10:59:01:890 GMT+08:00] 26d298eb SystemErr R at com.icss.resourceone.sdkimpl.datachannel.mqimpl.mqi.MQIProcessor.connMQ(MQIProcessor.java:106) [06-7-5 10:59:01:890 GMT+08:00] 26d298eb SystemErr R at com.icss.resourceone.sdkimpl.datachannel.mqimpl.MQTransport.<init>(MQTransport.java:63) [06-7-5 10:59:01:890 GMT+08:00] 26d298eb SystemErr R at sun.reflect.GeneratedConstructorAccessor95.newInstance(Unknown Source) [06-7-5 10:59:01:890 GMT+08:00] 26d298eb SystemErr R at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java(Compiled Code)) [06-7-5 10:59:01:890 GMT+08:00] 26d298eb SystemErr R at java.lang.reflect.Constructor.newInstance(Constructor.java(Compiled Code)) [06-7-5 10:59:01:890 GMT+08:00] 26d298eb SystemErr R at java.lang.Class.newInstance3(Class.java(Compiled Code)) [06-7-5 10:59:01:890 GMT+08:00] 26d298eb SystemErr R at java.lang.Class.newInstance(Class.java(Compiled Code)) [06-7-5 10:59:01:890 GMT+08:00] 26d298eb SystemErr R at com.icss.resourceone.sdk.datachannel.Transport.getInstance(Transport.java:42) [06-7-5 10:59:01:890 GMT+08:00] 26d298eb SystemErr R at com.icss.oa.exchange.service.impl.OaTransferServiceImpl.send(OaTransferServiceImpl.java:112) [06-7-5 10:59:01:890 GMT+08:00] 26d298eb SystemErr R at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [06-7-5 10:59:01:890 GMT+08:00] 26d298eb SystemErr R at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java(Compiled Code)) [06-7-5 10:59:01:890 GMT+08:00] 26d298eb SystemErr R at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java(Compiled Code)) [06-7-5 10:59:01:891 GMT+08:00] 26d298eb SystemErr R at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java(Compiled Code)) [06-7-5 10:59:01:891 GMT+08:00] 26d298eb SystemErr R at java.lang.reflect.Method.invoke(Method.java(Compiled Code)) [06-7-5 10:59:01:891 GMT+08:00] 26d298eb SystemErr R at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java(Inlined Compiled Code)) [06-7-5 10:59:01:891 GMT+08:00] 26d298eb SystemErr R at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java(Inlined Compiled Code)) [06-7-5 10:59:01:891 GMT+08:00] 26d298eb SystemErr R at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java(Compiled Code)) [06-7-5 10:59:01:891 GMT+08:00] 26d298eb SystemErr R at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java(Compiled Code)) [06-7-5 10:59:01:891 GMT+08:00] 26d298eb SystemErr R at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java(Compiled Code)) [06-7-5 10:59:01:891 GMT+08:00] 26d298eb SystemErr R at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java(Compiled Code)) [06-7-5 10:59:01:891 GMT+08:00] 26d298eb SystemErr R at $Proxy12.send(Unknown Source) [06-7-5 10:59:01:891 GMT+08:00] 26d298eb SystemErr R at com.icss.oa.workflow.service.DispatchEndingTool.exec(DispatchEndingTool.java:62) [06-7-5 10:59:01:891 GMT+08:00] 26d298eb SystemErr R at com.icss.starflow.engine.impl.WfActivityToolImplementation.runSyncService(WfActivityToolImplementation.java:160) [06-7-5 10:59:01:891 GMT+08:00] 26d298eb SystemErr R at com.icss.starflow.engine.impl.WfActivityToolImplementation.runServices(WfActivityToolImplementation.java:85) [06-7-5 10:59:01:891 GMT+08:00] 26d298eb SystemErr R at com.icss.starflow.engine.impl.WfActivityToolImplementation.run(WfActivityToolImplementation.java:46) [06-7-5 10:59:01:891 GMT+08:00] 26d298eb SystemErr R at com.icss.starflow.engine.impl.WfActivityImpl.start(WfActivityImpl.java:733) [06-7-5 10:59:01:891 GMT+08:00] 26d298eb SystemErr R at com.icss.starflow.engine.impl.WfActivityImpl.activate(WfActivityImpl.java:379) [06-7-5 10:59:01:891 GMT+08:00] 26d298eb SystemErr R at com.icss.starflow.engine.service.ActivateActivityJob.exec(ActivateActivityJob.java:43) [06-7-5 10:59:01:891 GMT+08:00] 26d298eb SystemErr R at com.icss.resourceone.service.job.JobInvoker.run(JobInvoker.java(Compiled Code)) [06-7-5 10:59:01:891 GMT+08:00] 26d298eb SystemErr R at java.lang.Thread.run(Thread.java:567)

查找 MQ 错误码

C:\Documents and Settings\heaven>mqrc 2009 2009 0x000007d9 MQRC_CONNECTION_BROKEN

基本确定为 MQ 环境问题。为了找到确定的原因，继续下一步——看 MQ 日志。

/var/mqm/errors/AMQERR01.LOG

没有对应时间段内的日志。

/var/mqm/qmgrs/QM_HN/errors/AMQERR01.LOG

找到对应时间段内的日志：

EXPLANATION:
Channel program 'DC.SVRCONN' ended abnormally.
ACTION:
Look at previous error messages for channel program 'DC.SVRCONN' in the error
files to determine the cause of the failure.
----- amqrmrsa.c : 467 --------------------------------------------------------
07/05/06 10:59:01
AMQ9513: Maximum number of channels reached.

EXPLANATION:
The maximum number of channels that can be in use simultaneously has been
reached. The number of permitted channels is a configurable parameter in the
queue manager configuration file.
ACTION:
Wait for some of the operating channels to close. Retry the operation when some
channels are available.
----- amqrcsia.c : 1007 -------------------------------------------------------
07/05/06 10:59:01
AMQ9999: Channel program ended abnormally.

确定原因：AMQ9513: Maximum number of channels reached.

修改 MQ 配置参数

/var/mqm/qmgrs/QM_HN/qm.ini

CHANNELS: MAXCHANNELS=300 MAXACTIVECHANNELS=300

重启 MQ 队列管理器等

su - mqm
endmqm -i QM_HN

使用命令
ps -ef | grep runmqlsr
找到类似如下的输出
mqm 974980 1 0 14:47:47 - 0:00 runmqlsr -t tcp -m QM_HN -p 1818

kill -9 974980

strmqm QM_HN
runmqlsr -t tcp -m QM_HN -p 1818 &
strmqcsv QM_HN