部署PinPoint分布式调用链系统监控

pinpoint是开源在github上的一款APM监控工具，JAVA语言开发用于大规模分布式系统监控，对性能的影响最小(只增加约3％资源利用率，吃内存)，安装agent是无侵入式的，只需要在被测试的Tomcat中加上配置下，打下探针就可以监控整套程序了

IP	OS	安装项	备注
172.16.10.170	Centos 7.3	pinpoint	pinpoint的web展示端，逻辑控制机，以及Hbase存储
172.16.10.170	Centos 7.3	pinpoint-agent	主要用来采集数据，发送给pinpoint处理

官网流程图：
PinPoint

环境：
　　　Centos 7.3
　　　pinpoint 1.6.2
　　　hbase 1.2.5

[root@localhost software]# tar -zxf ../apache-tomcat-7.0.79.tar.gz -C /data/pinpoint/
[root@localhost software]# cd /data/pinpoint/
[root@localhost pinpoint]# mv apache-tomcat-7.0.79/ pinpoint-collector
[root@localhost pinpoint]# cp -a pinpoint-collector/ pinpoint-web
[root@localhost software]# tar -zxf hbase-1.2.5-bin.tar.gz -C /data/pinpoint/

#配置环境变量
[root@localhost ~]# cat /etc/profile.d/java.sh 
#!/bin/bash
JAVA_HOME=/usr/local/jdk1.7
HBASE_HOME='/data/pinpoint/hbase-1.2.5'
PATH=$JAVA_HOME/bin:$HBASE_HOME/bin:$PATH
export PATH
[root@localhost ~]# source /etc/profile.d/java.sh

[root@localhost ~]# cd $HBASE_HOME/conf
[root@localhost conf]# grep -i 'export JAVA_HOME' hbase-env.sh
#手动指定正确的java环境变量值
export JAVA_HOME=/usr/local/jdk1.7
[root@localhost conf]#

[root@localhost pinpoint]# unzip /opt/tools/software/pinpoint-collector-1.6.1.war -d pinpoint-collector/webapps/ROOT/
[root@localhost pinpoint]# unzip /opt/tools/software/pinpoint-web-1.6.1.war -d pinpoint-web/webapps/ROOT/
[root@localhost pinpoint]# 

#安装数据采集Agent
[root@localhost pinpoint]# mkdir pinpoint-agent
[root@localhost pinpoint]# tar -zxf /opt/tools/software/pinpoint-agent-1.6.1.tar.gz -C ./pinpoint-agent/
[root@localhost ~]# grep -i 'PINPOINT_AGENT' /etc/profile.d/java.sh 
PINPOINT_AGENT='/data/pinpoint/pinpoint-agent'
PATH=$JAVA_HOME/bin:$HBASE_HOME/bin:$PINPOINT_AGENT:$PATH
[root@localhost ~]#
[root@localhost ~]# cd $PINPOINT_AGENT
[root@localhost pinpoint-agent]# grep -i '^profiler.collector.ip' pinpoint.config 
profiler.collector.ip=172.16.10.170
[root@localhost pinpoint-agent]#

#start.sh里面写绝对路径，不要使用变量
# ${AGENT_ID}:必须要唯一，推荐使用应用名称+本机ip，如：172.16.1.170.risk
# ${APPLICATION_NAME}:应用名称，如：soros-service-risk

#dubbo pinpoint配置
[root@localhost project]# grep -i 'POINPOINT' soros-service-risk/bin/start.sh 
POINPOINT="-javaagent:/data/pinpoint/pinpoint-agent/pinpoint-bootstrap-1.6.1.jar -Dpinpoint.agentId=172.16.1.170.risk -Dpinpoint.applicationName=soros-service-risk"
nohup java $JAVA_OPTS $JAVA_MEM_OPTS $JAVA_DEBUG_OPTS $JAVA_JMX_OPTS $POINPOINT -classpath $CONF_DIR:$LIB_JARS com.alibaba.dubbo.container.Main > $STDOUT_FILE 2>&1 &
[root@localhost project]#

#tomcat pinpoint配置
[jusys@localhost bin]$ pwd
/data/soft/julend/tomcat1/bin
[jusys@localhost bin]$ grep -iA2 "Xms1g" catalina.sh 
JAVA_OPTS='-server -Xms1g -Xmx2g -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:NewSize=512m -XX:MaxPermSize=256m -Dorg.apache.catalina.startup.EXIT_ON_INIT_FAILURE=tr
ue'
CATALINA_OPTS="-javaagent:/usr/local/pinpoint-agent/pinpoint-bootstrap-1.6.1-RC2.jar -Dpinpoint.agentId=10.10.2.101.webserver -Dpinpoint.applicationName=soros-webserver"
[jusys@localhost bin]$

[root@localhost pinpoint]# cd hbase-1.2.5/conf/
[root@localhost conf]# grep -iv ^$ hbase-site.xml | grep -iv '*'
#在hbase-site.xml下面加如下配置
<configuration>
  <property>
    <name>hbase.zookeeper.property.clientPort</name>
    <value>2181</value>
  </property>
  <property>
    <name>hbase.zookeeper.property.dataDir</name>
    <value>/data/pinpoint/data/hbase-${user.name}/zookeeper</value>
  </property>
  <property>
    <name>hbase.rootdir</name>
    <value>/data/pinpoint/data/hbase-${user.name}/hbase</value>
  </property>
  <property>
    <name>hbase.tmp.dir</name>
    <value>/data/pinpoint/data/hbase-${user.name}</value>
  </property>
  <property>
    <name>hbase.local.dir</name>
    <value>/data/pinpoint/data/hbase-${user.name}/local/</value>
  </property>
</configuration>
[root@localhost conf]#

#修改Hbase PID文件位置
[root@localhost conf]# grep -iB1 'HBASE_PID_DIR' hbase-env.sh 
# The directory where pid files are stored. /tmp by default.
export HBASE_PID_DIR=/data/pinpoint/data
[root@localhost conf]#

#必须要用普通用户启动，hbase用普通用户导入初始化数据
#hbase/pinpoint-collector/pinpoint-web需要部署到同一台服务器上
[root@localhost pinpoint]# chown jusys:jusys hbase-1.2.5/ pinpoint-agent/ pinpoint-collector/ pinpoint-web/ -R

[jusys@localhost pinpoint]$ sh hbase-1.2.5/bin/start-hbase.sh
[jusys@localhost ~]$ cd /data/pinpoint/hbase-1.2.5/bin/
#初始化数据表
[jusys@localhost bin]$ hbase shell < /opt/tools/software/hbase-create.hbase    #hbase用普通用户导入初始化数据
2017-08-14 14:52:47,175 WARN  [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 1.2.5, rd7b05f79dee10e0ada614765bb354b93d615a157, Wed Mar  1 00:34:48 CST 2017
create 'AgentInfo', { NAME => 'Info', TTL => 31536000, DATA_BLOCK_ENCODING => 'PREFIX' }
0 row(s) in 1.4230 seconds
Hbase::Table - AgentInfo
...
list
TABLE                                                                                                                                                            
AgentEvent
AgentInfo
AgentLifeCycle
AgentStat
AgentStatV2
ApiMetaData
ApplicationIndex
ApplicationMapStatisticsCallee_Ver2
ApplicationMapStatisticsCaller_Ver2
ApplicationMapStatisticsSelf_Ver2
ApplicationTraceIndex
HostApplicationMap_Ver2
SqlMetaData_Ver2
StringMetaData
TraceV2
Traces
16 row(s) in 0.0190 seconds
["AgentEvent", "AgentInfo", "AgentLifeCycle", "AgentStat", "AgentStatV2", "ApiMetaData", "ApplicationIndex", "ApplicationMapStatisticsCallee_Ver2", "ApplicationMapStatistic
sCaller_Ver2", "ApplicationMapStatisticsSelf_Ver2", "ApplicationTraceIndex", "HostApplicationMap_Ver2", "SqlMetaData_Ver2", "StringMetaData", "TraceV2", "Traces"]
exit
[jusys@localhost pinpoint]$ sh pinpoint-collector/bin/startup.sh
[jusys@localhost pinpoint]$ sh pinpoint-web/bin/startup.sh

如果pinpoint-agent日志输出这个，只能重新部署了，hbase初始化数据，启动pinpoint-collector和pinpoint-web

2017-08-14 16:57:07 [INFO ](.p.r.c.DefaultPinpointClientHandler) DefaultPinpointClientHandler@18f5940a initReconnect() completed.
2017-08-14 16:57:07 [INFO ](.p.r.c.DefaultPinpointClientHandler) DefaultPinpointClientHandler@18f5940a channelClosed() started.
2017-08-14 16:57:07 [INFO ](.p.r.c.DefaultPinpointClientHandler) DefaultPinpointClientHandler@18f5940a exceptionCaught() occurred. state:BEING_CONNECT, caused:Connection re
fused: /172.16.10.172:9994.2017-08-14 16:57:07 [INFO ](.n.p.r.c.PinpointClientHandlerState) DefaultPinpointClientHandler@18f5940a stateTo() completed. Socket state change success(updateWanted:CONNECT
_FAILED ,before:BEING_CONNECT ,current:CONNECT_FAILED).2017-08-14 16:57:07 [INFO ](c.n.p.r.c.PinpointClientHandshaker ) PinpointClientHandshaker@15bf565b handshakeAbort() started.
2017-08-14 16:57:07 [INFO ](c.n.p.r.c.PinpointClientHandshaker ) PinpointClientHandshaker@15bf565b unexpected state

如果出现这个，可能是PinPoint安装包(.war)损坏，去git上重新下载安装即可
PinPoint_set

#这个报错可以忽略，暂不影响pinpoint使用
java.lang.NullPointerException: calleeApplicationName must not be null
	at com.navercorp.pinpoint.collector.dao.hbase.HbaseMapStatisticsCallerDao.update(HbaseMapStatisticsCallerDao.java:88)
	at com.navercorp.pinpoint.collector.handler.StatisticsHandler.updateCaller(StatisticsHandler.java:59)
	at com.navercorp.pinpoint.collector.handler.SpanHandler.insertSpanEventStat(SpanHandler.java:183)
	at com.navercorp.pinpoint.collector.handler.SpanHandler.handleSimple(SpanHandler.java:90)
	at com.navercorp.pinpoint.collector.receiver.AbstractDispatchHandler.dispatchSendMessage(AbstractDispatchHandler.java:56)
	at com.navercorp.pinpoint.collector.receiver.DispatchHandlerWrapper.dispatchSendMessage(DispatchHandlerWrapper.java:49)
	at com.navercorp.pinpoint.collector.receiver.udp.BaseUDPHandlerFactory$DispatchPacket.receive(BaseUDPHandlerFactory.java:122)
	at com.navercorp.pinpoint.collector.receiver.udp.BaseUDPHandlerFactory$DispatchPacket.receive(BaseUDPHandlerFactory.java:101)
	at com.navercorp.pinpoint.collector.receiver.udp.PooledPacketWrap.run(PooledPacketWrap.java:51)
	at com.navercorp.pinpoint.collector.receiver.udp.UDPReceiver$1.run(UDPReceiver.java:186)
	at com.navercorp.pinpoint.collector.monitor.MonitoredExecutorService$InstrumentedRunnable.run(MonitoredExecutorService.java:208)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)