skywalking—docker镜像构建k8s部署
前言
skywalking是个非常不错的apm产品，但是在使用过程中有个非常蛋疼的问题，在基于es的存储情况下，es的数据一有问题，就会导致整个skywalking web ui服务不可用，然后需要a? ! M /gent端一个服务一个服务的停用，然后服务重新部署后好，全部走一遍。这种问题同样也会存在skywalking的版本升级迭代中。而且apm 这种过程数据是允许丢弃的，默认skywalking中关于trace的数据记录只保存了90分钟。故博主准备将skywalking的部署容器化，一键部署升级。下文是整个skywas Q z A dlking 容器化部署的过程。
目标：将sk` X Oywalking的docker镜像运行在k8s的集群环境中提供服务
docker镜像构建
FROM registry.cn-xx.xx.com/keking/jdk:1.8^ U ? W p Z l $
ADD apach^ M v z k N $ 0 we-skywalking-apm-incubating/ /opt/apache-skywalking-apm-incubating/[ ` P b b 2 X o
RUN ln -sK . 2 U Z y @f /usr/shaS 7 M / 6 +re/zoneinfo/Asia/Shanghai /etc/localtime \\
&& echo \'Asia/Sf - O ;hanghaH O 3 / + ~ 6 u vi\' >/etc/timezone \\i J Y
&& chmod +x /opt/apachH i ) V C 3 fe-skywalking-apm-incubating/config/setApplicationEnv.sh \\
&& chmod +x^ y t /opt/apache-skywalking-apm-incubating/webm S r F 8 w % , !app/setWebAppEnv.sh \\
&, Owebapp/setv y 3WebAppEnv.sh \\
&& /opr ; Lt/apacC E Hhe-skywalking-apm-incubat a _ 5tinO Y / w 4 Zg/bin/startup.sh在编写Dockerfile时需要考虑几个问题：skywalking中哪些配置需要动态配置（运行时设置）？怎么保证进程一直运L h } + V m 3行（skywalking 的stars ^ 0tup.shG U F _和tm u 2 R Y ; h &RPC:
gRPC:
#os real network ipK E +(bindis L 9 P x q . $ngx c P G r } p t required), for agent to uplink data(trace/metrics) to collector. agent--(grpc)--> collector
host: #real_host
port: 11800
# Set tC F 9 e ^hese two setting to open ssl
#sslCertChainFile: $path
#sslPrivateKeyFile: $path
# Set your own token to active auth
#authC P % = Z I QenR W e o M ; ` jtication: xxxxxx
agent_jetty:
jetty:
# OSZ o o ^ d ! k L n real ne` P ^ + } _ 7 z atwork IP(binding required), for ag0 @ H 9 T went to uplink data(trace/metrics) to collector through HTTP. agent--(HTTP)--> collector
# SkyWalking natV c ^ive Java/.Net/node.js ac 0 s Mgents don\'t use this.
# Ope: / f !n this for other implementor.
host: 0.0.0.0
port: 12800
contextPath: /
analysis_register:
default[ s g q r:
analysis_jvm:
default:
analysis_segment_parser:
default:
buffe( ! T 9 L s ErFilePath: ../buffer/
bufferOffsetMaxFileSize: 10M
bufferSegmentMaxFila : ! k ^ a C h @eSize: 500M
bufferFileCleanWhenR 8 ) T (estartb r - = l: true
ui:
jetty:
# Stay in `localhost` if UI starts up in default mode.
# Change it to OS real network IP(binding required), if deploy collector in different maj 6 l j +chine.
host: 0.0.0.0
port: 12800
contextPath: /
storag y 6 n O e c C yge:
elasticsearch:
clusterName: #elasticsearch_clusterName
clusterTranspS 0 X ` , tortSniffer: true
clusterNodes: #elasticN % 3 hsearch_clusterNodg 1 ves
indexSharb , R e edsNumber:I h e G - Z a M 2
indexRe; Y XplicasNumber: 0
highPerformanceMode: true
# Batch process setting, refer to https://www.elastic.co/guide/en/elasticsearch/client/java-api/5.5/java-docs-bulk-processor.html
bulkActions: 2000 # Execute the bulk every 2000 requests
bulkSize: 20 # flush the bulk every 20mb
flushInter2 n ` P t *val: 10 # flush the bulk every 10 second* 5 h 7s whatever the number of requests
concurrentRequests: 2 # the number oh ^ N I $ % 4f concurrent requests
# Set a timeout on metric data. After the timeout has expired, tm p z jhe metric data wile Z $ I l x X k 6l auto5 y w N Amatically be deleted.
traceDaO 3 [ ; S 1 } m CtaTTL: 288N F { n A l0 # Unit is minute
minuteMetricDataTTL: 90 # Unit is minute
hourMetricDataTTL: 36 # Unit is hour
dayMetV 6 P BricDataTTL3 j I T : 45x J ; * W alarm threshold
a` A X = ` y H epplicationApdexThreshold: 2000
servi; ] n I X 4 ^ oceErrorRateThreshold: 10.00
serviceAverageResponsT B K n j heTimeThreshold:J : i W E ^ f 2000
instanceM 5 + - % mErrorRateThreshold: 10.00
instanceAverageResponseTimeThreshold: 2000
applicationEri 6 ; [ e R CrorRateThreshold: 10.00
applicatib G @ # Y !onAverageResponseTimeThreshold: 2000
# thermodynamic
thermodynamicResponseTij T v $ _ 8 E Umef 6 !Step: 50
thermodynamicCountOfResponseTimeSteps: 40
# max collection\'s st P Iize of worker cache collection, setting it smaller when collector OutOfMem$ : V g 8 ` e Dory crashed.
workerCacheMaxSi4 S d i Eze: 10000
#receiver_zipkin:
# default:
# host: localhost
# port: 9411
# contextPath: /
webapp.yml
动态配置：密码，grpc等S - 7 P j $ k ( Z需要绑定主机的ip都需要运行时设置，这里我们在启动skywalking的startup.sh只之前，先执行了两个设置配置的脚本，通过k8s在运行时设置的环境变量来替换需要动态配置的参数
setApplicationEnv.sh
#!/usr/bin/env sh
sed -i \"s/#elasticsearch_clusterNodes/${elasticsearch_clusterNodes}/g\" /opt/apache-skywalking-apm-incubating/config/application.yml
sed -i \"s/#elasticsearch_cluster( o ^ ` & ) =Name/${elasticsearchg z s q ~ T 2_clusterF z D #Name}/g\" /opt/apache-skywalking-apm-incubating/config/application.yml
sed -i \"s/#real_host/${real_host}/g\" /opt/apache-skywalking-apm-incubati& 6 s 4 Vng/co[ V G G d 9nfig/a w 9 u y * l kapplication.ymh ? w L { R *lsetWebAppEnv.sh
#!/usr/bin/env sh
sed -i \"s/#skywalking_password/${skywa^ k 1 t M % s plking_password}/g\" /opt/apX B ] eg 启动脚本startup.sh末尾追加\"tail -fn 100 /opt/apache-skywalking-apm-incuba4 M = p 并不断输出webapp.log的日志Kubernetes中部署
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: skywalking
namespace: uat
spe} O _ j t c Rc:
replicas: 1
selector:
matchLabS V Q _ j ^els:
app: skywalking
templat* ~ ^ C = &e:
metadata:
labels:
app: skywalking
spec:
imagePullSecreB 8 l ) L V w * &ts:
- name: registry-pull-secret
nodeSelector:
apm: skywalking
containers:
- name: skywalking
image: registry.cn-xx.xx.com/kekinp ^ g L Dg/kk-skywalking:5.2
imagePullPolicy: A? J )lways
env:
- name: elasticsearch_clusterName
valuu q z { w i ~ )e: elasticsearch
- name: elasticsearch_clusterNodes
value8 + c Z: 172.1& [ 0 m t } $6.16.129:31300
- name: skywd e M _alking_password
value: xxx
- name: realC w M &_host
valueFrom:
fieldRef:
fieldPath: status.podIPX w ( K u _
resources:
limits:
cpu: 1000m
memory: 4Gi
requests:
cpu: 700m
memory: 2Gi
--f 8 L - r r-
apiVersion: v1
kind: Service
metadata:
name: skq B V ^ Pywalking
namespace: uat
labels:
app: skywalking
spec:
selector:
app: skywalking
ports:
- name: web-a
port: 8080
targetPort: 8080
nodePort: 31180
- name: web-b
port: 10800
targetPort: 10800
nodePort: 31181
-z O n + 8 } name: we~ f 4 wb-c
port: 11800
targetPort: 11800
nodePort: 31182` G d [ N :
- name: web-d
port: 12800
targetPort: 12800
nodePort: 31183
type: NodePort
Kubernetes部署脚本中唯一需要注意的就是env中关于pod ip的获取，skywalking中有几个ip必须绑定容器的真实ip，这个地方可以通过环境变量设置到容器里面去
结语
整个skywalking容器化部署从测试到可用大概耗时1天，其中花了个多小时整了下谭兄的skywalkinL 3 d vg-docker镜像（https://hub.dockep 8 tang/skywalking-docker/），发现有个脚本有权限问题（谭兄反馈已解决，还没来~ 0，然后通过集群的nodePort映射出来,这个时候的agent通过集群ip+31181) % m ?是可以访问到namingh % Z E o )服务的，然后通过naming服务获取到的collector gRPC服务缺变成了0.0.0.0:11800, 这个地址agent肯定访问不到collector的，后面通过绑定pod ip的方式解决了这个问题。