通过docker compose部署EFK
使用到的镜像:
docker pull store/elastic/filebeat:7.6.2
docker pull elasticsearch:7.6.2
docker pull kibana:7.6.2
为了更好理解docker编排文件下filebeat和kibana的配置方法,这里的Elasticsearch使用双节点部署。实际生产环境不会这样配置Elasticsearch集群。(官方的集群样例,请参考这里:https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html
Docker Compose文件里创建两个节点的Elasticsearch集群。节点es01在本地主机上侦听9200端口,es02通过Docker网络与es01对话。
volumes data01和data02是的Docker存储节点数据目录,以便数据在重新启动时保持不变。如果目录不存在,docker compose会在您启动集群时创建它们(如果不指定位置,默认位于/var/lib/docker/volumes/目录下)。
docker-compose.yml文件如下:
1
version: '3'
2
services:
3
es01:
4
image: elasticsearch:7.6.2
5
container_name: es01
6
environment:
7
- node.name=es01
8
- cluster.name=es-docker-cluster
9
- discovery.seed_hosts=es02
10
- cluster.initial_master_nodes=es01,es02
11
- bootstrap.memory_lock=true
12
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
13
ulimits:
14
memlock:
15
soft: -1
16
hard: -1
17
volumes:
18
- data01:/usr/share/elasticsearch/data
19
ports:
20
- 9200:9200
21
networks:
22
- elastic
23
es02:
24
image: elasticsearch:7.6.2
25
container_name: es02
26
environment:
27
- node.name=es02
28
- cluster.name=es-docker-cluster
29
- discovery.seed_hosts=es01
30
- cluster.initial_master_nodes=es01,es02
31
- bootstrap.memory_lock=true
32
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
33
ulimits:
34
memlock:
35
soft: -1
36
hard: -1
37
volumes:
38
- data02:/usr/share/elasticsearch/data
39
networks:
40
- elastic
41
filebeat:
42
image: store/elastic/filebeat:7.6.2
43
container_name: filebeat
44
user: root
45
environment:
46
- strict.perms=false
47
- output.elasticsearch.hosts=["es1:9200"]
48
volumes:
49
- ./filebeat.docker-compose.yml:/usr/share/filebeat/filebeat.yml:ro
50
- /var/lib/docker/containers:/var/lib/docker/containers:ro
51
- /var/run/docker.sock:/var/run/docker.sock
52
networks:
53
- elastic
54
depends_on:
55
- es01
56
kibana:
57
image: kibana:7.6.2
58
container_name: kibana
59
environment:
60
ELASTICSEARCH_HOSTS: http://es01:9200
61
networks:
62
- elastic
63
expose:
64
- "5601"
65
ports:
66
- "5601:5601"
67
depends_on:
68
- es01
69
70
volumes:
71
data01:
72
driver: local
73
data02:
74
driver: local
75
76
networks:
77
elastic:
78
driver: bridge
Copied!
filebeat.docker-compose.yml文件内容:
1
filebeat.config:
2
modules:
3
path: ${path.config}/modules.d/*.yml
4
reload.enabled: false
5
6
filebeat.autodiscover:
7
providers:
8
- type: docker
9
hints.enabled: true
10
11
processors:
12
- add_cloud_metadata: ~
13
14
output.elasticsearch:
15
hosts: '${ELASTICSEARCH_HOSTS:es01:9200}'
Copied!
常见问题:
1)vm.max_map_count设置问题
1
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
Copied!
解决办法:
这是由于默认vm.max_map_count=65530,因此缺省配置下,单个jvm能开启的最大线程数为其一半,即3w左右,大概32k的量。所以要调大到262144。(确保宿主机内存大于4G的情况下)
echo vm.max_map_count=262144 >> /etc/sysctl.conf && sysctl -p
检查配置是否生效:
1
#cat /proc/sys/vm/max_map_count
2
262144
Copied!
2)Data volume数据清除
Docker compose文件内配置,当容器启动后,会在/var/lib/docker/volumes/filebeat_data01/_data/目录下生成数据并持久化,在每次启动集群时都会复用。如果要删除这些数据,我们需要在停止集群时加上-v参数: docker-compose down -v.
1
# docker-compose down -v
2
Stopping es02 ... done
3
Stopping es01 ... done
4
Removing es02 ... done
5
Removing es01 ... done
6
Removing network filebeat_elastic
7
Removing volume filebeat_data02
8
Removing volume filebeat_data01
Copied!
3)手动创建的Kibana如何加入集群?
如果不使用docker compose同时创建的kibana,加入Elasticsearch集群时,需要指定集群的网络名,以及连接集群中的Elasticsearch节点名字,例如:
1
docker run -d --name kibana-cluster --net filebeat_elastic --link es01:elasticsearch -p 5601:5601 kibana:7.6.2
Copied!
Last modified 1yr ago
Copy link