应用场景
分析链路下的实时主机的流量交互
部署
安装ELK，这里选择7.8.1版本,下载安装包上传至服务器中
1
2
3
elasticsearch-7.8.1-x86_64.rpm
logstash-7.8.1.rpm
kibana-7.8.1-x86_64.rpm
使用 rpm 安装
1
rpm -ivh elasticsearch-7.3.2-x86_64.rpm logstash-7.3.2.rpm kibana-7.3.2-x86_64.rpm
重新加载系统服务并开机自启动
1
2
3
4
systemctl daemon-reload
systemctl enable elasticsearch.service
systemctl enable kibana.service
systemctl enable logstash.service
安装elasticsearch
修改文件句柄数
1
2
3
4
5
ulimit -n 65536
vim /etc/security/limits.conf
* soft nofile 65536
* hard nofile 65536
创建es的数据目录
1
2
mkdir -p /data/elk_data
chown elasticsearch:elasticsearch /data/elk_data
配置传输层TLS/SSL加密传输
1
2
3
4
5
6
7
8
/usr/share/elasticsearch/bin/elasticsearch-certutil ca
ENTER ENTER
/usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
ENTER ENTER ENTER
mkdir /etc/elasticsearch/certs
cp /usr/share/elasticsearch/elastic-* /etc/elasticsearch/certs
chown -R elasticsearch:elasticsearch /etc/elasticsearch/certs
运行 Elasticsearch 的密码配置工具，为各种内置用户生成随机的密码。
1
/usr/share/elasticsearch/bin/elasticsearch-setup-passwords auto
修改es配置文件
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
vim /etc/elasticsearch/elasticsearch.yml
cluster.name: my-elk-culster
node.name: node1
path.data: /data/elk_data
path.logs: /var/log/elasticsearch/
bootstrap.memory_lock: false
network.host: 0.0.0.0
http.port: 9200
cluster.initial_master_nodes: ["192.168.32.100:9300"]
discovery.zen.ping.unicast.hosts: ["192.168.32.100"]
xpack.security.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12
indices.query.bool.max_clause_count: 8192
search.max_buckets: 100000
启动elasticsearch
1
systemctl start elasticsearch
安装kibana
修改kibana配置文件
1
2
3
4
5
6
7
8
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://127.0.0.1:9200"]
kibana.index: ".kibana"
i18n.locale: "zh-CN"
xpack.security.enabled: true
elasticsearch.username: "kibana_system"
elasticsearch.password: "ChMu94Wyh6zRAfRlRyJM"
启动kibana
1
systemctl start kibana
安装elastiflow
修改logstash的jvm配置
1
2
3
vim /etc/logstash/jvm.options
-Xms4g
-Xmx4g
添加和更新所需的Logstash插件
1
2
3
4
5
6
7
/usr/share/logstash/bin/logstash-plugin install logstash-codec-sflow
/usr/share/logstash/bin/logstash-plugin update logstash-codec-netflow
/usr/share/logstash/bin/logstash-plugin update logstash-input-udp
/usr/share/logstash/bin/logstash-plugin update logstash-input-tcp
/usr/share/logstash/bin/logstash-plugin update logstash-filter-dns
/usr/share/logstash/bin/logstash-plugin update logstash-filter-geoip
/usr/share/logstash/bin/logstash-plugin update logstash-filter-translate
下载第三方elastiflow
1
2
cd /tmp/
git clone https://github.com/robcowart/elastiflow.git
将elastiflow文件夹复制到相关目录,并修改elastiflow.conf
1
2
3
4
5
6
7
8
9
10
// 处理数据所需要的配置文件
cp -a elastiflow-master/logstash/elastiflow /etc/logstash
mkdir -p /etc/systemd/system/logstash.service.d/
// 运行elastflow时所需要的环境变量
cp -a elastiflow-master/logstash.service.d/elastiflow.conf /etc/systemd/system/logstash.service.d/
修改 /etc/systemd/system/logstash.service.d/elastiflow.conf中的es集群认证信息
$ELASTIFLOW_ES_USER
$ELASTIFLOW_ES_PASSWD
配置logstash pipeline文件
1
2
3
4
5
6
7
8
9
10
vim /etc/logstash/pipelines.yml
# This file is where you define your pipelines. You can define multiple.
# For more information on multiple pipelines, see the documentation:
# https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html
- pipeline.id: main
path.config: "/etc/logstash/conf.d/*.conf"
- pipeline.id: elastiflow
path.config: "/etc/logstash/elastiflow/conf.d/*.conf"
执行系统脚本
1
/usr/share/logstash/bin/system-install
启动logstash服务
1
2
3
systemctl daemon-reload
systemctl enable logstash
systemctl start logstash
将elastiflow对应的kibana模板文件导入到kibana，elastiflow/kibana/elastiflow.kibana.7.5.x.ndjson
