Kibana Elasticsearch和Fluentd

发表于 阅读次数： 本文字数： 1.8k 阅读时长 ≈ 2 分钟

Elasticsearch

https://www.elastic.co/cn/downloads/elasticsearch

1
2
3
4
5
6
7
8
9
10
11
12
13
14
使用非root用户
vim config/jvm.options
-Xms1g
-Xmx2g
vim config/elasticsearch.yml
xpack.security.http.ssl:
  enabled: false # 关闭
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: false # 关闭

启动后看日志，密码在里面

Kibana

https://www.elastic.co/cn/downloads/kibana

1
2
3
4
5
6
7
使用非root用户
vim config/kibana.yml
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://localhost:9200"]
elasticsearch.username: "test"
elasticsearch.password: "123456"
elasticsearch.ssl.verificationMode: none

Fluentd

1
2
3
4
5
6
7
# fluent-package 5 (LTS)
curl -fsSL https://toolbelt.treasuredata.com/sh/install-redhat-fluent-package5-lts.sh | sh

查看fluetd配置文件是否正确：fluentd --dry-run -c fluent.conf

systemctl start fluentd.service
# 修改 /usr/lib/systemd/system/fluentd.service 改为root运行

编辑配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
vim /etc/fluent/fluentd.conf
<source>
  @type    tail
  format   /^(?<all>.*)$/
  path     /var/log/nginx/access.log*
  #pos_file /home/fluentd/nginx-access.pos
  tag      nginx.access
</source>


<match nginx.access>
  @type elasticsearch
  host 127.0.0.1
  port 9200
  flush_interval 5s
  logstash_format true
  logstash_prefix nginx-access
  scheme http
  user test
  password 123456
</match>


# 访问 http://172.12.1.150:5601/app/management/kibana/dataViews
# 这样能在kibana 创建数据视图时搜到nginx 的数据集了

多行匹配

1
2
3
4
5
6
7
8
9
10
11
12
<source>
  @type tail
  path /path/to/java.log
  pos_file /var/log/fluentd/java.log.pos
  tag java.logs
  <parse>
    @type multiline
    format_firstline /^\d{4}-\d{2}-\d{2}/
    format1 /^(?<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})\s+(?<message>.*)$/
    multiline_start_regexp /^\d{4}-\d{2}-\d{2}/
  </parse>
</source>