filebeat 6.4.3 采集 nginx日志
成都创新互联是专业的永州网站建设公司，永州接单;提供成都网站建设、
网站设计,网页设计,网站设计,建网站,PHP网站建设等专业做网站服务;采用PHP框架,可快速的进行永州网站开发网页制作和功能扩展;专业做搜索引擎喜爱的网站,专业的做网站团队,希望更多企业前来合作!
nginx 日志格式规范：
将日志配置组合添加至nginx主配置文件；
log_format eslog '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
ES集群安装插件
集群收集nginx日志需要安装 ingest-user-agent ，ingest-geoip ，安装完成之后需要重启es 服务，否则数据无法正常录入es集群。
/usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-user-agent
/usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-geoip
/usr/share/elasticsearch/bin/elasticsearch-plugin list
升级java至1.8
yum install java-1.8 -y
安装filebeat
导入key
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
创建repo 文件
more lostash.repo
[elastic-6.x]
name=Elastic repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
安装
yum install filebeat -y
修改filebeat主配置文件
more /etc/filebeat/filebeat.yml
filebeat.config:
prospectors:
path: ${path.config}/prospectors.d/.yml
reload.enabled: false
modules:
path: /etc/filebeat/modules.d/.yml
reload.enabled: false
output.elasticsearch:
hosts: ['10.2.3.30:9200']
setup.kibana:
host: "10.2.3.13:5601"
配置filebeat nginx 模块
Access 日志：/data/nginxlog/eslog/es-access.*
Error 日志：/data/nginxlog/eslog/es-error.*
more /etc/filebeat/modules.d/nginx.yml
- module: nginx
# Access logs
access:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths: ["/data/nginxlog/eslog/es-access.*"]
# Error logs
error:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths: ["/data/nginxlog/eslog/es-error.*"]
添加启动
systemctl enable filebeat
开启nginx 模块
cd /etc/filebeat
filebeat modules enable nginx
filebeat modules list
初始化环境
此操作会自动导入filebeat模板和nginx dashboard 到es 集群：
Set up the initial environment:
Loaded index template
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
Loaded machine learning job configurations
./filebeat setup -e
启动服务
systemctl start filebeat
启动服务报错
内容如下：
Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory /usr/share/filebeat/kibana: Failed to import index-pattern: Failed to load directory /usr/share/filebeat/kibana/6/index-pattern:
error loading /usr/share/filebeat/kibana/6/index-pattern/filebeat.json: returned 400 to import file: . Response: {"statusCode":400,"error":"Bad Request","message":"Request Timeout after 30000ms"}
2018-10-31T16:35:45.659+0900 INFO kibana/client.go:113 Kibana url: http://10.2.3.13:5601
2018-10-31T16:37:15.664+0900 ERROR instance/beat.go:743 Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory /usr/share/filebeat/kibana: Failed to import index-pattern: Failed to load directory /usr/share/filebeat/kibana/6/index-pattern:
error loading /usr/share/filebeat/kibana/6/index-pattern/filebeat.json: fail to execute the HTTP POST request: Post http://10.2.3.13:5601/api/kibana/dashboards/import?force=true: net/http: request canceled (Client.Timeout exceeded while awaiting headers). Response:
处理办法：
登陆kibana 进入Dev tools 删除
删除旧模板
#DELETE _template/filebeat-6.X.X
删除旧数据
#DELETE filebeat-6.4.2-*
重新导入模板：
cd /etc/filebeat/
filebeat setup
Loaded index template
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
Loaded machine learning job configurations
重启服务：
systemctl restart filebeat
Dashboard 效果：
另外有需要云服务器可以了解下创新互联scvps.cn，海内外云服务器15元起步，三天无理由+7*72小时售后在线，公司持有idc许可证，提供“云服务器、裸金属服务器、高防服务器、香港服务器、美国服务器、虚拟主机、免备案服务器”等云主机租用服务以及企业上云的综合解决方案，具有“安全稳定、简单易用、服务可用性高、性价比高”等特点与优势，专为企业上云打造定制，能够满足用户丰富、多元化的应用场景需求。
分享文章：filebeat6.4.3采集nginx日志-创新互联
网址分享：
http://cdkjz.cn/article/doopeg.html
