search
数据采集 采集源配置 Nginx 日志采集

Nginx 日志采集

简介

采集 nginx 日志上报到 DataFlux 中

前置条件

配置

进入 DataKit 安装目录下的 conf.d/log 目录,复制 nginxlog.conf.sample 并命名为 nginxlog.conf。示例如下:

使用 tailf 采集器 实现,已默认提供 grok 过滤规则,只需配置 nginx log 的文件路径。

[[inputs.tailf]]
    # 日志文件列表,可以指定绝对路径,支持使用 glob 规则进行批量指定
    # 推荐使用绝对路径
    logfiles = ["/var/log/nginx/*.log"]
    
    # 文件路径过滤,使用 glob 规则,符合任意一条过滤条件将不会对该文件进行采集
    ignore = [""]

    # 新增标记tag,如果为空,则默认使用"nginxlog"
    service = ""
    
    # 从文件首部开始采集,当此参数为 true 时,采集器会关闭自动发现文件的功能
    from_beginning = false
    
    ## 解释文件内容时所使用的的字符编码,如果设置为空,将不进行转码处理
    ## 注意不要选错编码,否则会导致数据无法正常显示
    ## ex: character_encoding = "utf-8"
    ##     character_encoding = "utf-16le"
    ##     character_encoding = "utf-16be"
    ##     character_encoding = "gbk"
    ##     character_encoding = "gb18030"
    ##     character_encoding = ""
    #character_encoding = ""
    
    # 自定义 tags
    # [inputs.tailf.tags]
    # tags1 = "value1"

配置好后,重启 DataKit 即可生效。

示例数据

access_log_1

  • 日志文本:

    127.0.0.1 - frank [13/Jul/2016:10:55:36 +0000] "GET /apache_pb.gif HTTP/1.0" 200 2326

  • 指标集:

指标 类型 单位
filename tags string
host tags string
service tags string
bytes fields int
client_ip fields string
http_auth fields string
http_ident fields string
http_method fields string
http_url fields string
http_version fields string
message fields string
status fields string
status_code fields int
  • 输出:

    nginxlog,filename=/var/log/nginx/access.log,host=ubuntu-server,service=nginxlog bytes=2326i,client_ip="127.0.0.1",http_auth="frank",http_method="GET",http_url="/apache_pb.gif",http_version="1.0",\
    message="127.0.0.1 - frank [13/Jul/2016:10:55:36 +0000] \"GET /apache_pb.gif HTTP/1.0\" 200 2326",status="OK",status_code=200i 1468407336000000000
    

    access_log_2

  • 日志文本:

    172.17.0.1 - - [06/Jan/2017:16:16:37 +0000] "GET /google/company?test=var12 HTTP/1.1" 200 612 "http://www.google.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36" "-"

  • 指标集:

指标 类型 单位
filename tags string
host tags string
service tags string
agent fields string
browser fields string
browserVer fields string
bytes fields int
client_ip fields string
engine fields string
engineVer fields string
http_auth fields string
http_ident fields string
http_method fields string
http_url fields string
http_version fields string
isBot fields bool
isMobile fields bool
message fields string
os fields string
referrer fields string
status fields string
status_code fields int
ua fields string
  • 输出:
nginxlog,filename=/var/log/nginx/access.log,host=ubuntu-server,service=nginxlog agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36\" \"-\"",\
browser="Chrome",browserVer="55.0.2883.87",bytes=612i,client_ip="172.17.0.1",engine="AppleWebKit",engineVer="537.36",http_method="GET",\http_url="/google/company?test=var12",http_version="1.1",isBot=false,isMobile=false,\
message="172.17.0.1 - - [06/Jan/2017:16:16:37 +0000] \"GET /google/company?test=var12 HTTP/1.1\" 401 612 \"http://www.google.com/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36\" \"-\"",\
os="Linux x86_64",referrer="http://www.google.com/",status="warnning",status_code=401i,ua="X11" 1483719397000000000

error_log

  • 日志文本:

    2017/09/26 14:36:50 [error] 8409#8409: *317058 "/usr/share/nginx/html/sql/sql-admin/index.html" is not found (2: No such file or directory), client: 217.92.148.44, server: localhost, request: "HEAD http://174.138.82.103:80/sql/sql-admin/ HTTP/1.1", host: "174.138.82.103"

  • 指标集:
指标 类型 单位
filename tags string
host tags string
service tags string
client_ip fields string
host fields string
http_method fields string
http_url fields string
http_version fields string
message fields string
msg fields string
server fields string
status fields string
  • 输出:
nginxlog,filename=/var/log/nginx/error.log,host=ubuntu-server,service=nginxlog client_ip="217.92.148.44",host="174.138.82.103",http_method="HEAD",http_url="http://174.138.82.103:80/sql/sql-admin/",http_version="1.1",\
message="2017/09/26 14:36:50 [error] 8409#8409: *317058 \"/usr/share/nginx/html/sql/sql-admin/index.html\" is not found (2: No such file or directory), client: 217.92.148.44, server: localhost, request: \"HEAD http://174.138.82.103:80/sql/sql-admin/ HTTP/1.1\", host: \"174.138.82.103\"",\
msg="8409#8409: *317058 \"/usr/share/nginx/html/sql/sql-admin/index.html\" is not found (2: No such file or directory)",server="localhost",status="error" 1506407810000000000