• Home
  • Photos
  • Work
  • Blog
  • Contact
  • Security
  • Links
  • Elasticsearch + Nginx Access Log using Kibana and FileBeat

    2017 - 11.27

    Long time no see. Very short post today, very fast howto to implement access log logging to elasticsearch using simple utilities.

    First of all, I expect you have already setup your elastic cluster with Kibana or Grafana or whatever.

    First of all, let’s begin with Nginx itself.

    Create file /etc/nginx/conf.d/json_log.conf with this content:

    log_format json_combined escape=json '{ "timestamp": "$time_iso8601", '
     '"remote_addr": "$remote_addr", '
     '"remote_user": "$remote_user", '
     '"request_url": "$request_uri", '
     '"req_status": "$status", '
     '"response_size_B": "$bytes_sent", '
     '"req_protocol": "$server_protocol",'
     '"req_method": "$request_method",'
     '"req_srvname": "$server_name",'
     '"req_time": "$request_time",'
     '"connection-id": "$request_id",'
     '"ssl_prot_version": "$ssl_protocol",'
     '"ssl_cipher": "$ssl_cipher",'
     '"ssl_conn_reused": "$ssl_session_reused",'
     '"ssl_session_id": "$ssl_session_id",'
     '"http_referrer": "$http_referer", '
     '"http_user_agent": "$http_user_agent", '
     '"http_x_referer": "$http_x_referer" }';
    
    
    access_log /var/log/nginx/access_json.log json_combined;
    
    

    This will create new access log in JSON format defined above.

    Time to restart nginx and check the file

    If you see something like this, you are ready for next step

    { "timestamp": "2017-11-27T17:28:47+01:00", "remote_addr": "5.164.120.117", "remote_user": "", "request_url": ...blablabla }
    

    Now go to Elastic.co FileBeat page and install filebeat to your server.

    After installing edit file /etc/filebeat/filebeat.yml

    filebeat.prospectors:
    
    - input_type: log
    
      # Paths that should be crawled and fetched. Glob based paths.
      paths:
        - /var/log/nginx/access_json.log
      json.message_key: remote_addr
      json.keys_under_root: true
    
    tags: ["whateveryouwant"]
    max_procs: 3
    
    #-------------------------- Elasticsearch output ------------------------------
    output.elasticsearch:
      # Array of hosts to connect to.
      hosts: ["your.elasticsearch.server:9200"]
    
      # Optional protocol and basic auth credentials.
      #protocol: "https"
      #username: "elastic"
      #password: "changeme"
      bulk_max_size: 2000
    

    Now restart filebeat and you’re DONE!

     

    Tags: , , , , , , ,

    Loading Facebook Comments ...

    Your Reply