Export JSON logs to ELK Stack

import-json-logs-to-elk-stack

Centralized logging can be very useful when attempting to identify problems with your servers or applications, as it allows you to search through all of your logs in a single place. It is also useful because it allows you to identify issues that span multiple servers by correlating their logs during a specific time frame.

ELK Stack is very good tool for indexing logs. However, changing mappings can be a big headache, that's where JSON can become handy.

These days JSON became a common format for exchanging data. It is both readable and reasonably compact, and it provides a standardized format for structuring data. Almost every programming language can parse it. JSON’s Unicode encoding makes it universally accessible, and its large and established user base provides an active community of helpful examples, patterns, and support.

In this blog post, we will examine the process of exporting your application JSON log into ELK stack.

1. Deploy an ELK Stack

If you don’t have one, you can deploy one using Cloud 66 EasyDeploy App Store.

2. Config your application to log in JSON.

Note: Sample below is in Rails

# Gemfile
gem "lograge"

then

# config/application.rb
# set up logging configuration
class Application < Rails::Application
   Logging.configure(config)
end

then

# config/logging
require 'lograge'
class Logging
	def self.configure(config)
		# LOGGING!
		log_level = Rails.env.production? ? :info : :debug
		log_level_number = ::Logger.const_get(log_level.to_s.upcase)

		# LOGRAGE CONFIGURATION
		config.lograge.enabled = true
		config.lograge.formatter = Lograge::Formatters::Json.new

		# APP LOGGING
		logger = ::Logger.new(File.join(Rails.root, 'log', "#{Rails.env}.log"))
		logger.formatter = proc do |severity, datetime, progname, msg|
			lograge_hash = nil
			lograge_hash = JSON.parse(msg) rescue nil if msg.is_a?(String)
			msg = 'ActiveRecord' if lograge_hash
			log_hash = {
				time: datetime.utc.strftime('%Y-%m-%dT%H:%M:%S.%3NZ'),
				# hostname: hostname,
				pid: $$,
				thread_id: Thread.current.object_id.to_s(36),
				severity: severity,
				progname: progname,
				msg: msg.strip.force_encoding('UTF-8')
			}
			log_hash = log_hash.merge(lograge_hash) if lograge_hash
			json_string = nil
			begin
				json_string = "#{log_hash.to_json}\n"
			rescue
				# handle non-utf8 encodings
				log_hash[:msg] = msg.strip.force_encoding('ISO-8859-1').encode('UTF-8')
				json_string = "#{log_hash.to_json}\n"
			end
			json_string
		end

    tag_log = ActiveSupport::TaggedLogging.new(logger)
		config.logger = tag_log
		config.log_level = log_level
	end
end

You can find full sample here.

3. Deploy/Run your application

With sample above you should see logs like below:

{"time":"2017-05-24T12:03:27.926Z","pid":22929,"thread_id":"ouypqenuo","severity":"INFO","progname":null,"msg":"","header":{"remote_ip":"92.168.124.59"}}

4. Install Filebeat on the application server

$ wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
$ sudo apt-get install apt-transport-https
$ echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
$ sudo apt-get update && sudo apt-get install filebeat
$ sudo update-rc.d filebeat defaults 95 10

More info here.

5. Config Filebeat on the application server

Edit /etc/filebeat/filebeat.yml

filebeat.prospectors:
- input_type: log
  paths:
    - <JSON_LOGFILE_PATH>
  json.keys_under_root: true
  json.overwrite_keys: false
  json.add_error_key: true
filebeat.registry_file: /var/lib/filebeat/registry
output.logstash:
  hosts: ["<YOUR_ELK_STACK_ADDRESS>:5044"]
  index: <YOUR_INDEX_NAME>

6. Start Filebeat

$ sudo /etc/init.d/filebeat start

7. Config ELK index pattern

Change the index pattern to <YOUR_INDEX_NAME>-* ELK will update fields base on logs that have received.

Note:
You can create different input_type when all of logs aren't in JSON.

filebeat.prospectors:
- input_type: log
  paths:
    - <JSON_LOGFILE_PATH>
  json.keys_under_root: true
  json.overwrite_keys: false
  json.add_error_key: true
- input_type: log
  paths:
    - <LOGFILES_PATH>/*.log
  exclude_files:
    - <JSON_LOGFILE>
  multiline.pattern: ^\[
  multiline.negate: true
  multiline.match: after
filebeat.registry_file: /var/lib/filebeat/registry
output.logstash:
  hosts: ["<YOUR_ELK_STACK_ADDRESS>:5044"]
  index: <YOUR_INDEX_NAME>

The biggest benefit of logging in JSON is that it’s a structured data format. This makes it possible for you to analyze your logs like big data. It’s not just readable text, but a database that can be queried. This allows summaries and analytics to take place and help you monitor your application and troubleshoot issues faster.