Centralized logging can be very useful when attempting to identify problems with your servers or applications, as it allows you to search through all of your logs in a single place. It is also useful because it allows you to identify issues that span multiple servers by correlating their logs during a specific time frame.
ELK Stack is very good tool for indexing logs. However, changing mappings can be a big headache, that's where JSON can become handy.
These days JSON became a common format for exchanging data. It is both readable and reasonably compact, and it provides a standardized format for structuring data. Almost every programming language can parse it. JSON’s Unicode encoding makes it universally accessible, and its large and established user base provides an active community of helpful examples, patterns, and support.
In this blog post, we will examine the process of exporting your application JSON log into ELK stack.
1. Deploy an ELK Stack
If you don’t have one, you can deploy one using Cloud 66 EasyDeploy App Store.
2. Config your application to log in JSON.
Note: Sample below is in Rails
# Gemfile
gem "lograge"
then
# config/application.rb
# set up logging configuration
class Application < Rails::Application
Logging.configure(config)
end
then
# config/logging
require 'lograge'
class Logging
def self.configure(config)
# LOGGING!
log_level = Rails.env.production? ? :info : :debug
log_level_number = ::Logger.const_get(log_level.to_s.upcase)
# LOGRAGE CONFIGURATION
config.lograge.enabled = true
config.lograge.formatter = Lograge::Formatters::Json.new
# APP LOGGING
logger = ::Logger.new(File.join(Rails.root, 'log', "#{Rails.env}.log"))
logger.formatter = proc do |severity, datetime, progname, msg|
lograge_hash = nil
lograge_hash = JSON.parse(msg) rescue nil if msg.is_a?(String)
msg = 'ActiveRecord' if lograge_hash
log_hash = {
time: datetime.utc.strftime('%Y-%m-%dT%H:%M:%S.%3NZ'),
# hostname: hostname,
pid: $$,
thread_id: Thread.current.object_id.to_s(36),
severity: severity,
progname: progname,
msg: msg.strip.force_encoding('UTF-8')
}
log_hash = log_hash.merge(lograge_hash) if lograge_hash
json_string = nil
begin
json_string = "#{log_hash.to_json}\n"
rescue
# handle non-utf8 encodings
log_hash[:msg] = msg.strip.force_encoding('ISO-8859-1').encode('UTF-8')
json_string = "#{log_hash.to_json}\n"
end
json_string
end
tag_log = ActiveSupport::TaggedLogging.new(logger)
config.logger = tag_log
config.log_level = log_level
end
end
You can find full sample here.
3. Deploy/Run your application
With sample above you should see logs like below:
{"time":"2017-05-24T12:03:27.926Z","pid":22929,"thread_id":"ouypqenuo","severity":"INFO","progname":null,"msg":"","header":{"remote_ip":"92.168.124.59"}}
4. Install Filebeat on the application server
$ wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
$ sudo apt-get install apt-transport-https
$ echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
$ sudo apt-get update && sudo apt-get install filebeat
$ sudo update-rc.d filebeat defaults 95 10
More info here.
5. Config Filebeat on the application server
Edit /etc/filebeat/filebeat.yml
filebeat.prospectors:
- input_type: log
paths:
- <JSON_LOGFILE_PATH>
json.keys_under_root: true
json.overwrite_keys: false
json.add_error_key: true
filebeat.registry_file: /var/lib/filebeat/registry
output.logstash:
hosts: ["<YOUR_ELK_STACK_ADDRESS>:5044"]
index: <YOUR_INDEX_NAME>
6. Start Filebeat
$ sudo /etc/init.d/filebeat start
7. Config ELK index pattern
Change the index pattern to <YOUR_INDEX_NAME>-* ELK will update fields base on logs that have received.
Note:
You can create different input_type when all of logs aren't in JSON.
filebeat.prospectors:
- input_type: log
paths:
- <JSON_LOGFILE_PATH>
json.keys_under_root: true
json.overwrite_keys: false
json.add_error_key: true
- input_type: log
paths:
- <LOGFILES_PATH>/*.log
exclude_files:
- <JSON_LOGFILE>
multiline.pattern: ^\[
multiline.negate: true
multiline.match: after
filebeat.registry_file: /var/lib/filebeat/registry
output.logstash:
hosts: ["<YOUR_ELK_STACK_ADDRESS>:5044"]
index: <YOUR_INDEX_NAME>
The biggest benefit of logging in JSON is that it’s a structured data format. This makes it possible for you to analyze your logs like big data. It’s not just readable text, but a database that can be queried. This allows summaries and analytics to take place and help you monitor your application and troubleshoot issues faster.
