Logging

Must: Make sure the log messages are in a processable format.

A processable format means that the log messages must conform to following rules:

• The log message must be valid JSON,
• the JSON must contain a field @timestamp,
• the value of the @timestamp field must be in ISO 8601 format
  • an example of a valid @timestamp value would be: 2017-01-30T08:56:47+00:00
• the JSON should contain a field @hostname containing the hostname the log message was produced on,
• the JSON should contain a field @service indicating the service/application the log messages belongs to,
• the JSON must not contain any sensitive information (e.g. secrets) for security reasons.

The JSON will always contain a field _type containing the environment name as defined via Ansible's environment_name variable (part of automation repository). This value is automatically injected by the filebeat agent as part of the logging pipeline.

Payload:

In general, the author of the log message is in full control of the additional payload of the log message. So all information considered helpful, important, necessary for whatever purposes (e.g. debugging, error tracking, metric extraction) should go into the log message. The content of the log message will not be altered or modified in any way by the logging pipeline.

Storage:

Each succesfully processed log message ends up in two different storage systems (at the same time, so the log message gets duplicated for redundancy reasons):

1. an AWS-managed ElasticSearch (ES) cluster with a Kibana4 frontend,
2. an AWS S3 bucket.

The specific location of the storage endpoints depends on the configuration of the logging pipeline as defined in the automation repository.

Note:

Because of the use of ES as on of the storage backends, additonal constraints are imposed on the log message payload.
ElasticSearch always applies some kind of mapping to the received log messages: In the default case - as long as no dedicated mapping definition is provided and feed into ES - that will be a dynamic mapping, where ES tries to figure out a schema of the JSON payload based on the already received JSON messages. The drawback of this convenient ES-behaviour is the following:
Let's say a log message's JSON payload contains a field @serviceError which holds a string value. ES will use this information in the dynamically created schema as manifested inside ES to define that the schema has a key @serviceError which has the type string. Now, for whatever reasons, another log message, later on feeded into ES, contains again the same JSON field @serviceError, this time however containing an array of strings as value. As soon as ES receives this JSON payload, the internal dynamically created schema will break because of the changed type of an already existing schema key. As a consequence, this will trigger a mapping exception inside ES, preventing the log message to be stored at all in ES. Ultimately the log message is lost due to the parsing exception. The log message will, however, still be stored in the S3 bucket (the second storage destination).
To prevent this kind of mapping exceptions, the type of a field's value in the message JSON payload should never change once introduced.