Log aggregation - definition & overview

Log aggregation is a software function that consolidates log data from throughout the IT infrastructure, including microservices, into a single centralized platform where it can be reviewed and analyzed. Log aggregation software tools may support additional functionality, such as data normalization, log search, and complex data analysis. Log aggregation is just one aspect of an overall log management process that produces real-time insights into application security and performance.

Log aggregation software tools capture event log files from applications and other sources within the IT infrastructure. Event logs are automatically computer-generated when certain events occur within the application. Event logs may also be classified according to the severity of the event and the required urgency of response. Use cases for event logs should fall into one of the following categories:

Information

Informational log documents changes in the state of the application or changes in entities within the application. Information logs are useful for determining what happened in the application during a specified time. An information log might be created when:

• A scheduled batch job completes

• The application successfully loaded

• New deployments

• A user copied some files

• A driver initialized correctly

Information logs focus on completed tasks, while other log classifications for reporting unsuccessful operations.

Errors/application failures

When the application experiences an error, it should automatically generate a log with the error categorization. An error means that the application is functioning incorrectly with no opportunity to recover. The error may affect users in the production environment, resulting in service interruptions and poor customer experience metrics. Error logs must be addressed immediately to minimize the impact of any application error on a critical service.

Warnings/application malfunctions

An application that fails to do an operation but still has the opportunity to recover and deliver the service may trigger a warning log.

Warnings and errors are relatively similar, so consider the following distinction:

• If the user performs an action that calls Database X and the application crashes, the result should be an error log.

• If the user performs an action that calls Database X and it takes 20 seconds longer than expected, the result should be a warning log.

Warning logs are not as urgent as errors but should be addressed relatively quickly to avoid negatively impacting customer service.

Positive security events

Most applications will generate a login response to complete a security event. This includes when a user logs onto the computer, a database, an application, or when the user answers a security question or completes another form of authentication.

Negative security events

In addition to logging success audits, log aggregation tools also keep track of failed security events. Anytime a user enters the wrong password, answers a security question incorrectly, or otherwise fails to authenticate access to the system, a log will be generated that documents the event.

In addition to the event type, each event log typically includes:

• The data that the event occurred

• The time that the event occurred

• A description of the event, including an error code if applicable

• The user profile that was active when the event occurred

• The name of the computer or network endpoint where the event occurred

• An event identification number for reference

• The source of the event

When professional software developers build an application, including open-source, they always include a built-in logging function that keeps track of events within the application. When an event happens within the application, the function automatically generates a log that records the event, along with additional metadata about the conditions surrounding the event, and writes the record into a log file. Programmers use log files in debugging to help determine the root cause of an error, but they can also be useful for users that wish to monitor the performance, security status and general behavior of an application.

As business IT environments increase in complexity and IT organizations deploy an increasing number of applications and infrastructure in public and hybrid cloud environments, there is a growing need to maintain central control of application security and performance. The IT organization may review the log file or log streams from each application as a way of monitoring application status, but it would be much more useful to bring all of that data into a common platform. Log aggregation is part of the overall log management process that helps IT organizations convert their log files into actionable insights in real-time or near real-time. The process can be described in five basic steps:

1. Instrument and collect - the first step of log management is collecting logs. IT organizations must implement log collector software tools that collect data from various parts of the software stack. Many devices across platforms generate logs using the Syslog message logging standard or other applications that can write logs directly into the log aggregation tool platform.

2. Centralize and index - log data needs to be normalized and indexed, making inputs easier to analyze and fully searchable for developers and security analysts.

3. Search and analyze - now that the log data is organized properly in the log aggregation tool. It can be searched and analyzed to discover patterns and identify any issues requiring IT operators' attention. Human or machine learning analysis can be used to identify patterns and anomalies.

4. Monitor and alert - effective log monitoring is critical to the log management process. An effective log management tool should integrate with message applications to deliver timely alerts when events occur that require a prompt response.

5. Report and dashboard - the final component of log management ensure that team members across departments have the necessary levels of access and visibility into application performance data.

Log aggregation allows IT organizations to combine data from public and hybrid cloud environments into a centralized location where it can be searched and analyzed. This scalable process increases the visibility of cloud-based computing environments, helps security analysts respond more quickly to security threats, and provides real-time insight into application performance.

Sumo Logic provides exceptional log aggregation and log parsing functionality for DevOps. It can collect logs from almost any system with timestamps and format and bring them together to support an end-to-end log management process. With our machine learning log analytics, IT organizations can turn millions of event log data points into a visualization of actionable insights that support application security and performance excellence.