Ad-Hoc Log Management in case of Emergency (SEC Defence)

defence

High visibility on all company systems is essential to ensure the correct detection, analysis and handling of any unauthorized or malicious activities. Nevertheless, organizations without an inhouse incident response team usually have a very poor visibility on their own company network (or none at all). This article focuses on detection, analysis and cleaning tools that can be set up at short notice in case of a security incident.

Cyber crime and Incident Management banner - SEC Consult

Introduction

Cybercrime is on the rise. From 2018 to 2019, the number of reported cyber incidents in Austria increased by 45% to a new total high of 28 439 cases[1] while the dwell time (the time that an attacker is active in the network until it is discovered) decreased continuously.

Based on FireEye’s annual security reports, the median global dwell time in 2018 (78 days) decreased to 56 days in 2019. This significant drop suggests that organizations can improve visibility in the network and detect attacks earlier. However, the percentage of companies notified by third parties about a possible compromise is 53% on average worldwide[2].

Cyber crime and Incident Management with data banner - SEC Consult

Of course, the number of externally reported incidents should be kept as low as possible in relation to those that have been noticed internally. The internal visibility is therefore not as good as that of attentive third parties. Many companies are often not adequately prepared and there is a lack of resources to handle a security incident successfully.

Unfortunately, a lack of visibility makes it difficult even for the best incident responders. External cybersecurity incident response teams (CSIRTs) usually start processing a network that is completely unknown to them. To provide analysts with the necessary information to investigate and remediate the attack, a basic level of visibility needs to be established.

This step consists of four tasks:

  1. Increase the level of detail of the log
  2. Set up a central log management system
  3. Transfer logs from the host to the log management system
  4. Increase log coverage

We will look into each one of these for tasks from the POV of rapid implementation in unprepared networks.

 

INCREASE THE LEVEL OF DETAIL IN THE LOG

You need to improve logging on the host to give the analyst as much information as possible about what is happening on a host, such as failed and successful login events. Started processes are important in order to get an idea of how the system is being used and to be able to track suspicious activities like an attacker. Keep on reading for explanations including best practices and available configurations for both Windows and Linux systems.

A.    Windows

With the right configuration, the Windows Event Log[17] does offer solid logging functionality. Sysmon from the SysInternal Tools[3] is another great possibility for logging on Windows systems, which is often used as an extensive extension to the normal Windows Event Logs.

1)   Windows Event Log

Since the default settings of the Windows Event Logs are not sufficient to give incident responders the necessary insight into the activities of a system, the log size needs to be adjusted. The standard settings have a relatively short rotation time at host level, which would be shortened even more if the level of log detail was increased. Please keep in mind that the right size depends on the environment and the system. The following recommendations are based on a publication by the Australian Cyber Security Center. [4]

Increase the maximum size of the Security Event Log to 2 GB and the size of the application and system event logs to 64 MB. The most important areas in log improvement are as follows:

  • User accounts (logins, lockouts, modifications, NTLM authentication)
  • Persistence mechanisms, credential access, and lateral movement techniques (scheduled tasks, services, file shares, WMI, object access, PowerShell)
  • Anti Virus (EMET, Windows Defender, logs from 3rd party AV solutions)
  • Process execution (process tracking – if Sysmon is not available, AppLocker, Windows Error Reporting, code integrity)

If an attacker were to manipulate the event logging, stop it, or delete the event logs, events in connection with the logging itself are also of great importance [4]. For a rapid implementation at the customer’s, the predefined changes to the audit policies can be done via GPOs.

2)   Sysmon

Sysmon is an abbreviation for System Monitor. It is a free tool developed by Microsoft that offers extended functionalities compared to the normal Windows Event Log. Sysmon can be adapted to the requirements of the network thanks to the flexible filter options. Also, the functionality of the IMPHASH calculation of the executed program is also worth a look. Individual parts, such as the called APIs, are hashed in this process, which enables the detection and correlation of modified malware to bypass anti-virus solutions. [3]

Ideally, the Sysmon configuration should be adapted to the requirements of the company network. However, a preconfigured configuration file is recommended for quick use. A publicly available configuration of good quality can be found in the Swift On Security GitHub repository [5]. Later, the configuration can be optimized iteratively. Sysmon is usually deployed via the software management platform established in the company, for example SCCM, or via GPOs[4]

 

 

B.    Linux

The Linux Auditing System (auditd), is the Linux counterpart to the Windows Event Log or Sysmon. It is highly configurable and offers excellent insight into the processes on Linux systems. Florian Roth’s configuration file [6] can be used as the starting point for rapid implementation. It enables high-quality logging which, like the Sysmon configuration, can also be optimized iteratively. Since the standard maximum log size is often too small for a highly active system, the optimal size depends on the activity, the level of logging, and the available resources of the system. You can find more information about logging on Linux systems here [7].

 

 

CENTRAL LOG MANAGEMENT SYSTEM

The next step for increasing visibility in the corporate network is the implementation of a central log management system. Central log management has several advantages. The size of the logs on the hosts can be kept smaller without disadvantages like visibility into the past. The storage is centralized and can be adapted to the desired retention time and handed over to long-term archiving. Access to central log management must be strictly secured and monitored.

Removing or manipulating the logs on the hosts – if they have already been sent to the log management system – has no effect and can be recognized. Further benefits of the system include enhancement of logs by submitting relevant information, correlation of related events, triggering of alarms in suspicious activities. Since the investigation can be carried out in one place and the analysts don’t have to physically move from system to system, system analysis is much easier and more efficient. [8]

Two well-known open-source log management systems are ELK Stack and Graylog.

A.    ELK Stack

The ELK Stack[9] is one of the most common open-source log management systems. It is a  fusion of the three open source projects ElasticsearchLogstash, and Kibana. The former is an engine for indexing, search, and analytics. The second is a pipeline for data processing, which simultaneously collects, transforms data from various sources, and forwards it to Elasticsearch. The latter provides the graphical user interface that analysts can use to search the data, create charts, graphs, and dashboards. Individual tools for transferring files, such as logs, were subsequently added. This functionality was called Beats and led to the renaming of the project to Elastic Stack. We will look into this in the next chapter.

Using Kibana, analysts can search the collected logs or create alerts, e.g. to draw attention to discovered Indicators of Compromise (IOCs). The following figure shows the output of a search on Kibana:

Kibana Search Results screen - SEC Consult
Figure 1 – Kibana Search Results [8]

B.    Graylog

Graylog started in 2009 as an open-source project in Hamburg. It became a very well-known log management system with headquarters in Texas, USA [10]. The functionalities are very similar to those of the ELK stack. The system itself consists of Graylog (as an application and user interface), MongoDB and Elasticsearch. In contrast to ELK as a Big Data Solution, Graylog was designed as a log management system right from the start and scores with better UX, simplicity of installation, and the authorization system [11]. The following figure shows search results in Graylog:

Graylog search results screen - SEC Consult
Figure 2 – Graylog search results [11]

Log Transfer

Last but not least, you’ll have to transfer the logs from the hosts into the central log management system. There are various methods for this, such as event forwarding under Windows. Log transfer functionalities are already supplied for the implementation using the ELK stack or Graylog. Winlogbeat for the transmission of Windows Event Logs as well as Sysmon Logs and Filebeat for the transmission of Auditd or other application logs under Linux are emphasized. [14] [15]

Graylog Sidecar is a convenient function for managing the beat configurations under Graylog. There are some features such as the central configuration management, the simple configuration deployment, the creation of groups with individual configurations as well as some security features. Graylog Sidecar also allows to query and control the status of the transfer agents[15]

 

Log Coverage

Log coverage is the share of systems that are integrated in the central log management. When central log management is introduced, it is hardly possible to integrate all systems into the log management system at the same time. Depending on the network size, the systems can only be gradually integrated into the log management or it will most likely overload. Based on the performance monitoring of the log management system, further steps can be taken to increase scalability, such as using a log broker like Kafka. [13]

The order of the systems used should be based on the business criticality and the probability of an infection. With a step-by-step onboarding into the log management system, the log coverage and the visibility within the company network can be gradually improved.

 

Conclusion

Complete log management systems can be set up using freely available software. The implementation overhead can be reduced by using open source configurations. The superficial, individual adjustment to the situation should be made during the incident and a more detailed adjustment is recommended after the incident. This can be done during normal operations e.g. by using an iterative improvement process. However, a rapid deployment of central log management systems is more complicated. Various sources and a brief look at configuration files for the automated deployment indicate that the Graylog implementation seems to be easier than the ELK stack.

The type of implementation is another complicated aspect. A log management system is usually placed in the internal network without access from the Internet, so the customer needs a suitable server with the appropriate resources. Deployment in the cloud would be easier and bears the advantage of the scalability and the better avoidance of blind spots, but the logs must be encrypted via TLS.

If client systems that are used in the home office without an active VPN connection, there is a risk that the logs on the clients will be overwritten before they are even sent to the central log management system. When deploying in the cloud, it is particularly important to clarify various aspects regarding the GDPR. Basic requirements include the encrypted transmission of the logs, to the log management system, the use of disk encryption, a strict access restriction for the log management system, pseudonymization of logs and the enforcement of retention times and secure deletion after the deadline. [16]

Discussing all the restrictions and influence of the GDPR on log management and especially cloud-hosted systems would go beyond the scope of this article.

A rapid introduction of extensive log management in the event of a security attack can reduce the time to resolve the incident drastically. This includes reducing the impact of the damage, costs and possible further legal or reputational consequences.

CSIRTaaS services benefit a lot from a rapid deployment plan for log management systems using automation. However, even with a rapid introduction of log management, past logs cannot be restored. Only the visibility in the network for current and future events is increased. The implementation of comprehensive log management in advance leads to better planning and preparation, as well as considerable time savings from detection to rectifying the incident and minimizing damage and costs in an emergency.

Q&A

An ad hoc log management system is used for rapid establishment or increase of visibility in the corporate network in the event of security incidents.

A central log management system is a central functional unit to which the logs of the systems are sent. It is responsible for the administration and indexing of the logs and serves as a central interface for analysis using search functions, dashboards or alert functionalities.

The introduction of a classic log management system takes a longer period of time. It is usually planned ahead and includes several test phases to guarantee an optimal introduction to the productive system, a logging strategy adapted to the company network. Use cases help to avoid typical pitfalls such as overloading an undersized log management system (usually caused by the flood of logs). An ad hoc log management is focused on the quick and at least rudimentary introduction of a log management system in order to clarify and remediate the security incident that has already occurred.

  • Increase the level of log detail
  • Set up a central log management system
  • Transfer of the logs from the host to the log management system
  • Increase log coverage in the network
  • ELK Stack / Elastic Stack
  • Graylog

Log coverage is a proportion of the systems that are connected to the central log management system and is monitored in the total of the systems in the company network.

The dwell time is the time an attacker is active in the network until the unauthorized access is discovered.

Security event logs should be configured to 2 GB and application and system event logs to 64 MB.

CSIRTaaS stands for Computer Security Incident Response Teams as a Service. This means nothing more than the implementation of incident response as a service, such as the SEC Defense Service from SEC Consult.

Both Winlogbeat and Filebeat are tools to send the logs from a host to a central log management system. Winlogbeat takes over the transmission of Windows Event Logs, while Filebeat transfers various logs, especially on Linux systems.

Sources

[1]ORF, „Starker Anstieg bei Cyberkriminalität,“ 8 Mai 2020. [Online]. Available: https://orf.at/stories/3164864/. [Accessed 05/13/2020].

[2]FireEye Mandiant Services, „M-Trends 2020,“ 2020. [Online]. Available: https://content.fireeye.com/m-trends/rpt-m-trends-2020. [Accessed 05/13/2020].

[3]Microsoft, „Sysmon v11.0,“ 28 April 2020. [Online]. Available: https://docs.microsoft.com/en-us/sysinternals/downloads/sysmon. [Accessed 05/14/2020].

[4]Australian Cyber Security Center, „Windows Event Logging and Forwarding,“ 2019. [Online]. Available: https://www.cyber.gov.au/sites/default/files/2019-03/Windows_Event_Logging_Technical_Guidance.pdf. [Accessed 05/14/2020].

[5]Swift On Security, „sysmon-config | A Sysmon configuration file for everybody to fork,“ [Online]. Available: https://github.com/SwiftOnSecurity/sysmon-config. [Accessed 05/14/2020].

[6]F. Roth, „auditd,“ [Online]. Available: https://github.com/Neo23x0/auditd. [Accessed 05/14/2020].

[7]National Security Agency, “Guide to the Secure Configuration of Red Hat Enterprise Linux 5”, 26. August 2011, [Online]. Available: https://apps.nsa.gov/iaarchive/library/ia-guidance/security-configuration/operating-systems/guide-to-the-secure-configuration-of-red-hat-enterprise.cfm. [Accessed 06/18/2020].

[8]N. Carstensen, „Graylog,“ 29 November 2019. [Online]. Available: https://www.graylog.org/post/what-is-log-management-a-complete-logging-guide. [Accessed 05/14/2020].

[9]Elastic, „The Elastic Stack,“ [Online]. Available: https://www.elastic.co/de/elastic-stack. [Accessed 05/14/2020].

[10]Graylog, „About Graylog,“ [Online]. Available: https://www.graylog.org/about. [Accessed 05/14/2020].

[11]D. Tkachenko, „Logicify – Graylog as a Tool for Technical Monitoring of Software Products,“ 19. Juni 2018. [Online]. Available: https://www.logicify.com/en/blog/graylog-as-a-tool-for-technical-monitoring-of-software-products-we-build/. [Accessed 05/14/2020].

[12]Graylog, „Graylog 3.1 Documentation: Searching,“ [Online]. Available: https://docs.graylog.org/en/3.1/pages/queries.html. [Accessed 05/14/2020].

[13]J. Kosik, „Large-Scale Log Management Deployment,“ June 13, 2018. [Online]. Available: https://uploads-ssl.webflow.com/5ac133a161e365de41ae40d3/5b439fa45a717805ecd1c1cf_Pan-Net%20Deployment%20with%20Graylog.pdf. [Accessed 05/14/2020].

[14]Elastic, „Beats overview,“ [Online]. Available: https://www.elastic.co/guide/en/beats/libbeat/current/beats-reference.html. [Accessed 05/14/2020].

[15]Graylog, „Introduction to Graylog Sidecar,“ [Online]. Available: https://www.graylog.org/features/sidecar. [Accessed 05/14/2020].

[16]Graylog, „Resources | What GDPR Means for Log Management,“ [Online]. Available: https://www.graylog.org/resources/what-gdpr-means-for-log-management. [Accessed 06/12/2020].

[17]Windows Event Log, „Event Viewer | Wikipedia,“ [Online]. Available: https://en.wikipedia.org/wiki/Event_Viewer . [Accessed 06/12/2020].