Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Edge Articles

08:00 AM
Terry Sweeney
Terry Sweeney
Edge Articles
Connect Directly

Back to Basics with Log Management, SIEMs & MSSPs

Not fully clear about why your organization collects any (or all) log data? Experts offer their tips on making better use of log data and alerts to improve your security profile.

For security professionals and forensics investigators, it doesn't get any more basic than the unrelenting flow of log data generated by countless machines attached to the enterprise backbone.

The activity, health, status, and anomalies of these endpoints and systems are time-stamped and delivered to some repository, usually a syslog, a security information and event management (SIEM) tool, or some managed security service provider's MSSP) archive in the cloud. The vast majority of these log records will never be reviewed or needed; for many organizations, they're simply checking a box by archiving log data to comply with state, federal, and international laws.

This fundamental collection of health and performance data has been going on for decades. Why, then, do so many organizations still manage to fumble the details of log data and actually undermine the very security they're charged with protecting?

The short answer: Log data can be really challenging to manage, and it requires some heavy-lifting on the part of the entire organization – not just the IT department and security professionals – to consider why its collecting log data and where. More on that in a minute.

The discomfiting example of Target isn't so flattened on the highway of infosec forensics that we can't run it over one more time. Overinstrumented and drowning in alert messages, Target's IT and security teams completely missed multiple intrusions, malware payloads, and massive theft of credit card data in 2013. It's noteworthy that in the bitterly competitive retail sector, none of Target's competitors rushed to proclaim, "That could never happen here." Their forbearance masks an acknowledgment that everyone can do a better job, not just with alerts but all their log data.

With that context in mind, here's some advice from experts and a CISO about how to get the basics right with log data, your SIEM vendor, or MSSP.

Pick Your Log Sources Carefully
Every source we talked to for this article agreed vehemently on two essential points: Collecting every single bit of log data is a mistake. And the log data you do collect must be mapped back to some business requirement or strategic issue. Otherwise, you end up boiling the ocean. 

"The most common mistake CISOs and others who want enterprise visibility make is they onboard every log source they can get their hands on," confirms James Carder, CISO at SIEM vendor LogRhythm and formerly the director of security informatics at the Mayo Clinic.

But more is not more in this instance; in log management, more leads to overload for those who must manage and analyze the data. It's also a dubious choice from a cost perspective: Though storage prices have dropped dramatically thanks to the cloud, it simply makes no financial sense to store all log data from every attached machine.

More logging data also means a greater volume of false-positives, the bane of any security pro's existence. Chasing down false-positives isn't an activity where any organization wants its security staff spending precious time. False-positives waste money, they're bad for morale, and, worst of all, they don't really improve the security profile of the organization.

Carder also encourages security pros to watch their assumptions, since they often have preconceptions or don't understand what's important to the business. In a previous job with the Mayo Clinic, Carder says he assumed clinical research data was critical to protect, only to find out that the organization gave away that data for free. "I made an incorrect assumption," he notes. And it changed how he collected log data.

Therefore, dig in and do the due diligence; otherwise your log data collection efforts are likely to fall short, if not fail. Carder's advice? Take a look at your business and threat sectors, and then map out end-to-end use cases of log data to get a clearer picture of the organization's needs. "What you get is a business case for what you're trying to solve," he adds. And you can leverage that information as you move from one division of the organization to another and get buy in to your log data initiative.

Another useful resource is NIST document 800-92, "Guide to Computer Security Log Management." It's a thorough, high-level discussion of logging concepts and technologies, not a step-by-step guide to implementing them. Though written in 2006, the document is time-tested and still useful, according to Mike Lloyd, CTO of security analytics vendor RedSeal. "It's not a holy bible, but 800-92 is still a really good document that precisely addresses security log management," he says.

Got the Time?
Once the business use cases are settled, there's the basic issue – frequently overlooked – of how to set the timestamping for each log message that gets sent and stored. This is particularly challenging for organizations that span multiple time zones, whether daylight saving time gets observed, and the inevitable internal politics and pushback ("Corporate wants us to do what?").

"Time is a big issue," notes RedSeal's Lloyd. But it's essential the organization specifies a time zone to use for syncing up its machinery, apps, and log alerts. He's a big advocate of standardizing on Coordinated Universal Time, or UTC, as it's referred to internationally with its French acronym. This is what gets used in most aeronautical applications and systems; most IT networking gear also has UTC defaults.

"UTC is what pilots use, otherwise the airways would be chaos. But it may be impossible to get everybody's laptop to use it," Lloyd says. He recommends an approach that's a variation of begging forgiveness rather than asking permission. "Go as far as you can [with UTC] till the organization stops you," he says.

Bad actors rarely break into just one machine, and deconstructing the sequence of their actions requires logs that can be easily correlated against a time line. "If you don't have a good measure of time, you can't reconstruct the right order of what they did," Lloyd adds.

Organizing Your Logging Sources
Another potential hurdle to effective log management is how you organize your sources of log data – hierarchical, geographical, by business unit? There are also details about logging data that can be customized, like which time zone you use for timestamping, an essential piece for correlating events as part of incident response and any forensics investigation.

LogRhythm's Carder is a big advocate of dividing up sources (and their software) by function:

--Endpoint sources: Servers, computers, laptops, and mobile.

--Network data: Routers, switches, Wi-Fi access points, and others.

--End-user behavior data: Networks and applications accessed, changes in authorizations, volumes of data uploaded or downloaded.

By monitoring log alerts in these categories, security pros can more quickly pinpoint where the trouble is in real time – and where it's not. "Even if end-point and network behaviors look normal, there are behavioral shifts that could still bubble up that indicate a person isn’t who they say they are," Carder explains.

But tracking these three areas can help isolate, mitigate, or prevent incidents if the security pros associated with each side of the triangle are monitoring alerts and communicating with each other.

Implementing SIEM Functions
Once the strategic aspects of logging data have been resolved, end-user organizations must then turn to another piece of heavy lifting: How will they manage log data? With the emergence of security information and event management (SIEM) software in the past 10 years, customers get some combination of intelligence and automation to analyze security alerts.

Typically, SIEMs collect and store log files from a variety of sources and systems, and perform some combination of real-time monitoring, event correlation, and analysis. SIEMs help move organizations closer to a meta view of their security IT infrastructure – that "single pane of glass" management vendors have promised for years.

And in the age of cloud computing and applications on demand, managed security service providers increasingly offer some version of SIEM as a service. While organizations in highly regulated, compliance-bound industries (financial services, healthcare) may be restricted from using third parties for security management, the MSSP model has nonetheless attracted plenty of customers who want to contain ballooning security costs.

Another model has SIEM vendors partnering with MSSPs to help customers get the most from their log data and keep their security posture strong. But some caveats apply here as well, according to Fred Kwong, CISO for Delta Dental Plans Association. To serve his member organizations and their customized requirements, he's working with both SIEM vendors and an MSSP to preserve members' options.

"Some SIEMs work better in the cloud than others, and some SIEM vendors are better partnered with cloud vendors," Kwong explains. Delta needed support for portability of SIEM as a Service, without a lot of operational overhead for its members to preserve transition options for the future.

One size for SIEM definitely doesn't fit all for Delta's members, and it underscores the need for understanding different business units' requirements and objectives. "Know the use case and what you need from the tool to make that happen," Kwong says. "It makes a big difference."

And while new or emerging technology can simplify and aggregate vast amounts of log data, it still costs money, notes LogRhythm's Carder. And that reinforces the imperative to understand what data you're collecting and why. "AI or machine learning may help you on the analytics side, but knowing which logs you need to be effective is the key," he adds.

Related Content:


Terry Sweeney is a Los Angeles-based writer and editor who has covered technology, networking, and security for more than 20 years. He was part of the team that started Dark Reading and has been a contributor to The Washington Post, Crain's New York Business, Red Herring, ... View Full Bio

Recommended Reading:

Comment  | 
Print  | 
More Insights
Oldest First  |  Newest First  |  Threaded View
User Rank: Ninja
8/6/2019 | 11:05:28 AM
We need to do more, the current methods are not working.

"AI or machine learning may help you on the analytics side, but knowing which logs you need to be effective is the key," he adds.

Currently we have seen the number of potential attacks increase by 200% this year, the existing method of doing business will have to change due to the number of attacks. There are a number of companies like Extrahop, CarbonBlack/Blue-Vector (CBBV) and Sophos are using ML to process large amounts of data. Some systems like CBBV are helping us to make and carry out those decisions.

To the point made earlier, we need to move to the Machine Learning (ML) instead of just accepting the status-quo because the existing methods are just not working.

User Rank: Ninja
8/14/2019 | 2:48:32 PM
Re: We need to do more, the current methods are not working.
Yes, we are working with companies like Blue-Vector/Carbon Black, Deep Learning and Sophos to help provides solutions to these threats because we have found they are not only identifying the threat, they are also eradicating the threat and sharing the information with the other offices in real-time (the solutions are becoming Prescriptive as opposed to Prescribed):

Big Data Business Model

This is good for business because we don't need to stop what we are doing from a Cloud, Engineering, Development standpoint to talk with the security team about what has been identified, it is being done through software interconnects and APIs, they see this instantaneously on a dashboard.

Also, we are finding more and more use cases (i.e. DNS Spoofing attacks, Randomware, and Application Vulnerabilities). By providing a work-bench (per-say), Sophos can start feeding information into the Machine Learning application but this application can be tied into a centralized DB where it learns constantly

Cyber Threats and ML


Sophos is doing that now with the purchase of Rook Security, a pioneer and leader in managed detection and response (MDR) services. Rook Security provides a 24/7 team of cyberthreat hunters and incident response experts who monitor, hunt for, analyze and respond to security incidents for businesses of all sizes. - Sophos Acquires Rook Security

I think the next level will be to take disparate applications and devices and tie them into a centralized data collector (i.e. SIEM); from there, ML can extract data points to help with the decision-making process, this adds a component to the cybersecurity mix because the system does not sleep so if a threat or vulnerability is found a decision can be made to address this potential or ongoing threat (i.e. by adding a rule to the firewall to stop external threat or isolate the application, server or workstation to a container-like environment where the threat is isolated until further notice from the security staff).

Vendors that may be of interest - https://blog.technavio.com/blog/top-10-deep-learning-companies
  • Crowdstrike
  • Sophos Intercept X
  • BluVector/Carbon Black
  • Deep Learning
  • Nvidia
  • Microsoft
  • Amazon
  • Google
  • Apple
  • IBM
  • Intel
  • Facebook
  • Sensory
  • Qualcomm

That is my take on it.

User Rank: Author
12/2/2019 | 10:47:25 PM
Re: We need to do more, the current methods are not working.
Terry, sorry to try and contact you in this way, but I do not think any of my replies to your recent emails are getting through to you, and it appears I can't message you in LinkedIn or on Twitter. 

Please check your email spam filter. Yes, I would be happy to answer some questions for you. Send the questions over, and then let me know the best way to provide you with my answers, if your email server is still blocking me. :) Thanks. Rebecca Herold
[email protected],
User Rank: Apprentice
12/18/2019 | 8:00:55 AM
Amazing Content
There are many elearning companies in India. E-learning refers to a learning system that we can obtain through the internet using an electronic device.

We also call it online learning or online education. ... Hence, the original term 'electronic learning.'
Making a right choice would be very difficult for anyone. I will make it easier for you to decide.
But I would strongly recommend one of the top elearning companies in India (https://zilliobit.com/). I am not sure whether you have heard about Zilliobit Interactive Private Limited (https://zilliobit.com/).

They design cutting edge, innovative and highly engaging learning and digital solutions. They believe in delivering learning experiences and digital solutions that transform businesses and change the mindset and behavior of learners. To know more you can visit them on

<a href="https://zilliobit.com/">Zilliobit Interactive Pvt Ltd </a> . If you have any questions, you can get back to me anything. Also they are the <a href="https://zilliobit.com/solutions/gamification/">top gamification companies in India</a> and also <a href="https://zilliobit.com/">top e learning companies in India </a> 
[email protected],
User Rank: Apprentice
12/18/2019 | 8:03:09 AM
Amazing Content

Wonderfull. Thanks for sharing.

Flash Poll