Log files are not so much overlooked as unappreciated. After all, it would be hard to overlook the mountain of data created each day by system hardware, network devices, PC hard drives, and IT security applications. In fact, most IT and security pros have so much log data that they typically only skim it, or ignore it altogether.
But log files can be the key to recognizing an attack, experts say. External attackers typically use methodical approaches that can be identified as log trends, enabling the IT organization to block or quarantine them. Internal attackers usually leave an audit trail in their logs that can be backtracked and exposed, enabling IT to catch the perpetrators red-handed.
The trick is learning how to analyze log files in a way that is thorough, yet not too time-consuming. For most IT organizations, this means using a combination of automated log file analyzers, security information management tools, and good old-fashioned detective work.
The automated tools for this task are improving, but they still aren't perfect, notes Eric Ogren, an analyst at Enterprise Strategy Group. Some tools offer network behavior anomaly detection (NBAD), which continuously monitors application traffic (destination, source, protocol) but forces IT to manually associate user names with IP addresses. On the other hand, security information management (SIM) does a decent job of collecting log file information but is generally geared more toward historical analysis, rather than identifying potential attacks in real time.
There are a slew of tools that fall somewhere along the NBAD-SIM spectrum, including products from Arcsight, LogLogic, netForensics, and Securify. These tools identify trends and warning signs, but in the end, it's usually a human analysis that identifies an attacker's trail -- and what to do about it.
In his paper "Five Mistakes of Security Log Analysis," netForensics security strategist Anton Chuvakin says that many IT analysts do analyze their logs, but they fail to normalize the data or study it for a long enough period of time. Other IT analysts have good data, but they focus too closely on trying to find specific attack patterns, he says.
"To fully realize the value of log data, one has to take it to the next level of log mining: actually discovering things of interest in log files without having any preconceived notion of 'what we need to find,'" Chuvakin says. "It sounds obvious -- how can we be sure that we know of all the possible malicious behavior in advance -- but it is disregarded so often."