Castles in the Cloud - Keeping control of your log files

It's all a bit nebulous

The roots of cloud computing can be traced back to the 1950s when the scientist Herb Grosch [1] postulated that “the entire world would operate ondumb terminals powered by about 15 large data centres”. Then the term "cloud" was hijacked as a metaphor for the Internet because of the cloud drawings used to represent first the telephone network, and later the Internet, as an abstraction of the underlying infrastructure it represented. Software-as-a-service (SaaS) may have been the first public manifestation of cloud computing, showing how easy IT services could be made available over the Web., but Cloud computing services come in three fundamental flavours: Infrastructure as a service (IaaS), platform as a service (PaaS), and software as a service (SaaS). Today cloud computing is generally understood as the delivery of IT as a service that provides computation, applications, data access, data management and storage resources without the need to worry about the supporting computing infrastructure.

Stormy weather

Dark days indeed for the poor, long-suffering log analysts. Still struggling to automate their inhouselog management and reporting processes, they must now also learn how to run those processes from the cloud. To make matters worse, those very problems, together with the dearth of real experts on log analysis, is forcing many organizations to consider whether to turn over log management to an cloud provider. SaaS is seen as a way to simplify log management because the provider is contracted to supply all the material resources and specialized staff to do a better job for less money. This makes sense not only for SMEs without the dedicated manpower, but also for larger enterprises whose IT resources are already stretched.

To cloud or not?

When making the decision whether to “cloud” or not there are a number of basic issues that need to be addressed, such as how valuable is the data to be stored in the cloud, how critical is easy access to and processing of that data, and of course bandwidth availability. Because while your local network may have more than enough available bandwidth for both standard traffic and log traffic, the bandwidth costs for transferring the massive amounts of log data to the cloud may prove too prohibitive. So, while many IT managers may find log management difficult, they are still not happy giving their log data to a third party provider because of data availability and security issues. In the cloud, it can also be complicated to track all the activity that occurs at different virtualized layers, which has a serious impact on identity and access management functions.

Some silver linings

Fortunately, moving log management to the cloud is not an all or nothing scenario. You have three approaches to choose from; a private cloud, a public cloud or a hybrid cloud. Logging in a private cloud,. where your company controls both the physical and virtual environments, allows your log management tools access to both of the environments within the cloud, and is essentially business as usual. In a public cloud, logging is much more challenging as the visibility of your log data can be severely reduced if system access and application controls are limited. Hybrid clouds can offer the best of both worlds, with the bulk of log data being created and managed in the private cloud, maximizing the operational and the forensics aspects of log management, while the estimated 5% of security events can be forwarded to the cloud. Whether you handle log management in-house or in the cloud, the bottom line is, as always, that reporting and correlation features should be easy to use and capable of meeting your company’s current and future business objectives.

The weather report

The increasing pressure for compliance is currently forcing organizations with smaller data management staffs to implement log management systems. Many larger enterprises are also suffering from shrinking budgets and stretched resources. In both these cases, the possible savings to be made, make a viable business case for moving log management to the cloud. However, before choosing a cloud vendor, companies must practice due diligence by examining the Service Level Agreements of the prospective vendor to check what is guaranteed and what isn't. And, since the competence a service provider is difficult to determine, as it is ultimately based on reputation, this also entails checking any publicly available data . Finally, despite all the obstacles, as a SANS Whitepaper of 2008 [2] concluded:

“If in-cloud providers can deliver prompt, secure, reliable service, cloud-based log management could be a growth sector over the next few years, particularly for the SME market”.

Recommended reading