Finding Business Insights in Machine Data

A large amount of data is now being created by Web servers, point of sale systems, RFID sensors, networks, and a variety of other systems that have long been used by people in the data center for diagnostic purposes and for performance analysis.

A large amount of data is now being created by Web servers, point of sale systems, RFID sensors, networks, and a variety of other systems that have long been used by people in the data center for diagnostic purposes and for performance analysis. What has gone unrecognized is that this machine data contains the ability to monitor various types of behavior.

For example, machine data is a window on the actions of customers or partners who are interacting with a company. If analyzed properly, machine data can be the source of business insights about what customers and partners are doing.

Machine data can also be used to track the details of various kinds of business processes. This problem statement examines the ways that machine data can be analyzed and the business insights that can result from that analysis.

Context and Background

Imagine a Web server log. What you see in it is millions of records that indicate that different URLs were served. Inside those records are indicators of all sorts of other specific information, and these Web server logs are often used to diagnose problems, such as which pages are loading slow or with errors or a variety of other things.

A variety of analysis tools is now available. These tools are making the process of analyzing machine data, such as a Web server log, much more straightforward, and in the process of analysis, it is possible now to look through this morass of technical information and get specific kinds of information about user behavior.

For example, in the case of a Web server log, it’s possible to track the number of times a user places an item into their shopping cart. You can now see, in real-time, the correlations between specific items that are being purchased together.

If you are a retailer, instead of having to wait for the purchasing data to be analyzed, perhaps weeks or months later, you can look at that data in real time and take responsive action – maybe you should be running a promotion to suggest that customers should buy both of those items.

In addition, you can watch the flow of users through a sequence of pages and graph their progress, setting alerts to that notify you if there’s an unnaturally high or unnaturally low number of users that flow through the sequence, so that you can be alerted to a problem sooner rather than later, and can move to fix the flow of pages in a critical path on a Website.

Web server logs are just one type of data. Analyzing the flow of traffic in an e-commerce application is just one sort of use case. It turns out that companies are using machine data all over.

For example, telecommunications providers are using machine data to quickly identify phone numbers that are being used for fraud by analyzing the call detail records. These companies are able to monitor, detect and eliminate fraud faster than they were before, because of close analysis of machine data.

Author: Dan Woods

My mission: Find technology for Early Adopters. Follow me on Twitter @danwoodsearly and on LinkedIn www.linkedin.com/in/danwoodsearly/. I am a CTO, writer, and consultant. For tech vendors, I help explain their technology. For users, I help find, select, and deploy new solutions that have explosive business value. I love to speak and share ideas.