Recent months have seen a great deal of excitement about the concept of Big Data, the idea that if we can apply massive amounts of computing power to massive amounts of data, we can solve problems or accomplish intellectual tasks that otherwise would not be possible.  And in particular there is a great deal of interest in how we can apply massive computing power to dealing with the stunning complexity that accompanies many attempts to categorize, measure, or abstract the physical world around us.

To some degree these are not new ideas.  Highly visible science projects like genome mapping, measurement of global climate change, or searching for Earth-like planets all depend on the ability to process very large data sets.  In fact, during my own time doing graduate work at MIT I contributed to the Human Speechome Project, which seeks to shed light on language development in children by capturing and analyzing the progressive speech patterns of a single boy over the first three years of his life.  This project attempted to capture as much of the child’s waking life as possible and boasts a data set of more than 100,000 hours of recorded material.  It’s not practical for researchers to sit and scrutinize these recordings.  The amount of input is simply too great.  Instead we need analytics to help categorize and make sense of what the digital video and audio have to tell us.

Now, what is new is the widespread sense that these same ideas can benefit lots and lots of the pragmatic and even mundane mechanisms that keep the wheels of society turning.  A great example is this interactive wind map of the continental United States.  As ITworld blogger Irfan Khan points out, a map like this can have simple practical benefits to firefighters, farmers, and power companies.

Another great example is analyzing and drawing actionable inferences from customers’ behavior in physical retail environments.  To do so a system must account for diverse factors like the physical location of shoppers and employees in the store (and be able to differentiate between the two); the layout, fixtures, and plan-o-gram of the store; complete detail on actual sales; staffing schedules; and even the weather.  Input sources can include in-store video cameras, Wi-Fi tracking tags, RFID, and other in-store systems like those for Point-of-Sale (POS), staffing, and task management.  Using these data sources, retailers presently on our system are collecting nearly 10,000 data points per store visit, and we expect that figure to go up over time as more and richer data sources become available.  Across our full customer set of more than fifty retail chains the RetailNext system is measuring more than 25 million store visits per month and collecting trillions of data points per year.