Big data: The trend that could change your business

It's a big opportunity for any business mining these networks to understand who likes what, where they are from and who is interacting with who.

Every Wal Mart store is different. It will look at all the data for the neighborhood – what kind of houses people are living in, what sort of jobs they have, earning capacity, family structures and cultural composition to name a few – and then stock their stores to match the neighborhood. Wal Mart handles more than one million customer transactions every hour. Its databases are estimated at more than 2.5 petabytes, the equivalent of 167 times the books in America's Library of Congress, and it monitors these assiduously to make sure each store fits in with its neighborhood. With this strategy, Wal Mart was able to break away from the pack.

Over at Qantas, for example, experts believe teams are now trawling through the internet, checking out every tweet, blog, Facebook entry talking about the airline's nation-rattling decision to ground its aircraft. Social media monitoring services like US-based Radian6, which has software that tracks conversations on the internet and which scours the web for any mention of a company and compiles reports on its findings, are now doing huge business because of big data.

Another piece of software heavily in demand is Hadoop. Started by Yahoo! and released as an open source project in 2007, Hadoop has operating system that can crunch through huge amounts of data that existing systems cannot process.

How much is all that worth? Salesforce.com this year bought the privately held Radian6 for $US326 million ($A314.8 million) to help its clients track customer trends on social media sites such as Facebook and Twitter

Retailers, for example, could use it to track customer likes, dislikes, influences and behaviours. All that will allow them to adjust prices in real time, re-order hot-selling items, shift items from store to store, and better manage their inventory.

Online businesses can use it to get a better understanding of customer preferences and social interactions. In financial services, it would create better risk assessment and allow the company to better tailor its offerings to the individual customer.