Thoughts on “Big Data”
I came across a recent Quora thread I thought I would share and comment on here: Q: Why the Current Obsession with Big Data? There are quite a few thoughtful responses from the data community including LinkedIn’s Peter Skomoroch and Cloudera’s Jeff Hammerbacher (formerly a Facebooker). Peter and Jeff think big data is interesting because we now have larger and more complete samples from which to work with and, thanks to declining storage costs, we’ll see the data go further back in time. Roger Ehrenberg who recently closed a new fund to invest in data startups, thinks that big data is only interesting with the ability to analyze and make actionable decisions from that data in a cost effective manner. This analysis is being made easier with open source technologies like Hadoop. We at Infochimps agree with Peter, Jeff and also Roger. Our marketplace exists to inform us where demand is, so that we can pre-compute or otherwise create data products from the massive amount of data we have and distribute them at a low cost to our customers. We believe the key to staying ahead of big data is understanding what questions are being asked of the data. When you know the questions you can figure out the answer. Otherwise, big data is just a liability and not an asset.
