We ran across a interesting blog post by a Chris Stucchio from last year. It can be found here:
It is a very entertaining article and one of the best quotes is “Too big for Excel is not “Big Data””. You would think since we offer Hadoop hosting that we would be against this article but in truth, we couldn’t agree more. Having been in the hosting and IT world for many years, the co-founders of Bit Refinery have seen many fads come and go. Big data is definitely here but how many people really have “big data” and even if they did, would the cost of analyzing it be worth the additional sales it would bring?
There are many RLDBs (relational databases) out there that can easily crunch hundreds of millions of rows with no issues. With the performance of most of the hardware out there, we’ve only run into a few situations where companies truly have a big data problem.
We had a blog post yesterday that points our readers to the Hortonworks site where they have some good examples of true big data. We even have a customer that is using SQL Server to house a large web retail database with a few tables over 1 billion. You could even take one of our Hadoop nodes with 64 or 128GB of Ram and use one of the many in-memory databases out there to crunch your data. Lots of different options out there…
So, in conclusion, we feel your pain Chris. We enjoy reading the hype of all the startups and new products coming out on a weekly basis but sometimes the obvious answers/tools have been around for 10 years. (ie. Perl, Python and of course good old Korn shell scripting. 🙂 )
– The Bit Refinery Team