I apologize in advance for all the hammertime memes. It was just too good to pass up. Surely he must be used to this. Or at least not surprised by it. =D
So, Big Data. What is it? Well, in simple terms, it’s the realization and acceptance of the fact that data is multi-model, multi-faceted, multi-sourced, and constantly growing. It’s the fact that the traditional RDBMS is no longer the be-all end-all source of truth and valuable information. It’s part of a larger ecosystem involving JSON document stores, CSV files, streaming volatile bits of data coming from random devices and user activity that loses its meaning and potential impact almost as quickly as it can be gathered and sifted and stored.
But what do we actually get out of it? As a small-medium enterprise NOT in the software business, I have to say, not as much as the hype would have us believe. And look, I’m not so jaded and crusty that I refuse to adapt new tech. I Just haven’t seen a meaningful transformative business use-case for it. Sure, we have Google Analytics telling us how our websites are doing, and someone in marketing knows something about trending our social media traffic. Does it really help us make more money? Heck if I know.
Here’s what I’d like to see from the thought leaders. Give me something I can chew on — a real-world, non-hypothetical, non-frivolous, impactful use-case for adopting and implementing something like Hadoop/Spark or Azure Data Lake. Show me how my business can realistically journey down the path of predictive analytics and what it’s going to take from our Devs, IT staff, and management to actually get there.
Because they don’t get it yet. I have managers still worrying about how much we’re spending on a dinky little flash storage array to support the growing needs of our on-prem converged infrastructure stack. Meanwhile the AWS bill continues to baffle, and Devs want to play with Docker and Lambda. But we can’t seem to convince the higher-ups that they’re short-staffed on the internal-apps team, even after a minor version upgrade takes 4 hours of Ops time and half a dozen end-users doing post-mortem testing just to be sure we didn’t break anything unexpected.
I’m not here to complain. Really. I do want to see something amazing, something inspiring, something that shows me what Big Data truly brings to the table. And sure, I’ve see the vendor demos; they’re all just a bit outlandish, no? I mean, they look really cool, sure — who doesn’t want to see a chord diagram of who’s killed who is GoT? — but does that really help my business improve sales and productivity?
My point is, there’s a gap. A chasm of misunderstanding and mis-matched expectations between what management thinks Big Data is/means, and what it takes to actually implement. They see the pretty pictures and the fancy demos, but they don’t see the toil and sweat (or at least, in the cloud, gobs of cash) that go into building & operating the underpinnings and pipelines that drive those nice graphics. Not to mention the fundamental issues of data quality and governance.
So do us a favor, Big Data pundits. Show us something real, something that “the little guy” can use to up his/her game in the market. Something that makes a positive impact on small non-startup non-software businesses with understaffed IT & Dev teams. But more importantly, stop glossing over the effort and resources that it takes to “do Big Data right“. Managers and executives need to understand that it’s not magic. And IT practitioners need to understand that it’s actually worth-while. Because I believe you — really — that the payoff in the end is there, and is good. But you need to convince the whole stack.
PS: I know this is a fully day late for T-SQL Tuesday, and as such, I wasn’t going to post a ping-back in the comments of the invite, but then I saw there were only 8 others, so I felt it would benefit the event if I did add my late contribution. I’ll tweet with a modified hash-tag instead of the standard #tsql2sday, to reflect my late-ness. Hopefully that’s a fair compromise to the community & the event’s intentions. =)