Hint: It’s not technology. It’s us.
?We have met the enemy and he is us.? Cartoonist Walt Kelly crafted this famous line for the American cartoon Pogo in 1970. But when we look at the adoption of Big Data, this turns into a truism. It?s not the flow of data holding us back. It’s not the technology used to process it. It’s us, the humans.
This might come as a surprising conclusion. After all, we humans produce acres of data every day, leaving a trail that spans social networking, shopping habits, web surfing, and lots of other areas. So you might think that dealing with the volume and variety of data would be challenging, let alone the velocity with which it keeps changing.
The fact is that Hadoop, Spark, MPP (Massively parallel processing) databases, and other Big Data tools can handle the data we are throwing down. In many cases, it is the mindset that can?t handle Big Data.
Why Aren’t Some Companies Embracing Big Data?
Some industries ?finance, healthcare, and media, for example have been early adopters of data analytics and have not shied away from the amped-up proportions of Big Data. Web giants like Amazon, Facebook and LinkedIn belong in this category as well. But many other large companies hesitate to make the leap or to fully utilize what they have. Why?
In my experience, one or more of the four things listed below hold organizations back from fully getting on board with Big Data:
- Lack of Skilled Users. Designing, implementing, and running an in-house analytics department takes a specific skill set that relatively few people have. In consequence, employees with analytics or data science experience are in high demand and are paid accordingly.
- Resource Availability. This is the hardware and software equivalent of the human problem described above: investing in Big Data is an investment indeed. Often, companies don?t have much extra room in the budget.
- Company Culture. Companies that are conservatively operated or that have an abundance of office politics can present a challenge for innovators regardless of what they are trying to innovate.
- Resistance to Change. This seems to be part of human makeup at the genetic level: we are suspicious of change. Even if it?s good change and we see how it is benefitting others, we may not be interested in altering what we do.
There are many other factors you could add to this list. But when perception shifts when people stop looking at the adoption of Big Data as an innovation issue and start seeing it as an issue of how we embrace innovation this is when progress can be made.
Ready or Not, Here It Comes
The insights that Big Data makes available are tremendous. It?s being used in everything from social graph analysis to credit risk management, from network monitoring to fraud detection. And it?s not going away any time soon; in fact, it?s advancing. Big Data is a key drive for artificial intelligence and machine learning, all the rage right now.
Very often businesses have the data they need, ready and waiting to be used. Data science has given us the tools to process mountains of data and find hidden treasures. The question is, what is stopping my organization from using them?
