A major paradigm shift is taking place in the data industry. Everyday we see massive amounts of new data being created; and unlike ever before, this data is moving fluidly in many directions. Data is being created every second and being accessed from all over the globe; and in order to truly harness all that data, a shift in the way that we think about data integration is essential.

THE OLD WORLD

The old paradigms of data integration stand archaic and obsolete in this new data landscape. Just think about where we’ve come in the last twenty years. When data integration was just emerging on the scene in the mid-to-late nineties, the sources for data were extremely limited. The targets were primarily data warehouses and data marts, with an occasional loop back to operational systems for closed loop business intelligence. Most of the prevailing integration tools we use today were created in a vastly different world with very different demands, and frankly, a lot less data.

A NEW WORLD WITH NEW CHALLENGES

We are well into the age of big data, armed with more data than ever before. The fact of the matter is that it will not let up anytime soon. EMC recently predicted that from 2013 to 2020, the digital universe will grow by a factor of 10 – from 4.4 trillion gigabytes to 44 trillion.

New data and new data types are emerging every day with very limited structure. In fact, 80% of all enterprise data is unstructured or semi-structured. Data is coming from all over the place.

Data is pouring out of thousands of new API’s that are created by apps and the Internet of Things. There are billions of sensors and devices creating trillions of time stamped events and the number of sensors will soon be in the trillions.

Hadoop adoption has skyrocketed in recent years, driven by three main factors:

  • It solves technology challenges like scalability, performance, and maintainability.
  • It empowers the business to forge new ground in both discovery and predictive analytics.
  • It is affordable enough to allow companies to keep all of their data and discover the value later.

While some still question the longevity of the Hadoop explosion, at this point, the game is over. Hadoop is here to stay and gaining more ground every day. The pace at which we are moving is no longer linear, but exponential; so now is the time to embrace all that we can do with big data.

Hadoop must also live within the existing environment, however, it is not the be-all and end-all of data platforms. There are many things that it does well, but there are also many things that can be done better on other kinds of platforms—like analytic platforms. Essentially, multiple platforms must live and work together. While Hadoop may end up being a big player in big data, today it is one of many players. As a result, some of the greatest challenges involve moving data into and out of the platform. It is both a landing spot for massive data coming in and a provisioning platform for processed data coming out.

NEW WINE IN OLD WINE SKINS

As the saying goes, “You cannot put new wine in old wineskins.” If you do, the wineskins burst. That’s exactly what is happening with companies who are trying to use the “old world” ETL technology to process and provision data in today’s interwoven world of data. These ETL servers become a huge bottleneck. While many of the legacy integration tools have been re-engineered to handle new kinds of data, there are only so many add-ons you can attach to old technology. It’s sort of like sewing patches onto the old wineskin, at a certain point the structure will just give out.

Now more than ever before, new challenges require new approaches rather than patched up old solutions. The future of data integration doesn’t have to be so overwhelming and complex, but enterprises must be armed with the tools accelerate analytics and adapt at business speed. It is obviously time for a paradigm shift in data integration, and over the coming months we are going to explore what it means to be an organization that leverages The Principles of Modern Data Integration. It’s time to make the move to modern data integration and Diyotta is here to help.