This blog is based on Episode 13 of our GovEd Talks Video series: Four Steps to Data Architecture for Analytics by Dave Wells, Director of Education at E Learning Curve.
The world of data architecture has seen significant changes in recent times, and its high time we rethought our approach to data architecture. In this blog post, we will explore the evolving landscape of data architecture, the challenges it presents, and the need for a fresh perspective to navigate this data-centric world.
Why Rethink Data Architecture?
Traditionally, data architecture has been rooted in the concepts of data warehousing from the 1990s. While these principles served us well for a long time, we have entered an era of what I call the "data quake." This seismic shift in the data landscape has disrupted the foundations of our traditional data management practices, much like an earthquake shakes the surface of the Earth.
The familiar landscape of relational database management systems, data warehousing, ETL processes, and business intelligence tools is now joined by a multitude of new data technologies and practices. Big data, Hadoop, data pipelines, schema-on-read, microservices, containers, edge computing, data science, AI, machine learning, NoSQL databases, and more have all contributed to this upheaval. The challenge we face is that many organizations are attempting to retrofit these new technologies into their existing 1990s data warehousing architectures using a patchwork of solutions. This duct-tape-and-band-aid approach is a short-term fix but is ultimately unsustainable.
The data management complexities of today are vastly different from what we once knew. We must adapt to the modern data landscape, which includes a diverse array of data sources, from traditional ones like ERP systems to big data, web, social, sensors, and open data. Managing data has evolved as well, with data cataloging taking the spotlight as the next generation of metadata management. The rise of semantic modeling, data governance, data providence, data lineage, and data connectors are all aspects we need to consider. Additionally, data pipelines have transformed from simple ETL processes to complex data synchronization, data stream ingestion, and change data capture. Data sharing now encompasses multiple data warehouses, data lakes, master data management, reference data management, and cataloging of user-created datasets.
Accessing data has also become more multifaceted, with a need for file access, APIs, and API management. The rise of analytic sandboxes for experimentation and exploration is changing how we work with data. Moreover, our applications have evolved beyond basic BI and reporting to include advanced analytics, data science, artificial intelligence, and machine learning. This evolution extends to the consumers of data, which now includes not only human analysts but also automated processes and applications.
The complexity of deployment platforms has also multiplied. It's no longer just about on-premises or the cloud; it's about on-premises, cloud, multi-cloud, and hybrid environments. Data management now spans across various cloud platforms and on-premises services, making the landscape more intricate than ever.
The Changing Data Landscape
The traditional BI data architecture depicted a simple flow of data from source systems to data warehouses, master data repositories, operational data stores, and finally to dashboards, OLAP, and reporting tools. This architecture primarily served a limited number of data consumers.
In contrast, the analytics data architecture of today has expanded to include a wide range of data sources, such including geospatial, machine, IoT, social, commercial, and more. This new architecture serves a diverse set of data consumers, from traditional business analysts to data scientists and automated processes.
The Path Forward
In this rapidly changing data landscape, it's clear that we cannot rely on outdated data architecture practices. We must embrace a fresh perspective on data architecture that aligns with the realities of the modern data-driven world.
This involves rethinking our approach to data management, metadata, data governance, data connectors, data pipelines, data sharing, data access, and data services. We need to build data connectors that enable flexible access to data, creating reusable components that support schema-on-read concepts without excessive manual effort.
In this evolving landscape, the role of a data architect becomes more critical than ever. Data architects should not be confined to traditional roles; they must adapt to the changing demands of data management. The key is to understand that data architecture is no longer about merely designing static data models; it's about creating dynamic, adaptable data ecosystems that can evolve with the data quake.
If you're interested in learning more about Data Analytics, consider enrolling in our upcoming course Data Analytics. Our course will help learners to ensure the integrity of the data, turn data into actionable information, and further use analytics to answer specific questions in a visual or another easy-to-use format.
