Lessons from a Data-Driven Organization
I joined eGroup | Enabling’s team after retiring as CIO for a top ENR Architecture & Engineering firm. Since then, I have had time to step back and reflect. After my former organization successfully replaced all our business applications with SaaS solutions, our data resided in the cloud. We consolidated where possible to minimize the number of business applications in our portfolio. Fewer business applications resulted in fewer data sources. These were positive steps and outcomes of our evolving digital transformation. This blog captures my main lessons learned along the way.
Designing the Enterprise Data Architecture
One of the significant advancements the organization made was adopting a crawl, walk, run approach to become a data driven organization. Our first goal was to develop an enterprise data architecture. We strove to eliminate data silos and begin to make trusted data widely available to individuals with proper authorization.
We began by creating a data warehouse that consolidated data from Dynamics 365 Finance & Operations, legacy financial data, Dynamics 365 Sales, and our Human Capital Management systems. Part of the design was data governance documenting the system of record for specific data. The data warehouse refresh was automated, utilizing Azure Data Factory, to update data as close to real time as possible from the respective systems of record.
Building the Foundation & Changing the Culture
Our data warehouse utilized an Azure SQL database. The use of Azure SQL is a good example of the benefit of elasticity of the cloud. We had a “normal” state for our operating Azure SQL database. Normal state included services such as database refreshes, incremental data loads, creation of Fact Tables and data analytics and reporting. The Fact Tables consisted of measurements, metrics, and facts about our business processes. On occasion, we needed to go outside the norm and process large quantities of data. At these times, we were able to elastically increase our Azure SQL resources to accommodate the temporary increased processing requirements.
Azure Data Factory (ADF) enabled us to perform cloud-based ETL (Extract, Transform and Load) and data integration services without custom software development for ETL functions. ADF saved significant time, enabling us to accomplish much with minimal effort and time. All ETL activities were set up to run on a schedule which updated the data warehouse to keep it as close to real time as possible.
When my former company began this journey in 2017, Power BI was new a new kid on the block, but we could see the potential it had for creating a unified data analytics and reporting platform. Utilizing the unified data warehouse of certified, trusted datasets, a portfolio of dashboards and reports were developed to provide executive leaders, operational managers, and project managers with near-real time data to make decisions. Their feedback was extremely positive, and we were able to rapidly expand the portfolio by incorporating their great ideas. Many of these leaders and managers discovered the Analyze in Excel capabilities available through Power BI and were able to self-serve to fine tune their data analysis to answer their “what if” questions.
With the use of Azure SQL, Azure Data Factory and Power BI, the organization’s appetite for analytics grew. The dashboards and reports portfolio continued to expand and make improvements. One impactful change was the use of push notifications to specific operational and project managers based on the KPI’s (Key Performance Indicators) falling outside of acceptable parameters. Operational managers and project managers were notified when potential actions were necessary. An example of push notifications creating a positive business impact is when our company DSO’s (Days Sales Outstanding) improved by over 30% by proactively alerting project managers and collections staff to take actions.
Executing this strategy provided significant financial and cultural improvements through the successful use of these Microsoft cloud services and technologies.
Harnessing Azure’s Innovations
Since the journey started, capabilities have continued to emerge, removing limitations that existed for years with legacy systems. Microsoft has continued to add capabilities to lower the barriers to begin organizations’ transformation to a data driven organization. One such addition was the availability of a Common Data Model. The Common Data Model is influenced by data schemas that are present in Dynamics 365. Common Data Model simplifies data management and app development by unifying data into a known form and applying structural and semantic consistency across multiple apps and deployments. Since my prior organization was operating Dynamics 365 Finance and Operations and Dynamics 365 Sales, the Common Data Model contained most of the schema items we needed. Additionally, the Common Data model is extensible.
The Common Data Model includes the following benefits:
Historically, the work to build an app has been tightly tied with data integration, but with the Common Data Model and the platforms that support it, the two can happen independently.
Other Microsoft innovations and releases have become available that support organizations implementing, managing, and governing an Enterprise Data Architecture. Notably Azure Purview and Azure Synapse.
Azure Purview is a unified data governance service that helps you manage and govern on-premises, multi-cloud, and software-as-a-service (SaaS) data. Purview creates a holistic, up-to-date map of your data landscape with automated data discovery, sensitive data classification, and end-to-end data lineage. The information curated by Purview empowers data consumers to find valuable, trustworthy data.
Azure Purview Data Map provides the foundation for data discovery and effective data governance. Purview Data Map is a cloud native PaaS service that captures metadata about enterprise data present in systems on-premises and in the cloud. Purview Data Map is automatically kept up to date with a built-in automated scanning and classification system. Business users can configure and use the Purview Data Map through an intuitive UI and developers can programmatically interact with the Data Map.
Azure Synapse is an enterprise analytics service that accelerates time to insight across data warehouses and big data systems. Azure Synapse brings together the best of SQL technologies used in enterprise data warehousing, Spark technologies used for big data, Data Explorer for log and time series analytics, Pipelines for data integration and ETL/ELT, and deep integration with other Azure services such as Power BI, CosmosDB, and AzureML.
Extracting Value from Data
With the enterprise data curated and organized, organizations now can enable and empower their digital natives and citizen developers with the low-code capabilities available in the Power Platform – Power BI, Power Apps, and Power Automate. Encouraging citizen developers to utilize the platform to analyze enterprise data will enable them to go faster, achieving greater creativity and innovation across their company without requiring highly skilled IT resources.
Once organizations have a solid enterprise data architecture, they can rapidly evolve into a data-driven organization. The enterprise data architecture increases the confidence in the quality of organizational data; they have trusted data sources. With these foundational changes in place, organizations have greater confidence in their operational metrics. Higher confidence in data yields higher-confidence business decisions.
Reflecting back, it was an exciting time to be in technology and data analytics. I departed my role as my organization was transitioning from the crawl phase and moving into the walk and run phases. I wish them well as they continue to accelerate their progress. It was extremely rewarding and gratifying that my team was able to initiate the journey and begin to transform the organization into a data driven organization.
Has your organization begun its enterprise data platform journey? Have you begun harnessing cloud innovations to improve your data management and data analytics? Hopefully these reflections give you much to think about as you evolve your data-driven organization.