Source URL: https://cloud.google.com/blog/products/data-analytics/how-virgin-media-o2-simplified-internal-data-sharing-with-bigquery-analytics-hub/
Source: Cloud Blog
Title: How Virgin Media O2 simplified internal data sharing with BigQuery’s Analytics Hub
Feedly Summary: Easily sharing data has become a critical asset for driving informed decisions across any business. Still, many organizations struggle with the complexities of sharing data in a way that’s effective and compliant. Data teams often navigate challenges like unclear governance, version control issues, data silos, access restrictions, and a lack of data management skills across the broader business.
Virgin Media O2 is a media and telecommunications company that relies on data sharing across the organization to optimize operations, drive strategy, and empower decision makers. From finance to marketing, all departments are supported by the data team for accurate, timely information.
As director of engineering, I was overseeing data platform teams who were running into a number of hurdles in sharing and documenting data. Communication overhead was high, with 20 to 30 hours a week spent on training, support, pipeline issues with deployment, and responding to data-related tickets — with even more time and resources spent on meetings to find solutions to access issues. Internal teams were also sharing and editing different versions of non-live data, which led to data duplication, latency, and slow time-to-value.
These obstacles and inefficient processes often slowed the flow of information and hindered decision-making across the business. Virgin Media O2 needed a solution that would make it easier to provide data access, as well as governance, across business divisions. Without it, we’d be stuck with no centralized data-sharing process, and unable to obtain org-level visibility and efficiency.
Navigating internal data-sharing challenges
As teams were already working on projects based in Google Cloud, it was important to ensure strong version control so the data was always accurate, consistent, and up-to-date, but this often delayed the time it took to produce insights further. Virgin Media O2 already had their enterprise data in BigQuery in order to support their enterprise and AI needs, so Analytics Hub — BigQuery’s data exchange capability — was one potential solution to build on their existing infrastructure.
The data platform team decided to pilot Analytics Hub after learning about its scalability, self-service capabilities, and simple governance model for data tags and quality; this last feature in particular aligned with improving the implementation of privacy by design.
After a successful pilot, Virgin Media O2 had established a clear onboarding and training process, appointing two dedicated owners for each new data exchange and two owners on the subscriber side to make it easier to track any actions in BigQuery. Over the next nine months, this process was scaled to 25 teams and more than 50 exchanges, 100 listings, and 500 tables, with around 300 daily users.
One of the biggest benefits the team found was that Analytics Hub doesn’t duplicate data, which avoids the network and storage costs associated with copying data. It does this by creating a real-time pointer, known as a Linked Dataset, to the source dataset that can be shared. So if someone edits the original data source, it is immediately available to all subscribers, and can easily be audited, tracked, and restored. This approach also means a disaster recovery safety net is also built in.
Analytics Hub also overcomes the complexity challenges with creating views, specifically accessing data through authorized views often results in lost metadata from the original table. However, with direct access to the source dataset, all table descriptions and columns are retained for the subscriber to see.
Now that Virgin Media O2 was linking data directly from the data publisher to a data subscriber, they were able to save time, reduce latency, and improve management and usability for both publishers and subscribers—all while the platform’s enhanced governance provides a centralized location for managing data access and quality.
Analytics Hub streamlines the process of sharing data between teams and business departments, reducing manual effort and errors. The platform has been particularly valuable to analytics engineers, data analysts, data engineers, data scientists, and software engineers within Virgin Media O2 — ensuring that everyone has real-time access to the data they need for their various roles.
Saving time with data that’s more accessible than ever
After rolling out Analytics Hub to around 25 squads, the solution helped save up to 30 hours a week in time spent on training, support, pipelines issues with deployment, and communication overhead from squads using the old solution. Now, the time spent each week across all teams is as low as 30 minutes, because there are almost zero issues. The team estimates this is around a 95% effort-saving result. Data is no longer kept in silos, making it widely available to the different departments that need it.
By building a dashboard, the team was able to democratize access to data for subscribers and their broader teams without needing to access Analytics Hub itself. Enabling users to subscribe to datasets enables self-service while still maintaining a strong governance model. Eliminating the middleman streamlines and speeds up this repetitive process while maintaining a robust governance model.
Key benefits realized from secure data sharing
1. Data Integrity and SecuritySecure, zero-copy sharing ensures consistent data integrity across departments, preventing metadata loss and unauthorized access.
2. Cost Efficiency and Simplified ManagementBy eliminating data movement, the platform reduces long-term costs and operational overhead, while a small team can manage data oversight effectively.
3. Centralized Monitoring and GovernanceA unified dashboard provides real-time control over data sharing activities, allowing quick identification of issues and enforcing strict access and permission policies.
Looking ahead, the team is focusing on streamlining four key areas: data ownership, data catalog, data quality metrics, and more effective tagging of sensitive data. The goal is to automate the entire data operation, and once data is certified, these four areas must be clearly defined and enforced as a policy through Analytics Hub.
The process above, known as “data certification,” offers two key benefits:
Fast identification of uncertified data assets and the tracing of data quality issues within minutes by leveraging data quality metrics (at the column level) and data lineage.
Real-time identification of sensitive data and the monitoring of where it’s being consumed, enabling proactive management of data privacy risks through live audit logs.
How you can get started
For customers new to Google Cloud, BigQuery is where to start your journey. For those already using BigQuery, Analytics Hub is the next step.
AI Summary and Description: Yes
Summary: The text discusses the challenges faced by Virgin Media O2 regarding data sharing and governance within their organization. The introduction of Google’s Analytics Hub has streamlined data access and improved management processes, leading to significant time savings and enhanced compliance with data governance. This case study highlights the importance of effective data sharing solutions in complex organizational structures.
Detailed Description:
The provided text emphasizes the critical role of effective data sharing in modern organizations and illustrates how Virgin Media O2 addressed their internal challenges by implementing Google Cloud’s Analytics Hub.
Key Insights and Points:
– **Internal Data Sharing Challenges**:
– Companies often face difficulties with governance, version control, and data silos, which complicate effective data sharing.
– Virgin Media O2 was significantly hindered by inefficient processes that wasted time through training and communication oversights.
– **Overview of Virgin Media O2’s Data Needs**:
– Data access was fundamental for departments ranging from finance to marketing, requiring real-time, accurate data for decision-making.
– Teams were working within Google Cloud but struggled with version control, which delayed insights.
– **Implementation of Analytics Hub**:
– The pilot of the Analytics Hub focused on scalability, self-service capabilities, and a governance model compatible with privacy standards.
– Key outcomes from the pilot included a well-defined onboarding process and dedicated ownership roles which facilitated better tracking and data management.
– **Results Achieved**:
– Centralized and real-time access to data was established, significantly reducing the time spent on operations from up to 30 hours a week to just 30 minutes.
– The solution minimized data silos by linking data directly from publishers to subscribers, enhancing organizational efficiency.
– **Key Benefits**:
– **Data Integrity and Security**: Zero-copy sharing maintains consistent data while preventing unauthorized access.
– **Cost Efficiency**: Reduction of data movement lowered long-term operational costs.
– **Centralized Governance**: A unified dashboard for real-time monitoring offered increased control and quick issue identification.
– **Future Focus**:
– The team aims to streamline processes concerning data ownership, quality metrics, and effective tagging of sensitive data, automating operations for efficiency.
– The concept of “data certification” will enable prompt detection of data quality issues and proactive management of data privacy risks.
Overall, this text illustrates the importance of modern solutions like Google Cloud’s Analytics Hub in enhancing operational efficiency, data governance, and security in data-sharing environments, making it relevant for professionals in information security, cloud computing, and compliance.