Source URL: https://cloud.google.com/blog/products/data-analytics/paypals-dataflow-migration-real-time-streaming-analytics/
Source: Cloud Blog
Title: PayPal’s Real-Time Revolution: Migrating to Google Cloud for Streaming Analytics
Feedly Summary: At PayPal, revolutionizing commerce globally has been a core mission for over 25 years. We create innovative experiences that make moving money, selling, and shopping simple, personalized, and secure, empowering consumers and businesses in approximately 200 markets. Ensuring the availability of services offered to both merchants and consumers is paramount.
PayPal’s journey with Dataflow has been a success – empowering the company to overcome streaming analytics challenges, unlock new opportunities, and build a more reliable, efficient, and scalable observability platform.
The observability platform team at PayPal is responsible for providing a telemetry platform for developers, technical account teams, and product managers. They own the SDKs, open telemetry collectors, and data streaming pipelines for receiving, processing, and exporting metrics and traces to their backend. PayPal developers rely on this observability platform for telemetry data to detect and fix problems in the shortest possible time. With applications running on diverse stacks like Java, Go, and Node.js, producing around three petabytes of logs per day, a robust, high-throughput, low-latency data streaming solution is critical for generating log-based metrics and traces.
Until 2023, PayPal’s observability platform used a self-managed Apache Flink-based infrastructure for streaming logs-based pipelines that generated metrics and spans. However, this solution presented several challenges:
Reliability: The system was highly unreliable, with no checkpointing in most pipelines, leading to data loss during restarts.
Efficiency: Managing the system was expensive and inefficient. Pipelines had to be planned for peak load, even if it occurred infrequently.
Security: The deployment needed to better conform to security guidelines.
Cluster management: Cluster creation and maintenance were manual tasks, requiring significant engineering time.
Community Support: The solution was proprietary, limiting community support and collaboration.
Software upgrades: Customizations required updating the binary, which was no longer supported.
Long-term support: The solution was an end-of-sale product, placing business continuity at risk.
PayPal needed a cloud-native solution that could address these challenges and unlock new opportunities. Their key requirements included:
Effortless scalability: Handling massive data volumes and fluctuating workloads with automatic scaling and resource optimization.
Cost reduction: Optimizing resource utilization and eliminating costly infrastructure management.
Seamless integration: Connecting with other data and AI tools within PayPal’s ecosystem.
Empowering real-time AI/ML: Leveraging advanced streaming ML capabilities for data enrichment, model training, and real-time inference.
After extensive research and a successful proof of concept, PayPal decided to migrate to Google Cloud’s Dataflow. Dataflow is a fully managed, serverless streaming analytics platform built on Apache Beam, offering unparalleled scalability, flexibility, and cost-effectiveness.
The migration process involved several key steps:
Initial POC: PayPal tested and validated Dataflow’s capabilities to meet their specific requirements.
Ingestion Layer Shift: They transitioned from Apache Pulsar to Apache Kafka for seamless integration with Dataflow.
Pipeline Optimization: Working with Google Cloud experts, PayPal fine-tuned pipelines for maximum efficiency, including redesigning the partitioning scheme and optimizing data shuffling.
Technical Benefits
Dataflow’s automatic scaling capabilities ensure consistent performance and cost efficiency by dynamically adjusting resources based on real-time data demands. Its robust state management capabilities enable accurate and reliable real-time insights from complex streaming operations, while its ability to process data with minimal latency provides up-to-the-minute insights for faster decision-making. Additionally, Dataflow’s comprehensive monitoring tools and integration with other Google Cloud services simplify troubleshooting and performance optimization.
Fig 2. An example image of the execution details tab showing data freshness by stage over time, providing anomaly warnings in data freshness.
Business benefits
The serverless architecture and dynamic resource allocation of Dataflow have significantly reduced infrastructure and operational costs for PayPal. They’ve also seen enhanced stability and uptime of critical streaming pipelines, leading to greater business continuity. Furthermore, Dataflow’s simplified programming model and rich tooling have accelerated development and deployment cycles, boosting developer productivity.
Implementing a high-throughput, low-latency streaming platform is critical to providing high cardinality analytics to business, developers and our command center teams. The dataflow integration has now empowered our engineering teams with a strong platform to monitor paypal.com 24 x 7 thereby ensuring PayPal is highly available for our consumers and merchants.
Varun Raju, Architect, Observability Platform, PayPal
Empowered Innovation
Perhaps most importantly, Dataflow has freed up PayPal’s engineering resources to focus on high-value initiatives. This includes integrating with Google BigQuery for real-time Failed Custom Interaction (FCI) analytics, providing the Site Reliability Engineering team with immediate insights. They’re also implementing real-time merchant monitoring, analyzing high-cardinality merchant API traffic for enhanced insights and risk management.
PayPal is excited to continue exploring Dataflow’s capabilities and further leverage its power to drive innovation and deliver exceptional experiences for their customers.
Learn more about getting started with Google Cloud Dataflow
AI Summary and Description: Yes
**Summary:** The text details PayPal’s transition to Google Cloud’s Dataflow for enhancing its observability platform amid challenges faced with its previous infrastructure. This migration highlights innovations in streaming analytics, scalability, and efficiency, which are critical for businesses looking to modernize their data management strategies.
**Detailed Description:**
The text provides a comprehensive overview of PayPal’s initiative to improve its data streaming and observability capabilities by migrating from a self-managed infrastructure to a cloud-native solution. Here are the major points of discussion:
– **Challenges with the Previous Infrastructure:**
– **Reliability Issues:** The legacy system lacked adequate checkpointing, leading to potential data loss.
– **Efficiency Problems:** Managing systems proved to be costly and inefficient, with peaks in data load requiring excessive resource planning.
– **Security Concerns:** Deployment of the previous infrastructure did not align well with established security guidelines.
– **Manual Cluster Management:** Significant engineering resources were required for cluster creation and maintenance.
– **Limited Community Support:** The proprietary nature of the solution restricted community-driven support and enhancements.
– **Obsolete Software Support:** The previous solution was an end-of-sale product, threatening business continuity.
– **Required Improvements:**
– PayPal sought a cloud-native solution that could offer:
– **Effortless Scalability:** Automatic adjustments to handle data volume variations without manual intervention.
– **Cost Efficiency:** Better resource optimization to minimize infrastructure expenses.
– **Integration Capabilities:** Seamless interactions with existing data and AI tools.
– **Real-time AI/ML Empowerment:** Enhanced streaming machine learning capabilities for quicker data processing and insights.
– **Migration to Google Cloud Dataflow:**
– After validating and testing Dataflow’s capabilities, PayPal transitioned to Google Cloud’s Dataflow, a serverless platform for streaming analytics.
– Key steps in the migration included:
– Conducting a successful proof of concept.
– Shifting the ingestion layer from Apache Pulsar to Apache Kafka for better integration.
– Collaborating with Google Cloud experts to optimize pipeline performance.
– **Technical Benefits of Dataflow:**
– **Automatic Scaling:** Adjusts resources based on real-time demands, enhancing performance and cost management.
– **Robust State Management:** Enables accurate insights from complex data operations.
– **Low Latency Processing:** Facilitates timely data insights for quicker decision-making.
– **Comprehensive Monitoring Tools:** Streamlines troubleshooting and performance enhancements.
– **Business Advantages:**
– Reduction in operational costs due to serverless architecture.
– Increased stability and uptime of critical systems supporting business continuity.
– Accelerated development processes, thus boosting productivity across teams.
– **Empowered Innovation:** The migration has allowed PayPal’s engineering teams to prioritize high-value projects like real-time analytics and merchant monitoring, significantly advancing their capabilities in insights and risk management.
The transition signifies a substantial evolution in how PayPal manages data, emphasizing cloud solutions’ role in enhancing security, scalability, and operational efficiency. This development is critical for professionals focusing on infrastructure security, cloud computing, and data management, as it illustrates effective strategies for modernizing legacy systems while addressing compliance and security concerns.