How LiveRamp is Empowering Data-Driven Decisions
In today’s data-driven world, businesses must harness the power of data to understand and engage with their customers effectively. Integrating and enriching disparate data sources to create a unified customer view is one of the biggest challenges companies face. This is where LiveRamp’s Offline Data Marketplace comes in, providing a seamless solution for data enrichment and integration.
Our product empowers clients to overcome these challenges by ingesting first-party PII-based data, appending third-party PII-based attributes, and returning the enriched data with PII intact. This process allows clients to gain a comprehensive view of their customers, leveraging this data for analytics and targeting audiences on known channels, such as email and direct mail. With enriched, accurate data, clients can personalize customer experiences, optimize marketing strategies, and create more effective campaigns.
The journey from “Hackweek” to innovation
Our journey began with a “Hackweek” project. What started as a proof of concept leveraging GCP-based tools quickly grew into a production-ready solution wrapped with orchestration frameworks. The initial system relied on a mix of different technologies for data processing, data shaping, and data movement tasks. This created significant complexity, leading to challenges in scaling and maintaining the system. These challenges made onboarding new data sellers and buyers a lengthy process, often taking weeks or even months. Additionally, processing and joining large datasets posed significant issues, resulting in performance bottlenecks.
Recognizing these limitations, we embarked on a mission to find a more streamlined and efficient solution. Enter the SQL-based architecture. This approach promised a simpler, more effective way to handle data processing, aligning with our needs. We envisioned a system where data could be ingested, processed, and managed with greater speed and reliability.
Overcoming hurdles
Our initial attempt to integrate a SQL-based system showed promise. We ingested data and processed it in batches using a single centralized process. While this approach worked for smaller datasets, it quickly became clear that it wasn’t scalable for larger workloads. Sequential processing was slow, and our pipelines often ran into out-of-memory issues as data volumes grew.
Undeterred, we refined our approach. We introduced intelligent orchestration pipelines capable of balancing workloads and distributing tasks efficiently. Data was partitioned and processed in parallel, significantly improving performance and scalability. We also implemented robust retry logic and exception handling to ensure that any issues were promptly addressed, minimizing disruptions.
Major operational improvements
To further enhance our operations, we made several key improvements:
- Real-time alerts: Providing instant updates on key tasks and failures, enabling quick responses
- Configurable design: Flexible resource allocation tailored to individual workflows and data providers
- Metadata monitoring: Comprehensive visibility into each workflow step
- Sequential task pipelines: Reduced memory load and prevented system stress
- Enhanced retry logic: Improved resilience with retries for individual queries and entire pipelines
The results of these efforts have been transformative. We reduced onboarding times dramatically, cutting down from weeks to hours in some cases. Data processing time was also reduced by half, providing more reliable data processing with no issues or errors encountered.
Looking ahead: Embracing new opportunities
As we continue to innovate and improve, we are excited about the future. Our experience with a SQL-based architecture has not only enhanced our current operations but also paved the way for future advancements. With the ability to onboard new customers faster, we are poised for significant growth and broader adoption of our product. This progress has also opened up new avenues for developing innovative product features, including potential integrations with other LiveRamp solutions. The opportunities this presents are truly exciting.