Migrating a Big Data Environment to the Cloud, Part 3

How do we get there? In part 2 of this series we discussed what we wanted our cloud MVP to look like.  The next question was — how do we get there without turning the company off for a month? We started with what we knew we needed.  For at minimum a few months, our […]

Migrating a Big Data Environment to the Cloud, Part 4

Copying to the cloud LiveRamp is in the midst of a massive migration of all of our infrastructure to GCP.  In our previous posts we talked about our migration and our decision to use Google as our cloud.  In this post, I want to zoom in on one major problem we needed to solve to […]

Migrating a Big Data Environment to the Cloud, Part 2

Starting the journey Last post we discussed why we were migrating to the cloud, GCP in particular.  Once we knew we were migrating, we started by asking ourselves three questions: What will our cloud architecture look like on Day 1?  We know there’s a lot of exciting stuff we could do in the cloud — […]

Migrating a Big Data Environment to the Cloud, Part 1

LiveRamp is a big data company. A lot of companies have big data.  Robust logging frameworks can generate a PB of logs before breakfast and stash it away forever on S3, just in case. A few companies even use their big data.  They have a product, and then use Hadoop and Spark to do some […]

[Opinion] Our Exciting Journey as Data Scientists Onwards to Higher Levels of Abstraction

Introduction Over the last few years, I’ve been consistently excited by the ever-increasing pace of new developments in the cloud computing space. By leveraging these tools, engineers and data scientists can now tackle bigger problems by outsourcing large parts of the work to cloud vendors. As a data scientist, I’ve been particularly focused on the […]

[Opinion] Report Guidelines

Reports are our chief vehicle for influencing organizations, especially as they grow to a larger number of people. Further, they are also how we assemble our collective knowledge about the problems we work on and the solutions to these problems. Because reports are so important, we’ve collected some guidelines about how to create amazing reports. […]