Once again, we’ve had another great Hackweek at LiveRamp, the time that engineers, customer associatives and product managers come together to work on whatever that they think will make LiveRamp better or make their life at LiveRamp better. Here is a recap of Hackweek projects. As this Hackweek coincided with Halloween, you’ll see LiveRampers hard-hacking in Halloween costumes:
Turnaround Tracker V2_V2
If you read our last blog post about Hackweek, you’ll remember this project that monitors the progress of customer data imports in LiveRamp’s matching pipeline. In this Hackweek, Aaron, Alfonso, Josh, Tina, Jeremy and Neil worked on the version 2, which is multiple times faster and much more user friendly to customer associatives.
The team achieved this by using more nimble database operations and bringing in feedback from customer associatives. Now customer associatives can see the statuses of their own imports and easily find links to other web apps that provide additional information about the import.
As LiveRamp grows, knowledge about our engineering workflows and process grow and it becomes difficult for everyone to keep up to date. While managers are thinking hard about how to tackle this day and night, LiveRamp engineer James Sun turned to robots for help. This Hackweek he developed Rampy, a chat robot that listen to all the LiveRamp chat rooms and auto replies to certain messages. He also built a backend platform to manage Rampy, which makes it easy for other people to add in their own message replying scripts. By moving certain messages to Rampy, LiveRamp is introducing a convenient information channel that saves unnecessary communication effort and prevents potential human error.
Workflow Notification / Pipeline UI
LiveRamp has been constantly working on better monitoring and notification for its workflows. Ben Podgursky made another huge step on that front by introducing a more granular workflow notification feature. People can now register themselves for notification of selected workflows through a new workflow UI, while in the past they had to hard-code the recipients’ email addresses in the code.
Possibly bored by the presentation preparation for that project, Ben decides to work on another project on the night of Hackday. And the result is Pipeline UI! Pipeline UI visualizes workflow dependencies for all Liveramp workflows based on data production and consumption relationships recorded in the database. It’s actually the first time we have insight into this programmatically, instead of human knowledge and communication.
Up till to now, LiveRamp engineers are still using less sophisticated way of running apps, i.e. deploying a certain app to a specific machine and configuring a cron job to run it there. Porter and Jared, tired of remembering which app runs on which machine, decides to try some modern job scheduling system this Hackweek. The thing they chose is Kubernetes.
Kubernetes is an open source system that clusterizes machines and orchestrates Docker containers in that cluster. It eliminates the idea of specific machine and thus by using it LiveRampers only needs to submit their jobs to the master of Kubernetes. During the Hackweek, Porter and Jared set up a Kubernetes cluster and integrated it with LiveRamp’s Chef framework. After that they successfully kicked off several apps wrapped in containers. We’re all quite excited by the possibility of soon writing our own containers.