During Hackweek, everyone at LiveRamp gets a week to work on projects they’re passionate about. Hackweek has a long history at Liveramp; we’ve gotten many efficiency tools, new products, interesting investigations, and other fun things from it. We are very proud of results of this Hackweek, too.
In case you missed our previous Hackweek recap, you can read it here.
Alex, Harry and Meng developed a convenient chrome extension that makes it simpler to search for articles in our internal resource website. It can also be used to add and remove labels to existing articles.
To make sure we’re delivering customer data on time, we closely monitor our workflows, and we want the right team to be alerted whenever something goes wrong. Although the logic for auto-detecting failures and delays is mostly simple, there used to be a lot of overhead to implementing alerts. Chris, Armaan and Shia removed this overhead by creating this great UI where you can easily register an alert. It auto-detects errors and sends an alert email to the right team. Canary also includes support for generating neat html formatted email bodies.
Slot Machine dashboard
Our mapreduce cluster is a heavily-utilized (~100%) resource that is shared among many workflows. When we add a new workflow or run an existing workflow more often, we impact other workflows by increasing the contention for cluster resources. This impact is not easily measurable and we’re left with many questions that aren’t easy to answer:
- How much processor resource are we utilizing?
- When should we invest more in the cluster?
- How many machines should we buy?.
Alfonso, Ben, Jeremy, Tenzing and Vishrut visualized this problem for us in their slot machine project (in mapreduce terminology, a slot is a portion of machine that can be reserved to run a map or reduce task) . In the slot machine dashboard, you can easily see how much cluster resource each workflow takes.