Author: David Greenberg
Presented at: MesosCon Europe, Dublin, Ireland
Abstract: Spark is a popular new platform for interactive high performance analytics, machine learning, and data processing. The trouble is, Spark tends to monopolize whatever Mesos cluster you run it on, so you either create completely separate Spark clusters for each user, or you otherwise limit the resources each user can use. Cook is an advanced fair-sharing, preemptive scheduling backend for Spark. You can run one instance of Cook on your Mesos cluster, and it will automatically adapt the capacity for every user and team on your cluster so that interactive jobs run immediately but utilization remains high. Cook also has a REST API and Java client, and it’s written in Clojure with Datomic.
