by Paroma Varma, Chris Ré, and other members of DAWN
“What do you spend time on while debugging machine learning pipelines?” Responses to this question at the Fall 2018 DAWN Retreat ranged from “finding the best way to use transfer learning” to “systematically sampling from raw data”. We identify three broad themes from our discussions and explore them in this post: (1) shaping training data, (2) exploiting log data, and (3) model introspection. Check out our other blogs related to debugging machine learning: using provenance to debug training sets and...
by Kexin Rong, Clara Yoon, Karianne Bergen, Hashem Elezabi, Peter Bailis, Philip Levis, Gregory Beroza
Worldwide, major earthquakes (magnitude 7+) occur approximately once a month, while magnitude 2 and smaller earthquakes can happen up to several thousand times a day. In fact, earthquake frequency is inversely proportional to magnitude, meaning most earthquakes are very small. An estimated 1% of these small-magnitude events are detected and recorded in public catalogs (Figure 1), yet these low magnitude earthquakes are used by scientists to uncover unknown seismic sources, understand earthquake mechanics and predict major seismic events. Figure 1:...
by Kai Sheng Tai, Vatsal Sharan, Peter Bailis, and Gregory Valiant
This post accompanies the paper “Sketching Linear Classifiers over Data Streams” by Kai Sheng Tai, Vatsal Sharan, Peter Bailis and Gregory Valiant, which was presented at SIGMOD 2018. Check out our code on GitHub. In online learning, we learn a predictor by continuously updating its weights according to a stream of labelled examples. For example, in spam classification, an online learning approach allows the spam classifier to dynamically adapt to newly-observed features, even those introduced by an adversary attempting to...
by Paroma Varma, Braden Hancock, Sahaana Suri, Jared Dunnmon, Chris Ré and members of Hazy Research
Training data is playing an increasingly important role in defining the performance of modern machine learning systems. The goal of this blog post is to maintain a “checklist” of the types of errors that can be introduced by unaccounted phenomena in the data and their labels, and simple ways to check for these errors. We would love to hear about other errors you have encountered and how you identify and correct them! Check out our previous blog on using the...
by Edward Gan, Jialin Ding, Kai Sheng Tai, Vatsal Sharan, and Peter Bailis
Quantiles or their equivalents (percentiles) are commonly used in data exploration workflows. However, they can be expensive to compute on increasingly high-volume, multi-dimensional datasets. In order to reduce query response times, data systems make use of sketch data structures to accelerate quantile computations and deliver approximate results. In this post, we show how a set of statistics including can be used to define a compact and efficient sketch: the moments sketch. The key property of this sketch is that it...
by Shoumik Palkar, Firas Abuzaid, Peter Bailis, and Matei Zaharia
Many big data applications often run on raw, unstructured or semi-structured data formats, such as JSON. Querying these files is often very time-consuming, especially for exploratory applications, where data scientists run queries to explore and better understand their data. Surprisingly, 80-90% of the execution time in these applications is actually spent on parsing the data, not on evaluating the actual query itself. Parsing is, in fact, the bottleneck. In this post, we introduce Sparser (code here), a recent research project...
by Shoumik Palkar, James Thomas, Deepak Narayanan, Pratiksha Thaker, Parimarjan Negi, Rahul Palamuttam, and Matei Zaharia
Weld is an open source project, with an initial prototype described in a CIDR 2017 paper. This blog describes the adaptive optimizer in Weld, which we present in our VLDB 2018 paper. Analytics applications compose a diverse mix of software libraries and functions, such as Pandas to manipulate tables, NumPy for numerical processing, and TensorFlow for machine learning. These libraries allow developers to combine fast, state-of-the art algorithms from a variety of domains into powerful processing pipelines. Unfortunately, even if...
by Cody Coleman, Deepak Narayanan, Daniel Kang, Peter Bailis, and Matei Zaharia
Following the successful completion of the DAWNBench v1 competition, we are re-opening DAWNBench to allow rolling submissions. We’re eager to see the community continue to innovate and improve on optimizing for time-to-accuracy in deep learning, so starting today, we will accept new pull requests to dawn-bench-entries. The tasks, thresholds, metrics, and instructions are still the same as DAWNBench v1, but with two changes to the reviewing process: We will only review submissions that are in the top 5 results for...
by Paroma Varma, Braden Hancock, Chris Ré
Debugging training set labels is challenging since they are often generated via black-box processes. We describe our work aggregating labels from user-defined heuristics[1], [2], machine-generated heuristics[1], and natural language explanations[1] as a step towards systematic debugging. Training sets are often aggregated from multiple imperfect sources, which can lead to systematic errors in the training set. Opening the black-box of how training labels are generated can help debug training sets and improve end model predictions. We look at how our work...
by Deepak Narayanan, Daniel Kang, Cody Coleman, Tian Zhao, Jian Zhang, Luigi Nardi, Peter Bailis, Kunle Olukotun, Chris Ré, and Matei Zaharia
As the cost of training deep learning models has increased, the community has proposed a range of hardware, software, and statistical optimizations to decrease this cost. While some of these optimizations simply run the same operations faster (e.g., upgrading from a K80 to a P100), others (e.g., asynchronous SGD, reduced precision) trade off statistical performance (number of iterations needed to obtain a certain accuracy) for improved hardware performance (time needed for each iteration). To understand these trade-offs, we created DAWNBench...
