by Braden Hancock, Clara McCreery, Ines Chami, Vincent S. Chen, Sen Wu, Jared Dunnmon, Paroma Varma, Max Lam, and Chris Ré
TL;DR: We use Snorkel MeTaL1 to construct a simple model (pretrained BERT + linear task heads) and incorporate a variety of supervision signals (traditional supervision, transfer learning, multi-task learning, weak supervision, and ensembling) in a Massive Multi-Task Learning (MMTL) setting, achieving a new state-of-the-art score on the GLUE Benchmark and four of its nine component tasks (CoLA, SST-2, MRPC, STS-B). Research is ongoing, with a code release of the MMTL package coming in Snorkel MeTaL v0.5 in April 2019. Designing...
by Daniel Kang, Deepti Raghavan, Peter Bailis, Matei Zaharia
Machine learning is increasingly being used in real-world domains, such as self driving cars or healthcare. However, ML models can fail in confusing or complicated ways. For example, autonomous vehicles have suffered multiple incidents where they accelerated into one type of highway lane divider. We believe it is critical to develop tools for ensure model quality and to improve models over time, especially as ML is deployed in mission-critical domains. Prior work on quality assurance for machine learning has focused...
We are excited to have DAWN Principal Investigator, Kunle Olukotun, presenting some of our latest research advancements in his keynote at the NeurIPS conference tomorrow. To accompany his keynote we are providing a reading list for some of the topics that will be covered during his talk. HALP: [Updated Manuscript (12/5/18)][Manuscript (3/9/18)][Blog] Stay tuned (updated results and code coming soon)![Initial Definition and Study of Hardware Versus Statistical Effiency] Snorkel: [Paper][Website] Multi-Task Learning (Snorkel MeTaL): [Paper][Code] Software 2.0: [Paper][Blog] Spatial: [Paper][Code]...
by Paroma Varma, Chris Ré, and other members of DAWN
“What do you spend time on while debugging machine learning pipelines?” Responses to this question at the Fall 2018 DAWN Retreat ranged from “finding the best way to use transfer learning” to “systematically sampling from raw data”. We identify three broad themes from our discussions and explore them in this post: (1) shaping training data, (2) exploiting log data, and (3) model introspection. Check out our other blogs related to debugging machine learning: using provenance to debug training sets and...
by Kexin Rong, Clara Yoon, Karianne Bergen, Hashem Elezabi, Peter Bailis, Philip Levis, Gregory Beroza
Worldwide, major earthquakes (magnitude 7+) occur approximately once a month, while magnitude 2 and smaller earthquakes can happen up to several thousand times a day. In fact, earthquake frequency is inversely proportional to magnitude, meaning most earthquakes are very small. An estimated 1% of these small-magnitude events are detected and recorded in public catalogs (Figure 1), yet these low magnitude earthquakes are used by scientists to uncover unknown seismic sources, understand earthquake mechanics and predict major seismic events. Figure 1:...
by Kai Sheng Tai, Vatsal Sharan, Peter Bailis, and Gregory Valiant
This post accompanies the paper “Sketching Linear Classifiers over Data Streams” by Kai Sheng Tai, Vatsal Sharan, Peter Bailis and Gregory Valiant, which was presented at SIGMOD 2018. Check out our code on GitHub. In online learning, we learn a predictor by continuously updating its weights according to a stream of labelled examples. For example, in spam classification, an online learning approach allows the spam classifier to dynamically adapt to newly-observed features, even those introduced by an adversary attempting to...
by Paroma Varma, Braden Hancock, Sahaana Suri, Jared Dunnmon, Chris Ré and members of Hazy Research
Training data is playing an increasingly important role in defining the performance of modern machine learning systems. The goal of this blog post is to maintain a “checklist” of the types of errors that can be introduced by unaccounted phenomena in the data and their labels, and simple ways to check for these errors. We would love to hear about other errors you have encountered and how you identify and correct them! Check out our previous blog on using the...
by Edward Gan, Jialin Ding, Kai Sheng Tai, Vatsal Sharan, and Peter Bailis
Quantiles or their equivalents (percentiles) are commonly used in data exploration workflows. However, they can be expensive to compute on increasingly high-volume, multi-dimensional datasets. In order to reduce query response times, data systems make use of sketch data structures to accelerate quantile computations and deliver approximate results. In this post, we show how a set of statistics including can be used to define a compact and efficient sketch: the moments sketch. The key property of this sketch is that it...
by Shoumik Palkar, Firas Abuzaid, Peter Bailis, and Matei Zaharia
Many big data applications often run on raw, unstructured or semi-structured data formats, such as JSON. Querying these files is often very time-consuming, especially for exploratory applications, where data scientists run queries to explore and better understand their data. Surprisingly, 80-90% of the execution time in these applications is actually spent on parsing the data, not on evaluating the actual query itself. Parsing is, in fact, the bottleneck. In this post, we introduce Sparser (code here), a recent research project...
by Shoumik Palkar, James Thomas, Deepak Narayanan, Pratiksha Thaker, Parimarjan Negi, Rahul Palamuttam, and Matei Zaharia
Weld is an open source project, with an initial prototype described in a CIDR 2017 paper. This blog describes the adaptive optimizer in Weld, which we present in our VLDB 2018 paper. Analytics applications compose a diverse mix of software libraries and functions, such as Pandas to manipulate tables, NumPy for numerical processing, and TensorFlow for machine learning. These libraries allow developers to combine fast, state-of-the art algorithms from a variety of domains into powerful processing pipelines. Unfortunately, even if...
