by Paroma Varma, Braden Hancock, Chris Ré
Debugging training set labels is challenging since they are often generated via black-box processes. We describe our work aggregating labels from user-defined heuristics[1], [2], machine-generated heuristics[1], and natural language explanations[1] as a step towards systematic debugging. Training sets are often aggregated from multiple imperfect sources, which can lead to systematic errors in the training set. Opening the black-box of how training labels are generated can help debug training sets and improve end model predictions. We look at how our work...
by Deepak Narayanan, Daniel Kang, Cody Coleman, Tian Zhao, Jian Zhang, Luigi Nardi, Peter Bailis, Kunle Olukotun, Chris Ré, and Matei Zaharia
As the cost of training deep learning models has increased, the community has proposed a range of hardware, software, and statistical optimizations to decrease this cost. While some of these optimizations simply run the same operations faster (e.g., upgrading from a K80 to a P100), others (e.g., asynchronous SGD, reduced precision) trade off statistical performance (number of iterations needed to obtain a certain accuracy) for improved hardware performance (time needed for each iteration). To understand these trade-offs, we created DAWNBench...
by Cody Coleman, Deepak Narayanan, Daniel Kang, Peter Bailis, and Matei Zaharia
April 20th, 2018 marked the end of our first iteration of DAWNBench, the first deep learning benchmark and competition that measures end-to-end performance: the time/cost required to achieve a state-of-the-art accuracy level for common deep learning tasks, as well as the latency/cost of inference at this state-of-the-art accuracy level. Focusing on end-to-end performance provided an objective means of normalizing across differences in computation frameworks, hardware, optimization algorithms, hyperparameter settings, and other factors that affect real-world performance. Thanks to innovative submissions...
by Michael Cafarella and Chris Ré
To go by Twitter and many hallway conversations, the database research community has been unsettled lately in a way that we have never seen before. Many people are unhappy with the review process, many types of useful work seem to be more difficult to pursue, and our relationship with adjacent fields such as machine learning is unclear. Turing award winner – and giant of the field – Mike Stonebraker made some (though not all) of these points in a recent...
by The Weld Developers
The Weld developers are happy to announce a new version of Weld, v0.2.0. Weld is a language and runtime for fast in-memory data analytics. It enables optimizations across operators within existing libraries as well as operators across Weld-enabled libraries. We have also released new versions of two Weld-enabled Python libraries: Grizzly v0.0.5 and weldnumpy v0.0.1. Grizzly is an accelerated subset of the Pandas data frame library, and weldnumpy accelerates the NumPy numerical computing library. What’s New in Weld v0.2.0 The...
by Chris De Sa, Albert Gu, Chris Ré, and Fred Sala
Check out our paper on arXiv, and our code on GitHub! Valuable knowledge is encoded in structured data such as carefully curated databases, graphs of disease interactions, and even low-level information like hierarchies of synonyms. Embedding these structured, discrete objects in a way that can be used with modern machine learning methods, including deep learning, is challenging. Fundamentally, the problem is that these objects are discrete and structured, while much of machine learning works on continuous and unstructured data. Recent...
by Chris De Sa, Megan Leszczynski, Jian Zhang, Alana Marzoev, Chris Aberger, Kunle Olukotun, and Chris Ré
Using fewer bits of precision to train machine learning models limits training accuracy—or does it? This post describes cases in which we can get high-accuracy solutions using low-precision computation via a technique called bit recentering, and our theory to explain what's going on. Low-precision computation has been gaining a lot of traction in machine learning. Companies have even started developing new hardware architectures that natively support and accelerate low-precision operations including Microsoft's Project Brainwave and Google's TPU. Even though using...
by Deepak Narayanan
The DAWN PIs recently helped start a new research conference called SysML that targets research at the intersection of Systems and Machine Learning. The first conference was very well-attended, with over 200 poster submissions and sold-out registration, demonstrating the huge interest in this new and evolving research area from both academia and industry. At SysML, many members of DAWN presented posters about our latest research; in this post, we highlight the work we presented. Accelerating Model Search with Model Batching...
by Cody Coleman, Deepak Narayanan, Daniel Kang, Peter Bailis, and Matei Zaharia
In December, we introduced DAWNBench, the first deep learning benchmark focused on end-to-end training and inference time at a state-of-the-art accuracy. Despite the successes of deep learning, achieving state-of-the-art accuracy remains surprisingly difficult with pitfalls hidden behind inconsistent evaluation, underspecified metrics, complex tuning, and conflicting implementations. This blog outlines several of the lessons we learned while building DAWNBench, which we hope will save researchers and practitioners time, and illustrate the various issues associated with using deep learning in practice. Lesson...
by Peter Bailis, Kai Sheng Tai, Pratiksha Thaker, and Matei Zaharia
There’s recently been a lot of excitement about a new proposal from authors at Google: to replace conventional indexing data structures like B-trees and hash maps by instead fitting a neural network to the dataset. The paper compares such learned indexes against several standard data structures and reports promising results. For range searches, the authors report up 3.2x speedups over B-trees while using 9x less memory, and for point lookups, the authors report up to 80% reduction of hash table...
