Debugging Training Data for Software 2.0

Training data is playing an increasingly important role in defining the performance of modern machine learning systems. The goal of this blog post is to maintain a “checklist” of the types of errors that can be introduced by unaccounted phenomena in the data and their labels, and simple ways to check for these errors. We would love to hear about other errors you have encountered and how you identify and correct them! Check out our previous blog on using the...

Moment-based quantile sketches for efficient aggregation

Quantiles or their equivalents (percentiles) are commonly used in data exploration workflows. However, they can be expensive to compute on increasingly high-volume, multi-dimensional datasets. In order to reduce query response times, data systems make use of sketch data structures to accelerate quantile computations and deliver approximate results. In this post, we show how a set of statistics including \(\sum x^2, \sum x^3,...\) can be used to define a compact and efficient sketch: the moments sketch. The key property of this...

Filter Before You Parse: Faster Analytics on Raw Data with Sparser

Many big data applications often run on raw, unstructured or semi-structured data formats, such as JSON. Querying these files is often very time-consuming, especially for exploratory applications, where data scientists run queries to explore and better understand their data. Surprisingly, 80-90% of the execution time in these applications is actually spent on parsing the data, not on evaluating the actual query itself. Parsing is, in fact, the bottleneck. In this post, we introduce Sparser (code here), a recent research project...

End-to-End Optimization for Data Analytics with Weld

Weld is an open source project, with an initial prototype described in a CIDR 2017 paper. This blog describes the adaptive optimizer in Weld, which we present in our VLDB 2018 paper. Analytics applications compose a diverse mix of software libraries and functions, such as Pandas to manipulate tables, NumPy for numerical processing, and TensorFlow for machine learning. These libraries allow developers to combine fast, state-of-the art algorithms from a variety of domains into powerful processing pipelines. Unfortunately, even if...

Announcing Rolling Submissions for DAWNBench

Following the successful completion of the DAWNBench v1 competition, we are re-opening DAWNBench to allow rolling submissions. We’re eager to see the community continue to innovate and improve on optimizing for time-to-accuracy in deep learning, so starting today, we will accept new pull requests to dawn-bench-entries. The tasks, thresholds, metrics, and instructions are still the same as DAWNBench v1, but with two changes to the reviewing process: We will only review submissions that are in the top 5 results for...

Using Provenance to Debug Training Data for Software 2.0

Debugging training set labels is challenging since they are often generated via black-box processes. We describe our work aggregating labels from user-defined heuristics[1], [2], machine-generated heuristics[1], and natural language explanations[1] as a step towards systematic debugging. Training sets are often aggregated from multiple imperfect sources, which can lead to systematic errors in the training set. Opening the black-box of how training labels are generated can help debug training sets and improve end model predictions. We look at how our work...

An Analysis of DAWNBench v1, a Time-to-Accuracy Benchmark for Deep Learning

As the cost of training deep learning models has increased, the community has proposed a range of hardware, software, and statistical optimizations to decrease this cost. While some of these optimizations simply run the same operations faster (e.g., upgrading from a K80 to a P100), others (e.g., asynchronous SGD, reduced precision) trade off statistical performance (number of iterations needed to obtain a certain accuracy) for improved hardware performance (time needed for each iteration). To understand these trade-offs, we created DAWNBench...

DAWNBench v1 Deep Learning Benchmark Results

April 20th, 2018 marked the end of our first iteration of DAWNBench, the first deep learning benchmark and competition that measures end-to-end performance: the time/cost required to achieve a state-of-the-art accuracy level for common deep learning tasks, as well as the latency/cost of inference at this state-of-the-art accuracy level. Focusing on end-to-end performance provided an objective means of normalizing across differences in computation frameworks, hardware, optimization algorithms, hyperparameter settings, and other factors that affect real-world performance. Thanks to innovative submissions...

The last decade of database research and its blindingly bright future. or Database Research: A love song.

To go by Twitter and many hallway conversations, the database research community has been unsettled lately in a way that we have never seen before. Many people are unhappy with the review process, many types of useful work seem to be more difficult to pursue, and our relationship with adjacent fields such as machine learning is unclear. Turing award winner – and giant of the field – Mike Stonebraker made some (though not all) of these points in a recent...

Weld v0.2.0 Released with New Features and Improved Performance

The Weld developers are happy to announce a new version of Weld, v0.2.0. Weld is a language and runtime for fast in-memory data analytics. It enables optimizations across operators within existing libraries as well as operators across Weld-enabled libraries. We have also released new versions of two Weld-enabled Python libraries: Grizzly v0.0.5 and weldnumpy v0.0.1. Grizzly is an accelerated subset of the Pandas data frame library, and weldnumpy accelerates the NumPy numerical computing library. What’s New in Weld v0.2.0 The...