Finding Errors in Perception Data With Learned Observation Assertions

Machine learning (ML) is increasingly being deployed in mission-critical settings, where errors can have disastrous consequences: autonomous vehicles have already been involved in fatal accidents. As a result, auditing ML deployments is becoming increasingly important. An emerging body of work has shown the importance of one aspect of the ML deployment pipeline: training data. Much work in ML assumes that provided labels are ground truth and measure performance against this data. For example, benchmarks on domains ranging from autonomous vehicles...

Accelerating Queries over Unstructured Data with ML, Part 5 (Semantic Indexes for Machine Learning-based Queries over Unstructured Data)

In this blog post, we’ll describe our recent work on constructing indexes for unstructured data. As a sneak preview, our index can reduce ingest costs by 10x while simultaneously improving query costs by up to 24x over prior work! While this blog post will be self-contained, please see our other blog posts (part 1, part 2, part 3, part 4 for other exciting developments and more context! As we described in part 1, unstructured data records (e.g., videos, text) are...

Accelerating Queries over Unstructured Data with ML, Part 4 (Accelerating Aggregation Queries with Expensive Predicates)

In this blog post, we’ll describe our recent work on accelerating approximate aggregation queries with expensive predicates. While this blog post will be self-contained, please see our other blog posts for other exciting developments and more context (part 1, part 2, part 3)! Analysts and researchers are increasingly interested in using powerful machine learning models and human labeling services (which we’ll refer to as “oracles”) to compute statistics over their datasets. For example, a media studies researcher may want to...

Accelerating Queries over Unstructured Data with ML, Part 3 (Preprocessing-aware Optimizations for Visual Analytics)

In this blog post, we’ll describe our recent work on benchmarking recent progress on deep neural network (DNN) execution and optimizing end-to-end DNN inference for visual DNNs. While this blog post will be self-contained, please see our other blog posts (part 1, part 2) for other exciting developments and more context! DNNs now power a range of visual analytics applications because they can produce high quality annotations over visual data. However, they can cost billions of float-point operations to execute...

Accelerating Queries over Unstructured Data with ML, Part 2 (Approximate Selection Queries with Statistical Guarantees)

In this blog post, we’ll describe our recent work on approximate selection queries with statistical guarantees. While this blog post will be self-contained, please see our other blog posts for other exciting developments and more context (part 1)! As we described in part 1, unstructured data records (e.g., videos, text) are becoming increasingly possible to automatically query with the proliferation of powerful deep neural networks (DNNs) and human-powered labeling services (which we collectively refer to as “oracle methods”). As a...

Accelerating Queries over Unstructured Data with ML, Part 1 (Accelerating Aggregation and Limit Queries with BlazeIt)

Unstructured data (e.g., videos, text) has become increasingly possible to automatically query with the proliferation of powerful deep neural networks (DNNs) and human-powered labeling services (which we collectively refer to as “oracle methods”). For example, an urban planner may query videos of street cameras to count vehicles to understand traffic patterns. As another example, a lawyer may be interested in extracting emails mentioning employee/employer information (“relation extraction”) for legal discovery. One naive method to execute such queries is to use...

How do MLPerf v0.7 entries compare on cost?

MLPerf announced its v0.7 results recently, with sub-minute submissions from companies such as Nvidia, Google, Alibaba, Intel, and others. While this is exciting, many of these entries are fundamentally hard to compare, since they use different numbers and types of accelerators. Consequently, it is still challenging for an end user to determine the right accelerator type to use for their model and with what scale factor, particularly if the user has time or budget constraints. In this blogpost, we use...

Selection via Proxy: Efficient Data Selection for Deep Learning

Given massive amounts of data available to train deep networks for many tasks, how can we quickly determine which data should actually be used in training? Data selection methods like active learning and core-set selection techniques are powerful ways to curate data for training, but these approaches can be computationally expensive and struggle to scale. In recent work at ICLR 2020, we show how to speed up data selection by up to 41.9x: we use a small, less accurate model...

Willump: A Statistically-Aware End-to-end Optimizer for Machine Learning Inference

Machine learning inference, which involves making predictions from machine learning models, is an increasingly important problem today, crucial for many systems like spam detection and content recommendation. Serving ML predictions is fundamentally different from serving other workloads, such as web pages or database queries, because, unlike those workloads, ML applications have unique statistical properties, like an amenability to approximation. However, existing systems for ML inference serving, like Clipper or AWS Sagemaker, neglect these statistical properties and approach ML inference workloads...

DAWN at MLSys 2020

We are excited to present some of our latest research at the MLSys 2020 conference in Austin next week! DAWN researchers are involved in five conference papers and several workshop papers, and on top of that, our PI Chris Ré is giving a keynote on Monday, and PI Matei Zaharia is co-organizing the MLOps workshop. Be sure to check out the the following talks on our papers at MLSys next week: Willump: A Statistically-Aware End-to-end Optimizer for Machine Learning Inference...