DAWNBench

An End-to-End Deep Learning Benchmark and Competition

SQuAD Inference

Submission Date Model 1-example Latency (milliseconds) 10,000 batch answering cost (USD) Max F1 Score Hardware Framework

Oct 2017

BiDAF

Stanford DAWN

source

100.0000 $0.15 0.7579 60 GB / 16 CPU (Google Cloud [n1-standard-16]) TensorFlow v1.2

Oct 2017

BiDAF

Stanford DAWN

source

638.1000 N/A 0.7530 1 P100 / 512 GB / 56 CPU (DAWN Internal Cluster) TensorFlow v1.2

Oct 2017

BiDAF

Stanford DAWN

source

590.0000 $1.58 0.7524 1 K80 / 30 GB / 8 CPU (Google Cloud) TensorFlow v1.2

Feb 2019

FastFusionNet

Wu et al. (Cornell, SayMosaic, Google)

source

7.9000 N/A 82.5209 1 NVidia GTX-1080 Ti Pytorch v0.3.1

Oct 2017

BiDAF

Stanford DAWN

source

705.9000 $1.76 0.7533 1 K80 / 61 GB / 4 CPU (Amazon EC2 [p2.xlarge]) TensorFlow v1.2
Disclosure: The Stanford DAWN research project is a five-year industrial affiliates program at Stanford University and is financially supported in part by founding members including Intel, Microsoft, NEC, Teradata, VMWare, and Google. For more information, including information regarding Stanford’s policies on openness in research and policies affecting industrial affiliates program membership, please see DAWN's membership page.