DAWNBench

An End-to-End Deep Learning Benchmark and Competition

SQuAD Training

Submission Date Model Time to 0.75 F1 Cost (USD) Max F1 Score Hardware Framework

Dec 2018

 DrQA

Runqi Yang, Facebook ParlAI, Brett Koonce

source

0:50:21 $0.76 0.7606 1 T4 / GCP Pytorch 1.0.0

Dec 2018

 DrQA

Runqi Yang, Facebook ParlAI, Brett Koonce

source

0:56:43 $0.57 0.7620 1 P4 / GCP Pytorch 1.0.0

Apr 2018

 QANet

Google

source

0:45:56 N/A 0.7637 1 TPUv2 TensorFlow v1.8

Dec 2018

 DrQA

Runqi Yang, Facebook ParlAI, Brett Koonce

source

0:27:07 N/A 0.7829 1 NVidia 2080 RTX (dev box) Pytorch 1.0.0

Oct 2017

 BiDAF

Stanford DAWN

source

10:50:22 $5.78 0.7579 60 GB / 16 CPU (Google Cloud [n1-standard-16]) TensorFlow v1.2

Sep 2018

 DrQA

Runqi Yang, Facebook ParlAI, Brett Koonce

source

1:00:35 $3.09 0.7569 1 V100 / AWS p3.2xlarge Pytorch 0.4.1

Oct 2017

 BiDAF

Stanford DAWN

source

7:51:22 N/A 0.7530 1 P100 / 512 GB / 56 CPU (DAWN Internal Cluster) TensorFlow v1.2

Oct 2017

 BiDAF

Stanford DAWN

source

8:43:40 $8.44 0.7524 1 K80 / 30 GB / 8 CPU (Google Cloud) TensorFlow v1.2

Mar 2019

 FastFusionNet

Wu et al. (Cornell, SayMosaic, Google)

source

0:18:46 N/A 0.8236 1 NVidia GTX-1080 Ti Pytorch v0.3.1

Oct 2017

 BiDAF

Stanford DAWN

source

7:38:10 $6.87 0.7533 1 K80 / 61 GB / 4 CPU (Amazon EC2 [p2.xlarge]) TensorFlow v1.2

Sep 2018

 DrQA

Runqi Yang, Facebook ParlAI, Brett Koonce

source

1:21:55 $1.23 0.7584 1 K80 / AWS p2.xlarge Pytorch 0.4.1
Disclosure: The Stanford DAWN research project is a five-year industrial affiliates program at Stanford University and is financially supported in part by founding members including Intel, Microsoft, NEC, Teradata, VMWare, and Google. For more information, including information regarding Stanford’s policies on openness in research and policies affecting industrial affiliates program membership, please see DAWN's membership page.