DAWNBench

An End-to-End Deep Learning Benchmark and Competition

SQuAD Inference

Submission Date Model 1-example Latency (milliseconds) 10,000 batch answering cost (USD) Max F1 Score Hardware Framework

Oct 2017

BiDAF

Stanford DAWN

source

100.0000 $0.15 0.7579 60 GB / 16 CPU (Google Cloud [n1-standard-16]) TensorFlow v1.2

Oct 2017

BiDAF

Stanford DAWN

source

638.1000 N/A 0.7530 1 P100 / 512 GB / 56 CPU (DAWN Internal Cluster) TensorFlow v1.2

Oct 2017

BiDAF

Stanford DAWN

source

590.0000 $1.58 0.7524 1 K80 / 30 GB / 8 CPU (Google Cloud) TensorFlow v1.2

Oct 2017

BiDAF

Stanford DAWN

source

705.9000 $1.76 0.7533 1 K80 / 61 GB / 4 CPU (Amazon EC2 [p2.xlarge]) TensorFlow v1.2