SQuAD Inference

Submission Date	Model	1-example Latency (milliseconds)	10,000 batch answering cost (USD)	Max F1 Score	Hardware	Framework
Jul 2019	PA-Occam-Bert Ping An Technology Occam Platform source	7.5790	N/A	0.7589	1 NVidia Tesla V100	Tensorflow 1.13.0
Feb 2019	FastFusionNet Wu et al. (Cornell, SayMosaic, Google) source	7.9000	N/A	82.5209	1 NVidia GTX-1080 Ti	Pytorch v0.3.1
Oct 2017	BiDAF Stanford DAWN source	100.0000	$0.15	0.7579	60 GB / 16 CPU (Google Cloud [n1-standard-16])	TensorFlow v1.2
Oct 2017	BiDAF Stanford DAWN source	590.0000	$1.58	0.7524	1 K80 / 30 GB / 8 CPU (Google Cloud)	TensorFlow v1.2
Oct 2017	BiDAF Stanford DAWN source	638.1000	N/A	0.7530	1 P100 / 512 GB / 56 CPU (DAWN Internal Cluster)	TensorFlow v1.2
Oct 2017	BiDAF Stanford DAWN source	705.9000	$1.76	0.7533	1 K80 / 61 GB / 4 CPU (Amazon EC2 [p2.xlarge])	TensorFlow v1.2