DAWNBench

An End-to-End Deep Learning Benchmark and Competition

ImageNet Inference

Submission Date Model 1-example Latency (milliseconds) 10,000 batch classification cost (USD) Max Accuracy Hardware Framework

Apr 2018

ResNet50

Intel(R) Corporation

source

9.9600 N/A 93.07% Amazon EC2 [c5.18xlarge] Intel(R) Optimized Caffe

Oct 2019

ResNet26d

Apsara AI Acceleration(AIACC) team in Alibaba Cloud

source

0.6373 $0.00 93.03% Alibaba Cloud [ecs.gn6i-c8g1.2xlarge] Pytorch+AIACC-Inference

Feb 2020

ResNet101

AI Cognitive Computing team in Alipay Group

source

0.3880 N/A 93.38% Alibaba Cloud Npu tensorflow+NpuInference

Nov 2017

ResNet 152

Stanford DAWN

source

26.8200 $0.13 93.00% 1 P100 / 30 GB / 8 CPU (Google Compute) MXNet 0.11.0

Mar 2019

ResNet50

ModelArts Service of Huawei Cloud

source

2.4500 N/A 93.40% Huawei Cloud [pi1.2xlarge.4] TensorFlow 1.8.0

Jun 2019

ResNet50

InferenceX Team of Didi Cloud

source

1.5439 $0.00 93.35% Didi Cloud [1 P4 / 16 GB / 8 vCPU] ifx

Mar 2020

ResNet26d

Apsara AI Acceleration(AIACC) team in Alibaba Cloud && Alibaba T-Head

source

0.0739 N/A 93.16% Alibaba Cloud [ecs.ebman1.26xlarge] Pytorch+AIACC-Inference+HGAI

Jul 2019

ResNet50

HaifanData

source

1.8571 N/A 93.08% 1 NVidia GTX-1080 Ti TensorFlow 1.13.1

Nov 2017

ResNet 152

Stanford DAWN

source

29.2400 $0.07 93.00% 1 K80 / 61 GB / 4 CPU (Amazon EC2 [p2.xlarge]) MXNet 0.11.0

Apr 2018

ResNet50

Intel(R) Corporation

source

17.3800 $0.02 93.07% Amazon EC2 [c5.2xlarge] Intel(R) Optimized Caffe

Jun 2019

ResNet50

InferenceX Team of Didi Cloud

source

0.9695 N/A 93.35% Didi Cloud [1 T4 / 16 GB / 8 vCPU] ifx

Nov 2017

ResNet 152

Stanford DAWN

source

49.5100 $0.12 93.35% 1 K80 / 61 GB / 4 CPU (Amazon EC2 [p2.xlarge]) TensorFlow v1.2

May 2018

ResNet50

Perseus AI Cloud Acceleration team in Alibaba Cloud

source

2.2700 $0.01 93.25% Alibaba Cloud [ecs.gn5i-c8g1.2xlarge] TensorFlow 1.12.2

Jun 2019

ResNet50

Iluvatar CoreX, P.S.R team: Gang Xu, Yu Song, Tao Yang, Fanwu Han

source

1.4328 N/A 93.73% 1 P4 / 128 GB / 32 CPU TensorRT 5.1.5

Oct 2019

ResNet26d

Iluvatar CoreX, X Lab, P.S.R team: Gang Xu, Yu Song, Tao Yang, Fanwu Han

source

0.5792 N/A 93.03% 1 T4 / 128 GB / 40 CPU TensorRT 6.0.1.4

Apr 2018

ResNet50

Intel(R) Corporation

source

12.4000 $0.02 93.07% Amazon EC2 [c5.4xlarge] Intel(R) Optimized Caffe

Dec 2018

ResNet50

Perseus AI Cloud Acceleration team in Alibaba Cloud

source

4.2180 $0.02 93.16% Alibaba Cloud [ecs.gn5i-c8g1.2xlarge] TensorFlow 1.10.0

Feb 2020

ResNet26

PAI: Platform of A.I. in Alibaba Cloud

source

0.4662 N/A 93.08% Alibaba Cloud [ecs.gn6i-c8g1.2xlarge] PAI-Blade + TensorRT

Nov 2017

ResNet 152

Stanford DAWN

source

22.2700 $0.11 93.35% 1 P100 / 30 GB / 8 CPU (Google Compute) TensorFlow v1.2

Jul 2019

ResNet50

PingAn GammaLab & PingAn Cloud team

source

0.8200 N/A 93.01% PingAn Cloud {1 T4 / 128 GB / Xeon(G) Gold 6130} Caffe 1.0

Nov 2019

ResNet26

ModelArts Service of Huawei Cloud

source

0.4945 N/A 93.01% Huawei Cloud [pi2.2xlarge.4] ModelArts-AIBOX + TensorRT

Mar 2020

MIVT-NET-v2

Machine Intelligence in Alibaba Cloud

source

0.3926 N/A 93.12% Alibaba Cloud [ecs.gn6i-c8g1.2xlarge] HIE
Disclosure: The Stanford DAWN research project is a five-year industrial affiliates program at Stanford University and is financially supported in part by founding members including Intel, Microsoft, NEC, Teradata, VMWare, and Google. For more information, including information regarding Stanford’s policies on openness in research and policies affecting industrial affiliates program membership, please see DAWN's membership page.