神经网络推理_分析神经网络推理性能的新工具

神经网络推理

Measuring the inference time of a trained deep neural model on different hardware devices is a critical task when making deployment decisions. Should you deploy your inference on 8 Nvidia V100s, on 12 P100s, or perhaps you can use 64 CPU cores?

在制定部署决策时，测量经过训练的深度神经模型在不同硬件设备上的推理时间是一项关键任务。 您应该在8个Nvidia V100、12个P100上部署推理，还是可以使用64个CPU内核？

When it comes to inference timing, apple-to-apple comparisons among devices do not require rocket-science. Nevertheless, the process is a true time consuming burden that is prone to errors and requires expertise to perform correctly.

当涉及到推理时间时，设备之间的苹果对苹果比较不需要火箭科学。 然而，该过程是一个真正耗时的负担，容易出错，并且需要专业知识才能正确执行。

Fortunately, DeciAI released a free service that does it for you. The Deci Inference Performance Simulator (DIPS) can help practitioners analyze their inference performance. DIPS can measure model throughput, latency, cloud cost, model memory usage, and other important performance metrics. It provides a full analysis on how your model will behave and perform across various production environments — at no cost.

幸运的是，DeciAI发布了一项免费服务，可以为您完成这项工作。 Deci推理性能模拟器(DIPS)可以帮助从业人员分析其推理性能。 DIPS可以测量模型吞吐量，延迟，云成本，模型内存使用情况以及其他重要的性能指标。 它免费提供了有关模型在各种生产环境中的行为和性能的完整分析。

为什么测量运行时性能很痛苦？ (Why is measuring run-time performance painful?)

In how to measure deep learning performance, we provide practical guidelines for inference evaluation that include the following steps: (1) Write a latency measurement script (2) Write a script to compute the optimal batch size for inference (3) Write a throughput measurement script (4) Launch several machines on the cloud to run the all these scripts and (5) Summarize the obtained metrics and analyze the results.

在如何衡量深度学习性能方面 ，我们提供了用于推理评估的实用指南，包括以下步骤：(1)编写延迟测量脚本(2)编写脚本以计算推理的最佳批处理大小(3)编写吞吐量度量脚本(4)在云上启动多台计算机以运行所有这些脚本，以及(5)汇总获得的指标并分析结果。

Performing these steps is not only time consuming, it is also highly error prone. For example, issues may arise when it comes to timing on the CPU, measuring the transfer of data to and from the acceleration device, me