add fast inference tutorial

[yiheng-wang-nv]

Part of #1865 .

Description

A few sentences describing the changes proposed in this pull request.

Checks

• Avoid including large-size files in the PR.
• Clean up long text outputs from code cells in the notebook.
• For security purposes, please check the contents and remove any sensitive info such as user names and private key.
• Ensure (1) hyperlinks and markdown anchors are working (2) use relative paths for tutorial repo files (3) put figure and graphs in the ./figure folder
• Notebook runs automatically ./runner.sh -t <path to .ipynb file>

[review-notebook-app]

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[ericspod]

This addresses #1865 I assume.

[yiheng-wang-nv]

HI @ericspod@Nic-Ma@KumoLiu , the tutorial is almost ready, could you help to review the layout first?
I will add analyze and visualization results later

[yiheng-wang-nv]

HI @ericspod@Nic-Ma@KumoLiu , the tutorial is almost ready, could you help to review the layout first? I will add analyze and visualization results later

tutorial is ready.

[KumoLiu]

Do you think we can also include the .nii.gz benchmark result in the notebook since the original data is nii.gz format.

Reply via ReviewNB

[yiheng-wang-nv]

Hi @KumoLiu , thanks for the suggestion. .nii.gz files have to be decompressed in CPU, thus using GDS may not have acceleration. I added a section to introduces the limitations on each feature, could you help to review the updates? Thanks!

[Nic-Ma]

Thanks for adding the detailed tutorial, it overall looks good to me.
Do you plan to add the INT8/INT4 quantization in this PR or a separate PR later?

Thanks.

[yiheng-wang-nv]

acceleration/fast_inference_tutorial/fast_inference_tutorial.ipynb

Hi @Nic-Ma , thanks for the suggestion. I think we can consider adding quantization in a separate PR. Before adding it, it may need some time to:

1. prove it's faster
2. prove there will not have too much accuracy loss.

[Nic-Ma]

Plan sounds good to me.

Thanks.

[ericspod]

You could instead put this into a cell with %%bash at the top to allow users to run command, or you could do it with Python more directly for those that don't have bash:

forbenchmark_typein ("original", "trt", "trt_gpu_transforms", "trt_gds_gpu_transforms"): !pythonrun_benchmark.py--benchmark_type{benchmark_type}

[ericspod]

You should also state here that the script contains the same code as what's in this notebook, running it will generate a csv with the results for each type, but if the user wants to run the benchmark here in this notebook then can run the following cell with the commented lines uncommented.

[ericspod]

I've looked at the tutorial and it all looks good to me, however I am wondering about what the results show. It seems to me that GDS has the most impact so the example is just IO bound, using TRT or not has little impact. This is good to demonstrate how to overcome such issues, but it seems to me that the model is so small that it's not relevant to the benchmarks you're showing. If you used a much larger model with many more parameters the actual inference time itself would be significant. Since the inference results aren't considered you could just use a randomly initialised model so you don't need to load pre-trained weights. Thoughts?

[yiheng-wang-nv]

I've looked at the tutorial and it all looks good to me, however I am wondering about what the results show. It seems to me that GDS has the most impact so the example is just IO bound, using TRT or not has little impact. This is good to demonstrate how to overcome such issues, but it seems to me that the model is so small that it's not relevant to the benchmarks you're showing. If you used a much larger model with many more parameters the actual inference time itself would be significant. Since the inference results aren't considered you could just use a randomly initialised model so you don't need to load pre-trained weights. Thoughts?

Thanks @ericspod for the suggestions, I will use a more suitable model to show these features, and then update the PR

[ericspod]

Hi @yiheng-wang-nv we'd like to get this tutorial through, do we have any progress on using a different model to demonstrate speedup better? Thanks!

[yiheng-wang-nv]

Hi @yiheng-wang-nv we'd like to get this tutorial through, do we have any progress on using a different model to demonstrate speedup better? Thanks!

Hi @ericspod , thanks for the notice. Sorry for late reply, I will do some updates later