TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.

CUDA Core Compute Libraries

Holodeck is a project to create test environments optimised for GPU projects.

Ongoing research training transformer models at scale

KAI Scheduler is an open source Kubernetes Native scheduler for AI workloads at large scale

JAX-Toolbox

NVIDIA fork of tianocore/edk2

RAPIDS Accelerator JNI For Apache Spark

A unified library of SOTA model optimization techniques like quantization, pruning, distillation, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.