在 Grace Hopper 上安装#

  1. 安装 TensorRT-LLM（已在 Ubuntu 24.04 上测试）。

    pip3 install torch==2.6.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126

sudo apt-get -y install libopenmpi-dev && pip3 install --upgrade pip setuptools && pip3 install tensorrt_llm

    如果使用 PyTorch NGC Container 镜像，则不需要安装启用 CUDA 的 PyTorch 包的先决条件步骤。

  2. 通过在 Python 中运行以下代码来验证安装（已在 Python 3.12 上测试）

     1from tensorrt_llm import LLM, SamplingParams
 2
 3
 4def main():
 5
 6    prompts = [
 7        "Hello, my name is",
 8        "The president of the United States is",
 9        "The capital of France is",
10        "The future of AI is",
11    ]
12    sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
13
14    llm = LLM(model="TinyLlama/TinyLlama-1.1B-Chat-v1.0")
15
16    outputs = llm.generate(prompts, sampling_params)
17
18    # Print the outputs.
19    for output in outputs:
20        prompt = output.prompt
21        generated_text = output.outputs[0].text
22        print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
23
24
25# The entry point of the program need to be protected for spawning processes.
26if __name__ == '__main__':
27    main()