enableBlockReuse option is not available for tensorrt_llm.runtime.ModelRunner

[yupbank]

System Info

Nothing to do with hardware.

Who can help?

@kaiyux or @ncomly-nvidia

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

nothing to reproduce

Expected behavior

we want to enable common prefix caching, which seems this keyword enableBlockReuse is doing that for other runtime.

actual behavior

we are able to specify enableBlockReuse when using tensorrt_llm.runtime.ModelRunner

additional notes

no

[yupbank]

e.g. for https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/multimodal/run.py#L136

there is no way i can enable enableBlockReuse

[dcampora]

Thanks for the report @yupbank . We're on it and it will be fixed in the next release.