vLLM

https://github.com/vllm-project/vllm

vLLM https://blog.vllm.ai/2023/06/20/vllm.html

Quantization https://docs.vllm.ai/en/latest/features/quantization/fp8.html#offline-quantization