serving-llms-vllm - AI Research Skills
by Orchestra Research
Cross-platformSkillskillai-researchAI-Research-SKILLs-main
Description
Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference latency/throughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism.
Type
Skill
Ecosystem
Cross-platform
Trust Score
85%
Related Skills
Stable Diffusion WebUI
Feature-rich web interface for Stable Diffusion image generation by AUTOMATIC1111.
Cross-platformai-tool
LangChain
Comprehensive framework for building LLM-powered applications with chains, agents, and retrieval.
Cross-platformframework
LobeChat
Modern, extensible AI chat framework with plugin ecosystem and multi-model support.
Cross-platformopen-source
Open WebUI
Self-hosted web UI for LLMs with multi-model support, RAG, and plugin system.
Cross-platformopen-source