This post argues for separating CPU-side orchestration from GPU inference in LLM serving, using a model gateway architecture to manage routing, lifecycle, and compatibility across backends. It is most useful for teams operating heterogeneous deployments that need tighter control over traffic, tooling, and privacy-sensitive workflows.
Shepherd Model Gateway (SMG) is a high-performance model-routing gateway for large-scale LLM deployments. It centralizes worker lifecycle management, balances traffic across HTTP/gRPC/OpenAI-compatible backends, and provides enterprise-ready control over history storage, MCP tooling, and privacy-sensitive workflows. SMG has full OpenAI and Anthropic API compatibility across SGLang, vLLM, TRT-LLM, OpenAI, Gemini, and more. This post discusses the underlying architecture behind the gateway.