Client-facing OpenAPI 3.0 REST spec + generated Java code. Internal node-to-node gRPC definitions (inference.proto). All other modules depend on this one for shared contracts.

Inference Node — runs on each GPU machine, cuBLAS SGEMM via Panama FFI, pipeline forward pass, gRPC activation transfer

Model Registry and Shard Loader — GGUF weight loading, IMQ-inspired seed election, dynamic VRAM-aware sharding via Hazelcast

KV Cache Manager — three-tier GPU/CPU/Disk storage, prefix Trie cache, configurable eviction, Hazelcast metadata index

Tokenizer — DJL SentencePiece wrapper, chat template formatter, BPE support for LLaMA 2/3, Mistral, Gemma

Sampler — pure Java, zero external deps, temperature/top-k/top-p/repetition-penalty pipeline, configurable presets

Coordinator and Scheduler — continuous batching, leader election, gRPC streaming, Java 25 virtual threads