Kog builds a co-designed inference stack for real-time AI agents on standard datacenter GPUs, spanning model architecture, inference engine, and low-level GPU kernels.
On the model side, we developed Laneformer 2B, our current coding model, and Delayed Tensor Parallelism (DTP), a Transformer architecture that delays communication so it overlaps with useful computation and weight streaming.
On the systems side, the Kog Inference Engine runs this co-designed stack on standard AMD and NVIDIA datacenter GPUs.
Kog generates 3,500 tokens/s per request on 8 AMD MI300X GPUs and 2,100 tokens/s per request on 8 NVIDIA H200 GPUs, in FP16 at batch size 1, with quantization and speculative decoding disabled.
Our approach is to design the model and its execution together, so architecture, communication, memory behavior, and GPU execution can be optimized as parts of the same system.
The team has 11 people, including 10 engineers and researchers and 5 PhDs.
Test it at playground.kog.ai. Read the technical details on the Kog Labs blog.