AI Inference Engineer

Note:- This is not a RAG engineer or AI Agents development role. Apply only if you have worked on optimising inference layer e.g vllm, sglang, kv cache, disaggregation, rocm, tensorRT, cuda etc.

Roles and Responsibility

Own the Inference Runtime:

Own the serving stack for LLMs and multimodal models: engine configuration, parallelism (tensor, expert, data), and deployment across NVIDIA and AMD GPUs.

Bring new open-weight models to production quickly and correctly: chat templates, tool-calling and reasoning parsers, structured outputs, quantized checkpoints.

Maintain and contribute patches to open-source inference engines (vLLM, SGLang) and upstream fixes where it makes sense.

Drive Performance and Cost Efficiency:

Profile and optimize throughput, latency (time to first token, inter-token latency) and cost per token, from kernels to the request path.

Apply and evaluate techniques such as quantization (FP8, FP4), speculative decoding, KV-cache reuse and offloading, prefix caching, and prefill/decode disaggregation.

Build first-principles performance models of models and hardware to know what is achievable, then close the gap with measured, reproducible benchmarks.

Make Serving Reliable at Scale:

Run inference on Kubernetes across multiple clusters and clouds, with autoscaling, zero-downtime rollouts and graceful failure handling.

Build observability into the serving path: engine metrics, request tracing, and correctness checks that catch silent quality regressions before customers do.

Build the tooling for benchmarking, load testing, evaluation and model release.