Distinguished Engineer, Scaled Out Inferencing

NVIDIA is leading the way in groundbreaking developments in Artificial Intelligence, High-Performance Computing and Visualization. The GPU, our invention, serves as the visual cortex of modern computers and is at the heart of our products and services. As a technology leader at NVIDIA, you will lead the development of our global strategy for scaled-out AI inferencing. You will architect the high-throughput, low-latency distributed pipelines and model serving strategies required for massive scale and production reliability. You will define and drive the technical roadmap for full-lifecycle, from deployment and versioning to automated scaling, across enterprise and cloud environments. Working with NVIDIA leadership, you will establish the systems and orchestration layers that enable the world’s most advanced AI models to run with peak performance and efficiency. This is a high-impact role where your technical decisions will influence the trajectory of AI systems globally. What You’ll Be Doing Lead the architecture and development of distributed, high-performance inferencing pipelines for large-scale AI model serving across enterprise and cloud environments. Define the technical strategy for model deployment, versioning, and scaling to support hundreds of thousands of requests per second with low latency and high reliability. Drive hardware-software co-optimization, ensuring that NVIDIA’s AI stack—including Dynamo and TensorRT-LLM—delivers peak performance at scale. Guide open-source efforts and community collaboration around Dynamo, TensorRT-LLM, and inference-related projects, shaping industry standards. Develop and maintain relationships with key stakeholders, including product managers, hardware engineers, and research teams, to align inferencing strategy with broader business goals. Stay at the forefront of inferencing technologies, evaluating emerging techniques and integrating them into NVIDIA’s offerings. What We Need to See MS or PhD in Computer Science, Electrical Engineering, or a related field (or equivalent experience). 15+ years of experience in distributed systems, cloud infrastructure, or AI/ML engineering, with a focus on large-scale model serving and inference. Deep expertise in AI inferencing frameworks, including TensorRT-LLM, vLLM, and other model serving technologies. Strong understanding of GPU architecture, CUDA programming, and performance optimization techniques. Proven track record of leading cross-functional teams and driving technical strategy at an organizational level. Experience with Kubernetes, Docker, and cloud-native technologies for deploying and scaling AI workloads. Excellent communication skills and the ability to influence technical direction across the company. Ways to Stand Out From the Crowd Contributions to open-source inference projects, such as Dynamo, TensorRT-LLM, or vLLM. Experience with large-scale AI deployments in production environments, including multi-region and multi-cloud setups. Deep knowledge of model optimization techniques, including quantization, pruning, and distillation. Published research or patents in AI inference, distributed systems, or related fields. NVIDIA is leading the way in groundbreaking developments in Artificial Intelligence, High-Performance Computing and Visualization. The GPU, our invention, serves as the visual cortex of modern computers and is at the heart of our products and services. Our work opens up new universes to explore, enables amazing creativity and discovery, and powers what were once science fiction inventions from artificial intelligence to autonomous cars. NVIDIA is looking for great people like you to help us accelerate the next wave of artificial intelligence. The base salary range for this role is 320,000 USD – 488,750 USD. Your base salary will be determined based on your location, experience, and the pay of employees in similar positions. You will also be eligible for equity and benefits. NVIDIA accepts applications on an ongoing basis. Learn more about NVIDIA benefits at www.nvidiabenefits.com . We are committed to fostering a diverse work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law. This posting is for an existing vacancy. The use of AI tools in our application and hiring processes is limited to the following specific purposes: – Drafting, summarizing, and/or reviewing candidate resumes and cover letters – Translating candidate information – Drafting summaries of candidate interview notes or assessments We do not use AI tools to make, or assist with, employment-related decisions.