TechPrufer
Get started
NVIDIA
NVIDIA logoNVIDIA · Feature trail

Automated Model Fine-Tuning and Deployment with TAO Agent Skills

13 posts · May 2026 – Oct 2026

What this trail covers

This feature thread tracks the advancement of automated model fine-tuning and deployment, focusing on streamlining the process for NVIDIA's foundation models. Initial efforts likely involved manual fine-tuning and deployment pipelines. Subsequent developments have introduced tools and frameworks like NVIDIA TAO and agent skills to automate these processes, significantly reducing engineering overhead and time-to-deployment. This includes advancements in techniques like LoRA for efficient adaptation, and now extends to fine-tuning large language models like Nemotron for specific dialects and languages, incorporating techniques like minimal curation, weighted replay mixing, and partial encoder unfreezing to adapt models while retaining existing language capabilities. The workflow is being extended to include diarization and is being made reproducible via notebooks and agent skills.

All NVIDIA trails
  1. Fine-Tuning NVIDIA Nemotron for Saudi Arabic Dialects, with a Path to Other Languages | NVIDIA Technical BlogThis post details a specific workflow for fine-tuning NVIDIA Nemotron 3.5 ASR for Saudi Arabic dialects. It introduces a practical recipe involving minimal data curation, weighted replay mixing with FLEURS data, duration-based bucketing, and partial encoder unfreezing. The post quantifies the reduction in Word Error Rate (WER) achieved for Saudi dialects and demonstrates that English performance is maintained or slightly improved. It also discusses the trade-offs between unfreezing all encoder layers versus a partial unfreeze for different resource constraints. Furthermore, it touches upon extending the workflow to include speaker diarization and provides links to reproducible notebooks and agent skills.
  2. AI Native by Design: Lessons Learned from Building NVIDIA TensorRT Model Connect | NVIDIA Technical BlogThis post details the engineering lessons learned from building NVIDIA TensorRT Model Connect, an open-source project designed around AI agents. Key contributions include: 1. **AI-Native Design Principles:** Treating AI outputs as modular, verifiable units of work to prevent error cascades. 2. **Horizontal Scalability:** Selecting problems that can be decomposed into parallel tasks, exemplified by supporting 128 model families independently. 3. **Outcome-Based Agent Interaction:** Providing agents with high-level goals and objective references rather than prescriptive recipes, allowing them to explore implementation paths. 4. **Isolation as a Scaling Unit:** Separating components with different evolution speeds (TensorRT/CUDA foundation, TensorRT Model Connect integration layer, model-family implementations) to reduce coupling and error impact. 5. **Reversibility:** Prioritizing changes that are easy to evaluate and revert. 6. **Evidence-Centric Validation:** Emphasizing human-legible evidence, agent-native self-improving tests, reproducible CI, and adversarial collaboration between QA and developers, recognizing that as candidate code becomes cheaper, evidence becomes more expensive. The project shifts human judgment upstream to system design and accountability.
  3. Developing NVIDIA Holoscan Applications with CLI, Skills, and AI Coding Agents | NVIDIA Technical BlogThis post details the use of AI coding agents (specifically Codex with GPT-5.6) to develop a real-time endoscopic tool segmentation application using NVIDIA Holoscan and HoloHub. It describes an iterative development workflow where an engineer defines goals and constraints, and the AI agent uses the Holoscan CLI, HoloHub documentation, and development skills to implement, test, and benchmark the application. Key contributions include demonstrating an agent-guided workflow for creating a minimal working application, adding repeatable benchmarking modes (visual, smoke, benchmark) with data flow tracking, and showing how the combination of CLI, skills, and documentation leads to efficient development.
  4. Evaluating AI Agent Skill Performance with NVIDIA SkillEvaluator | NVIDIA Technical BlogThis post introduces NVIDIA SkillEvaluator, an open-source tool for measuring how skills affect AI agent performance through static checks and live task runs with and without skills. It details the three-tier evaluation methodology (Safety and structure, Distinctiveness, Live evaluation) and presents the first benchmark results for over 300 verified skills across 30+ NVIDIA products, showing significant Skill Lift in Correctness, Discoverability, Effectiveness, and Efficiency.
  5. Start Customizing NVIDIA Nemotron 3 Nano with Prime Intellect Lab in Minutes | NVIDIA Technical BlogThis post details the practical application of customizing NVIDIA Nemotron 3 Nano using Prime Intellect Lab's hosted reinforcement learning environment. It demonstrates a complete workflow: establishing a baseline accuracy on a Python math task, training a LoRA adapter using RLVR, and re-evaluating to show improvement. The post highlights the accessibility of model customization through platforms that abstract away infrastructure management and emphasizes the importance of open model resources (weights, data, recipes) for reproducibility and adaptation.
  7. Lessons From the Leaderboard: What 5,000+ Kagglers Taught Us About Improving AI Reasoning | NVIDIA Technical BlogThis post details lessons learned from a large-scale Kaggle competition focused on improving AI reasoning with NVIDIA's Nemotron models. It highlights practical engineering workflows for enhancing model reasoning, including making chain-of-thought data verifiable, designing reasoning to fit token budgets, and separating reusable knowledge from problem-solving. These insights contribute to the broader theme of automating and optimizing the fine-tuning and deployment of foundation models by providing concrete strategies for improving their practical reasoning capabilities.
  8. How to Run an Autoresearch Workflow with RL Agent Skills and NVIDIA NeMo | NVIDIA Technical BlogThis post details the practical application of agentic AI for automating reinforcement learning research workflows. It demonstrates how a coding agent (Codex with GPT 5.5) can autonomously set up the full software stack (NeMo RL, NeMo Gym), resolve dependencies, manage resources, launch experiments, monitor execution, analyze metrics, and summarize results. Key contributions include the use of specialized agent skills (Brev-etiquette, Session-memory, Autoresearch) for reproducibility and robust long-running ML workflows, enabling autonomous hypothesis generation, paper-to-code translation, and iterative model optimization. The post showcases a novel NeMo Gym visual counting environment creation and training of a Qwen3-VL-2B-Instruct model, increasing accuracy from 25% to 96.9%, and the implementation of an off-policy RL algorithm from a research paper.
  9. Post-Train NVIDIA Cosmos 3 in One Day Using Agent Skills | NVIDIA Technical BlogThis post details the use of NVIDIA TAO agent skills and LoRA for post-training the NVIDIA Cosmos 3 Nano model for video question answering. It demonstrates how agent skills automate data handling, baseline evaluation, LoRA configuration, and hyperparameter optimization via TAO AutoML, condensing a multi-day engineering effort into a single day. The post highlights the Cosmos 3 Mixture-of-Transformers (MoT) architecture and contrasts LoRA with full-parameter SFT for domain specialization, emphasizing LoRA's efficiency in terms of GPU hours. It also introduces the Cosmos 3 Reasoner NIM for streamlined production deployment of post-trained LoRA adapters.
  10. Accelerating End-to-End Co-Folding Performance with NVIDIA BioNeMo Agent Toolkit | NVIDIA Technical BlogThis post details the acceleration of end-to-end co-folding performance using the NVIDIA BioNeMo Agent Toolkit. It highlights specific optimizations for each stage of the biomolecular structure prediction pipeline: MMseqs2-GPU for MSA generation (up to 177x faster than CPU JackHMMER), cuEquivariance for co-folding inference (up to 3x faster and extending sequence length to ~5.9k tokens), and further optimizations via the OpenFold3 NIM (up to 6,400 tokens on a single B300). It also introduces Fold-CP for context-parallel inference, enabling the modeling of large assemblies (up to 32,000 tokens on 64 B300 GPUs). These components are orchestrated by the BioNeMo Agent Toolkit, demonstrating an end-to-end accelerated workflow for drug discovery and protein design.
  11. Fine-Tuning Biological Foundation Models with LoRA Using NVIDIA BioNeMo Recipes | NVIDIA Technical BlogThis post details the application of Low-Rank Adaptation (LoRA) for parameter-efficient fine-tuning of large biological foundation models (ESM2-3B for protein secondary structure prediction and Evo2-1B for DNA splice-site classification) using NVIDIA BioNeMo Recipes. It demonstrates achieving state-of-the-art performance on these tasks with minimal trainable parameters (~1% of total) and training on a single NVIDIA RTX 6000 Blackwell Workstation Edition GPU. Key technical contributions include the integration of LoRA with Transformer Engine (TE) for accelerated training, sequence packing (THD format) for improved throughput and memory efficiency, and the use of PEFT library for adapter configuration. The post provides code snippets for loading TE-compatible ESM2 models with LoRA adapters and presents performance metrics comparing the LoRA-tuned models against established baselines.
  12. Run DiffusionGemma on NVIDIA for Developer-Ready, High-Throughput Text Generation | NVIDIA Technical BlogThis post details the deployment and optimization of DiffusionGemma, a diffusion-based text generation model, on NVIDIA platforms. It highlights the model's parallel token generation capability for high throughput, achieving up to 1000 tokens/sec on an H100 GPU. The post provides practical guidance on accessing and deploying DiffusionGemma through Hugging Face Transformers, NVIDIA NIM (with containerized inference and OpenAI-compatible API), and NVIDIA NeMo AutoModel for fine-tuning. It also specifies supported hardware (H100, DGX Spark, DGX Station, RTX/RTX PRO) and precision formats (BF16, NVFP4), and includes code examples for NIM deployment and testing.
  13. Evaluate Clinical ASR Models Faster with Agent Skills and NVIDIA Nemotron Speech | NVIDIA Technical BlogThis post details a specific application of agent skills within the automated model fine-tuning and deployment thread. It introduces a workflow for generating pronunciation-aware synthetic audio for clinical ASR evaluation using NVIDIA agent skills, NeMo Data Designer, and Nemotron Speech. The contribution lies in demonstrating how agent skills can orchestrate a complex, iterative process involving profile definition, term ingestion, pronunciation QA, synthetic audio generation via SSML and phoneme markup, and ASR evaluation. This workflow addresses the challenges of clinical terminology, data privacy, and the need for repeatable feedback loops in ASR model development.
  14. Train Models Faster with JAX and MaxText Using NVFP4 on NVIDIA Blackwell | NVIDIA Technical BlogThis post introduces the NVFP4 training recipe for JAX, implemented in MaxText, which enables high-throughput, 4-bit mixed-precision pre-training on NVIDIA Blackwell and Rubin platforms. It details the five core techniques used to preserve convergence with negligible accuracy loss: micro block scaling, E4M3 block scale factors, selective Random Hadamard Transform for WGRAD inputs, 2D FP8 scaling per 16x16 weight block, and stochastic rounding for unbiased quantization. The post also provides guidance on enabling NVFP4 in MaxText and presents performance results showing up to 1.73x speedup over FP8 baselines on GB300 hardware.
  15. What’s New for Game Developers in NVIDIA RTX: DLSS 4.5 for UE5 and Multilingual AI Characters | NVIDIA Technical BlogThis post introduces multilingual models (Qwen 3.5 4B, Riva Parakeet TDT 600M ASR, Chatterbox Multilingual 500M) and the NVIDIA In-Game Inferencing (NVIGI) SDK 1.6 update, enabling multilingual, autonomous game characters. It also details the DLSS 4.5 Unreal Engine plugin with Dynamic Multi Frame Generation, a 6x mode, and the second-generation transformer for Super Resolution. Updates to the NVIDIA RTX Branch of Unreal Engine (NvRTX) 5.7.4 are also covered, focusing on stability and compatibility improvements.