How to choose an AI model: lessons from Notion and Gamma

This post details strategies for choosing and integrating AI models, emphasizing that a single, model-agnostic harness is suboptimal. It advocates for modifying native harnesses per model to preserve their specific behaviors and context management. The post highlights the engineering investment required for model switching, enabling flexibility for reliability, cost optimization, and faster adoption of new models. It introduces A/B testing with real users as a crucial method for evaluating subjective model outputs, complementing automated evals. The core message is to select models based on specific workflow requirements and cost-effectiveness rather than leaderboard rankings. It also notes the increasing parity of open-weight models with closed ones, making them viable for diverse markets and offering greater control over pricing. Finally, it frames fine-tuning not just for margin but for market expansion, with RL being a powerful tool for task-specific optimization, contingent on explosive product-market fit.