Review and Debug an AI Chatbot

Review the design of an AI chatbot and propose improvements. Then, explain how you would investigate reports of incorrect, unsafe, slow, or inconsistent responses, without assuming every failure originates from the language model itself.

Constraints & Assumptions:

• The chatbot architecture includes a language model, application prompts, retrieval mechanisms, optional tool integrations, safety controls, and a user-facing service.
• Conversation data may contain sensitive information.
• Model calls are probabilistic, potentially expensive, and dependent on external services.
• Product quality is defined by task success, safety, latency, reliability, and user trust.

Hints:

• Reconstruct a failing request across the entire pipeline.
• Build an error taxonomy before modifying prompts or models.
• Utilize a representative evaluation set and conduct controlled comparisons.