Build an Ollama Eval Harness
Ollama Evaluation Harness Design
Design and implement a lightweight evaluation harness for a language model served locally through Ollama.
The harness should fulfill the following requirements:
- Dataset Ingestion: Read a dataset of evaluation cases. Each case will include an
id, aprompt, and optionally areference_answeror anexpected label. - Model Interaction: Send each
promptto an Ollama-hosted model and capture its generated output. - Metadata Collection: Record essential metadata for each interaction, such as the model name used, latency of the response, any failures encountered, and the number of retry attempts.
- Metric Computation: Calculate useful evaluation metrics. This includes exact match for deterministic tasks, overall pass rate, average latency across all prompts, and the error rate.
- Reporting: Save both individual per-example results and an aggregate summary report.
- Configuration: Expose configurable parameters for the model name, request timeout, number of retries, model temperature, and concurrency level.
- Extensibility: Be architected in a way that allows for the addition of another model backend (e.g., a different local LLM server or a remote API) with minimal modifications to the existing codebase.
Your explanation should cover the overall system design, identify major components, detail the data flow, and describe how you would ensure the harness is both reliable and extensible.
AI interview prep
Core dilemma
Balancing the need for a 'lightweight' and efficient local evaluation harness with robust error handling, comprehensive metric collection, and a highly extensible architecture for future model backends and evaluation types.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Build an Ollama Eval Harness
Ollama Evaluation Harness Design
Design and implement a lightweight evaluation harness for a language model served locally through Ollama.
The harness should fulfill the following requirements:
- Dataset Ingestion: Read a dataset of evaluation cases. Each case will include an
id, aprompt, and optionally areference_answeror anexpected label. - Model Interaction: Send each
promptto an Ollama-hosted model and capture its generated output. - Metadata Collection: Record essential metadata for each interaction, such as the model name used, latency of the response, any failures encountered, and the number of retry attempts.
- Metric Computation: Calculate useful evaluation metrics. This includes exact match for deterministic tasks, overall pass rate, average latency across all prompts, and the error rate.
- Reporting: Save both individual per-example results and an aggregate summary report.
- Configuration: Expose configurable parameters for the model name, request timeout, number of retries, model temperature, and concurrency level.
- Extensibility: Be architected in a way that allows for the addition of another model backend (e.g., a different local LLM server or a remote API) with minimal modifications to the existing codebase.
Your explanation should cover the overall system design, identify major components, detail the data flow, and describe how you would ensure the harness is both reliable and extensible.