Metadata-Version: 2.4
Name: novaeval
Version: 0.6.3
Summary: A comprehensive, open-source LLM evaluation framework for testing and benchmarking AI models
Author-email: Noveum AI <team@noveum.ai>
Maintainer-email: Noveum AI <team@noveum.ai>
License:                                  Apache License
                                   Version 2.0, January 2004
                                http://www.apache.org/licenses/
        
           TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
        
           1. Definitions.
        
              "License" shall mean the terms and conditions for use, reproduction,
              and distribution as defined by Sections 1 through 9 of this document.
        
              "Licensor" shall mean the copyright owner or entity granting the License.
        
              "Legal Entity" shall mean the union of the acting entity and all
              other entities that control, are controlled by, or are under common
              control with that entity. For the purposes of this definition,
              "control" means (i) the power, direct or indirect, to cause the
              direction or management of such entity, whether by contract or
              otherwise, or (ii) ownership of fifty percent (50%) or more of the
              outstanding shares, or (iii) beneficial ownership of such entity.
        
              "You" (or "Your") shall mean an individual or Legal Entity
              exercising permissions granted by this License.
        
              "Source" shall mean the preferred form for making modifications,
              including but not limited to software source code, documentation
              source, and configuration files.
        
              "Object" shall mean any form resulting from mechanical
              transformation or translation of a Source form, including but
              not limited to compiled object code, generated documentation,
              and conversions to other media types.
        
              "Work" shall mean the work of authorship, whether in Source or
              Object form, made available under the License, as indicated by a
              copyright notice that is included in or attached to the work
              (which shall not include communications that are solely for the
              purpose of providing information about the License).
        
              "Derivative Works" shall mean any work, whether in Source or Object
              form, that is based upon (or derived from) the Work and for which the
              editorial revisions, annotations, elaborations, or other modifications
              represent, as a whole, an original work of authorship. For the purposes
              of this License, Derivative Works shall not include works that remain
              separable from, or merely link (or bind by name) to the interfaces of,
              the Work and derivative works thereof.
        
              "Contribution" shall mean any work of authorship, including
              the original version of the Work and any modifications or additions
              to that Work or Derivative Works thereof, that is intentionally
              submitted to Licensor for inclusion in the Work by the copyright owner
              or by an individual or Legal Entity authorized to submit on behalf of
              the copyright owner. For the purposes of this definition, "submitted"
              means any form of electronic, verbal, or written communication sent
              to the Licensor or its representatives, including but not limited to
              communication on electronic mailing lists, source code control
              systems, and issue tracking systems that are managed by, or on behalf
              of, the Licensor for the purpose of discussing and improving the Work,
              but excluding communication that is conspicuously marked or otherwise
              designated in writing by the copyright owner as "Not a Contribution."
        
              "Contributor" shall mean Licensor and any individual or Legal Entity
              on behalf of whom a Contribution has been received by Licensor and
              subsequently incorporated within the Work.
        
           2. Grant of Copyright License. Subject to the terms and conditions of
              this License, each Contributor hereby grants to You a perpetual,
              worldwide, non-exclusive, no-charge, royalty-free, irrevocable
              copyright license to use, reproduce, modify, distribute, and prepare
              Derivative Works of, publicly display, publicly perform, sublicense,
              and distribute the Work and such Derivative Works in Source or Object
              form.
        
           3. Grant of Patent License. Subject to the terms and conditions of
              this License, each Contributor hereby grants to You a perpetual,
              worldwide, non-exclusive, no-charge, royalty-free, irrevocable
              (except as stated in this section) patent license to make, have made,
              use, offer to sell, sell, import, and otherwise transfer the Work,
              where such license applies only to those patent claims licensable
              by such Contributor that are necessarily infringed by their
              Contribution(s) alone or by combination of their Contribution(s)
              with the Work to which such Contribution(s) was submitted. If You
              institute patent litigation against any entity (including a
              cross-claim or counterclaim in a lawsuit) alleging that the Work
              or a Contribution incorporated within the Work constitutes direct
              or contributory patent infringement, then any patent licenses
              granted to You under this License for that Work shall terminate
              as of the date such litigation is filed.
        
           4. Redistribution. You may reproduce and distribute copies of the
              Work or Derivative Works thereof in any medium, with or without
              modifications, and in Source or Object form, provided that You
              meet the following conditions:
        
              (a) You must give any other recipients of the Work or
                  Derivative Works a copy of this License; and
        
              (b) You must cause any modified files to carry prominent notices
                  stating that You changed the files; and
        
              (c) You must retain, in the Source form of any Derivative Works
                  that You distribute, all copyright, trademark, patent,
                  attribution and other notices from the Source form of the Work,
                  excluding those notices that do not pertain to any part of
                  the Derivative Works; and
        
              (d) If the Work includes a "NOTICE" text file as part of its
                  distribution, then any Derivative Works that You distribute must
                  include a readable copy of the attribution notices contained
                  within such NOTICE file, excluding those notices that do not
                  pertain to any part of the Derivative Works, in at least one
                  of the following places: within a NOTICE text file distributed
                  as part of the Derivative Works; within the Source form or
                  documentation, if provided along with the Derivative Works; or,
                  within a display generated by the Derivative Works, if and
                  wherever such third-party notices normally appear. The contents
                  of the NOTICE file are for informational purposes only and
                  do not modify the License. You may add Your own attribution
                  notices within Derivative Works that You distribute, alongside
                  or as an addendum to the NOTICE text from the Work, provided
                  that such additional attribution notices cannot be construed
                  as modifying the License.
        
              You may add Your own copyright notice to Your modifications and
              may provide additional or different license terms and conditions
              for use, reproduction, or distribution of Your modifications, or
              for any such Derivative Works as a whole, provided Your use,
              reproduction, and distribution of the Work otherwise complies with
              the conditions stated in this License.
        
           5. Submission of Contributions. Unless You explicitly state otherwise,
              any Contribution intentionally submitted for inclusion in the Work
              by You to the Licensor shall be under the terms and conditions of
              this License, without any additional terms or conditions.
              Notwithstanding the above, nothing herein shall supersede or modify
              the terms of any separate license agreement you may have executed
              with Licensor regarding such Contributions.
        
           6. Trademarks. This License does not grant permission to use the trade
              names, trademarks, service marks, or product names of the Licensor,
              except as required for reasonable and customary use in describing the
              origin of the Work and reproducing the content of the NOTICE file.
        
           7. Disclaimer of Warranty. Unless required by applicable law or
              agreed to in writing, Licensor provides the Work (and each
              Contributor provides its Contributions) on an "AS IS" BASIS,
              WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
              implied, including, without limitation, any warranties or conditions
              of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
              PARTICULAR PURPOSE. You are solely responsible for determining the
              appropriateness of using or redistributing the Work and assume any
              risks associated with Your exercise of permissions under this License.
        
           8. Limitation of Liability. In no event and under no legal theory,
              whether in tort (including negligence), contract, or otherwise,
              unless required by applicable law (such as deliberate and grossly
              negligent acts) or agreed to in writing, shall any Contributor be
              liable to You for damages, including any direct, indirect, special,
              incidental, or consequential damages of any character arising as a
              result of this License or out of the use or inability to use the
              Work (including but not limited to damages for loss of goodwill,
              work stoppage, computer failure or malfunction, or any and all
              other commercial damages or losses), even if such Contributor
              has been advised of the possibility of such damages.
        
           9. Accepting Warranty or Support. You may choose to offer, and to
              charge a fee for, warranty, support, indemnity or other liability
              obligations and/or rights consistent with this License. However, in
              accepting such obligations, You may act only on Your own behalf and on
              Your sole responsibility, not on behalf of any other Contributor, and
              only if You agree to indemnify, defend, and hold each Contributor
              harmless for any liability incurred by, or claims asserted against,
              such Contributor by reason of your accepting any such warranty or support.
        
           END OF TERMS AND CONDITIONS
        
           APPENDIX: How to apply the Apache License to your work.
        
              To apply the Apache License to your work, attach the following
              boilerplate notice, with the fields enclosed by brackets "[]"
              replaced with your own identifying information. (Don't include
              the brackets!)  The text should be enclosed in the appropriate
              comment syntax for the file format. We also recommend that a
              file or class name and description of purpose be included on the
              same page as the copyright notice for easier identification within
              third-party archives.
        
           Copyright 2024 Noveum
        
           Licensed under the Apache License, Version 2.0 (the "License");
           you may not use this file except in compliance with the License.
           You may obtain a copy of the License at
        
               http://www.apache.org/licenses/LICENSE-2.0
        
           Unless required by applicable law or agreed to in writing, software
           distributed under the License is distributed on an "AS IS" BASIS,
           WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
           See the License for the specific language governing permissions and
           limitations under the License.
        
Project-URL: Homepage, https://github.com/Noveum/NovaEval
Project-URL: Documentation, https://novaeval.readthedocs.io
Project-URL: Repository, https://github.com/Noveum/NovaEval
Project-URL: Bug Tracker, https://github.com/Noveum/NovaEval/issues
Project-URL: Changelog, https://github.com/Noveum/NovaEval/blob/main/CHANGELOG.md
Keywords: llm,evaluation,ai,machine-learning,benchmarking,testing,rag,agents,conversational-ai,g-eval
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic>=2.5.0
Requires-Dist: httpx>=0.25.0
Requires-Dist: tenacity>=8.2.0
Requires-Dist: rich>=13.0.0
Requires-Dist: typer>=0.9.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: jinja2>=3.1.0
Requires-Dist: numpy>=1.24.0
Requires-Dist: pandas>=2.0.0
Requires-Dist: datasets>=2.14.0
Requires-Dist: openai>=1.3.0
Requires-Dist: anthropic>=0.8.0
Requires-Dist: tiktoken>=0.5.0
Requires-Dist: sentence-transformers>=2.2.0
Requires-Dist: tqdm>=4.66.0
Requires-Dist: click>=8.1.0
Requires-Dist: requests>=2.31.0
Requires-Dist: google-genai>=0.1.0
Requires-Dist: ijson>=3.4.0
Requires-Dist: noveum-trace>=0.3.5
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: ollama>=0.5.3
Requires-Dist: transformers>=4.20.0
Requires-Dist: scikit-learn>=1.0.0
Requires-Dist: plotly>=5.0.0
Provides-Extra: dev
Requires-Dist: pytest>=7.4.0; extra == "dev"
Requires-Dist: pytest-cov>=4.1.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
Requires-Dist: pytest-mock>=3.11.0; extra == "dev"
Requires-Dist: pytest-xdist>=3.3.0; extra == "dev"
Requires-Dist: responses>=0.23.0; extra == "dev"
Requires-Dist: black>=23.9.0; extra == "dev"
Requires-Dist: isort>=5.12.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Requires-Dist: mypy>=1.6.0; extra == "dev"
Requires-Dist: typing_extensions>=4.7.0; extra == "dev"
Requires-Dist: pre-commit>=3.5.0; extra == "dev"
Requires-Dist: bandit>=1.7.5; extra == "dev"
Requires-Dist: safety>=2.3.0; extra == "dev"
Requires-Dist: coverage>=7.3.0; extra == "dev"
Requires-Dist: sphinx>=7.2.0; extra == "dev"
Requires-Dist: sphinx-rtd-theme>=1.3.0; extra == "dev"
Requires-Dist: myst-parser>=2.0.0; extra == "dev"
Requires-Dist: jupyter>=1.0.0; extra == "dev"
Requires-Dist: ipykernel>=6.25.0; extra == "dev"
Requires-Dist: commitizen>=3.12.0; extra == "dev"
Provides-Extra: docs
Requires-Dist: sphinx>=7.2.0; extra == "docs"
Requires-Dist: sphinx-rtd-theme>=1.3.0; extra == "docs"
Requires-Dist: myst-parser>=2.0.0; extra == "docs"
Requires-Dist: sphinx-autodoc-typehints>=1.24.0; extra == "docs"
Requires-Dist: sphinx-copybutton>=0.5.2; extra == "docs"
Provides-Extra: azure
Requires-Dist: azure-ai-inference>=1.0.0; python_version < "3.11" and extra == "azure"
Provides-Extra: aws
Requires-Dist: boto3>=1.34.0; extra == "aws"
Provides-Extra: gcp
Requires-Dist: google-cloud-aiplatform>=1.38.0; extra == "gcp"
Provides-Extra: api
Requires-Dist: fastapi>=0.110.0; extra == "api"
Requires-Dist: uvicorn[standard]>=0.25.0; extra == "api"
Requires-Dist: python-dotenv>=1.0.0; extra == "api"
Requires-Dist: python-multipart>=0.0.6; extra == "api"
Requires-Dist: pydantic-settings>=2.0.0; extra == "api"
Requires-Dist: prometheus_client>=0.17.0; extra == "api"
Provides-Extra: all
Requires-Dist: pytest>=7.4.0; extra == "all"
Requires-Dist: pytest-cov>=4.1.0; extra == "all"
Requires-Dist: pytest-asyncio>=0.21.0; extra == "all"
Requires-Dist: pytest-mock>=3.11.0; extra == "all"
Requires-Dist: pytest-xdist>=3.3.0; extra == "all"
Requires-Dist: responses>=0.23.0; extra == "all"
Requires-Dist: black>=23.9.0; extra == "all"
Requires-Dist: isort>=5.12.0; extra == "all"
Requires-Dist: ruff>=0.1.0; extra == "all"
Requires-Dist: mypy>=1.6.0; extra == "all"
Requires-Dist: typing_extensions>=4.7.0; extra == "all"
Requires-Dist: pre-commit>=3.5.0; extra == "all"
Requires-Dist: bandit>=1.7.5; extra == "all"
Requires-Dist: safety>=2.3.0; extra == "all"
Requires-Dist: coverage>=7.3.0; extra == "all"
Requires-Dist: sphinx>=7.2.0; extra == "all"
Requires-Dist: sphinx-rtd-theme>=1.3.0; extra == "all"
Requires-Dist: myst-parser>=2.0.0; extra == "all"
Requires-Dist: jupyter>=1.0.0; extra == "all"
Requires-Dist: ipykernel>=6.25.0; extra == "all"
Requires-Dist: commitizen>=3.12.0; extra == "all"
Requires-Dist: sphinx-autodoc-typehints>=1.24.0; extra == "all"
Requires-Dist: sphinx-copybutton>=0.5.2; extra == "all"
Requires-Dist: azure-ai-inference>=1.0.0; python_version < "3.11" and extra == "all"
Requires-Dist: boto3>=1.34.0; extra == "all"
Requires-Dist: google-cloud-aiplatform>=1.38.0; extra == "all"
Dynamic: license-file

# NovaEval by Noveum.ai

[![CI](https://github.com/Noveum/NovaEval/actions/workflows/ci.yml/badge.svg)](https://github.com/Noveum/NovaEval/actions/workflows/ci.yml)
[![Release](https://github.com/Noveum/NovaEval/actions/workflows/release.yml/badge.svg)](https://github.com/Noveum/NovaEval/actions/workflows/release.yml)
[![codecov](https://codecov.io/gh/Noveum/NovaEval/branch/main/graph/badge.svg)](https://codecov.io/gh/Noveum/NovaEval)
[![PyPI version](https://badge.fury.io/py/novaeval.svg)](https://badge.fury.io/py/novaeval)
[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)

A comprehensive, extensible AI model evaluation framework designed for production use. NovaEval provides a unified interface for evaluating language models across various datasets, metrics, and deployment scenarios.

> **We're looking for contributors!** See the [Contributing](#-contributing) section below for ways to help.

## 🤝 We Need Your Help!

NovaEval is an open-source project that thrives on community contributions. Whether you're a seasoned developer or just getting started, there are many ways to contribute:

### 🎯 High-Priority Contribution Areas

We're actively looking for contributors in these key areas:

- **🧪 Unit Tests**: Help us improve our test coverage (currently 23% overall, 90%+ for core modules)
- **📚 Examples**: Create real-world evaluation examples and use cases
- **📝 Guides & Notebooks**: Write evaluation guides and interactive Jupyter notebooks
- **📖 Documentation**: Improve API documentation and user guides
- **🔍 RAG Metrics**: Add more metrics specifically for Retrieval-Augmented Generation evaluation
- **🤖 Agent Evaluation**: Build frameworks for evaluating AI agents and multi-turn conversations

### 🚀 Getting Started as a Contributor

1. **Start Small**: Pick up issues labeled `good first issue` or `help wanted`
2. **Join Discussions**: Share your ideas in [GitHub Discussions](https://github.com/Noveum/NovaEval/discussions)
3. **Review Code**: Help review pull requests and provide feedback
4. **Report Issues**: Found a bug? Report it in [GitHub Issues](https://github.com/Noveum/NovaEval/issues)
5. **Spread the Word**: Star the repository and share with your network

## 🚀 Features

- **Multi-Model Support**: Evaluate models from OpenAI, Anthropic, AWS Bedrock, and custom providers
- **Extensible Scoring**: Built-in scorers for accuracy, semantic similarity, code evaluation, and custom metrics
- **Dataset Integration**: Support for MMLU, HuggingFace datasets, custom datasets, and more
- **Production Ready**: Docker support, Kubernetes deployment, and cloud integrations
- **Comprehensive Reporting**: Detailed evaluation reports, artifacts, and visualizations
- **Secure**: Built-in credential management and secret store integration
- **Scalable**: Designed for both local testing and large-scale production evaluations
- **Cross-Platform**: Tested on macOS, Linux, and Windows with comprehensive CI/CD

## 📦 Installation

### From PyPI (Recommended)

```bash
pip install novaeval
```

### From Source

```bash
git clone https://github.com/Noveum/NovaEval.git
cd NovaEval
pip install -e .
```

### Docker

```bash
docker pull noveum/novaeval:latest
```

## 🏃‍♂️ Quick Start

### Basic Evaluation

```python
from novaeval import Evaluator
from novaeval.datasets import MMLUDataset
from novaeval.models import OpenAIModel
from novaeval.scorers import AccuracyScorer

# Configure for cost-conscious evaluation
MAX_TOKENS = 100  # Adjust based on budget: 5-10 for answers, 100+ for reasoning

# Initialize components
dataset = MMLUDataset(
    subset="elementary_mathematics",  # Easier subset for demo
    num_samples=10,
    split="test"
)

model = OpenAIModel(
    model_name="gpt-4o-mini",  # Cost-effective model
    temperature=0.0,
    max_tokens=MAX_TOKENS
)

scorer = AccuracyScorer(extract_answer=True)

# Create and run evaluation
evaluator = Evaluator(
    dataset=dataset,
    models=[model],
    scorers=[scorer],
    output_dir="./results"
)

results = evaluator.run()

# Display detailed results
for model_name, model_results in results["model_results"].items():
    for scorer_name, score_info in model_results["scores"].items():
        if isinstance(score_info, dict):
            mean_score = score_info.get("mean", 0)
            count = score_info.get("count", 0)
            print(f"{scorer_name}: {mean_score:.4f} ({count} samples)")
```

### Configuration-Based Evaluation

```python
from novaeval import Evaluator

# Load configuration from YAML/JSON
evaluator = Evaluator.from_config("evaluation_config.yaml")
results = evaluator.run()
```

### Command Line Interface

NovaEval provides a comprehensive CLI for running evaluations:

```bash
# Run evaluation from configuration file
novaeval run config.yaml

# Quick evaluation with minimal setup
novaeval quick -d mmlu -m gpt-4 -s accuracy

# List available datasets, models, and scorers
novaeval list-datasets
novaeval list-models
novaeval list-scorers

# Generate sample configuration
novaeval generate-config sample-config.yaml
```

📖 **[Complete CLI Reference](docs/cli-reference.md)** - Detailed documentation for all CLI commands and options

### Example Configuration

```yaml
# evaluation_config.yaml
dataset:
  type: "mmlu"
  subset: "abstract_algebra"
  num_samples: 500

models:
  - type: "openai"
    model_name: "gpt-4"
    temperature: 0.0
  - type: "anthropic"
    model_name: "claude-3-opus"
    temperature: 0.0

scorers:
  - type: "accuracy"
  - type: "semantic_similarity"
    threshold: 0.8

output:
  directory: "./results"
  formats: ["json", "csv", "html"]
  upload_to_s3: true
  s3_bucket: "my-eval-results"
```

## 🌐 HTTP API

NovaEval provides a FastAPI-based HTTP API for programmatic access to evaluation capabilities. This enables easy integration with web applications, microservices, and CI/CD pipelines.

### Quick API Start

```bash
# Install API dependencies
pip install -e ".[api]"

# Run the API server
uvicorn app.main:app --host 0.0.0.0 --port 8000

# Access interactive documentation
open http://localhost:8000/docs
```

### Core API Endpoints

- **Health Check**: `GET /health` - Service health status
- **Component Discovery**: `GET /api/v1/components/` - List available models, datasets, scorers
- **Model Operations**: `POST /api/v1/models/{model}/predict` - Generate predictions
- **Dataset Operations**: `POST /api/v1/datasets/{dataset}/load` - Load and query datasets
- **Scorer Operations**: `POST /api/v1/scorers/{scorer}/score` - Score predictions
- **Evaluation Jobs**: `POST /api/v1/evaluations/submit` - Submit async evaluation jobs

### Example API Usage

```python
import requests

# Submit evaluation via API
evaluation_config = {
    "name": "api_evaluation",
    "models": [{"provider": "openai", "identifier": "gpt-3.5-turbo"}],
    "datasets": [{"name": "mmlu", "split": "test", "limit": 10}],
    "scorers": [{"name": "accuracy"}]
}

response = requests.post(
    "http://localhost:8000/api/v1/evaluations/submit",
    json=evaluation_config
)

task_id = response.json()["task_id"]
print(f"Evaluation started: {task_id}")
```

### Deployment Options

- **Docker**: `docker run -p 8000:8000 novaeval-api:latest`
- **Kubernetes**: Full manifests provided in `kubernetes/`
- **Cloud Platforms**: Supports AWS, GCP, Azure with environment variable configuration

📖 **[Complete API Documentation](app/README.md)** - Detailed API reference, examples, and deployment guide

## 🌐 Noveum Platform Integration

NovaEval includes comprehensive integration with the Noveum Platform API, providing 26 methods for traces, datasets, and scorer results to ease access to the Noveum platform. The `NoveumClient` provides a unified interface for all platform operations with complete type safety and error handling.

📖 **[Complete Platform API Documentation](src/novaeval/noveum_platform/README.md)** - Detailed API reference, examples, and usage patterns

## 🏗️ Architecture

NovaEval is built with extensibility and modularity in mind:

```
src/novaeval/
├── datasets/          # Dataset loaders and processors
├── evaluators/        # Core evaluation logic
├── integrations/      # External service integrations
├── models/           # Model interfaces and adapters
├── noveum_platform/  # Noveum Platform API client and integration
├── reporting/        # Report generation and visualization
├── scorers/          # Scoring mechanisms and metrics
└── utils/            # Utility functions and helpers
```

### Core Components

- **Datasets**: Standardized interface for loading evaluation datasets
- **Models**: Unified API for different AI model providers
- **Scorers**: Pluggable scoring mechanisms for various evaluation metrics
- **Evaluators**: Orchestrates the evaluation process
- **Reporting**: Generates comprehensive reports and artifacts
- **Integrations**: Handles external services (S3, credential stores, etc.)
- **Noveum Platform**: Complete API client for traces, datasets, and scorer results

## 📊 Supported Datasets

- **MMLU**: Massive Multitask Language Understanding
- **HuggingFace**: Any dataset from the HuggingFace Hub
- **Custom**: JSON, CSV, or programmatic dataset definitions
- **Code Evaluation**: Programming benchmarks and code generation tasks
- **Agent Traces**: Multi-turn conversation and agent evaluation

## 🤖 Supported Models

- **OpenAI**: GPT-3.5, GPT-4, and newer models
- **Anthropic**: Claude family models
- **AWS Bedrock**: Amazon's managed AI services
- **Noveum AI Gateway**: Integration with Noveum's model gateway
- **Custom**: Extensible interface for any API-based model

## 📏 Built-in Scorers & Metrics

NovaEval provides a comprehensive suite of scorers organized by evaluation domain. All scorers implement the `BaseScorer` interface and support both synchronous and asynchronous evaluation.

### 🎯 Accuracy & Classification Metrics

#### **ExactMatchScorer**
- **Purpose**: Performs exact string matching between prediction and ground truth
- **Features**:
  - Case-sensitive/insensitive matching options
  - Whitespace normalization and stripping
  - Perfect for classification tasks with exact expected outputs
- **Use Cases**: Multiple choice questions, command validation, exact answer matching
- **Configuration**: `case_sensitive`, `strip_whitespace`, `normalize_whitespace`

#### **AccuracyScorer**
- **Purpose**: Advanced classification accuracy with answer extraction capabilities
- **Features**:
  - Intelligent answer extraction from model responses using multiple regex patterns
  - Support for MMLU-style multiple choice questions (A, B, C, D)
  - Letter-to-choice text conversion
  - Robust parsing of various answer formats
- **Use Cases**: MMLU evaluations, multiple choice tests, classification benchmarks
- **Configuration**: `extract_answer`, `answer_pattern`, `choices`

#### **F1Scorer**
- **Purpose**: Token-level F1 score for partial matching scenarios
- **Features**:
  - Calculates precision, recall, and F1 score
  - Configurable tokenization (word-level or character-level)
  - Case-sensitive/insensitive options
- **Use Cases**: Question answering, text summarization, partial credit evaluation
- **Returns**: Dictionary with `precision`, `recall`, `f1`, and `score` values

### 💬 Conversational AI Metrics

#### **KnowledgeRetentionScorer**
- **Purpose**: Evaluates if the LLM retains information provided by users throughout conversations
- **Features**:
  - Sophisticated knowledge extraction from conversation history
  - Sliding window approach for relevant context (configurable window size)
  - Detects when LLM asks for previously provided information
  - Tracks knowledge items with confidence scores
- **Use Cases**: Chatbots, virtual assistants, multi-turn conversations
- **Requirements**: LLM model for knowledge extraction, conversation context

#### **ConversationRelevancyScorer**
- **Purpose**: Measures response relevance to recent conversation context
- **Features**:
  - Sliding window context analysis
  - LLM-based relevance assessment (1-5 scale)
  - Context coherence evaluation
  - Conversation flow maintenance tracking
- **Use Cases**: Dialogue systems, context-aware assistants
- **Configuration**: `window_size` for context scope

#### **ConversationCompletenessScorer**
- **Purpose**: Assesses whether user intentions and requests are fully addressed
- **Features**:
  - Extracts user intentions from conversation history
  - Evaluates fulfillment level of each intention
  - Comprehensive coverage analysis
  - Outcome-based evaluation
- **Use Cases**: Customer service bots, task-oriented dialogue systems

#### **RoleAdherenceScorer**
- **Purpose**: Evaluates consistency with assigned persona or role
- **Features**:
  - Role consistency tracking throughout conversations
  - Character maintenance assessment
  - Persona adherence evaluation
  - Customizable role expectations
- **Use Cases**: Character-based chatbots, role-playing AI, specialized assistants
- **Configuration**: `expected_role` parameter

#### **ConversationalMetricsScorer**
- **Purpose**: Comprehensive conversational evaluation combining multiple metrics
- **Features**:
  - Combines knowledge retention, relevancy, completeness, and role adherence
  - Configurable metric inclusion/exclusion
  - Weighted aggregation of individual scores
  - Detailed per-metric breakdown
- **Use Cases**: Holistic conversation quality assessment
- **Configuration**: Enable/disable individual metrics, window sizes, role expectations

### 🔍 RAG (Retrieval-Augmented Generation) Metrics

#### **AnswerRelevancyScorer**
- **Purpose**: Evaluates how relevant answers are to given questions
- **Features**:
  - Generates questions from answers using LLM
  - Semantic similarity comparison using embeddings (SentenceTransformers)
  - Multiple question generation for robust evaluation
  - Cosine similarity scoring
- **Use Cases**: RAG systems, Q&A applications, knowledge bases
- **Configuration**: `threshold`, `embedding_model`

#### **FaithfulnessScorer**
- **Purpose**: Measures if responses are faithful to provided context without hallucinations
- **Features**:
  - Extracts factual claims from responses
  - Verifies each claim against source context
  - Three-tier verification: SUPPORTED/PARTIALLY_SUPPORTED/NOT_SUPPORTED
  - Detailed claim-by-claim analysis
- **Use Cases**: RAG faithfulness, fact-checking, source attribution
- **Configuration**: `threshold` for pass/fail determination

#### **ContextualPrecisionScorer**
- **Purpose**: Evaluates precision of retrieved context relevance
- **Features**:
  - Splits context into chunks for granular analysis
  - Relevance scoring per chunk (1-5 scale)
  - Intelligent context segmentation
  - Average relevance calculation
- **Use Cases**: Retrieval system evaluation, context quality assessment
- **Requirements**: Context must be provided for evaluation

#### **ContextualRecallScorer**
- **Purpose**: Measures if all necessary information for answering is present in context
- **Features**:
  - Extracts key information from expected outputs
  - Checks presence of each key fact in provided context
  - Three-tier presence detection: PRESENT/PARTIALLY_PRESENT/NOT_PRESENT
  - Comprehensive information coverage analysis
- **Use Cases**: Retrieval completeness, context sufficiency evaluation
- **Requirements**: Both context and expected output required

#### **RAGASScorer**
- **Purpose**: Composite RAGAS methodology combining multiple RAG metrics
- **Features**:
  - Integrates Answer Relevancy, Faithfulness, Contextual Precision, and Contextual Recall
  - Configurable weighted aggregation
  - Parallel execution of individual metrics
  - Comprehensive RAG pipeline evaluation
- **Use Cases**: Complete RAG system assessment, benchmark evaluation
- **Configuration**: Custom weights for each metric component

### 🤖 LLM-as-Judge Metrics

#### **GEvalScorer**
- **Purpose**: Uses LLMs with chain-of-thought reasoning for custom evaluation criteria
- **Features**:
  - Based on G-Eval research paper methodology
  - Configurable evaluation criteria and steps
  - Chain-of-thought reasoning support
  - Multiple evaluation iterations for consistency
  - Custom score ranges and thresholds
- **Use Cases**: Custom evaluation criteria, human-aligned assessment, complex judgments
- **Configuration**: `criteria`, `use_cot`, `num_iterations`, `threshold`

#### **CommonGEvalCriteria** (Predefined Criteria)
- **Correctness**: Factual accuracy and completeness assessment
- **Relevance**: Topic adherence and query alignment evaluation
- **Coherence**: Logical flow and structural consistency analysis
- **Helpfulness**: Practical value and actionability assessment

#### **PanelOfJudgesScorer**
- **Purpose**: Multi-LLM evaluation with diverse perspectives and aggregation
- **Features**:
  - Multiple LLM judges with individual weights and specialties
  - Configurable aggregation methods (mean, median, weighted, consensus, etc.)
  - Consensus requirement and threshold controls
  - Parallel judge evaluation for efficiency
  - Detailed individual and aggregate reasoning
- **Use Cases**: High-stakes evaluation, bias reduction, robust assessment
- **Configuration**: Judge models, weights, specialties, aggregation method

#### **SpecializedPanelScorer** (Panel Configurations)
- **Diverse Panel**: Different models with varied specialties (accuracy, clarity, completeness)
- **Consensus Panel**: High-consensus requirement for agreement-based decisions
- **Weighted Expert Panel**: Domain experts with expertise-based weighting

### 🎭 Agent Evaluation Metrics

#### **Tool Relevancy Scoring**
- **Purpose**: Evaluates appropriateness of tool calls given available tools
- **Features**: Compares selected tools against available tool catalog
- **Use Cases**: Agent tool selection assessment, action planning evaluation

#### **Tool Correctness Scoring**
- **Purpose**: Compares actual tool calls against expected tool calls
- **Features**: Detailed tool call comparison and correctness assessment
- **Use Cases**: Agent behavior validation, expected action verification

#### **Parameter Correctness Scoring**
- **Purpose**: Evaluates correctness of parameters passed to tool calls
- **Features**: Parameter validation against tool call results and expectations
- **Use Cases**: Tool usage quality, parameter selection accuracy

#### **Task Progression Scoring**
- **Purpose**: Measures agent progress toward assigned tasks
- **Features**: Analyzes task completion status and advancement quality
- **Use Cases**: Agent effectiveness measurement, task completion tracking

#### **Context Relevancy Scoring**
- **Purpose**: Assesses response appropriateness given agent's role and task
- **Features**: Role-task-response alignment evaluation
- **Use Cases**: Agent behavior consistency, contextual appropriateness

#### **Role Adherence Scoring**
- **Purpose**: Evaluates consistency with assigned agent role across actions
- **Features**: Comprehensive role consistency across tool calls and responses
- **Use Cases**: Agent persona maintenance, role-based behavior validation

#### **Goal Achievement Scoring**
- **Purpose**: Measures overall goal accomplishment using complete interaction traces
- **Features**: End-to-end goal evaluation with G-Eval methodology
- **Use Cases**: Agent effectiveness assessment, outcome-based evaluation

#### **Conversation Coherence Scoring**
- **Purpose**: Evaluates logical flow and context maintenance in agent conversations
- **Features**: Conversational coherence and context tracking analysis
- **Use Cases**: Agent dialogue quality, conversation flow assessment

#### **AgentScorers** (Convenience Class)
- **Purpose**: Unified interface for all agent evaluation metrics
- **Features**: Single class providing access to all agent scorers with consistent LLM model
- **Methods**: Individual scoring methods plus `score_all()` for comprehensive evaluation

### 🔧 Advanced Features

#### **BaseScorer Interface**
All scorers inherit from `BaseScorer` providing:
- **Statistics Tracking**: Automatic score history and statistics
- **Batch Processing**: Efficient batch scoring capabilities
- **Input Validation**: Robust input validation and error handling
- **Configuration Support**: Flexible configuration from dictionaries
- **Metadata Reporting**: Detailed scoring metadata and information

#### **ScoreResult Model**
Comprehensive scoring results include:
- **Numerical Score**: Primary evaluation score
- **Pass/Fail Status**: Threshold-based binary result
- **Detailed Reasoning**: Human-readable evaluation explanation
- **Rich Metadata**: Additional context and scoring details

### 📊 Usage Examples

```python
# Basic accuracy scoring
scorer = AccuracyScorer(extract_answer=True)
score = scorer.score("The answer is B", "B")

# Advanced conversational evaluation
conv_scorer = ConversationalMetricsScorer(
    model=your_llm_model,
    include_knowledge_retention=True,
    include_relevancy=True,
    window_size=10
)
result = await conv_scorer.evaluate(input_text, output_text, context=conv_context)

# RAG system evaluation
ragas = RAGASScorer(
    model=your_llm_model,
    weights={"faithfulness": 0.4, "answer_relevancy": 0.3, "contextual_precision": 0.3}
)
result = await ragas.evaluate(question, answer, context=retrieved_context)

# Panel-based evaluation
panel = SpecializedPanelScorer.create_diverse_panel(
    models=[model1, model2, model3],
    evaluation_criteria="overall quality and helpfulness"
)
result = await panel.evaluate(input_text, output_text)

# Agent evaluation
agent_scorers = AgentScorers(model=your_llm_model)
all_scores = agent_scorers.score_all(agent_data)
```

## 🚀 Deployment

### Local Development

```bash
# Install dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Run example evaluation
python examples/basic_evaluation.py
```

### Docker

```bash
# Build image
docker build -t nova-eval .

# Run evaluation
docker run -v $(pwd)/config:/config -v $(pwd)/results:/results nova-eval --config /config/eval.yaml
```

### Kubernetes

```bash
# Deploy to Kubernetes
kubectl apply -f kubernetes/

# Check status
kubectl get pods -l app=nova-eval
```

## 🔧 Configuration

NovaEval supports configuration through:

- **YAML/JSON files**: Declarative configuration
- **Environment variables**: Runtime configuration
- **Python code**: Programmatic configuration
- **CLI arguments**: Command-line overrides

### Environment Variables

```bash
export NOVA_EVAL_OUTPUT_DIR="./results"
export NOVA_EVAL_LOG_LEVEL="INFO"
export OPENAI_API_KEY="your-api-key"
export AWS_ACCESS_KEY_ID="your-aws-key"
```

### CI/CD Integration

NovaEval includes optimized GitHub Actions workflows:
- **Unit tests** run on all PRs and pushes for quick feedback
- **Integration tests** run on main branch only to minimize API costs
- **Cross-platform testing** on macOS, Linux, and Windows

## 📈 Reporting and Artifacts

NovaEval generates comprehensive evaluation reports:

- **Summary Reports**: High-level metrics and insights
- **Detailed Results**: Per-sample predictions and scores
- **Visualizations**: Charts and graphs for result analysis
- **Artifacts**: Model outputs, intermediate results, and debug information
- **Export Formats**: JSON, CSV, HTML, PDF

### Example Report Structure

```
results/
├── summary.json              # High-level metrics
├── detailed_results.csv      # Per-sample results
├── artifacts/
│   ├── model_outputs/        # Raw model responses
│   ├── intermediate/         # Processing artifacts
│   └── debug/               # Debug information
├── visualizations/
│   ├── accuracy_by_category.png
│   ├── score_distribution.png
│   └── confusion_matrix.png
└── report.html              # Interactive HTML report
```

## 🔌 Extending NovaEval

### Custom Datasets

```python
from novaeval.datasets import BaseDataset

class MyCustomDataset(BaseDataset):
    def load_data(self):
        # Implement data loading logic
        return samples

    def get_sample(self, index):
        # Return individual sample
        return sample
```

### Custom Scorers

```python
from novaeval.scorers import BaseScorer

class MyCustomScorer(BaseScorer):
    def score(self, prediction, ground_truth, context=None):
        # Implement scoring logic
        return score
```

### Custom Models

```python
from novaeval.models import BaseModel

class MyCustomModel(BaseModel):
    def generate(self, prompt, **kwargs):
        # Implement model inference
        return response
```

## 🤝 Contributing

We welcome contributions! NovaEval is actively seeking contributors to help build a robust AI evaluation framework. Please see our [Contributing Guide](CONTRIBUTING.md) for detailed guidelines.

### 🎯 Priority Contribution Areas

As mentioned in the [We Need Your Help](#-we-need-your-help) section, we're particularly looking for help with:

1. **Unit Tests** - Expand test coverage beyond the current 23%
2. **Examples** - Real-world evaluation scenarios and use cases
3. **Guides & Notebooks** - Interactive evaluation tutorials
4. **Documentation** - API docs, user guides, and tutorials
5. **RAG Metrics** - Specialized metrics for retrieval-augmented generation
6. **Agent Evaluation** - Frameworks for multi-turn and agent-based evaluations

### Development Setup

```bash
# Clone repository
git clone https://github.com/Noveum/NovaEval.git
cd NovaEval

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install development dependencies
pip install -e ".[dev]"

# Install pre-commit hooks
pre-commit install

# Run tests
pytest

# Run with coverage
pytest --cov=src/novaeval --cov-report=html
```

### 🏗️ Contribution Workflow

1. **Fork** the repository
2. **Create** a feature branch (`git checkout -b feature/amazing-feature`)
3. **Make** your changes following our coding standards
4. **Add** tests for your changes
5. **Commit** your changes (`git commit -m 'Add amazing feature'`)
6. **Push** to the branch (`git push origin feature/amazing-feature`)
7. **Open** a Pull Request

### 📋 Contribution Guidelines

- **Code Quality**: Follow PEP 8 and use the provided pre-commit hooks
- **Testing**: Add unit tests for new features and bug fixes
- **Documentation**: Update documentation for API changes
- **Commit Messages**: Use conventional commit format
- **Issues**: Reference relevant issues in your PR description

### 🎉 Recognition

Contributors will be:
- Listed in our contributors page
- Mentioned in release notes for significant contributions
- Invited to join our contributor Discord community

## 📄 License

This project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.

## 🙏 Acknowledgments

- Inspired by evaluation frameworks like DeepEval, Confident AI, and Braintrust
- Built with modern Python best practices and industry standards
- Designed for the AI evaluation community

## 📞 Support

- **Documentation**: [https://noveum.github.io/NovaEval](https://noveum.github.io/NovaEval)
- **Issues**: [GitHub Issues](https://github.com/Noveum/NovaEval/issues)
- **Discussions**: [GitHub Discussions](https://github.com/Noveum/NovaEval/discussions)
- **Email**: support@noveum.ai

---

Made with ❤️ by the Noveum.ai team
