Metadata-Version: 2.4
Name: cce
Version: 0.2.0
Summary: Confidence-Consistency Evaluation for Time Series Anomaly Detection
Home-page: https://github.com/EmorZz1G/CCE
Author: EmorZz1G
Author-email: EmorZz1G <csemor@mail.scut.edu.cn>
Maintainer: EmorZz1G
Maintainer-email: EmorZz1G <csemor@mail.scut.edu.cn>
License-Expression: MIT
Project-URL: Homepage, https://github.com/EmorZz1G/CCE
Project-URL: Documentation, https://github.com/EmorZz1G/CCE#readme
Project-URL: Repository, https://github.com/EmorZz1G/CCE.git
Project-URL: Issues, https://github.com/EmorZz1G/CCE/issues
Project-URL: Changelog, https://github.com/EmorZz1G/CCE/blob/main/CHANGELOG.md
Keywords: time-series,anomaly-detection,evaluation,metrics,machine-learning,confidence-consistency
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=1.8.0
Requires-Dist: numpy>=1.19.0
Requires-Dist: scipy>=1.7.0
Requires-Dist: scikit-learn>=1.0.0
Requires-Dist: pandas>=1.3.0
Requires-Dist: matplotlib>=3.3.0
Requires-Dist: seaborn>=0.11.0
Requires-Dist: tqdm>=4.60.0
Requires-Dist: pyyaml>=5.4.0
Requires-Dist: deprecated>=1.2.0
Provides-Extra: dev
Requires-Dist: pytest>=6.0; extra == "dev"
Requires-Dist: pytest-cov>=2.0; extra == "dev"
Requires-Dist: black>=21.0; extra == "dev"
Requires-Dist: flake8>=3.8; extra == "dev"
Requires-Dist: mypy>=0.800; extra == "dev"
Requires-Dist: pre-commit>=2.0; extra == "dev"
Provides-Extra: docs
Requires-Dist: sphinx>=4.0; extra == "docs"
Requires-Dist: sphinx-rtd-theme>=1.0; extra == "docs"
Requires-Dist: myst-parser>=0.15; extra == "docs"
Provides-Extra: jupyter
Requires-Dist: jupyter>=1.0.0; extra == "jupyter"
Requires-Dist: ipython>=7.0.0; extra == "jupyter"
Requires-Dist: plotly>=5.0.0; extra == "jupyter"
Requires-Dist: bokeh>=2.4.0; extra == "jupyter"
Requires-Dist: streamlit>=1.20.0; extra == "jupyter"
Provides-Extra: all
Requires-Dist: cce[dev,docs,jupyter]; extra == "all"
Dynamic: author
Dynamic: home-page
Dynamic: license-file
Dynamic: maintainer
Dynamic: requires-python

# CCE: Confidence-Consistency Evaluation for Time Series Anomaly Detection

[![Python](https://img.shields.io/badge/Python-3.8%2B-blue.svg)](https://www.python.org/downloads/)
[![License](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
[![PyPI](https://img.shields.io/badge/PyPI-CCE-red.svg)](https://pypi.org/project/cce/)

A comprehensive evaluation framework for time series anomaly detection metrics, focusing on confidence-consistency evaluation, robustness assessment, and discriminative power analysis.

## 🚀 Features

- **Multi-metric Evaluation**: Support for various anomaly detection metrics (F1, AUC-ROC, VUS-PR, etc.)
- **Performance Benchmarking**: Latency analysis and theoretical ranking validation
- **Robustness Assessment**: Noise-resistant evaluation with variance consideration
- **Discriminative Power Analysis**: Both ranking-based and value-change-ratio-based approaches
- **Automated Testing**: Streamlined evaluation pipeline for new metrics
- **Real-world Dataset Support**: Comprehensive testing on multiple datasets

## 📦 Installation

### Option 1: Install from PyPI (Recommended)

```bash
pip install cce
```

### Option 2: Install from Source

```bash
# Clone the repository
git clone https://github.com/EmorZz1G/CCE.git
cd CCE

# Install dependencies
pip install -r requirements.txt

# Install in development mode
pip install -e .
```

**Note**: Build-related files are located in the `docs` directory. For detailed build instructions, please refer to `docs/*.md`.

## 🔧 Requirements

- Python 3.8+
- PyTorch
- NumPy
- Other dependencies (see `requirements.txt`)

## ⚙️ Configuration

After installation, you may need to configure the datasets path:

```bash
# Create a configuration file
cce config create

# Set your datasets directory
cce config set-datasets-path /path/to/your/datasets

# View current configuration
cce config show
```

For detailed configuration options, see [Configuration Guide](docs/CONFIGURATION_GUIDE.md).

## 📚 Quick Start

### Basic Usage

```bash
# Run baseline evaluation
. scripts/run_baseline.sh

# Run real-world dataset evaluation
. scripts/run_real_world.sh
```

### Adding New Metrics

1. **Implement the metric function** in `src/metrics/basic_metrics.py`:
   ```python
   def metric_NewMetric(labels, scores, **kwargs):
       # Your metric implementation
       return metric_value
   ```

2. **Add evaluation logic** in `src/evaluation/eval_metrics/eval_latency_baselines.py`:
   ```python
   elif baseline == 'NewMetric':
       with timer(case_name, model_name, case_seed_new, score_seed_new, model, metric_name='NewMetric') as data_item:
           result = metricor.metric_NewMetric(labels, scores)
           data_item['val'] = result
   ```

3. **Run the evaluation**:
   ```bash
   python src/evaluation/eval_metrics/eval_latency_baselines.py --baseline NewMetric
   ```

4. **View results** in `logs/NewMetric/`

## 🏗️ Project Structure

```
CCE/
├── src/                    # Source code
│   ├── metrics/           # Metric implementations
│   ├── evaluation/        # Evaluation framework
│   ├── models/            # Model implementations
│   ├── data_utils/        # Data processing utilities
│   ├── utils/             # Helper functions
│   └── scripts/           # Execution scripts
├──                   # Build and installation files
│   ├── setup.py           # Package setup configuration
│   ├── pyproject.toml     # Modern Python package config
│   ├── MANIFEST.in        # Package file inclusion
│   ├── BUILD.md           # Detailed build instructions
│   └── INSTALL.md         # Quick install guide
├── datasets/              # Dataset storage
├── logs/                  # Evaluation results
├── tests/                 # Test files
├── docs/                  # Documentation
├── requirements.txt       # Dependencies
├── setup.py               # Simple setup entry point
└── pyproject.toml         # Basic build configuration
```

## 📊 Supported Evaluations

- **Latency Analysis**: Metric computation time measurement
- **Theoretical Ranking**: Validation against theoretical expectations
- **Robustness Assessment**: Noise resistance evaluation
- **Discriminative Power**: Ranking-based and value-change-ratio analysis

## 🔄 Updates
- **2025-08-26**: Core evaluation framework implementation
- **2025-08-26**: Multi-metric support and benchmarking

## 📋 TODO List

- [ ] Automated standard evaluation pipeline
- [ ] Enhanced robustness assessment
- [ ] Advanced discriminative power analysis
- [ ] CI/CD integration for metric testing

## 🤝 Contributing

We welcome contributions! Please feel free to submit issues and pull requests.

## 📄 License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## 🙏 Acknowledgments

- **FTSAD**: For providing the time series anomaly detection evaluation framework
- **SimAD**: For dataset load.
- **TSB-AD**: For model implementation code
- **Community**: For feedback and contributions

## 📞 Contact

For questions and support, please open an issue on GitHub or contact the maintainers.

---

**CCE** - Making time series anomaly detection evaluation more reliable and comprehensive.
