Metadata-Version: 2.3
Name: tap-dune
Version: 0.1.0
Summary: Singer tap for extracting data from Dune Analytics API
License: Apache-2.0
Keywords: singer,tap,dune,analytics,meltano
Author: Blueprint Data
Requires-Python: >=3.8.1,<4.0
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.8
Requires-Dist: requests (>=2.31.0,<3.0.0)
Requires-Dist: singer-sdk (>=0.36.0,<0.37.0)
Project-URL: Homepage, https://github.com/blueprint-data/tap-dune
Project-URL: Repository, https://github.com/blueprint-data/tap-dune
Description-Content-Type: text/markdown

# tap-dune

This is a [Singer](https://singer.io) tap that produces JSON-formatted data following the [Singer spec](https://hub.meltano.com/spec).

This tap:
- Pulls data from the [Dune Analytics API](https://dune.com/docs/api/)
- Extracts data from specified Dune queries
- Produces [Singer](https://github.com/singer-io/getting-started/blob/master/docs/SPEC.md) formatted data following the Singer spec
- Supports incremental replication using query parameters

## Installation

```bash
pipx install poetry
git clone https://github.com/your-username/tap-dune.git
cd tap-dune
poetry install
```

## Configuration

### Accepted Config Options

A full list of supported settings and capabilities is available by running:

```bash
poetry run tap-dune --about
```

### Config File Setup

1. Copy the example config file:
   ```bash
   cp config.json.example config.json
   ```

2. Edit `config.json` with your settings:

```json
{
    "api_key": "YOUR_DUNE_API_KEY",
    "query_id": "YOUR_QUERY_ID",
    "performance": "medium",
    "query_parameters": [
        {
            "key": "date_from",
            "value": "2025-08-01",
            "replication_key": true
        }
    ],
    "schema": {
        "properties": {
            "day": {"type": "string", "format": "date"},
            "network": {"type": "string"},
            "total_mana": {"type": "number"},
            "total_usd": {"type": "number"}
        }
    }
}
```

### Configuration Fields

| Field | Required | Description |
|-------|----------|-------------|
| `api_key` | Yes | Your Dune Analytics API key |
| `query_id` | Yes | The ID of the Dune query to execute |
| `performance` | No | Query execution performance tier: 'medium' (10 credits) or 'large' (20 credits). Defaults to 'medium' |
| `query_parameters` | No | Array of parameters to pass to your Dune query |
| `schema` | Yes | JSON Schema definition of your query's output fields |

#### Query Parameters

Each query parameter object can have:
- `key`: Parameter name in your Dune query
- `value`: Parameter value
- `replication_key`: Set to `true` for the parameter that should be used for incremental replication

#### Schema Configuration

The `schema` section must define all fields that your Dune query returns. Each field should specify:
- `type`: The data type ('string', 'number', 'integer', 'boolean', 'object', 'array')
- `format` (optional): Special format for string fields (e.g., 'date', 'date-time')

### Source Authentication and Authorization

1. Visit [Dune Analytics](https://dune.com)
2. Create an account and obtain an API key
3. Add the API key to your config file

## Usage

### Basic Usage

1. Generate a catalog file:
   ```bash
   poetry run tap-dune --config config.json --discover > catalog.json
   ```

2. Run the tap:
   ```bash
   poetry run tap-dune --config config.json --catalog catalog.json
   ```

### Incremental Replication

To use incremental replication:

1. Mark one of your query parameters with `"replication_key": true`
2. Ensure the parameter value is in a format that can be ordered (e.g., dates, timestamps)
3. The tap will track the last value processed and resume from there in subsequent runs

### Pipeline Usage

You can easily run `tap-dune` in a pipeline using [Meltano](https://meltano.com/) or any other Singer-compatible tool.

Example with `target-jsonl`:
```bash
poetry run tap-dune --config config.json --catalog catalog.json | target-jsonl
```

## Development

### Initialize your Development Environment

```bash
pipx install poetry
poetry install
```

### Testing

```bash
poetry run pytest
```

### SDK Dev Guide

See the [dev guide](https://sdk.meltano.com/en/latest/dev_guide.html) for more instructions on how to use the SDK to develop your own taps and targets.
