Metadata-Version: 2.3
Name: anystore
Version: 0.3.9
Summary: Store and cache things anywhere
License: AGPLv3+
Author: Simon Wörpel
Author-email: simon.woerpel@pm.me
Requires-Python: >=3.11,<4
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Provides-Extra: redis
Provides-Extra: sql
Requires-Dist: banal (>=1.0.6,<2.0.0)
Requires-Dist: cloudpickle (>=3.1.0,<4.0.0)
Requires-Dist: dateparser (>=1.2.2,<2.0.0)
Requires-Dist: fakeredis (>=2.26.2,<3.0.0) ; extra == "redis"
Requires-Dist: fsspec (>2023.10,<2026)
Requires-Dist: httpx (>=0.28.1,<0.29.0)
Requires-Dist: orjson (>=3.10.18,<4.0.0)
Requires-Dist: pyaml (>=25.5.0,<26.0.0)
Requires-Dist: pydantic (>=2.11.7,<3.0.0)
Requires-Dist: pydantic-settings (>=2.7.1,<3.0.0)
Requires-Dist: redis (>=4.0.0) ; extra == "redis"
Requires-Dist: rich (>=10.0.0,<15.0.0)
Requires-Dist: rigour (>=1.0.0,<2.0.0)
Requires-Dist: s3fs (>2023.10,<2026)
Requires-Dist: sqlalchemy (>=2.0.36,<3.0.0) ; extra == "sql"
Requires-Dist: structlog (>=22.0.0,<26.0.0)
Requires-Dist: tqdm (>=4.67.1,<5.0.0)
Requires-Dist: typer (>=0.16.0,<0.17.0)
Requires-Dist: uuid7 (>=0.1.0,<0.2.0)
Project-URL: Documentation, https://docs.investigraph.dev/lib/anystore
Project-URL: Homepage, https://docs.investigraph.dev/lib/anystore
Project-URL: Issues, https://github.com/dataresearchcenter/anystore/issues
Project-URL: Repository, https://github.com/dataresearchcenter/anystore
Description-Content-Type: text/markdown

[![anystore on pypi](https://img.shields.io/pypi/v/anystore)](https://pypi.org/project/anystore/)
[![Python test and package](https://github.com/dataresearchcenter/anystore/actions/workflows/python.yml/badge.svg)](https://github.com/dataresearchcenter/anystore/actions/workflows/python.yml)
[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit)](https://github.com/pre-commit/pre-commit)
[![Coverage Status](https://coveralls.io/repos/github/dataresearchcenter/anystore/badge.svg?branch=main)](https://coveralls.io/github/dataresearchcenter/anystore?branch=main)
[![AGPLv3+ License](https://img.shields.io/pypi/l/anystore)](./LICENSE)

# anystore

Store anything anywhere. `anystore` provides a high-level storage and retrieval interface for various supported _store_ backends, such as `redis`, `sql`, `file`, `http`, cloud-storages and anything else supported by [`fsspec`](https://filesystem-spec.readthedocs.io/en/latest/index.html).

Think of it as a `key -> value` store, and `anystore` acts as a cache backend. And when _keys_ become filenames and _values_ become byte blobs, `anystore` becomes actually a file-like storage backend – but always with the same and interchangeable interface.

### Why?

[In our several data engineering projects](https://dataresearchcenter.org/projects) we always wrote boilerplate code that handles the featureset of `anystore` but not in a reusable way. This library shall be a stable foundation for data wrangling related python projects.

### Examples

#### Base cli interface:

```shell
anystore -i ./local/foo.txt -o s3://mybucket/other.txt

echo "hello" | anystore -o sftp://user:password@host:/tmp/world.txt

anystore -i https://dataresearchcenter.org > index.html

anystore --store sqlite:///db keys <prefix>

anystore --store redis://localhost put foo "bar"

anystore --store redis://localhost get foo  # -> "bar"
```
#### Use in your applications:

```python
from anystore import smart_read, smart_write

data = smart_read("s3://mybucket/data.txt")
smart_write(".local/data", data)
```

#### Simple cache example via decorator:

Use case: [`@anycache` is used for api view cache in `ftmq-api`](https://github.com/dataresearchcenter/ftmq-api/blob/main/ftmq_api/views.py)

```python
from anystore import get_store, anycache

cache = get_store("redis://localhost")

@anycache(store=cache, key_func=lambda q: f"api/list/{q.make_key()}", ttl=60)
def get_list_view(q: Query) -> Response:
    result = ... # complex computing will be cached
    return result
```

#### Mirror file collections:

```python
from anystore import get_store

source = get_store("https://example.org/documents/archive1")  # directory listing
target = get_store("s3://mybucket/files", backend_config={"client_kwargs": {
    "aws_access_key_id": "my-key",
    "aws_secret_access_key": "***",
    "endpoint_url": "https://s3.local"
}})  # can be configured via ENV as well

for path in source.iterate_keys():
    # streaming copy:
    with source.open(path) as i:
        with target.open(path, "wb") as o:
            i.write(o.read())
```

## Documentation

Find the docs at [docs.investigraph.dev/lib/anystore](https://docs.investigraph.dev/lib/anystore)

## Used by

- [ftmq](https://github.com/dataresearchcenter/ftmq), a query interface layer for [followthemoney](https://followthemoney.tech) data
- [investigraph](https://github.com/dataresearchcenter/investigraph),  a framework to manage collections of structured [followthemoney](https://followthemoney.tech) data
- [ftmq-api](https://github.com/dataresearchcenter/ftmq-api), a simple api on top off `ftmq` built with [FastApi](https://fastapi.tiangolo.com/)
- [ftm-geocode](https://github.com/dataresearchcenter/ftm-geocode), batch parse and geocode addresses from followthemoney entities
- [ftm-datalake](https://github.com/dataresearchcenter/ftm-datalake), a library to crawl, sync and move around document collections (in progress)


## Development

This package is using [poetry](https://python-poetry.org/) for packaging and dependencies management, so first [install it](https://python-poetry.org/docs/#installation).

Clone this repository to a local destination.

Within the repo directory, run

    poetry install --with dev

This installs a few development dependencies, including [pre-commit](https://pre-commit.com/) which needs to be registered:

    poetry run pre-commit install

Before creating a commit, this checks for correct code formatting (isort, black) and some other useful stuff (see: `.pre-commit-config.yaml`)

### testing

`anystore` uses [pytest](https://docs.pytest.org/en/stable/) as the testing framework.

    make test

## License and Copyright

`anystore`, (C) 2024 investigativedata.io

`anystore`, (C) 2025 investigativedata.io, Data and Researc Center – DARC

`anystore` is licensed under the AGPLv3 or later license.

Prior to version 0.3.0, `anystore` was released under the GPL-3.0 license.

see [NOTICE](./NOTICE) and [LICENSE](./LICENSE)

