Metadata-Version: 2.1
Name: hfutils
Version: 0.0.4
Summary: Useful utilities for huggingface
Home-page: https://github.com/deepghs/hfutils
Author: narugo1992
Author-email: narugo992@gmail.com
License: Apache License, Version 2.0
Keywords: Utilities of images.
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: hbutils>=0.9.0
Requires-Dist: huggingface_hub>=0.16
Requires-Dist: tqdm
Requires-Dist: requests
Requires-Dist: click>=7
Provides-Extra: 7z
Requires-Dist: py7zr; extra == "7z"
Provides-Extra: test
Requires-Dist: coverage>=5; extra == "test"
Requires-Dist: mock>=4.0.3; extra == "test"
Requires-Dist: flake8~=3.5; extra == "test"
Requires-Dist: testfixtures>=6.18.5; extra == "test"
Requires-Dist: pytest~=6.2.5; extra == "test"
Requires-Dist: pytest-cov~=3.0.0; extra == "test"
Requires-Dist: pytest-mock~=3.6.1; extra == "test"
Requires-Dist: pytest-xdist>=1.34.0; extra == "test"
Requires-Dist: pytest-rerunfailures~=10.2; extra == "test"
Requires-Dist: pytest-timeout~=2.0.2; extra == "test"
Requires-Dist: pytest-benchmark~=3.4.0; extra == "test"
Requires-Dist: easydict<2,>=1.7; extra == "test"
Requires-Dist: testtools>=2; extra == "test"
Requires-Dist: where>=1.0.2; extra == "test"
Requires-Dist: responses>=0.20.0; extra == "test"
Requires-Dist: natsort; extra == "test"
Provides-Extra: doc
Requires-Dist: Jinja2>=3.0.0; extra == "doc"
Requires-Dist: sphinx>=3.2.0; extra == "doc"
Requires-Dist: sphinx_rtd_theme>=0.4.3; extra == "doc"
Requires-Dist: enum_tools>=0.9.0; extra == "doc"
Requires-Dist: sphinx-toolbox; extra == "doc"
Requires-Dist: plantumlcli>=0.0.2; extra == "doc"
Requires-Dist: packaging; extra == "doc"
Requires-Dist: sphinx-multiversion>=0.2.4; extra == "doc"
Requires-Dist: where>=1.0.2; extra == "doc"
Requires-Dist: easydict<2,>=1.7; extra == "doc"
Requires-Dist: responses>=0.20.0; extra == "doc"
Requires-Dist: natsort; extra == "doc"
Requires-Dist: nbsphinx>=0.8.8; extra == "doc"
Requires-Dist: ipython>=7.16.3; extra == "doc"
Requires-Dist: psutil>=5.8.0; extra == "doc"
Requires-Dist: ipykernel>=6.15; extra == "doc"
Requires-Dist: py-cpuinfo>=8.0.0; extra == "doc"
Requires-Dist: click>=7.0.0; extra == "doc"
Requires-Dist: pandas; extra == "doc"
Provides-Extra: rar
Requires-Dist: rarfile; extra == "rar"
Provides-Extra: transfer
Requires-Dist: hf-transfer; extra == "transfer"
Provides-Extra: build
Requires-Dist: pyinstaller<5,>=4.7; extra == "build"

# hfutils

[![PyPI](https://img.shields.io/pypi/v/hfutils)](https://pypi.org/project/hfutils/)
![PyPI - Python Version](https://img.shields.io/pypi/pyversions/hfutils)
![Loc](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/narugo1992/b98a0fd1468c4858abf2a355bc9b4039/raw/loc.json)
![Comments](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/narugo1992/b98a0fd1468c4858abf2a355bc9b4039/raw/comments.json)

[![Code Test](https://github.com/deepghs/hfutils/workflows/Code%20Test/badge.svg)](https://github.com/deepghs/hfutils/actions?query=workflow%3A%22Code+Test%22)
[![Package Release](https://github.com/deepghs/hfutils/workflows/Package%20Release/badge.svg)](https://github.com/deepghs/hfutils/actions?query=workflow%3A%22Package+Release%22)
[![codecov](https://codecov.io/gh/deepghs/hfutils/branch/main/graph/badge.svg?token=XJVDP4EFAT)](https://codecov.io/gh/deepghs/hfutils)

![GitHub Org's stars](https://img.shields.io/github/stars/deepghs)
[![GitHub stars](https://img.shields.io/github/stars/deepghs/hfutils)](https://github.com/deepghs/hfutils/stargazers)
[![GitHub forks](https://img.shields.io/github/forks/deepghs/hfutils)](https://github.com/deepghs/hfutils/network)
![GitHub commit activity](https://img.shields.io/github/commit-activity/m/deepghs/hfutils)
[![GitHub issues](https://img.shields.io/github/issues/deepghs/hfutils)](https://github.com/deepghs/hfutils/issues)
[![GitHub pulls](https://img.shields.io/github/issues-pr/deepghs/hfutils)](https://github.com/deepghs/hfutils/pulls)
[![Contributors](https://img.shields.io/github/contributors/deepghs/hfutils)](https://github.com/deepghs/hfutils/graphs/contributors)
[![GitHub license](https://img.shields.io/github/license/deepghs/hfutils)](https://github.com/deepghs/hfutils/blob/master/LICENSE)

Useful utilities for huggingface

## Quick Start

To get started with `hfutils`, you can install it using PyPI:

```shell
pip install hfutils

```

Alternatively, you can install it from the source code:

```shell
git clone https://github.com/deepghs/hfutils.git
cd hfutils
pip install .

```

Verify the installation by checking the version:

```shell
hfutils -v

```

If Python is not available in your local environment, we recommend downloading the pre-compiled executable version from
the [releases](https://github.com/deepghs/hfutils/releases).

## Upload Data

Upload data to repositories using the following commands:

```shell
# Upload a single file to the repository
hfutils upload -r your/repository -i /your/local/file -f file/in/your/repo

# Upload files in a directory as an archive file to the repository
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
hfutils upload -r your/repository -i /your/local/directory -a archive/file/in/your/repo.zip

# Upload files in a directory as a directory tree to the repository
hfutils upload -r your/repository -i /your/local/directory -d dir/in/your/repo

```

You can achieve the same using the Python API:

```python
from hfutils.operate import upload_file_to_file, upload_directory_as_archive, upload_directory_as_directory

# Upload a single file to the repository
upload_file_to_file(
    local_file='/your/local/file',
    repo_id='your/repository',
    file_in_repo='file/in/your/repo'
)

# Upload files in a directory as an archive file to the repository
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
upload_directory_as_archive(
    local_directory='/your/local/directory',
    repo_id='your/repository',
    archive_in_repo='archive/file/in/your/repo.zip'
)

# Upload files in a directory as a directory tree to the repository
upload_directory_as_directory(
    local_directory='/your/local/directory',
    repo_id='your/repository',
    path_in_repo='dir/in/your/repo'
)
```

Explore additional options for uploading:

```shell
hfutils upload -h

```

## Download Data

Download data from repositories using the following commands:

```shell
# Download a single file from the repository
hfutils download -r your/repository -o /your/local/file -f file/in/your/repo

# Download an archive file from the repository and extract it to the given directory
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
hfutils download -r your/repository -o /your/local/directory -a archive/file/in/your/repo.zip

# Download files from the repository as a directory tree
hfutils download -r your/repository -o /your/local/directory -d dir/in/your/repo

```

Use the Python API for the same functionality:

```python
from hfutils.operate import download_file_to_file, download_archive_as_directory, download_directory_as_directory

# Download a single file from the repository
download_file_to_file(
    local_file='/your/local/file',
    repo_id='your/repository',
    file_in_repo='file/in/your/repo'
)

# Download an archive file from the repository and extract it to the given directory
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
download_archive_as_directory(
    local_directory='/your/local/directory',
    repo_id='your/repository',
    file_in_repo='archive/file/in/your/repo.zip',
)

# Download files from the repository as a directory tree
download_directory_as_directory(
    local_directory='/your/local/directory',
    repo_id='your/repository',
    dir_in_repo='dir/in/your/repo'
)
```

Explore additional options for downloading:

```shell
hfutils download -h

```

## Supported Formats

By default, we support the `zip` and `tar` formats, including `.zip`, `.tar`, `.tar.gz`, `.tar.bz2`, and `.tar.xz`.

If you require support for `rar` and `7z` files, install the extra dependencies using the following command:

```shell
pip install hfutils[rar,7z]
```

**NOTE:** Creating RAR archive files is not supported. We use the [rarfile](https://github.com/markokr/rarfile) library,
which lacks the functionality for creating RAR files.

## How to Access Private Repositories

Simply configure the `HF_TOKEN` environment variable by using your HuggingFace access token.
Note that write permissions are required if you plan to upload any content.

## How to Use Hf-Transfer for Acceleration

If you are using the PyPI CLI, you need to install `hfutils` with the following command:

```shell
pip install hfutils[transfer]

```

If you are using a precompiled executable file, the transfer module is integrated inside; simply use it.

Enable Hf-Transfer acceleration by setting the environment variable `HF_HUB_ENABLE_HF_TRANSFER` to `1`.

## How to Change the Temporary Directory

The temporary directory is utilized for storing partially downloaded files, consuming a considerable amount of disk
space.

If your disk, especially the C drive on Windows, does not have sufficient space, simply set the `TMPDIR` to designate
another directory as the temporary directory.
