Examples

This section provides a series of examples demonstrating basic and common functionalities of fsspec-utils. Each example includes an executable Python code block and a clear explanation of its purpose. For more advanced features, performance optimizations, and integrations with specialized data systems, please refer to the Advanced Usage documentation.

1. Flexible Storage Configuration

fsspec-utils simplifies configuring connections to various storage systems, including local filesystems, AWS S3, Azure Storage, and Google Cloud Storage, using StorageOptions classes. These options can then be converted into fsspec filesystems.

Local Storage Example

This example demonstrates how to initialize LocalStorageOptions and use it to interact with the local filesystem.

Step-by-step walkthrough:

  1. Create a temporary directory for our test
  2. Create a test file and write content to it
  3. List files in the directory to verify our file was created
  4. Read the content back to verify it was written correctly
  5. Clean up the temporary directory

StorageOptions classes simplify configuration for different storage systems and provide a consistent interface for creating fsspec filesystem objects.

import os
import tempfile
import shutil
from fsspec_utils.storage_options import LocalStorageOptions

print("=== LocalStorageOptions Example ===\n")

local_options = LocalStorageOptions(auto_mkdir=True)
local_fs = local_options.to_filesystem()

temp_dir = tempfile.mkdtemp()
print(f"Working in temporary directory: {temp_dir}")

temp_file = os.path.join(temp_dir, "test_file.txt")
with local_fs.open(temp_file, "w") as f:
    f.write("Hello, LocalStorageOptions!")
print(f"Created test file: {temp_file}")

files = local_fs.ls(temp_dir)
print(f"Files in {temp_dir}: {[os.path.basename(f) for f in files]}")

with local_fs.open(temp_file, "r") as f:
    content = f.read()
print(f"File content: '{content}'")

shutil.rmtree(temp_dir)
print(f"Cleaned up temporary directory: {temp_dir}")
print("Local storage example completed.\n")

Conceptual AWS S3 Configuration

This example demonstrates the configuration pattern for AwsStorageOptions. It is expected to fail when attempting to connect to actual cloud services because it uses dummy credentials.

Note: The to_filesystem() method converts StorageOptions into fsspec-compatible objects, allowing seamless integration with any fsspec-compatible library.

from fsspec_utils.storage_options import AwsStorageOptions

print("=== Conceptual AwsStorageOptions Example (using a dummy endpoint) ===\n")

aws_options = AwsStorageOptions(
    endpoint_url="http://s3.dummy-endpoint.com",
    access_key_id="DUMMY_KEY",
    secret_access_key="DUMMY_SECRET",
    allow_http=True,
    region="us-east-1"
)

aws_fs = aws_options.to_filesystem()
print(f"Created fsspec filesystem for S3: {type(aws_fs).__name__}")
print("AWS storage example completed.\n")

Conceptual Azure Configuration

This example shows how to configure AzureStorageOptions. It is expected to fail when attempting to connect to actual cloud services because it uses dummy credentials.

from fsspec_utils.storage_options import AzureStorageOptions

print("=== Conceptual AzureStorageOptions Example (using a dummy connection string) ===\n")
azure_options = AzureStorageOptions(
    protocol="az",
    account_name="demoaccount",
    connection_string="DefaultEndpointsProtocol=https;AccountName=demoaccount;AccountKey=demokey==;EndpointSuffix=core.windows.net"
)

azure_fs = azure_options.to_filesystem()
print(f"Created fsspec filesystem for Azure: {type(azure_fs).__name__}")
print("Azure storage example completed.\n")

Conceptual GCS Configuration

This example shows how to configure GcsStorageOptions. It is expected to fail when attempting to connect to actual cloud services because it uses dummy credentials.

StorageOptions classes provide a simplified, consistent interface for configuring connections to various storage systems. They abstract away the complexity of different storage backends and provide a unified way to create fsspec filesystem objects.

The to_filesystem() method converts these options into fsspec compatible objects, enabling seamless integration with any fsspec-compatible library or tool.

Important Note: The AWS, Azure, and GCS examples use dummy credentials and are for illustrative purposes only. These examples are expected to fail when attempting to connect to actual cloud services because:

  1. The endpoint URLs are not real service endpoints
  2. The credentials are placeholder values that don’t correspond to actual accounts
  3. The connection strings and tokens are examples, not valid credentials

This approach allows you to understand the configuration pattern without needing actual cloud credentials. When using these examples in production, you would replace the dummy values with your real credentials and service endpoints.

```python from fsspec_utils.storage_options import GcsStorageOptions

print(“=== Conceptual GcsStorageOptions Example (using a dummy token path) ===”) gcs_options = GcsStorageOptions( protocol=“gs”, project=“demo-project”, token=“path/to/dummy-service-account.json” )

gcs_fs = gcs_options.to_filesystem() print(f”Created fsspec filesystem for GCS: {type(gcs_fs).__name__}“) print(”GCS storage example completed.“)