Metadata-Version: 2.1
Name: spetlr
Version: 4.1.23
Summary: A python ETL libRary (SPETLR) for Databricks powered by Apache SPark.
Home-page: https://github.com/spetlr-org/spetlr
Author: Spetlr.Org
Author-email: spetlr.org@gmail.com
License: UNKNOWN
Project-URL: Documentation, https://github.com/spetlr-org/spetlr
Project-URL: Bug Reports, https://github.com/spetlr-org/spetlr/issues
Project-URL: Source Code, https://github.com/spetlr-org/spetlr
Description: # spetlr
        
        A python ETL libRary (SPETLR) for Databricks powered by Apache SPark.
        
        Visit SPETLR official webpage: [https://spetlr.com/](https://spetlr.com/)
        
        # NEWS
        
        Support for LTS9.1 is ending. [See the issue for discussions.](https://github.com/spetlr-org/spetlr/issues/61)
        
        TransformerNC class will be removed permanently. [Follow the PR.](https://github.com/spetlr-org/spetlr/pull/82)
        
        # Table of Contents
        
        - [Description](#description)
        - [Important Notes](#important-notes)
        - [Installation](#installation)
        - [Development Notes](#development-notes)
        - [Testing](#testing)
        - [General Project Info](#general-project-info)
        - [Contributing](#contributing)
        - [Build Status](#build-status)
        - [Releases](#releases)
        - [Requirements](#requirements-and-dependencies)
        - [Contact](#contact)
        
        # Description
        
        SPETLR has a lot of great tools for working with ETL in Databricks. But to make it easy for you to consider why you need
        SPETLR here is a list of the core features:
        
        * ETL framework: A common ETL framework that enables reusable transformations in an object-oriented manner. Standardized
          structures facilitate cooperation in large teams.
        
        * Integration testing: A framework for creating test databases and tables before deploying to production in order to
          ensure reliable and stable data platforms. An additional layer of data abstraction allows full integration testing.
        
        * Handlers: Standard connectors with commonly used options reduce boilerplate.
        
        For more information, visit SPETLR official webpage: [https://spetlr.com/](https://spetlr.com/)
        
        # Important Notes
        
        This package can not be run or tested without access to `pyspark`.
        However, installing `pyspark` as part of our installer gave issues when
        other versions of `pyspark` were needed. Hence we took out the dependency
        from our installer.
        
        # Installation
        
        Install SPETLR from PyPI:
        [![PyPI version](https://badge.fury.io/py/spetlr.svg)](https://pypi.org/project/spetlr/)
        [![PyPI](https://img.shields.io/pypi/dm/spetlr)](https://pypi.org/project/spetlr/)
        
        ```    
        pip install spetlr
        ```
        
        # Development Notes
        
        To prepare for development, please install these additional requirements:
        
        - Java 8
        - `pip install -r test_requirements.txt`
        
        Then install the package locally
        
            python setup.py develop
        
        ## Testing
        
        ### Local tests
        
        After installing the dev-requirements, execute tests by running:
        
            pytest tests
        
        These tests are located in the `./tests/local` folder and only require a Python interpreter. Pull requests will not be
        accepted if these tests do not pass. If you add new features, please include corresponding tests.
        
        ### Cluster tests
        
        Tests in the `./tests/cluster` folder are designed to run on a Databricks cluster.
        The [Pre-integration Test](https://github.com/spetlr-org/spetlr/blob/main/.github/workflows/pre-integration.yml)
        utilizes Azure Resource deployment - and can only be run by the spetlr-org admins.
        
        To deploy the necessary Azure resources to your own Azure Tenant, run the following command:
        
        ```powershell
        .\.github\deploy\deploy.ps1 -uniqueRunId "yourUniqueId"
        ```
        Be aware that the applied name for *uniqueRunId* should only contain lower case and numbers, and its length should not
        exceed 12 characters.
        
        Afterward, execute the following commands:
        
        ```powershell 
        .\.github\submit\build.ps1
        .\.github\submit\submit_test_job.ps1
        ```
        
        # General Project Info
        
        [![Github top language](https://img.shields.io/github/languages/top/spetlr-org/spetlr)](https://github.com/spetlr-org/spetlr)
        [![Github stars](https://img.shields.io/github/stars/spetlr-org/spetlr)](https://github.com/spetlr-org/spetlr)
        [![Github forks](https://img.shields.io/github/forks/spetlr-org/spetlr)](https://github.com/spetlr-org/spetlr)
        [![Github size](https://img.shields.io/github/repo-size/spetlr-org/spetlr)](https://github.com/spetlr-org/spetlr)
        [![Issues Open](https://img.shields.io/github/issues/spetlr-org/spetlr.svg?logo=github)](https://github.com/spetlr-org/spetlr/issues)
        [![PyPI spetlr badge](https://img.shields.io/pypi/v/spetlr)](https://pypi.org/project/spetlr/)
        
        # Contributing
        
        Feel free to contribute to SPETLR. Any contributions are appreciated - not only new features, but also if you find a way
        to improve SPETLR.
        
        If you have a suggestion that can enhance SPETLR, please fork the repository and create a pull request. Alternatively,
        you can open an issue with the "enhancement" tag.
        
        1. Fork the Project
        2. Create your Feature Branch (`git checkout -b feature/NewSPETLRFeature`)
        3. Commit your Changes (`git commit -m 'Add some SEPTLRFeature'`)
        4. Push to the Branch (`git push origin feature/NewSPETLRFeature`)
        5. Open a Pull Request
        
        # Build Status
        
        [![Post-Integration](https://github.com/spetlr-org/spetlr/actions/workflows/post-integration.yml/badge.svg)](https://github.com/spetlr-org/spetlr/actions/workflows/post-integration.yml)
        
        # Releases
        
        Releases to PyPI is an Github Action which needs to be manually triggered.
        
        [![Release](https://github.com/spetlr-org/spetlr/actions/workflows/release.yml/badge.svg)](https://github.com/spetlr-org/spetlr/actions/workflows/release.yml)
        [![PyPI spetlr badge](https://img.shields.io/pypi/v/spetlr)](https://pypi.org/project/spetlr/)
        
        
        # Requirements and dependencies
        The library has three txt-files at the root of the repo. These files defines three levels of requirements:
        - `requirements_install.txt` - this file contains the required libraries to be able to install spetlr.
        - `requirements_test.txt` - libraries required to run unit- and integration tests
        - `requirements_dev.txt` - libraries required in the development process in order to contribute to the repo
        
        All libraries and their dependencies are added with a fixed version to the configuration file `setup.cfg` using the defined requirements from `requirements_install.txt`.
        
        To __upgrade__ the the dependencies in the `setup.cfg` file do the following:
        
        1. Create a new branch
        2. Run `upgrade_requirements.ps1` in your terminal
        3. Commit the changes the script has made to the cfg file. If there are no changes, everything is up to date.
        4. The PR runs all tests and ensure that the library is compliant with any updates
        
        Note that if it is desired to upgrade a dependency, but not to its newest version, it is possible to set the desired version in the `requirements_install.txt`, then this will be respected by the upgrade script.
        
        # Contact
        
        For any inquiries, please use the [SPETLR Discord Server](https://discord.gg/p9bzqGybVW).
        
Keywords: databricks,pyspark
Platform: UNKNOWN
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: Topic :: Software Development :: Build Tools
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.8
Description-Content-Type: text/markdown
