Metadata-Version: 2.1
Name: baytune
Version: 0.3.4
Summary: Bayesian Tuning and Bandits
Home-page: https://github.com/HDI-Project/BTB
Author: MIT Data To AI Lab
Author-email: dailabmit@gmail.com
License: MIT license
Description: <p align="left">
        <img width="15%" src="https://dai.lids.mit.edu/wp-content/uploads/2018/06/Logo_DAI_highres.png" alt="BTB" />
        <i>An open source project from Data to AI Lab at MIT.</i>
        </p>
        
        ![](https://raw.githubusercontent.com/HDI-Project/BTB/master/docs/images/BTB-Icon-small.png)
        
        A simple, extensible backend for developing auto-tuning systems.
        
        [![PyPi Shield](https://img.shields.io/pypi/v/baytune.svg)](https://pypi.python.org/pypi/baytune)
        [![Travis CI Shield](https://travis-ci.org/HDI-Project/BTB.svg?branch=master)](https://travis-ci.org/HDI-Project/BTB)
        [![Coverage Status](https://codecov.io/gh/HDI-Project/BTB/branch/master/graph/badge.svg)](https://codecov.io/gh/HDI-Project/BTB)
        [![Downloads](https://pepy.tech/badge/baytune)](https://pepy.tech/project/baytune)
        
        
        * Free software: MIT license
        * Documentation: https://HDI-Project.github.io/BTB
        * Homepage: https://github.com/HDI-Project/BTB
        
        # Overview
        
        Bayesian Tuning and Bandits is a simple, extensible backend for developing auto-tuning systems such as AutoML systems. It is currently being used in [ATM](https://github.com/HDI-Project/ATM) (an AutoML system that allows tuning of classifiers) and MIT's system for the DARPA [Data driven discovery of models program](https://www.darpa.mil/program/data-driven-discovery-of-models).
        
        **BTB** is under active development. If you come across any issues, please report them [here](https://github.com/HDI-Project/BTB/issues/new).
        
        # Install
        
        ## Requirements
        
        **BTB** has been developed and tested on [Python 3.5, 3.6 and 3.7](https://www.python.org/downloads)
        
        Also, although it is not strictly required, the usage of a
        [virtualenv](https://virtualenv.pypa.io/en/latest/) is highly recommended in order to avoid
        interfering with other software installed in the system where **BTB** is run.
        
        These are the minimum commands needed to create a virtualenv using python3.6 for **BTB**:
        
        ```bash
        pip install virtualenv
        virtualenv -p $(which python3.6) btb-venv
        ```
        
        Afterwards, you have to execute this command to have the virtualenv activated:
        
        ```bash
        source btb-venv/bin/activate
        ```
        
        Remember about executing it every time you start a new console to work on **BTB**!
        
        ## Install using Pip
        
        After creating the virtualenv and activating it, we recommend using
        [pip](https://pip.pypa.io/en/stable/) in order to install **BTB**:
        
        ```bash
        pip install baytune
        ```
        
        This will pull and install the latest stable release from [PyPi](https://pypi.org/).
        
        ## Install from Source
        
        With your virtualenv activated, you can clone the repository and install it from
        source by running `make install` on the `stable` branch:
        
        ```bash
        git clone git@github.com:HDI-Project/BTB.git
        cd BTB
        git checkout stable
        make install
        ```
        
        ## Install for Development
        
        If you want to contribute to the project, a few more steps are required to make the project ready
        for development.
        
        Please head to the [Contributing Guide](https://HDI-Project.github.io/BTB/contributing.html#get-started)
        for more details about this process.
        
        # Quickstart
        
        ## Tuners
        
        Tuners are specifically designed to speed up the process of selecting the
        optimal hyper parameter values for a specific machine learning algorithm.
        
        `btb.tuning.tuners` defines Tuners: classes with a fit/predict/propose interface for
        suggesting sets of hyperparameters.
        
        This is done by following a Bayesian Optimization approach and iteratively:
        
        * letting the tuner propose new sets of hyper parameter
        * fitting and scoring the model with the proposed hyper parameters
        * passing the score obtained back to the tuner
        
        At each iteration the tuner will use the information already obtained to propose
        the set of hyper parameters that it considers that have the highest probability
        to obtain the best results.
        
        To instantiate a ``Tuner`` all we need is a ``Tunable`` class with a collection of
        ``hyperparameters``.
        
        ``` python
        >>> from btb.tuning import Tunable
        >>> from btb.tuning.tuners import GPTuner
        >>> from btb.tuning.hyperparams import IntHyperParam
        >>> hyperparams = {
        ...     'n_estimators': IntHyperParam(min=10, max=500),
        ...     'max_depth': IntHyperParam(min=10, max=500),
        ... }
        >>> tunable = Tunable(hyperparams)
        >>> tuner = GPTuner(tunable)
        ```
        
        Then we perform the following three steps in a loop.
        
        1. Let the Tuner propose a new set of parameters:
        
            ``` python
            >>> parameters = tuner.propose()
            >>> parameters
            {'n_estimators': 297, 'max_depth': 3}
            ```
        
        2. Fit and score a new model using these parameters:
        
            ``` python
            >>> model = RandomForestClassifier(**parameters)
            >>> model.fit(X_train, y_train)
            RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',
                        max_depth=3, max_features='auto', max_leaf_nodes=None,
                        min_impurity_decrease=0.0, min_impurity_split=None,
                        min_samples_leaf=1, min_samples_split=2,
                        min_weight_fraction_leaf=0.0, n_estimators=297, n_jobs=1,
                        oob_score=False, random_state=None, verbose=0,
                        warm_start=False)
            >>> score = model.score(X_test, y_test)
            >>> score
            0.77
            ```
        
        3. Pass the used parameters and the score obtained back to the tuner:
        
            ``` python
            tuner.record(parameters, score)
            ```
        
        At each iteration, the ``Tuner`` will use the information about the previous tests
        to evaluate and propose the set of parameter values that have the highest probability
        of obtaining the highest score.
        
        ### Selectors
        
        The selectors are intended to be used in combination with tuners in order to find
        out and decide which model seems to get the best results once it is properly fine tuned.
        
        In order to use the selector we will create a ``Tuner`` instance for each model that
        we want to try out, as well as the ``Selector`` instance.
        
        ```python
        >>> from sklearn.ensemble import RandomForestClassifier
        >>> from sklearn.svm import SVC
        >>> from btb.selection import UCB1
        >>> from btb.tuning.hyperparams import FloatHyperParam
        >>> models = {
        ...     'RF': RandomForestClassifier,
        ...     'SVC': SVC
        ... }
        >>> selector = UCB1(['RF', 'SVC'])
        >>> rf_hyperparams = {
        ...     'n_estimators': IntHyperParam(min=10, max=500),
        ...     'max_depth': IntHyperParam(min=3, max=20)
        ... }
        >>> rf_tunable = Tunable(rf_hyperparams)
        >>> svc_hyperparams = {
        ...     'C': FloatHyperParam(min=0.01, max=10.0),
        ...     'gamma': FloatHyperParam(0.000000001, 0.0000001)
        ... }
        >>> svc_tunable = Tunable(svc_hyperparams)
        >>> tuners = {
        ...     'RF': GPTuner(rf_tunable),
        ...     'SVC': GPTuner(svc_tunable)
        ... }
        ```
        
        Then we perform the following steps in a loop.
        
        1. Pass all the obtained scores to the selector and let it decide which model to test.
        
            ``` python
            >>> next_choice = selector.select({
            ...     'RF': tuners['RF'].scores,
            ...     'SVC': tuners['SVC'].scores
            ... })
            >>> next_choice
            'RF'
            ```
        
        2. Obtain a new set of parameters from the indicated tuner and create a model instance.
        
            ``` python
            >>> parameters = tuners[next_choice].propose()
            >>> parameters
            {'n_estimators': 289, 'max_depth': 18}
            >>> model = models[next_choice](**parameters)
            ```
        
        3. Evaluate the score of the new model instance and pass it back to the tuner
        
            ``` python
            >>> model.fit(X_train, y_train)
            RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',
                        max_depth=18, max_features='auto', max_leaf_nodes=None,
                        min_impurity_decrease=0.0, min_impurity_split=None,
                        min_samples_leaf=1, min_samples_split=2,
                        min_weight_fraction_leaf=0.0, n_estimators=289, n_jobs=1,
                        oob_score=False, random_state=None, verbose=0,
                        warm_start=False)
            >>> score = model.score(X_test, y_test)
            >>> score
            0.89
            >>> tuners[next_choice].record(parameters, score)
            ```
        
        ## What's next?
        For more details about **BTB** and all its possibilities and features, please check the
        [project documentation site](https://HDI-Project.github.io/BTB/)!
        
        ## Citing BTB
        
        If you use BTB, please consider citing our related papers.
        
        For the current design of BTB and its usage within the larger *Machine Learning Bazaar* project at the MIT Data To AI Lab, please see:
        
        Micah J. Smith, Carles Sala, James Max Kanter, and Kalyan Veeramachaneni. ["The Machine Learning Bazaar: Harnessing the ML Ecosystem for Effective System Development."](https://arxiv.org/abs/1905.08942) arXiv Preprint 1905.08942. 2019.
        
        ``` bibtex
        @article{smith2019mlbazaar,
          author = {Smith, Micah J. and Sala, Carles and Kanter, James Max and Veeramachaneni, Kalyan},
          title = {The Machine Learning Bazaar: Harnessing the ML Ecosystem for Effective System Development},
          journal = {arXiv e-prints},
          year = {2019},
          eid = {arXiv:1905.08942},
          pages = {arXiv:1905.08942},
          archivePrefix = {arXiv},
          eprint = {1905.08942},
        }
        ```
        
        For the initial design of BTB, usage of Recommenders, and initial evaluation, please see:
        
        Laura Gustafson. ["Bayesian Tuning and Bandits: An Extensible, Open Source Library for AutoML."](https://dai.lids.mit.edu/wp-content/uploads/2018/05/Laura_MEng_Final.pdf) Masters thesis, MIT EECS, June 2018.
        
        ``` bibtex
          @mastersthesis{gustafson2018bayesian,
            author = {Gustafson, Laura},
            title = {Bayesian Tuning and Bandits: An Extensible, Open Source Library for AutoML},
            month = {May},
            year = {2018},
            url = {https://dai.lids.mit.edu/wp-content/uploads/2018/05/Laura_MEng_Final.pdf},
            type = {M. Eng Thesis},
            school = {Massachusetts Institute of Technology},
            address = {Cambridge, MA},
          }
        ```
        
        
        # History
        
        ## 0.3.4 - 2019-12-24
        
        With this release we introduce a `BTBSession` class. This class represents the process of selecting
        and tuning several tunables until the best possible configuration fo a specific `scorer` is found.
        We also have improved and fixed some minor bugs arround the code (described in the issues below).
        
        ### New Features
        
        * `BTBSession` that makes `BTB` more user friendly.
        
        ### Internal Improvements
        
        Improved unittests, removed old dependencies, added more `MLChallenges` and fixed an issue with
        the bound methods.
        
        ### Resolved Issues
        
        * Issue #145: Implement `BTBSession`.
        * Issue #155: Set defaut to `None` for `CategoricalHyperParam` is not possible.
        * Issue #157: Metamodel `_MODEL_KWARGS_DEFAULT` becomes mutable.
        * Issue #158: Remove `mock` dependency from the package.
        * Issue #160: Add more Machine Learning Challenges and more estimators.
        
        
        ## 0.3.3 - 2019-12-11
        
        Fix a bug where creating an instance of `Tuner` ends in an error.
        
        ### Internal Improvements
        
        Improve unittests to use `spec_set` in order to detect errors while mocking an object.
        
        ### Resolved Issues
        
        * Issue #153: Bug with tunner logger message that avoids creating the Tunner.
        
        ## 0.3.2 - 2019-12-10
        
        With this release we add the new `benchmark` challenge `MLChallenge` which allows users to
        perform benchmarking over datasets with machine learning estimators, and also some new
        features to make the workflow easier.
        
        ### New Features
        
        * New `MLChallenge` challenge that allows performing crossvalidation over datasets and machine
        learning estimators.
        * New `from_dict` function for `Tunable` class in order to instantiate from a dictionary that
        contains information over hyperparameters.
        * New `default` value for each hyperparameter type.
        
        ### Resolved Issues
        
        * Issue #68: Remove `btb.tuning.constants` module.
        * Issue #120: Tuner repr not helpful.
        * Issue #121: HyperParameter repr not helpful.
        * Issue #141: Imlement propper logging to the tuning section.
        * Issue #150: Implement Tunable `from_dict`.
        * Issue #151: Add default value for hyperparameters.
        * Issue #152: Support `None` as a choice in `CategoricalHyperPrameters`.
        
        ## 0.3.1 - 2019-11-25
        
        With this release we introduce a `benchmark` module for `BTB` which allows the users to perform
        a benchmark over a series of `challenges`.
        
        ### New Features
        
        * New `benchmark` module.
        * New submodule named `challenges` to work toghether with `benchmark` module.
        
        ### Resolved Issues
        
        * Issue #139: Implement a Benchmark for BTB
        
        ## 0.3.0 - 2019-11-11
        
        With this release we introduce an improved `BTB` that has a major reorganization of the project
        with emphasis on an easier way of interacting with `BTB` and an easy way of developing, testing and
        contributing new acquisition functions, metamodels, tuners  and hyperparameters.
        
        ### New project structure
        
        The new major reorganization comes with the `btb.tuning` module. This module provides everything
        needed for the `tuning` process and comes with three new additions `Acquisition`, `Metamodel` and
        `Tunable`. Also there is an update to the `Hyperparamters` and `Tuners`. This changes are meant
        to help developers and contributors to easily develop, test and contribute new `Tuners`.
        
        ### New API
        
        There is a slightly new way of using `BTB` as the new `Tunable` class is introduced, that is meant
        to be the only requiered object to instantiate a `Tuner`. This `Tunable` class represents a
        collection of `HyperParams` that need to be tuned as a whole, at once. Now, in order to create a
        `Tuner`, a `Tunable` instance must be created first with the `hyperparameters` of the
        `objective function`.
        
        ### New Features
        
        * New `Hyperparameters` that allow an easier interaction for the final user.
        * New `Tunable` class that manages a collection of `Hyperparameters`.
        * New `Tuner` class that is a python mixin that requieres of `Acquisition` and `Metamodel` as
        parents. Also now works with a single `Tunable` object.
        * New `Acquisition` class, meant to implement an acquisition function to be inherit by a `Tuner`.
        * New `Metamodel` class, meant to implement everything that a certain `model` needs and be inherit
        by the `Tuner`.
        * Reorganization of the `selection` module to follow a similar `API` to `tuning`.
        
        ### Resolved Issues
        
        * Issue #131: Reorganize the project structure.
        * Issue #133: Implement Tunable class to control a list of hyperparameters.
        * Issue #134: Implementation of Tuners for the new structure.
        * Issue #140: Reorganize selectors.
        
        ## 0.2.5
        
        ### Bug Fixes
        
        * Issue #115: HyperParameter subclass instantiation not working properly
        
        ## 0.2.4
        
        ### Internal Improvements
        
        * Issue #62: Test for `None` in `HyperParameter.cast` instead of `HyperParameter.__init__`
        
        ### Bug fixes
        
        * Issue #98: Categorical hyperparameters do not support `None` as input
        * Issue #89: Fix the computation of `avg_rewards` in `BestKReward`
        
        ## 0.2.3
        
        ### Bug Fixes
        
        * Issue #84: Error in GP tuning when only one parameter is present bug
        * Issue #96: Fix pickling of HyperParameters
        * Issue #98: Fix implementation of the GPEi tuner
        
        ## 0.2.2
        
        ### Internal Improvements
        
        * Updated documentation
        
        ### Bug Fixes
        
        * Issue #94: Fix unicode `param_type` caused error on python 2.
        
        ## 0.2.1
        
        ### Bug fixes
        
        * Issue #74: `ParamTypes.STRING` tunables do not work
        
        ## 0.2.0
        
        ### New Features
        
        * New Recommendation module
        * New HyperParameter types
        * Improved documentation and examples
        * Fully tested Python 2.7, 3.4, 3.5 and 3.6 compatibility
        * HyperParameter copy and deepcopy support
        * Replace print statements with logging
        
        ### Internal Improvements
        
        * Integrated with Travis-CI
        * Exhaustive unit testing
        * New implementation of HyperParameter
        * Tuner builds a grid of real values instead of indices
        * Resolve Issue #29: Make args explicit in `__init__` methods
        * Resolve Issue #34: make all imports explicit
        
        ### Bug Fixes
        
        * Fix error from mixing string/numerical hyperparameters
        * Inverse transform for categorical hyperparameter returns single item
        
        ## 0.1.2
        
        * Issue #47: Add missing requirements in v0.1.1 setup.py
        * Issue #46: Error on v0.1.1: 'GP' object has no attribute 'X'
        
        ## 0.1.1
        
        * First release.
        
Keywords: machine learning hyperparameters tuning classification
Platform: UNKNOWN
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Natural Language :: English
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Requires-Python: >=3.5
Description-Content-Type: text/markdown
Provides-Extra: examples
Provides-Extra: test
Provides-Extra: dev
