Adding a classification method

ATM includes several classification methods out of the box, but it’s possible to add custom ones too.

From 10,000 feet, a “method” in ATM comprises the following:

  1. A Python class which defines a fit-predict interface;

  2. A set of hyperparameters that are (or may be) passed to the class’s constructor, and the range of values that each hyperparameter may take;

  3. A conditional parameter tree that defines how hyperparameters depend on one another; and

  4. A JSON file in atm/methods/ that describes all of the above.

1. Valid method classes

Every method must be implemented by a python class that has the following instance methods:

  1. fit: accepts training data and labels (X and y) and trains a predictive model.

  2. predict: accepts a matrix of unlabeled feature vectors (X) and returns predictions for the corresponding labels (y).

This follows the convention used by scikit-learn, and most of the classifier methods already included with ATM are sklearn classes. However, any custom python class that implements the fit/predict interface can be used with ATM.

Once you have a class, you need to configure the relevant hyperparameters and tell ATM about your class.

2. Creating the JSON file

All configuration for a classification method must be described in a json file with the following format:

{
    "name": "bnb",
    "class": "sklearn.naive_bayes.BernoulliNB",
    "hyperparameters": {...},
    "root_hyperparameters": [...],
    "conditions": {...}
}

  • “name” is a short string (or “code”) which ATM uses to refer to the method.

  • “class” is an import path to the class which Python can interpret.

  • “hyperparameters” is a list of hyperparameters which ATM will attempt to tune.

Defining hyperparameters

Most parameter definitions have two fields: “type” and either “range” or “values”. The “type” is one of [“float”, “float_exp”, “float_cat”, “int”, “int_exp”, “int_cat”, “string”, “bool”]. Types ending in “_cat” are categorical types, and those ending in “_exp” are exponential types.

  • If the type is ordinal or continuous (e.g. “int” or “float”), “range” defines the upper and lower bound on possible values for the parameter. Ranges are inclusive: [0.0, 1.0] includes both 0.0 and 1.0.

  • If the type is categorical (e.g. “string” or “float_cat”), “values” defines the list of all possible values for the parameter.

Example categorical types:

"nu": {
    "type": "float_cat",
    "values": [0.5, 1.5, 3.5]  // will select one of the listed values
}

"kernel": {
    "type": "string",
    "values": ["constant", "rbf", "matern"]  // will select one of the listed strings
}

Example (uniform) numeric type:

"max_depth": {
    "type": "int",
    "range": [2, 10]   // will select integer values uniformly at random between 2 and 10, inclusive
}

Example exponential numeric type:

"length_scale": {
    "type": "float_exp",
    "range": [1e-5, 1e5]  // will select floating-point values from an exponential distribution between 10^-5 and 10^5, inclusive
}

Defining the Conditional Parameter Tree

There are two kinds of hyperparameters: root hyperparameters (also referred to as “method hyperparameters” in the paper) and conditional parameters. Root parameters must be passed to the method class’s constructor no matter what, and conditional parameters are only passed if specific values for other parameters are set. For example, the GaussianProcessClassifier configuration has a single root parameter: kernel. This must be set no matter what. Depending on how it’s set, other parameters might need to be set as well. The format for conditions is as follows:

{
    "root_parameter_name": {
        "value1": ["conditional_parameter_name", ...],
        "value2": ["other_conditional_parameter_name", ...]
    }
}

In gaussian_process.json, there are three sets of parameters which are conditioned on the value of the root parameter kernel:

"root_parameters": ["kernel"],

"conditions": {
    "kernel": {
        "matern": ["nu"],
        "rational_quadratic": ["length_scale", "alpha"],
        "exp_sine_squared": ["length_scale", "periodicity"]
    }
}

If kernel is set to “matern”, it means nu must also be set. If it’s set to “rational_quadratic” instead, length_scale and alpha must be set instead. Conditions can overlap – for instance, length_scale must be set if kernel is either “rational_quadratic” or “exp_sine_squared”, so it’s included in both conditional lists. The only constraint is that any parameter which is set as a result of a condition (i.e. a conditional parameter) must not be listed in “root_parameters”.

The example above defines a conditional parameter tree that looks something like this:

kernel-----------------------
|        \                   \
matern    rational_quadratic  exp_sine_squared
|         |           |       |             |
nu      length_scale  alpha   length_scale  periodicity

3. (Optional) Adding a new method to the ATM library

We are always looking for new methods to add to ATM’s core! If your method is implemented as part of a publicly-available Python library which is compatible with ATM’s other dependencies, you can submit it for permanent inclusion in the library.

Save a copy of your configuration json in the atm/methods/ directory. Then, in in the METHODS_MAP dictionary in atm/constants.py, enter a mapping from a short string representing your method’s name to the name of its json file. For example, 'dt': 'decision_tree.json'. If necessary, add the library where your method lives to requirements.txt.

Test out your method with python scripts/test_method.py --method <your_method_code>. If all hyperpartitions run error-free, you’re probably good to go. Commit your changes to a separate branch, then open up a pull request in the main repository. Explain why your method is a useful addition to ATM, and we’ll merge it in if we agree!