Optimisation¶

Hypertunity ships with three types of hyperparameter space exploration algorithms. A Bayesian optimisation, random and grid search. While the first one is sequential in nature and requires evaluations to update its internal model of the objective function, so that more informed sample suggestions are generated, the latter two are able to generate all samples in parallel and do not require updating. In this section we will give a brief overview of each.

Bayesian optimisation¶

BayesianOptimisation in Hypertunity is a wrapper around GPyOpt.methods.BayesianOptimization which uses Gaussian Process regression to build a surrogate model of the objective function. It is initialised from a Domain object:

bo = BayesianOptimization(domain)

The BayesianOptimisation optimiser is highly customisable during sampling. This enables the user to dynamically refine the model during calling run_step(). This approach introduces however the computational burden of recomputing the surrogate model at each query. In the following example we show how one can set the GP model using readily available ones from GPy.models, e.g. a GPHeteroschedasticRegression:

bo = BayesianOptimisation(domain=domain, seed=7)                    # initialise BO optimiser
kernel = GPy.kern.RBF(1) + GPy.kern.Bias(1)                         # create a custom kernel
custom_model = GPy.models.GPHeteroscedasticRegression(..., kernel)  # create a custom model
samples = bayes_opt.run_step(model=custom_model)                    # generate samples

Random search¶

This class is a wrapper around the Domain.sample() method. It has the API of an Optimiser class and yields samples which are uniformly drawn from the domain. There is no limitation on the number of samples that can be returned in a single call of run_step(), even if this leads to repetitions.

Grid search¶

GridSearch is a wrapper around the iteration over a domain. It goes over each point in the Cartesian-product of all discrete subdomains. If one of the subdomains is continuous GridSearch will sample uniformly from this interval. Once the domain is exhausted, further iteration will be prevented by raising an ExhaustedSearchSpaceError. To iterate again the GridSearch optimiser must be reset by calling the reset() method.

>>> domain = Domain({"x": {1, 2, 3}, "y": {"a", "b"}, "z": [0, 1]})
>>> gs = GridSearch(domain, sample_continuous=True)
>>> gs.run_step(batch_size=6)
[
    {'x': 1, 'y': 'b', 'z': 0.054781406913364084},
    {'x': 2, 'y': 'b', 'z': 0.7006391867439882},
    {'x': 3, 'y': 'b', 'z': 0.9674445624792569},
    {'x': 1, 'y': 'a', 'z': 0.7837727333178091},
    {'x': 2, 'y': 'a', 'z': 0.17240297136803384},
    {'x': 3, 'y': 'a', 'z': 0.844465575155033}
]
>>> gs.reset()

Custom optimiser¶

If neither of the predefined optimiser are useful for your problem, you can easily roll out a custom one. Only thing you have to do is to inherit from the base Optimiser class and implement the run_step() method.

class CustomOptimiser(Optimiser):
    def __init__(self, domain, *args, **kwargs):
        super(CustomOptimiser, self).__init__(domain)
        ...

    def run_step(batch_size, *args, **kwargs):
        ...
        return [samples]