Reports

Saving and visualising progress can be accomplished by using Reporter instance. The reporter is supplied with data using the log() method which takes a tuple of a sample and score. Optionally one can store additional information about the current experiment, e.g. the output directory or the job id, using the meta keyword argument:

for s, e, m in zip(samples, evaluations, meta_infos):
    reporter.log((s, e), meta=m)

Table

Hypertunity comes with a built-in reporter which organises the experiment results into an ascii table. It is initialised from a domain and a list of metrics and can be viewed as a formatted string table by calling str on the object. The table can be sorted in ascending or descending order and the best results can be emphasised:

>>> domain = ht.Domain({"x": [-5., 6.], "y": {"sin", "cos"}, "z": set(range(4))})
>>> reporter = ht.Table(domain, metrics=["score"])
>>> # run experiment and call reporter.log(...)
...
>>> print(reporter.format(order="descending"))
+=====+========+=====+===+==============+
| No. |   x    |  y  | z |    score     |
+=====+========+=====+===+==============+
|  6  | -4.35  | cos | 1 | 15.921 ± 0.0 |
+-----+--------+-----+---+--------------+
|  5  | -4.232 | cos | 3 | 8.906 ± 0.0  |
+-----+--------+-----+---+--------------+
|  4  | -4.588 | sin | 3 | 6.134 ± 0.0  |
+-----+--------+-----+---+--------------+
|  2  |  2.16  | cos | 0 | 4.667 ± 0.0  |
+-----+--------+-----+---+--------------+
|  3  | -0.977 | cos | 1 | -2.045 ± 0.0 |
+-----+--------+-----+---+--------------+
|  1  | -1.438 | cos | 3 | -6.933 ± 0.0 |
+-----+--------+-----+---+--------------+

Tensorboard

If Hypertunity is installed with the tensorboard option, a suitable version of Tensorflow and Tensorboard will be installed. This will enable a Tensorboard reporter which, using the HParams plugin, will generate live visualisations as experiments are being logged. One can start the Tensorboard dashboard in the browser as usual, using the logdir supplied at initialisation.

Note that to create a Tensorboard reporter one will have to import hypertunity.reports.tensorboard explicitly:

import hypertunity.reports.tensorboard as tb
tb_reporter = tb.Tensorboard(domain, metrics=["score"], logdir="./logs")

See the Quick start guide for a preview of the dashboard visualisation.