We are happy to release the new version of DeepHyper with significant updates that improve the consistency of our package and make it interoperable with more machine-learning frameworks.

The most significant updates are for deephyper.hpo and deephyper.ensemble.

We have included for you below the details about the update.

deephyper.analysis

• deephyper.analysis.hpo (link to documentation)includes new utility functions:
  • filter_failed_objectives
  • parameters_at_max
  • parameters_at_topk
  • parameters_from_row
  • read_results_from_csv

deephyper.ensemble

The ensemble subpackage (link to documentation) was refactored to expand its applicability. It is now compatible with any ML framework such as Tensorflow/Keras2, PyTorch, and Scikit-Learn models. The only requirement is to follow the new deephyper.predictor interface that represents a frozen machine learning model on which inferences can be run.

To help you start with the new ensemble feature we prepared a tutorial that optimizes the hyperparameter of a Decision Tree and build an ensemble to improve accuracy, probabilistic calibration, and uncertainty estimates: Hyperparameter Optimization for Tree Ensemble with Uncertainty Quantification (Scikit-Learn)

The ensemble API is mainly built around the follow classes:

• Predictor that represents a predictive function (see doc on predictor).
• Aggregator: to aggregate the predictions from a set of predictors (see doc on aggregator).
• Loss: loss and scoring functions adapted to classification, regression, or uncertainty quantification (see doc on loss).
• Selector: a selection algorithm that selects a subset of predictors (see doc on selector) and weight them.

The API already provides the tools to build:

1. Simple ensembles to improve the predictions of models with variability (when retrained). Our tutorial on Decision trees is a good example of that.
2. Ensembles with Epistemic uncertainty.
3. Ensembles with decomposed aleatoric and epistemic uncertainty.

For classification, you can use the MixedCategoricalAggregator (see doc) that can use confidence or entropy for uncertainty.

For regression, you can use the MixedNormalAggregator (see doc) that uses variance for uncertainty.

deephyper.hpo

The deephyper.hpo comes in replacement of both deephyper.search.hps and deephyper.search.nas. For consistency, we decided to refactor neural architecture search and hyperparameter optimization together. The main algorithms to explore hyperparameters are:

• Bayesian optimization (see doc on CBO).
• Experimental design (factorial/grid, randomized, quasi-monte carlo) (see doc on ExperimentalDesignSearch).
• Aging/regularized evolution (see doc on RegularizedEvolution).

All search algorithms are now following the _ask/_tell interface. Good and simple examples to follow if you want to implement your own algorithm are RandomSearch and RegularizedEvolution).

Therefore all search algorithms are now compatible with decentralization. An example for decentralized search with subprocess is available here.

A new tutorial on how to do neural architecture search was published: Neural Architecture Search with Tensorflow/Keras2 (Basic).

deephyper.predictor

Utility classes are provided to reload checkpointed models from Scikit-Learn, Tensorflow/Keras2 and Pytorch.

deephyper.stopper

This sub-package provides algorithm that can help speed-up hyperparameter optimization by observing the training of machine learning models.

The main update is to expose the BayesianLearningCurveRegressor which helps extrapolate with uncertainty the future performance of a training curve based on parametric models.

The currently available algorithms are:

• Learning Curve Extrapolation (see doc on LCModelStopper).
• Median Stopper (see doc on MedicanStopper).
• Asynchronous Successive Halving (see doc on SuccessiveHalvingStopper).

deephyper.search

• If a results.csv file already exists in the log_dir folder, it is renamed instead of overwritten.
• Parallel and asynchronous standard experimental design can now be used through DeepHyper to perform Random, Grid, or Quasi-Monte-Carlo evaluations: Example on Standard Experimental Design (Grid Search).

Bayesian optimization (CBO and MPIDistributedBO)

• New optimizers of the acquisition function: the acquisition function for non-derivable surrogate models (e.g., "RF", "ET") can now be optimized with acq_optimizer="ga" or acq_optimizer="mixedga". This makes BO iterations more efficient but implies an additional overhead (negligible if the evaluated function is slow). The acq_optimizer_freq=2 parameter can be used to amortize this overhead.
• New exploration/exploitation scheduler: the periodic exponential decay scheduler can now be specified with its initial and final values CBO(..., kappa=10, scheduler={"type": "periodic-exp-decay", "period": 25, "kappa_final": 1.96}). This mechanism allows to escape local optimum.
• New family of acquisition functions for Random-Forests surrogate models. acq_func="UCBd" or "EId" or "PId". The "d" postfix stands for "deterministic" because in this case, the acquisition function will only use epistemic uncertainty from the black-box function to evaluate the acquisition function and it will ignore aleatoric uncertainty (i.e., noise estimation).
• The default surrogate model was renamed "ET" which stands for "Extremely Randomized Trees" to better match the machine learning literature. This surrogate model provides better epistemic uncertainty estimates than the standard "RF" which stands for "Random Forest". It is also a type of randomized ensemble of trees but it uses a randomized split decision rule instead of an optimized split decision rule.
• HpProblem based on ConfigSpace objects using constraints now uses the lower bound of each hyperparameter as a slack value.

deephyper.stopper

• The stopper based on learning curve extrapolation has an improved fit and speed.

• The documentation site theme was updated.
• PyPI Release: https://pypi.org/project/deephyper/0.6.0/
• New BibTeX citation for DeepHyper to include our growing community:

@misc{deephyper_software,
title = {"DeepHyper: A Python Package for Scalable Neural Architecture and Hyperparameter Search"},
author = {{DeepHyper Development Team}},
organization = {DeepHyper Team},
year = 2018,
url = {https://github.com/deephyper/deephyper}
} 

deephyper.evaluator

• @profile(memory=True) decorator can now profile memory using tracemalloc (adding an overhead).
• RayStorage is now available for the ray parallel backend. It is based on remote actors and is a wrapper around the base MemoryStorage. This allows to use deephyper.stopper in parallel only with ray backend requirements.

deephyper.search

• Multi-objective optimization (MOO) has been upgraded for better optimization performance. A new tutorial to discover this feature is available at Multi-Objective Optimization - 101.
  • A minimum-lower bound performance can be specified to avoid exploring not interesting trade-offs moo_lower_bounds=....
  • A new objective scaler is available to normalize objectives (e.g., accuracy and latency) more efficiently objective_scaler="quantile-uniform".
  • The results.csv or DataFrame now contains a new information pareto_efficient which indicates the optimal solution in a multi-objective problem (i.e., Pareto-set/front).
• Random-Forest (RF) surrogate model predictions are faster by about x1.5 factor, speeding up the Bayesian optimization process.
• Added a dynamic prior update for Bayesian optimization: update_prior=..., update_prior_quantile=... This allows to increase the density of sampling in areas of interest and makes "random"-sampling-based optimization of the surrogate model more competitive (against more expensive optimizers like gradient-based or genetic algorithms).

deephyper.stopper

• SuccessiveHalvingStopper is now compatible with failures. If a "failure" is observed during training (i.e., observation starting with "F") then previous observations are replaced in shared memory to notify other competitors of the failure.

deephyper.analysis

• Creation of a new module to provide utilities for the analysis of experiments.

deephyper.evaluator

• removed SubprocessEvaluator evaluator due to limited features and confusion with ProcessPoolEvaluator. The ProcessPoolEvaluator seamed to be enough for our different use cases (hyperparameter optimization, auto-tuning).
• retrieving and returning remote exception when using MPICommEvaluator.
• timeout of the search is now handled through thread-based timeout instead of signal handlers.
• timeout with MPICommEvaluator is now handled through refreshed "remaining time" and thread-based timeout in each rank.
• exceptions happening in remote rank when using MPICommEvaluator are now traced-back and printed out in root rank.
• possibility to return metadata logged in the results DataFrame with the run-function.

def run(job):
    config = job.parameters
    return {"objective": config["x"], "metadata": {"time": time.time()}}

• new in the resutls.csv or returned pd.DataFrame, the hyperparameters will start with p: prefix and metadata will start with m: to allow for easier filtration of the columns.
• new deephyper.evaluator.storage. The interface is defined by Storage with two basic implementations MemoryStorage (local memory) and RedisStorage (in-memory key-value database).
• new RunningJob API. The run-function is now passed a RunningJob instance instead of dict. The RunningJob.parameters corresponds to the former dictionary passed to the run-function. The RunningJob object should implement the dictionary interface to be backward compatible with the previous standard argument of the run-function.

deephyper.search

• new preprocessing of the objective values when using surrogate="RF" in CBO. The objective is now preprocessed with Min-Max and Log to improve the fitting of the surrogate model on small values and improve Bayesian optimisation convergence. (see paper)
• new cyclic exponential decay of the exploration-exploitation trade-off in CBO this is particularly when using MPIDistributedDBO and scaling the number of BO instances to avoid "over-exploration".
• new distributed Bayesian optimization through MPI and Storage available. See Tutorial - Distributed Bayesian Optimization with MPI and Redis.

deephyper.stopper

A new module in DeepHyper to allow for Multi-Fidelity Bayesian Optimization. A stopper can observe the evolving performance of an iterative algorithm and decide to continue its evaluation or stop it early. This can allow to the search to be more effective when the time-ressource or computational ressource is a bottleneck. However, it can also converge to sub-optimal solution. Different, multi-fidelity schemes are now proposed and documented at DeepHyper Documentation - Stopper.

deephyper.evaluator

• patched ThreadPoolEvaluator to remove extra overheads of pool initialisation

deephyper.search

• resolved constant hyperparameter when hyperparameter is discrete with log-uniform in df01040d44a8f3b80700f2f853a6b452680e1112
• patch id to job_id in neural architecture search history saver
• adding multi objectives optimisation to CBO a run-function can now return multiple objectives as a tuple to be maximised

def run(config):
    ...
    return objective_0, objective_1

deephyper.ensemble

• ensemble with uncertainty quantification UQBaggingEnsembleRegressor is now compatible with predictions of arbitrary shapes

deephyper dashboard

• adding dashboard with deephyper-analytics dashboard paired with results stored in local deephyper database managed through DBManager
• adding dataframe visualization
• adding scatter plot visualization

global updates

• contributors of DeepHyper now appear on a dedicated page, see DeepHyper Authors, submit a PR if we forgot you!
• lighter installation via pip install deephyper packed with the minimum requirements for hyperparameter search.
• update API documentation
• removed deephyper.benchmark
• make neural architecture search features optional with pip install deephyper[nas]
• make auto-sklearn features optional with pip install deephyper[popt] (Pipeline OPTimization)
• improve epistemic uncertainty quantification for Random Forest surrogate model in Bayesian Optimisation
• moved deephyper/scikit-optimize as a sub package deephyper.skopt
• new tutorials dedicated to ALCF systems, see Tutorials - Argonne Leadership Computing Facility

deephyper.search

• renamed AMBS to CBO (Centralised Bayesian Optimization) at deephyper.search.hps.CBO
• added new scalable Distributed Bayesian Optimization algorithm at deephyper.search.hps.DBO (experimented with up to 4,096 parallel workers)
• moved problem.add_starting_point of HpProblem to CBO(..., initial_points=[...])
• added generative-model based transfer-learning for hyper-parameter optimisation (Example - Transfer Learning for Hyperparameter Search)
• added filtration of duplicated configurations in CBO CBO(..., filter_duplicated=True)
• notify failures to the optimiser for it to learn them (Example - Notify Failures in Hyperparameter optimization)
• added new multi-point acquisition strategy for better scalability in CBO CBO(..., acq_func="UCB, multi_point_strategy="qUCB", ...)
• added the possibility to switch between synchronous/asynchronous communication in CBO CBO(..., sync_communication=True, ...)

deephyper.evaluator

• added MPI-based Evaluators (better scaling, lower initialisation overhead): MPICommEvaluator and MPIPoolEvaluator
• added @profile(run_function) decorator for run-function to collect execution times/durations of the black-box, this allow to profile the worker utilisation (Example - Profile the Worker Utilisation)
• added @queued(Evaluator) decorator for any evaluator class to manage a queue of resources
• added SerialEvalutor to adapt to serial-search (one by one)
• added deephyper.evaluator.callback.TqdmCallback to display progress bar when running a search
• the run-function can now return other values than the objective to be logged in the results.csv for example {"objective": 0.9, "num_parameters": 20000, ...}
• asyncio is patched automatically when using notebooks/ipython

deephyper.ensemble

• an ensemble API is provided to have uncertainty quantification estimates after running a neural architecture search or hyperparameter search (Tutorial - From Neural Architecture Search to Automated Deep Ensemble with Uncertainty Quantification)

• Now compatible with Python >=3.7, <3.10
• Fixed log_dir argument in search
• Added logging for command line HPS/NAS

• All the search algorithms were tested to have a correct behaviour when random_state is set.
• Callbacks (deephyper.evaluator.callback) can now be used to extend the behavior of the existing Evaluator. A LoggerCallback, ProfilingCallback, SearchEarlyStopping are already available (see example below).
• All search algorithms are now importable from their hps or nas package. For example, from deephyper.search.hps import AMBS and from deephyper.search.nas import AgEBO.
• HpProblem and NaProblem do not have a seed parameter anymore. The random_state has to be set when instantiating a Search(random_state=...).

Examlpe: SearchEarlyStopping

from deephyper.problem import HpProblem
from deephyper.search.hps import AMBS
from deephyper.evaluator import Evaluator
from deephyper.evaluator.callback import LoggerCallback, SearchEarlyStopping

problem = HpProblem()
problem.add_hyperparameter((0.0, 10.0), "x")

def f(config):
    return config["x"]
    
evaluator = Evaluator.create(f, 
                             method="ray",
                             method_kwargs={
                                 "num_cpus": 1,
                                 "num_cpus_per_task": 0.25,
                                 "callbacks": [LoggerCallback(), SearchEarlyStopping(patience=10)]
                             })
print(f"Num. Workers {evaluator.num_workers}")

search = AMBS(problem, evaluator, filter_duplicated=False)

results = search.search(max_evals=500)

Gives the following output:

Num. Workers 4
[00001] -- best objective: 3.74540 -- received objective: 3.74540
[00002] -- best objective: 6.38145 -- received objective: 6.38145
Objective has improved from 3.74540 -> 6.38145
[00003] -- best objective: 6.38145 -- received objective: 3.73641
[00004] -- best objective: 7.29998 -- received objective: 7.29998
Objective has improved from 6.38145 -> 7.29998
[00005] -- best objective: 7.29998 -- received objective: 2.98912
[00006] -- best objective: 7.29998 -- received objective: 5.52077
[00007] -- best objective: 7.29998 -- received objective: 4.59535
[00008] -- best objective: 7.29998 -- received objective: 5.28775
[00009] -- best objective: 7.29998 -- received objective: 5.52099
[00010] -- best objective: 9.76781 -- received objective: 9.76781
Objective has improved from 7.29998 -> 9.76781
[00011] -- best objective: 9.76781 -- received objective: 7.48943
[00012] -- best objective: 9.76781 -- received objective: 7.42981
[00013] -- best objective: 9.76781 -- received objective: 9.30103
[00014] -- best objective: 9.76781 -- received objective: 8.22588
[00015] -- best objective: 9.76781 -- received objective: 8.96084
[00016] -- best objective: 9.76781 -- received objective: 8.96303
[00017] -- best objective: 9.96415 -- received objective: 9.96415
Objective has improved from 9.76781 -> 9.96415
[00018] -- best objective: 9.96415 -- received objective: 9.58723
[00019] -- best objective: 9.96415 -- received objective: 9.93599
[00020] -- best objective: 9.96415 -- received objective: 9.35591
[00021] -- best objective: 9.96415 -- received objective: 9.90210
[00022] -- best objective: 9.97627 -- received objective: 9.97627
Objective has improved from 9.96415 -> 9.97627
[00023] -- best objective: 9.98883 -- received objective: 9.98883
Objective has improved from 9.97627 -> 9.98883
[00024] -- best objective: 9.98883 -- received objective: 9.97969
[00025] -- best objective: 9.98883 -- received objective: 9.96051
[00026] -- best objective: 9.98883 -- received objective: 9.86835
[00027] -- best objective: 9.98883 -- received objective: 9.80940
[00028] -- best objective: 9.98883 -- received objective: 9.84498
[00029] -- best objective: 9.98883 -- received objective: 9.86562
[00030] -- best objective: 9.99664 -- received objective: 9.99664
Objective has improved from 9.98883 -> 9.99664
[00031] -- best objective: 9.99664 -- received objective: 9.99541
[00032] -- best objective: 9.99790 -- received objective: 9.99790
Objective has improved from 9.99664 -> 9.99790
[00033] -- best objective: 9.99790 -- received objective: 9.99640
[00034] -- best objective: 9.99790 -- received objective: 9.98190
[00035] -- best objective: 9.99790 -- received objective: 9.98854
[00036] -- best objective: 9.99790 -- received objective: 9.98335
[00037] -- best objective: 9.99790 -- received objective: 9.99303
[00038] -- best objective: 9.99790 -- received objective: 9.99271
[00039] -- best objective: 9.99790 -- received objective: 9.99164
[00040] -- best objective: 9.99790 -- received objective: 9.99313
[00041] -- best objective: 9.99790 -- received objective: 9.99236
[00042] -- best objective: 9.99875 -- received objective: 9.99875
Objective has improved from 9.99790 -> 9.99875
[00043] -- best objective: 9.99875 -- received objective: 9.99735
[00044] -- best objective: 9.99969 -- received objective: 9.99969
Objective has improved from 9.99875 -> 9.99969
[00045] -- best objective: 9.99969 -- received objective: 9.99755
[00046] -- best objective: 9.99969 -- received objective: 9.99742
[00047] -- best objective: 9.99995 -- received objective: 9.99995
Objective has improved from 9.99969 -> 9.99995
[00048] -- best objective: 9.99995 -- received objective: 9.99725
[00049] -- best objective: 9.99995 -- received objective: 9.99746
[00050] -- best objective: 9.99995 -- received objective: 9.99990
[00051] -- best objective: 9.99995 -- received objective: 9.99915
[00052] -- best objective: 9.99995 -- received objective: 9.99962
[00053] -- best objective: 9.99995 -- received objective: 9.99930
[00054] -- best objective: 9.99995 -- received objective: 9.99982
[00055] -- best objective: 9.99995 -- received objective: 9.99985
[00056] -- best objective: 9.99995 -- received objective: 9.99851
[00057] -- best objective: 9.99995 -- received objective: 9.99794
Stopping the search because it did not improve for the last 10 evaluations!

Tutorials

• [NEW] Hyperparameter search for text classification (Pytorch)
• [NEW] Neural Architecture Search with Multiple Input Tensors
• [NEW] From Neural Architecture Search to Automated Deep Ensemble with Uncertainty Quantification
• [UPDATED] Execution on the Theta supercomputer/N-evaluation per 1-node

Hyperparameter search

• [NEW] Filtering duplicated samples: New parameters filter_duplicated and n_points appeared for deephyper.search.hps.AMBS. By default filter_duplicated = True implies that the search space filters duplicated values until it cannot sample new unique values (and therefore will re-sample existing configurations of hyperparameters). This filtering behaviour and sampling speed are sensitive to the n_points parameter which corresponds to the number of samples drawn from the search space before being filtered by the surrogate model. By default n_points = 10000. If filter_duplicated = False then the filtering of duplicated points will be skipped but n_points will still impact sampling speed.
• Arguments of AMBS were adapted to match the maximisation setting of DeepHyper: "LCB" -> "UCB", cl_min -> cl_max, "cl_max" -> "cl_min".

Neural architecture search

The package deephyper.nas was restructured. All the neural architecture search space should now be subclasses of deephyper.nas.KSearchSpace:

import tensorflow as tf

from deephyper.nas import KSearchSpace
from deephyper.nas.node import ConstantNode, VariableNode
from deephyper.nas.operation import operation, Identity

Dense = operation(tf.keras.layers.Dense)
Dropout = operation(tf.keras.layers.Dropout)

class ExampleSpace(KSearchSpace):
    
    def build(self):
        
        # input nodes are automatically built based on `input_shape`
        input_node = self.input_nodes[0] 
        
        # we want 4 layers maximum (Identity corresponds to not adding a layer)
        for i in range(4):
            node = VariableNode()
            self.connect(input_node, node) 

            # we add 3 possible operations for each node
            node.add_op(Identity())
            node.add_op(Dense(100, "relu"))
            node.add_op(Dropout(0.2))
            
            input_node = node
            
        output = ConstantNode(op=Dense(self.output_shape[0]))
        self.connect(input_node, output)

        return self

space = ExampleSpace(input_shape=(1,), output_shape=(1,)).build()
space.sample().summary()

will output:

Model: "model_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_0 (InputLayer)         [(None, 1)]               0         
_________________________________________________________________
dense_3 (Dense)              (None, 100)               200       
_________________________________________________________________
dense_4 (Dense)              (None, 100)               10100     
_________________________________________________________________
dropout_2 (Dropout)          (None, 100)               0         
_________________________________________________________________
dense_5 (Dense)              (None, 1)                 101       
=================================================================
Total params: 10,401
Trainable params: 10,401
Non-trainable params: 0
_________________________________________________________________

To have a complete example follow the Neural Architecture Search (Basic) tutorial.

The main changes were the following:

• AutoKSearchSpace, SpaceFactory, Dense, Dropout and others were removed. Operations like Dense can now be created directly using the operation(tf.keras.layers.Dense) to allow for lazy tensor allocation.
• The search space class should now be passed directly to the NaProblem.search_space(KSearchSpaceSubClass).
• deephyper.nas.space is now deephyper.nas
• All operations are now under deephyper.nas.operation
• Nodes are now under deephyper.nas.node

Documentation

• API Reference: A new section on the documentation website to give details about all usable functions/classes of DeepHyper.

Suppressed

• Notebooks generated with deephyper-analytics were removed.
• deephyper ray-submit
• deephyper ray-config
• Some unused dependencies were removed: balsam-flow, deap.

This new release help us move toward a more stable version of DeepHyper.

• Refactored the DeepHyper Documentation
• Developed notebook tutorials
• Decoupled the command line and Python interfaces
• Refactored the Evaluator interface with evaluator.submit/gather
• Added deephyper.ensemble for ensembles with uncertainty quantification
• Removed deephyper.post

General

Full API documentation

The DeepHyper API is now fully documented at DeepHyper API

Tensorflow-Probability as a new dependency

TensorFlow Probability is now part of DeepHyper default set of dependencies

Automated submission with Ray at ALCF

It is now possible to directly submit with deephyper ray-submit ... for DeepHyper at the ALCF. This feature is only available on ThetaGPU for now but can be extended to other systems by following this script.

ThetaGPU at ALCF

• New installation documentation is available at Installation ThetaGPU (ALCF)
• A new user guide is available at Running on ThetaGPU (ALCF) to understand how to run manually and automatically DeepHyper on ThetaGPU.

New documentation for auto-sklearn search with DeepHyper

The access to auto-sklearn features was changed to deephyper.sklearn and a new documentation is available for this feature at User guide: AutoSklearn

New command lines for DeepHyper Analytics

The deephyper-analytics command was modified and enhanced with new features. The see the full updated documentation follow DeepHyper Analytics Tools.

The topk command is now available to have quick feedback from the results of an experiment:

$ deephyper-analytics topk combo_8gpu_8_agebo/infos/results.csv -k 2
'0':
arch_seq: '[229, 0, 22, 1, 1, 53, 29, 1, 119, 1, 0, 116, 123, 1, 273, 0, 1, 388]'
batch_size: 59
elapsed_sec: 10259.2741303444
learning_rate: 0.0001614947
loss: log_cosh
objective: 0.9236862659
optimizer: adam
patience_EarlyStopping: 22
patience_ReduceLROnPlateau: 10
'1':
arch_seq: '[229, 0, 22, 0, 1, 235, 29, 1, 313, 1, 0, 116, 123, 1, 37, 0, 1, 388]'
batch_size: 51
elapsed_sec: 8818.2674164772
learning_rate: 0.0001265946
loss: mae
objective: 0.9231553674
optimizer: nadam
patience_EarlyStopping: 23
patience_ReduceLROnPlateau: 14

Neural architecture search

New documentation for the problem definition

A new documentation for the neural architecture search problem setup can be found here.

It is now possible to defined auto-tuned hyperparameters in addition of the architecture in a NAS Problem.

New Algorithms for Joint Hyperparameter and Neural Architecture Search

Three new algorithms are available to run a joint Hyperparameter and neural architecture search. The Hyperparameter optimisation is defined as HPO and neural architecture search as NAS.

• agebo (Aging Evolution for NAS with Bayesian Optimisation for HPO)
• ambsmixed (an extension of Asynchronous Model-Based Search for HPO + NAS)
• regevomixed (an extension of regularised evolution for HPO + NAS)

A run function to use data-parallelism with Tensorflow

A new run function to use data-parallelism during neural architecture search is available (link to code)

To use this function pass it to the run argument of the command line such as:

deephyper nas agebo ... --run deephyper.nas.run.tf_distributed.run ... --num-cpus-per-task 2 --num-gpus-per-task 2 --evaluator ray --address auto ...

This function allows for new hyperparameters in the Problem.hyperparameters(...):

...
Problem.hyperparameters(
    ...
    lsr_batch_size=True,
    lsr_learning_rate=True,
    warmup_lr=True,
    warmup_epochs=5,
    ...
)
...

Optimization of the input pipeline for the training

The data-ingestion pipeline was better optimised to reduce the overheads on GPU instances:

self.dataset_train = (
  self.dataset_train.cache()            
  .shuffle(self.train_size, reshuffle_each_iteration=True)            
  .batch(self.batch_size)
  .prefetch(tf.data.AUTOTUNE)
  .repeat(self.num_epochs)        
)

Easier model generation from Neural Architecture Search results

A new method is now available from the Problem object Problem.get_keras_model(arch_seq) to easily build a Keras model instance from an arch_seq (list encoding a neural network).

Minor bug corrections

• Compatible with Tensorflow 2.
• Horovod compatibility with Balsam evaluator for Theta.
• Horovod and Balsam are now optional installations.
• Update of the AMBS algorithm for Hyperparameter search for better scalability.
• Removing the PPO search for Neural Architecture Search.
• Creating the SpaceFactory interface for the deepspace package which provides ready to go neural architecture search spaces.
• Local distribution of jobs with Ray and multiprocessors CPUs.

DeepHyper 0.1.13

New NAS Algorithm

• Aging Evolution with Bayesian Optimization (AgEBO)

New AMBS implementation

• Previous AMBS renamed to ambsv1
• New implementation of AMBS for better scaling capabilities

Data-Parallelism settings for Balsam and Horovod

Graph convolution layers with message passin

A release for the creation of a DOI on Zeno.

Changelog - DeepHyper 0.1.2

DeepHyper 0.1.2 is now forward-compatible with Python 3.7+ and Balsam 0.3.8+ after removing the async reserved keyword.

Changelog - DeepHyper 0.1.1

This release is mostly introducing features for Neural Architecture Search with DeepHyper.

DeepHyper command-line interface

For hyperparameter search use deephyper hps ... here is an example for the
hyperparameter polynome2 benchmark:

deephyper hps ambs --problem deephyper.benchmark.hps.polynome2.Problem --run deephyper.benchmark.hps.polynome2.run

For neural architecture search use deephyper nas ... here is an example for the
neural architecture search linearReg benchmark:

deephyper nas regevo --problem deephyper.benchmark.nas.linearReg.Problem

Use commands such as deephyper --help, deephyper nas --help or deephyper nas regevo --help to find out more about the command-line interface.

Create an Operation from a Keras Layer

• Create a new Operation directly from tensorflow.keras.layers:

>>> import tensorflow as tf
>>> from deephyper.search.nas.model.space.node import VariableNode
>>> from deephyper.search.nas.model.space.op import Operation
>>> vnode = VariableNode()
>>> vnode.add_op(Operation(layer=tf.keras.layers.Dense(10)))

Trainer default CSVLogger callback

• TrainerTrainValid now has a default callback: tf.keras.callbacks.CSVLogger(...)

Ray evaluator

The ray evaluator is now available through ... --evaluator ray... for both hyperparameter
and neural architecture search.

Seeds for reproducibility

To use a seed for any run do Problem(seed=seed) while creating your problem object.

AMBS learner distributed

Use the .. --n-jobs ... to define how to distribute the learner computation in AMBS.

MimeNode to replicate actions

The goal of MimeNode is to replicate the action applied to the targeted variable node.

import tensorflow as tf

from deephyper.search.nas.model.space.node import VariableNode, MimeNode
from deephyper.search.nas.model.space.op.op1d import Dense

vnode = VariableNode()
dense_10_op = Dense(10)
vnode.add_op(dense_10_op)
vnode.add_op(Dense(20))

mnode = MimeNode(vnode)
dense_30_op = Dense(30)
mnode.add_op(dense_30_op)
mnode.add_op(Dense(40))

# The first operation "Dense(10)" has been choosen
# for the mimed node: vnode
vnode.set_op(0)

assert vnode.op == dense_10_op

# mnode is miming the choice made for vnode as you can see
# the first operation was choosen as well
assert mnode.op == dense_30_op

MirrorNode to reuse the same operation

The goal of MirroNode is to replicate the action applied to the targeted VariableNode,
ConstantNode or MimeNode.

import tensorflow as tf

from deephyper.search.nas.model.space.node import VariableNode, MirrorNode
from deephyper.search.nas.model.space.op.op1d import Dense

vnode = VariableNode()
dense_10_op = Dense(10)
vnode.add_op(dense_10_op)
vnode.add_op(Dense(20))

mnode = MirrorNode(vnode)

# The operation "Dense(10)" is being set for vnode.
vnode.set_op(0)

# The same operation (i.e. same instance) is now returned by both vnode and mnode.
assert vnode.op == dense_10_op
assert mnode.op == dense_10_op

Tensorboard and Beholder callbacks available for post-training

Tensorboard and Beholder callbacks can now be used during the post-training. Beholder is
a Tensorboard which enable you to visualize the evolution of the trainable parameters of
a model during the training.

Problem.post_training(
    ...
    callbacks=dict(
        TensorBoard={
            'log_dir':'tb_logs',
            'histogram_freq':1,
            'batch_size':64,
            'write_graph':True,
            'write_grads':True,
            'write_images':True,
            'update_freq':'epoch',
            'beholder': True
        })
)