DeepHyper: Scalable Asynchronous Neural Architecture and Hyperparameter Search for Deep Neural Networks
BSD-3-CLAUSE License
Bot releases are hidden (Show)
We are happy to release the new version of DeepHyper with significant updates that improve the consistency of our package and make it interoperable with more machine-learning frameworks.
The most significant updates are for deephyper.hpo
and deephyper.ensemble
.
We have included for you below the details about the update.
deephyper.analysis.hpo
(link to documentation)includes new utility functions:
filter_failed_objectives
parameters_at_max
parameters_at_topk
parameters_from_row
read_results_from_csv
The ensemble subpackage (link to documentation) was refactored to expand its applicability. It is now compatible with any ML framework such as Tensorflow/Keras2, PyTorch, and Scikit-Learn models. The only requirement is to follow the new deephyper.predictor
interface that represents a frozen machine learning model on which inferences can be run.
To help you start with the new ensemble feature we prepared a tutorial that optimizes the hyperparameter of a Decision Tree and build an ensemble to improve accuracy, probabilistic calibration, and uncertainty estimates: Hyperparameter Optimization for Tree Ensemble with Uncertainty Quantification (Scikit-Learn)
The ensemble API is mainly built around the follow classes:
Predictor
that represents a predictive function (see doc on predictor).Aggregator
: to aggregate the predictions from a set of predictors (see doc on aggregator).Loss
: loss and scoring functions adapted to classification, regression, or uncertainty quantification (see doc on loss).Selector
: a selection algorithm that selects a subset of predictors (see doc on selector) and weight them.The API already provides the tools to build:
For classification, you can use the MixedCategoricalAggregator
(see doc) that can use confidence
or entropy
for uncertainty.
For regression, you can use the MixedNormalAggregator
(see doc) that uses variance
for uncertainty.
The deephyper.hpo
comes in replacement of both deephyper.search.hps
and deephyper.search.nas
. For consistency, we decided to refactor neural architecture search and hyperparameter optimization together. The main algorithms to explore hyperparameters are:
All search algorithms are now following the _ask/_tell
interface. Good and simple examples to follow if you want to implement your own algorithm are RandomSearch and RegularizedEvolution).
Therefore all search algorithms are now compatible with decentralization. An example for decentralized search with subprocess is available here.
A new tutorial on how to do neural architecture search was published: Neural Architecture Search with Tensorflow/Keras2 (Basic).
Utility classes are provided to reload checkpointed models from Scikit-Learn, Tensorflow/Keras2 and Pytorch.
This sub-package provides algorithm that can help speed-up hyperparameter optimization by observing the training of machine learning models.
The main update is to expose the BayesianLearningCurveRegressor which helps extrapolate with uncertainty the future performance of a training curve based on parametric models.
The currently available algorithms are:
Published by Deathn0t 7 months ago
results.csv
file already exists in the log_dir
folder, it is renamed instead of overwritten."RF", "ET"
) can now be optimized with acq_optimizer="ga"
or acq_optimizer="mixedga"
. This makes BO iterations more efficient but implies an additional overhead (negligible if the evaluated function is slow). The acq_optimizer_freq=2
parameter can be used to amortize this overhead.CBO(..., kappa=10, scheduler={"type": "periodic-exp-decay", "period": 25, "kappa_final": 1.96})
. This mechanism allows to escape local optimum.acq_func="UCBd"
or "EId"
or "PId"
. The "d"
postfix stands for "deterministic" because in this case, the acquisition function will only use epistemic uncertainty from the black-box function to evaluate the acquisition function and it will ignore aleatoric uncertainty (i.e., noise estimation)."ET"
which stands for "Extremely Randomized Trees" to better match the machine learning literature. This surrogate model provides better epistemic uncertainty estimates than the standard "RF"
which stands for "Random Forest". It is also a type of randomized ensemble of trees but it uses a randomized split decision rule instead of an optimized split decision rule.HpProblem
based on ConfigSpace
objects using constraints now uses the lower bound of each hyperparameter as a slack value.Published by Deathn0t about 1 year ago
@misc{deephyper_software,
title = {"DeepHyper: A Python Package for Scalable Neural Architecture and Hyperparameter Search"},
author = {{DeepHyper Development Team}},
organization = {DeepHyper Team},
year = 2018,
url = {https://github.com/deephyper/deephyper}
}
@profile(memory=True)
decorator can now profile memory using tracemalloc
(adding an overhead).RayStorage
is now available for the ray
parallel backend. It is based on remote actors and is a wrapper around the base MemoryStorage
. This allows to use deephyper.stopper
in parallel only with ray
backend requirements.moo_lower_bounds=...
.objective_scaler="quantile-uniform"
.results.csv
or DataFrame now contains a new information pareto_efficient
which indicates the optimal solution in a multi-objective problem (i.e., Pareto-set/front).update_prior=..., update_prior_quantile=...
This allows to increase the density of sampling in areas of interest and makes "random"-sampling-based optimization of the surrogate model more competitive (against more expensive optimizers like gradient-based or genetic algorithms).SuccessiveHalvingStopper
is now compatible with failures. If a "failure" is observed during training (i.e., observation starting with "F"
) then previous observations are replaced in shared memory to notify other competitors of the failure.Published by Deathn0t over 1 year ago
SubprocessEvaluator
evaluator due to limited features and confusion with ProcessPoolEvaluator
. The ProcessPoolEvaluator
seamed to be enough for our different use cases (hyperparameter optimization, auto-tuning).MPICommEvaluator
.MPICommEvaluator
is now handled through refreshed "remaining time" and thread-based timeout in each rank.MPICommEvaluator
are now traced-back and printed out in root rank.def run(job):
config = job.parameters
return {"objective": config["x"], "metadata": {"time": time.time()}}
resutls.csv
or returned pd.DataFrame
, the hyperparameters will start with p:
prefix and metadata will start with m:
to allow for easier filtration of the columns.deephyper.evaluator.storage
. The interface is defined by Storage
with two basic implementations MemoryStorage
(local memory) and RedisStorage
(in-memory key-value database).RunningJob
API. The run-function is now passed a RunningJob
instance instead of dict
. The RunningJob.parameters
corresponds to the former dictionary passed to the run-function. The RunningJob
object should implement the dictionary interface to be backward compatible with the previous standard argument of the run-function.surrogate="RF"
in CBO
. The objective is now preprocessed with Min-Max
and Log
to improve the fitting of the surrogate model on small values and improve Bayesian optimisation convergence. (see paper)CBO
this is particularly when using MPIDistributedDBO
and scaling the number of BO instances to avoid "over-exploration".A new module in DeepHyper to allow for Multi-Fidelity Bayesian Optimization. A stopper can observe the evolving performance of an iterative algorithm and decide to continue its evaluation or stop it early. This can allow to the search to be more effective when the time-ressource or computational ressource is a bottleneck. However, it can also converge to sub-optimal solution. Different, multi-fidelity schemes are now proposed and documented at DeepHyper Documentation - Stopper.
Published by Deathn0t over 2 years ago
ThreadPoolEvaluator
to remove extra overheads of pool initialisationid
to job_id
in neural architecture search history saverCBO
a run
-function can now return multiple objectives as a tuple to be maximiseddef run(config):
...
return objective_0, objective_1
UQBaggingEnsembleRegressor
is now compatible with predictions of arbitrary shapesdeephyper-analytics dashboard
paired with results stored in local deephyper database managed through DBManager
Published by Deathn0t over 2 years ago
pip install deephyper
packed with the minimum requirements for hyperparameter search.deephyper.benchmark
pip install deephyper[nas]
pip install deephyper[popt]
(Pipeline OPTimization)deephyper/scikit-optimize
as a sub package deephyper.skopt
deephyper.search.hps.CBO
deephyper.search.hps.DBO
(experimented with up to 4,096 parallel workers)problem.add_starting_point
of HpProblem
to CBO(..., initial_points=[...])
CBO(..., filter_duplicated=True)
CBO(..., acq_func="UCB, multi_point_strategy="qUCB", ...)
CBO(..., sync_communication=True, ...)
MPICommEvaluator
and MPIPoolEvaluator
@profile(run_function)
decorator for run-function to collect execution times/durations of the black-box, this allow to profile the worker utilisation (Example - Profile the Worker Utilisation)@queued(Evaluator)
decorator for any evaluator class to manage a queue of resourcesSerialEvalutor
to adapt to serial-search (one by one)deephyper.evaluator.callback.TqdmCallback
to display progress bar when running a searchresults.csv
for example {"objective": 0.9, "num_parameters": 20000, ...}
Published by Deathn0t almost 3 years ago
Python >=3.7, <3.10
log_dir
argument in searchPublished by Deathn0t almost 3 years ago
random_state
is set.deephyper.evaluator.callback
) can now be used to extend the behavior of the existing Evaluator
. A LoggerCallback
, ProfilingCallback
, SearchEarlyStopping
are already available (see example below).hps
or nas
package. For example, from deephyper.search.hps import AMBS
and from deephyper.search.nas import AgEBO
.HpProblem
and NaProblem
do not have a seed
parameter anymore. The random_state
has to be set when instantiating a Search(random_state=...)
.Examlpe: SearchEarlyStopping
from deephyper.problem import HpProblem
from deephyper.search.hps import AMBS
from deephyper.evaluator import Evaluator
from deephyper.evaluator.callback import LoggerCallback, SearchEarlyStopping
problem = HpProblem()
problem.add_hyperparameter((0.0, 10.0), "x")
def f(config):
return config["x"]
evaluator = Evaluator.create(f,
method="ray",
method_kwargs={
"num_cpus": 1,
"num_cpus_per_task": 0.25,
"callbacks": [LoggerCallback(), SearchEarlyStopping(patience=10)]
})
print(f"Num. Workers {evaluator.num_workers}")
search = AMBS(problem, evaluator, filter_duplicated=False)
results = search.search(max_evals=500)
Gives the following output:
Num. Workers 4
[00001] -- best objective: 3.74540 -- received objective: 3.74540
[00002] -- best objective: 6.38145 -- received objective: 6.38145
Objective has improved from 3.74540 -> 6.38145
[00003] -- best objective: 6.38145 -- received objective: 3.73641
[00004] -- best objective: 7.29998 -- received objective: 7.29998
Objective has improved from 6.38145 -> 7.29998
[00005] -- best objective: 7.29998 -- received objective: 2.98912
[00006] -- best objective: 7.29998 -- received objective: 5.52077
[00007] -- best objective: 7.29998 -- received objective: 4.59535
[00008] -- best objective: 7.29998 -- received objective: 5.28775
[00009] -- best objective: 7.29998 -- received objective: 5.52099
[00010] -- best objective: 9.76781 -- received objective: 9.76781
Objective has improved from 7.29998 -> 9.76781
[00011] -- best objective: 9.76781 -- received objective: 7.48943
[00012] -- best objective: 9.76781 -- received objective: 7.42981
[00013] -- best objective: 9.76781 -- received objective: 9.30103
[00014] -- best objective: 9.76781 -- received objective: 8.22588
[00015] -- best objective: 9.76781 -- received objective: 8.96084
[00016] -- best objective: 9.76781 -- received objective: 8.96303
[00017] -- best objective: 9.96415 -- received objective: 9.96415
Objective has improved from 9.76781 -> 9.96415
[00018] -- best objective: 9.96415 -- received objective: 9.58723
[00019] -- best objective: 9.96415 -- received objective: 9.93599
[00020] -- best objective: 9.96415 -- received objective: 9.35591
[00021] -- best objective: 9.96415 -- received objective: 9.90210
[00022] -- best objective: 9.97627 -- received objective: 9.97627
Objective has improved from 9.96415 -> 9.97627
[00023] -- best objective: 9.98883 -- received objective: 9.98883
Objective has improved from 9.97627 -> 9.98883
[00024] -- best objective: 9.98883 -- received objective: 9.97969
[00025] -- best objective: 9.98883 -- received objective: 9.96051
[00026] -- best objective: 9.98883 -- received objective: 9.86835
[00027] -- best objective: 9.98883 -- received objective: 9.80940
[00028] -- best objective: 9.98883 -- received objective: 9.84498
[00029] -- best objective: 9.98883 -- received objective: 9.86562
[00030] -- best objective: 9.99664 -- received objective: 9.99664
Objective has improved from 9.98883 -> 9.99664
[00031] -- best objective: 9.99664 -- received objective: 9.99541
[00032] -- best objective: 9.99790 -- received objective: 9.99790
Objective has improved from 9.99664 -> 9.99790
[00033] -- best objective: 9.99790 -- received objective: 9.99640
[00034] -- best objective: 9.99790 -- received objective: 9.98190
[00035] -- best objective: 9.99790 -- received objective: 9.98854
[00036] -- best objective: 9.99790 -- received objective: 9.98335
[00037] -- best objective: 9.99790 -- received objective: 9.99303
[00038] -- best objective: 9.99790 -- received objective: 9.99271
[00039] -- best objective: 9.99790 -- received objective: 9.99164
[00040] -- best objective: 9.99790 -- received objective: 9.99313
[00041] -- best objective: 9.99790 -- received objective: 9.99236
[00042] -- best objective: 9.99875 -- received objective: 9.99875
Objective has improved from 9.99790 -> 9.99875
[00043] -- best objective: 9.99875 -- received objective: 9.99735
[00044] -- best objective: 9.99969 -- received objective: 9.99969
Objective has improved from 9.99875 -> 9.99969
[00045] -- best objective: 9.99969 -- received objective: 9.99755
[00046] -- best objective: 9.99969 -- received objective: 9.99742
[00047] -- best objective: 9.99995 -- received objective: 9.99995
Objective has improved from 9.99969 -> 9.99995
[00048] -- best objective: 9.99995 -- received objective: 9.99725
[00049] -- best objective: 9.99995 -- received objective: 9.99746
[00050] -- best objective: 9.99995 -- received objective: 9.99990
[00051] -- best objective: 9.99995 -- received objective: 9.99915
[00052] -- best objective: 9.99995 -- received objective: 9.99962
[00053] -- best objective: 9.99995 -- received objective: 9.99930
[00054] -- best objective: 9.99995 -- received objective: 9.99982
[00055] -- best objective: 9.99995 -- received objective: 9.99985
[00056] -- best objective: 9.99995 -- received objective: 9.99851
[00057] -- best objective: 9.99995 -- received objective: 9.99794
Stopping the search because it did not improve for the last 10 evaluations!
filter_duplicated
and n_points
appeared for deephyper.search.hps.AMBS
. By default filter_duplicated = True
implies that the search space filters duplicated values until it cannot sample new unique values (and therefore will re-sample existing configurations of hyperparameters). This filtering behaviour and sampling speed are sensitive to the n_points
parameter which corresponds to the number of samples drawn from the search space before being filtered by the surrogate model. By default n_points = 10000
. If filter_duplicated = False
then the filtering of duplicated points will be skipped but n_points
will still impact sampling speed.AMBS
were adapted to match the maximisation setting of DeepHyper: "LCB" -> "UCB"
, cl_min -> cl_max
, "cl_max" -> "cl_min"
.The package deephyper.nas
was restructured. All the neural architecture search space should now be subclasses of deephyper.nas.KSearchSpace
:
import tensorflow as tf
from deephyper.nas import KSearchSpace
from deephyper.nas.node import ConstantNode, VariableNode
from deephyper.nas.operation import operation, Identity
Dense = operation(tf.keras.layers.Dense)
Dropout = operation(tf.keras.layers.Dropout)
class ExampleSpace(KSearchSpace):
def build(self):
# input nodes are automatically built based on `input_shape`
input_node = self.input_nodes[0]
# we want 4 layers maximum (Identity corresponds to not adding a layer)
for i in range(4):
node = VariableNode()
self.connect(input_node, node)
# we add 3 possible operations for each node
node.add_op(Identity())
node.add_op(Dense(100, "relu"))
node.add_op(Dropout(0.2))
input_node = node
output = ConstantNode(op=Dense(self.output_shape[0]))
self.connect(input_node, output)
return self
space = ExampleSpace(input_shape=(1,), output_shape=(1,)).build()
space.sample().summary()
will output:
Model: "model_1"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_0 (InputLayer) [(None, 1)] 0
_________________________________________________________________
dense_3 (Dense) (None, 100) 200
_________________________________________________________________
dense_4 (Dense) (None, 100) 10100
_________________________________________________________________
dropout_2 (Dropout) (None, 100) 0
_________________________________________________________________
dense_5 (Dense) (None, 1) 101
=================================================================
Total params: 10,401
Trainable params: 10,401
Non-trainable params: 0
_________________________________________________________________
To have a complete example follow the Neural Architecture Search (Basic) tutorial.
The main changes were the following:
AutoKSearchSpace
, SpaceFactory
, Dense
, Dropout
and others were removed. Operations like Dense
can now be created directly using the operation(tf.keras.layers.Dense)
to allow for lazy tensor allocation.NaProblem.search_space(KSearchSpaceSubClass)
.deephyper.nas.space
is now deephyper.nas
deephyper.nas.operation
deephyper.nas.node
deephyper-analytics
were removed.deephyper ray-submit
deephyper ray-config
balsam-flow
, deap
.Published by Deathn0t about 3 years ago
This new release help us move toward a more stable version of DeepHyper.
Evaluator
interface with evaluator.submit/gather
deephyper.ensemble
for ensembles with uncertainty quantificationdeephyper.post
Published by Deathn0t over 3 years ago
The DeepHyper API is now fully documented at DeepHyper API
TensorFlow Probability is now part of DeepHyper default set of dependencies
It is now possible to directly submit with deephyper ray-submit ...
for DeepHyper at the ALCF. This feature is only available on ThetaGPU for now but can be extended to other systems by following this script.
The access to auto-sklearn features was changed to deephyper.sklearn
and a new documentation is available for this feature at User guide: AutoSklearn
The deephyper-analytics
command was modified and enhanced with new features. The see the full updated documentation follow DeepHyper Analytics Tools.
The topk
command is now available to have quick feedback from the results of an experiment:
$ deephyper-analytics topk combo_8gpu_8_agebo/infos/results.csv -k 2
'0':
arch_seq: '[229, 0, 22, 1, 1, 53, 29, 1, 119, 1, 0, 116, 123, 1, 273, 0, 1, 388]'
batch_size: 59
elapsed_sec: 10259.2741303444
learning_rate: 0.0001614947
loss: log_cosh
objective: 0.9236862659
optimizer: adam
patience_EarlyStopping: 22
patience_ReduceLROnPlateau: 10
'1':
arch_seq: '[229, 0, 22, 0, 1, 235, 29, 1, 313, 1, 0, 116, 123, 1, 37, 0, 1, 388]'
batch_size: 51
elapsed_sec: 8818.2674164772
learning_rate: 0.0001265946
loss: mae
objective: 0.9231553674
optimizer: nadam
patience_EarlyStopping: 23
patience_ReduceLROnPlateau: 14
A new documentation for the neural architecture search problem setup can be found here.
It is now possible to defined auto-tuned hyperparameters in addition of the architecture in a NAS Problem.
Three new algorithms are available to run a joint Hyperparameter and neural architecture search. The Hyperparameter optimisation is defined as HPO and neural architecture search as NAS.
agebo
(Aging Evolution for NAS with Bayesian Optimisation for HPO)ambsmixed
(an extension of Asynchronous Model-Based Search for HPO + NAS)regevomixed
(an extension of regularised evolution for HPO + NAS)A new run function to use data-parallelism during neural architecture search is available (link to code)
To use this function pass it to the run argument of the command line such as:
deephyper nas agebo ... --run deephyper.nas.run.tf_distributed.run ... --num-cpus-per-task 2 --num-gpus-per-task 2 --evaluator ray --address auto ...
This function allows for new hyperparameters in the Problem.hyperparameters(...)
:
...
Problem.hyperparameters(
...
lsr_batch_size=True,
lsr_learning_rate=True,
warmup_lr=True,
warmup_epochs=5,
...
)
...
The data-ingestion pipeline was better optimised to reduce the overheads on GPU instances:
self.dataset_train = (
self.dataset_train.cache()
.shuffle(self.train_size, reshuffle_each_iteration=True)
.batch(self.batch_size)
.prefetch(tf.data.AUTOTUNE)
.repeat(self.num_epochs)
)
A new method is now available from the Problem object Problem.get_keras_model(arch_seq)
to easily build a Keras model instance from an arch_seq
(list encoding a neural network).
Published by Deathn0t almost 4 years ago
Minor bug corrections
Published by Deathn0t almost 4 years ago
SpaceFactory
interface for the deepspace
package which provides ready to go neural architecture search spaces.Published by Deathn0t almost 4 years ago
ambsv1
Published by Deathn0t about 4 years ago
A release for the creation of a DOI on Zeno.
Published by Deathn0t almost 5 years ago
DeepHyper 0.1.2 is now forward-compatible with Python 3.7+ and Balsam 0.3.8+ after removing the async reserved keyword.
Published by Deathn0t about 5 years ago
This release is mostly introducing features for Neural Architecture Search with DeepHyper.
For hyperparameter search use deephyper hps ...
here is an example for the
hyperparameter polynome2 benchmark:
deephyper hps ambs --problem deephyper.benchmark.hps.polynome2.Problem --run deephyper.benchmark.hps.polynome2.run
For neural architecture search use deephyper nas ...
here is an example for the
neural architecture search linearReg benchmark:
deephyper nas regevo --problem deephyper.benchmark.nas.linearReg.Problem
Use commands such as deephyper --help
, deephyper nas --help
or deephyper nas regevo --help
to find out more about the command-line interface.
Operation
directly from tensorflow.keras.layers
:>>> import tensorflow as tf
>>> from deephyper.search.nas.model.space.node import VariableNode
>>> from deephyper.search.nas.model.space.op import Operation
>>> vnode = VariableNode()
>>> vnode.add_op(Operation(layer=tf.keras.layers.Dense(10)))
TrainerTrainValid
now has a default callback: tf.keras.callbacks.CSVLogger(...)
The ray evaluator is now available through ... --evaluator ray...
for both hyperparameter
and neural architecture search.
To use a seed for any run do Problem(seed=seed)
while creating your problem object.
Use the .. --n-jobs ...
to define how to distribute the learner computation in AMBS.
The goal of MimeNode
is to replicate the action applied to the targeted variable node.
import tensorflow as tf
from deephyper.search.nas.model.space.node import VariableNode, MimeNode
from deephyper.search.nas.model.space.op.op1d import Dense
vnode = VariableNode()
dense_10_op = Dense(10)
vnode.add_op(dense_10_op)
vnode.add_op(Dense(20))
mnode = MimeNode(vnode)
dense_30_op = Dense(30)
mnode.add_op(dense_30_op)
mnode.add_op(Dense(40))
# The first operation "Dense(10)" has been choosen
# for the mimed node: vnode
vnode.set_op(0)
assert vnode.op == dense_10_op
# mnode is miming the choice made for vnode as you can see
# the first operation was choosen as well
assert mnode.op == dense_30_op
The goal of MirroNode
is to replicate the action applied to the targeted VariableNode
,
ConstantNode
or MimeNode
.
import tensorflow as tf
from deephyper.search.nas.model.space.node import VariableNode, MirrorNode
from deephyper.search.nas.model.space.op.op1d import Dense
vnode = VariableNode()
dense_10_op = Dense(10)
vnode.add_op(dense_10_op)
vnode.add_op(Dense(20))
mnode = MirrorNode(vnode)
# The operation "Dense(10)" is being set for vnode.
vnode.set_op(0)
# The same operation (i.e. same instance) is now returned by both vnode and mnode.
assert vnode.op == dense_10_op
assert mnode.op == dense_10_op
Tensorboard and Beholder callbacks can now be used during the post-training. Beholder is
a Tensorboard which enable you to visualize the evolution of the trainable parameters of
a model during the training.
Problem.post_training(
...
callbacks=dict(
TensorBoard={
'log_dir':'tb_logs',
'histogram_freq':1,
'batch_size':64,
'write_graph':True,
'write_grads':True,
'write_images':True,
'update_freq':'epoch',
'beholder': True
})
)