FLAML

What's Changed

• Update Task-Oriented-AutoML.md by @vvijayalakshmi21 in https://github.com/microsoft/FLAML/pull/446
• Update Task-Oriented-AutoML.md by @vvijayalakshmi21 in https://github.com/microsoft/FLAML/pull/447
• Update Tune-User-Defined-Function.md by @vvijayalakshmi21 in https://github.com/microsoft/FLAML/pull/448
• corrected typo in example xgboost documentation by @MichaelMarien in https://github.com/microsoft/FLAML/pull/449
• bump ray version to 1.10 by @sonichi in https://github.com/microsoft/FLAML/pull/450
• fix a bug when using ray & update ray on aml by @sonichi in https://github.com/microsoft/FLAML/pull/455

New Contributors

• @vvijayalakshmi21 made their first contribution in https://github.com/microsoft/FLAML/pull/446

Full Changelog: https://github.com/microsoft/FLAML/compare/v0.9.6...v0.9.7

What's Changed

• reducing AutoConfig.from_pretrained by @liususan091219 in https://github.com/microsoft/FLAML/pull/411
• Set use_ray to True for logging to databricks by @liususan091219 in https://github.com/microsoft/FLAML/pull/414
• Bump nanoid from 3.1.30 to 3.2.0 in /website by @sonichi in https://github.com/microsoft/FLAML/pull/420
• bump version of node-fetch to 3.1.1 in website/ by @sonichi in https://github.com/microsoft/FLAML/pull/423
• Use Ray _BackwardsCompatibleNumpyRng if possible by @Yard1 in https://github.com/microsoft/FLAML/pull/421
• remove FLAML sample size from config by @sonichi in https://github.com/microsoft/FLAML/pull/418
• max_iter < 2 -> no search; sign in metric constraints; test and example for forecasting by @sonichi in https://github.com/microsoft/FLAML/pull/415
• remove redundant imports by @liususan091219 in https://github.com/microsoft/FLAML/pull/426
• Support time series forecasting for discrete target variable by @int-chaos in https://github.com/microsoft/FLAML/pull/416
• homepage update by @sonichi in https://github.com/microsoft/FLAML/pull/425
• fix a broken link in README.md by @m13uz in https://github.com/microsoft/FLAML/pull/439
• adding catch for HTTP error by @liususan091219 in https://github.com/microsoft/FLAML/pull/432
• Change the upper bound for "lags" hyperparameter for sklearn forecast models by @int-chaos in https://github.com/microsoft/FLAML/pull/437
• Gpu support for xgboost by @sonichi in https://github.com/microsoft/FLAML/pull/442
• data in csv by @sonichi in https://github.com/microsoft/FLAML/pull/430
• note about preview feature by @sonichi in https://github.com/microsoft/FLAML/pull/431

New Contributors

• @m13uz made their first contribution in https://github.com/microsoft/FLAML/pull/439

Full Changelog: https://github.com/microsoft/FLAML/compare/v0.9.5...v0.9.6

What's Changed

• fixing load best model at the end by @liususan091219 in https://github.com/microsoft/FLAML/pull/389
• Regression forecast debug by @int-chaos in https://github.com/microsoft/FLAML/pull/391
• set verbose for transformers by @liususan091219 in https://github.com/microsoft/FLAML/pull/392
• Logging multiple checkpoints by @liususan091219 in https://github.com/microsoft/FLAML/pull/394
• postcss version update by @sonichi in https://github.com/microsoft/FLAML/pull/385
• fixing default metric for regression + change verbosity for transformers by @liususan091219 in https://github.com/microsoft/FLAML/pull/397
• fix issues in logging, bug in space.py, constraint sign, and improve code coverage by @sonichi in https://github.com/microsoft/FLAML/pull/388
• moving intermediate_results logging from model.py to huggingface/trainer.py by @liususan091219 in https://github.com/microsoft/FLAML/pull/403
• Update flaml/nlp/README.md by @liususan091219 in https://github.com/microsoft/FLAML/pull/404
• Logo by @qingyun-wu in https://github.com/microsoft/FLAML/pull/399
• update browser icon by @qingyun-wu in https://github.com/microsoft/FLAML/pull/407
• adding logging of training loss by @liususan091219 in https://github.com/microsoft/FLAML/pull/406
• Bump shelljs from 0.8.4 to 0.8.5 in /website by @sonichi in https://github.com/microsoft/FLAML/pull/402
• Sklearn api x by @MichaelMarien in https://github.com/microsoft/FLAML/pull/405

New Contributors

• @MichaelMarien made their first contribution in https://github.com/microsoft/FLAML/pull/405

Full Changelog: https://github.com/microsoft/FLAML/compare/v0.9.4...v0.9.5

This release enables regression models for time series forecasting. It also fixes bugs in nlp tasks, such as serialization of transformer models and automatic metrics.

What's Changed

• citation file by @sonichi in https://github.com/microsoft/FLAML/pull/364
• Fix several issues for nlp tasks by @sonichi in https://github.com/microsoft/FLAML/pull/380
• serialize TransformerEstimator by @sonichi in https://github.com/microsoft/FLAML/pull/381
• Time series forecasting with sklearn regressors by @int-chaos in https://github.com/microsoft/FLAML/pull/362
• fixing auto metric bug by @liususan091219 in https://github.com/microsoft/FLAML/pull/387

Full Changelog: https://github.com/microsoft/FLAML/compare/v0.9.3...v0.9.4

What's Changed

• Finish the Multiple Choice Classification by @oberonbot in https://github.com/microsoft/FLAML/pull/367
• logging by @sonichi in https://github.com/microsoft/FLAML/pull/371
• adding token classification by @liususan091219 and @siddheshshaji in https://github.com/microsoft/FLAML/pull/376

New Contributors

• @oberonbot and @siddheshshaji made their first contribution in https://github.com/microsoft/FLAML/pull/367

Full Changelog: https://github.com/microsoft/FLAML/compare/v0.9.2...v0.9.3

New Features:

• New task: text summarization
• Reproducibility of hyperparameter search sequence
• Run flaml in azureml + ray

What's Changed

• url update for doc edit by @sonichi in https://github.com/microsoft/FLAML/pull/345
• Adding the NLP task summarization by @liususan091219 @XinZofStevens @GideonWu0105 in https://github.com/microsoft/FLAML/pull/346
• reproducibility for random sampling by @sonichi in https://github.com/microsoft/FLAML/pull/349
• doc update by @sonichi in https://github.com/microsoft/FLAML/pull/352
• azureml + ray by @sonichi in https://github.com/microsoft/FLAML/pull/344
• Fixing the bug in custom metric by @liususan091219 in https://github.com/microsoft/FLAML/pull/356
• Simplify lgbm example by @ruizhuanguw in https://github.com/microsoft/FLAML/pull/358
• fixing custom metric by @liususan091219 in https://github.com/microsoft/FLAML/pull/357
• Example by @sonichi in https://github.com/microsoft/FLAML/pull/359

New Contributors

• @ruizhuanguw @XinZofStevens @GideonWu0105 made their first contribution in https://github.com/microsoft/FLAML/pull/358

Full Changelog: https://github.com/microsoft/FLAML/compare/v0.9.1...v0.9.2

This release contains several feature improvements and bug fixes. For example,

• support for custom data splitter.
• evaluation_function can receive incumbent result in local search and perform domain-specific early stopping by comparing with the incumbent result. As long as the comparison result (better or worse) is known, the evaluation can be stopped.
• support and automate huggingface metrics.
• use cfo in tune.run if bs is not installed.
• fixed a bug in modifying n_estimators to satisfy constraints.
• new documentation website.

What's Changed

• Update flaml_pytorch_cifar10.ipynb by @sonichi in https://github.com/microsoft/FLAML/pull/328
• adding HF metrics by @liususan091219 in https://github.com/microsoft/FLAML/pull/335
• train at least one iter when not trained by @sonichi in https://github.com/microsoft/FLAML/pull/336
• use cfo in tune.run if bs is not installed by @sonichi in https://github.com/microsoft/FLAML/pull/334
• Makes the evaluation_function could receive the incumbent best result as input in Tune by @Shao-kun-Zhang in https://github.com/microsoft/FLAML/pull/339
• support for customized splitters by @wuchihsu in https://github.com/microsoft/FLAML/pull/333
• Deploy a new doc website by @sonichi, @qingyun-wu and @Shao-kun-Zhang in https://github.com/microsoft/FLAML/pull/338
• version update by @sonichi in https://github.com/microsoft/FLAML/pull/341

New Contributors

• @Shao-kun-Zhang made their first contribution in https://github.com/microsoft/FLAML/pull/339

Full Changelog: https://github.com/microsoft/FLAML/compare/v0.9.0...v0.9.1

1. Revise flaml.tune API

• Add a “scheduler” argument (a user can choose from “flaml”, “asha” or a customized scheduler)
• Rename "prune_attr" to "resource_attr"
• Rename “training_function” to “evaluation_function”
• Remove the “report_intermediate_result” argument (covered by “scheduler” instead)
• Add tests for the supported schedulers
• Re-run notebooks that use schedulers

2. Add save_best_config() to save best config in a json file

What's Changed

• add save_best_config() by @sonichi in https://github.com/microsoft/FLAML/pull/324
• tune api for schedulers by @qingyun-wu in https://github.com/microsoft/FLAML/pull/322
• add init.py in nlp by @sonichi in https://github.com/microsoft/FLAML/pull/325
• rename training_function by @qingyun-wu in https://github.com/microsoft/FLAML/pull/327

Full Changelog: https://github.com/microsoft/FLAML/compare/v0.8.2...v0.9.0

What's Changed

• include default value in rf search space by @sonichi in https://github.com/microsoft/FLAML/pull/317
• adding TODOs for NLP module, so students can implement other tasks easier by @liususan091219 in https://github.com/microsoft/FLAML/pull/321
• pred_time_limit clarification and logging by @sonichi in https://github.com/microsoft/FLAML/pull/319
• bug fix in confg2params by @sonichi in https://github.com/microsoft/FLAML/pull/323

Full Changelog: https://github.com/microsoft/FLAML/compare/v0.8.1...v0.8.2

What's Changed

• Update test_regression.py by @fengsxy in https://github.com/microsoft/FLAML/pull/306
• Add conda forge minimal test by @MichalChromcak in https://github.com/microsoft/FLAML/pull/309
• fixing config2params for transformersestimator by @liususan091219 in https://github.com/microsoft/FLAML/pull/316
• Code quality improvement based on #275 by @abnsy and @sonichi in https://github.com/microsoft/FLAML/pull/313
• skip cv preparation if eval_method is holdout by @sonichi in https://github.com/microsoft/FLAML/pull/314

New Contributors

• @fengsxy made their first contribution in https://github.com/microsoft/FLAML/pull/306
• @abnsy made their first contribution in https://github.com/microsoft/FLAML/pull/313

Full Changelog: https://github.com/microsoft/FLAML/compare/v0.8.0...v0.8.1

In this release, we add two nlp tasks: sequence classification and sequence regression to flaml.AutoML, using transformer-based neural networks. Previously the nlp module was detached from flaml.AutoML with a separate API. We redesigned the API such that the nlp tasks can be accessed from the same API as other tasks, and adding more nlp tasks in future would be easy. Thanks for the hard work @liususan091219 !

We've also continued to make more performance & feature improvements. Examples:

• We added a variation of XGBoost search space which uses limited max_depth. It includes the default configuration from XGBoost library. The new search space leads to significantly better performance for some regression datasets.
• We allow arguments for flaml.AutoML to be passed to the constructor. This enables multioutput regression by combining sklearn's MultioutputRegressor and flaml's AutoML.
• We made more memory optimization, while allowing users to keep the best model per estimator in memory through the "model_history" option.

What's Changed

• Unify regression and classification for XGBoost by @sonichi in https://github.com/microsoft/FLAML/pull/276
• when max_iter=1, skip search only if retrain_final by @sonichi in https://github.com/microsoft/FLAML/pull/280
• example update by @sonichi in https://github.com/microsoft/FLAML/pull/281
• Merge exp into flaml by @liususan091219 in https://github.com/microsoft/FLAML/pull/210
• add best_loss_per_estimator by @qingyun-wu in https://github.com/microsoft/FLAML/pull/286
• model_history -> save_best_model_per_estimator by @sonichi in https://github.com/microsoft/FLAML/pull/283
• datetime feature engineering by @sonichi in https://github.com/microsoft/FLAML/pull/285
• add warmstart test by @qingyun-wu in https://github.com/microsoft/FLAML/pull/298
• empty search space by @sonichi in https://github.com/microsoft/FLAML/pull/295
• multioutput regression by @sonichi in https://github.com/microsoft/FLAML/pull/292
• add max_depth to xgboost search space by @sonichi in https://github.com/microsoft/FLAML/pull/282
• custom metric function clarification by @sonichi in https://github.com/microsoft/FLAML/pull/300
• checkpoint naming in nonray mode, fix ray mode, delete checkpoints in nonray mode by @liususan091219 in https://github.com/microsoft/FLAML/pull/293

Full Changelog: https://github.com/microsoft/FLAML/compare/v0.7.1...v0.8.0

What's Changed

• make default verbose level > 0 when using ray by @sonichi in https://github.com/microsoft/FLAML/pull/272
• default to cfo for single estimator by @sonichi in https://github.com/microsoft/FLAML/pull/273
• update docstr by @sonichi and @qingyun-wu in https://github.com/microsoft/FLAML/pull/274
• fixed a bug in #278 by @sonichi in https://github.com/microsoft/FLAML/pull/274

Full Changelog: https://github.com/microsoft/FLAML/compare/v0.7.0...v0.7.1

New feature: multivariate time series forecasting.

What's Changed

• Fix exception in CFO's _create_condition if all candidate start points didn't return yet by @Yard1 in https://github.com/microsoft/FLAML/pull/263
• Integrate multivariate time series forecasting by @int-chaos in https://github.com/microsoft/FLAML/pull/254
• Update Dockerfile by @wuchihsu in https://github.com/microsoft/FLAML/pull/269
• limit time and memory consumption by @sonichi in https://github.com/microsoft/FLAML/pull/264

New Contributors

• @wuchihsu made their first contribution in https://github.com/microsoft/FLAML/pull/269

Full Changelog: https://github.com/microsoft/FLAML/compare/v0.6.9...v0.7.0

What's Changed

• no search when max_iter < 2 by @sonichi in https://github.com/microsoft/FLAML/pull/253
• fix bug in subspace identification by @sonichi in https://github.com/microsoft/FLAML/pull/257
• Add documentation for warm-start by @qingyun-wu in https://github.com/microsoft/FLAML/pull/255

Full Changelog: https://github.com/microsoft/FLAML/compare/v0.6.8...v0.6.9

What's Changed

• fix the bug in hierarchical search space (#248); make dependency on lgbm and xgboost optional (#252) by @sonichi in https://github.com/microsoft/FLAML/pull/250
• Add conda forge badge by @MichalChromcak in https://github.com/microsoft/FLAML/pull/251

New Contributors

• @MichalChromcak made their first contribution in https://github.com/microsoft/FLAML/pull/251

Full Changelog: https://github.com/microsoft/FLAML/compare/v0.6.7...v0.6.8

What's Changed

• remove big objects after fit by @sonichi in https://github.com/microsoft/FLAML/pull/176
• remove catboost training dir by @sonichi in https://github.com/microsoft/FLAML/pull/178
• Forecast v2 by @int-chaos in https://github.com/microsoft/FLAML/pull/182
• Fix decide_split_type bug. by @gianpDomiziani in https://github.com/microsoft/FLAML/pull/184
• Cleanml by @qingyun-wu in https://github.com/microsoft/FLAML/pull/185
• warmstart blendsearch by @sonichi in https://github.com/microsoft/FLAML/pull/186
• variable name by @sonichi in https://github.com/microsoft/FLAML/pull/187
• notebook example by @sonichi in https://github.com/microsoft/FLAML/pull/189
• make flaml work without catboost by @sonichi in https://github.com/microsoft/FLAML/pull/197
• package name in setup by @sonichi in https://github.com/microsoft/FLAML/pull/198
• clean up forecast notebook by @sonichi in https://github.com/microsoft/FLAML/pull/202
• consider num_samples in bs thread priority by @sonichi in https://github.com/microsoft/FLAML/pull/207
• accommodate nni usage pattern by @sonichi in https://github.com/microsoft/FLAML/pull/209
• random search by @sonichi in https://github.com/microsoft/FLAML/pull/213
• add consistency test by @qingyun-wu in https://github.com/microsoft/FLAML/pull/216
• set converge flag when no trial can be sampled by @sonichi in https://github.com/microsoft/FLAML/pull/217
• seed for hpo method by @sonichi in https://github.com/microsoft/FLAML/pull/224
• update config if n_estimators is modified by @sonichi in https://github.com/microsoft/FLAML/pull/225
• warning -> info for low cost partial config by @sonichi in https://github.com/microsoft/FLAML/pull/231
• Consistent California by @cdeil in https://github.com/microsoft/FLAML/pull/245
• Package by @sonichi in https://github.com/microsoft/FLAML/pull/244

New Contributors

• @cdeil made their first contribution in https://github.com/microsoft/FLAML/pull/245

Full Changelog: https://github.com/microsoft/FLAML/compare/v0.6.0...v0.6.7

What's Changed

• remove big objects after fit by @sonichi in https://github.com/microsoft/FLAML/pull/176
• remove catboost training dir by @sonichi in https://github.com/microsoft/FLAML/pull/178
• Forecast v2 by @int-chaos in https://github.com/microsoft/FLAML/pull/182
• Fix decide_split_type bug. by @gianpDomiziani in https://github.com/microsoft/FLAML/pull/184
• Cleanml by @qingyun-wu in https://github.com/microsoft/FLAML/pull/185
• warmstart blendsearch by @sonichi in https://github.com/microsoft/FLAML/pull/186
• variable name by @sonichi in https://github.com/microsoft/FLAML/pull/187
• notebook example by @sonichi in https://github.com/microsoft/FLAML/pull/189
• make flaml work without catboost by @sonichi in https://github.com/microsoft/FLAML/pull/197
• package name in setup by @sonichi in https://github.com/microsoft/FLAML/pull/198
• clean up forecast notebook by @sonichi in https://github.com/microsoft/FLAML/pull/202
• consider num_samples in bs thread priority by @sonichi in https://github.com/microsoft/FLAML/pull/207
• accommodate nni usage pattern by @sonichi in https://github.com/microsoft/FLAML/pull/209
• random search by @sonichi in https://github.com/microsoft/FLAML/pull/213
• add consistency test by @qingyun-wu in https://github.com/microsoft/FLAML/pull/216
• set converge flag when no trial can be sampled by @sonichi in https://github.com/microsoft/FLAML/pull/217
• seed for hpo method by @sonichi in https://github.com/microsoft/FLAML/pull/224
• update config if n_estimators is modified by @sonichi in https://github.com/microsoft/FLAML/pull/225
• warning -> info for low cost partial config by @sonichi in https://github.com/microsoft/FLAML/pull/231

Full Changelog: https://github.com/microsoft/FLAML/compare/v0.6.0...v0.6.6

In this release, we added support for time series forecasting task and NLP model fine tuning. Also, we have made a large number of feature & performance improvements.

• data split by 'time' for time-ordered data, and by 'group' for grouped data.
• support parallel trials and random search in AutoML.fit() API.
• support warm-start in AutoML.fit() by using previously found start points.
• support constraints on training/prediction time per model.
• new optimization metric: ROC_AUC for multi-class classification, MAPE for time series forecasting.
• utility functions for getting normalized confusion matrices and multi-class ROC or precision-recall curves.
• automatically retrain models after search by default; options to disable retraining or enforce time limit.
• CFO supports hierarchical search space and uses points_to_evaluate more effectively.
• variation of CFO optimized for unordered categorical hps.
• BlendSearch improved for better performance in parallel setting.
• memory overhead optimization.
• search space improvements for random forest and lightgbm.
• make stacking ensemble work for categorical features.
• python 3.9 support.
• experimental support for automated fine-tuning of transformer models from huggingface.
• experimental support for time series forecasting.
• warnings to suggest increasing time budget, and warning to inform users there is no performance improvement for a long time.

Minor updates

• make log file name optional.
• notebook for time series forecasting.
• notebook for using AutoML in sklearn pipeline.
• bug fix when training_function returns a value.
• support fixed random seeds to improve reproducibility.
• code coverage improvement.
• exclusive upper bounds for hyperparameter type randint and lograndint.
• experimental features in BlendSearch.
• documentation improvement.
• bug fixes for multiple logged metrics in cv.
• adjust epsilon when time per trial is very fast.

Contributors

• @sonichi
• @qingyun-wu
• @int-chaos
• @liususan091219
• @Yard1
• @bnriiitb
• @su2umaru
• @eduardobull
• @sek788432
• @ekzhu
• @anshumandutt
• @yue-msr
• @sadtaf
• @fzanartu
• @dsbyprateekg
• @hanhanwu
• @PardeepRassani
• @gianpDomiziani
• @stepthom
• @anhnht3
• @zzheng93
• @flippercy
• @luizhemelo
• @nabalamu
• @lostmygithubaccount
• @suryajayaraman

Major update:

• Online automl. For example, we support tuning online machine learning library vowpal wabbit.

Minor updates:

• log best model in mlflow
• utility functions to produce normalized confusion matrix and roc or precision-recall curves for each class in multi-class tasks

Support for general config constraints and metric constraints in hyperparameter tuning