Releases: autogluon/autogluon
v0.6.0
Version 0.6.0
We're happy to announce the AutoGluon 0.6 release. 0.6 contains major enhancements to Tabular, Multimodal, and Time Series
modules, along with many quality of life improvements and fixes.
As always, only load previously trained models using the same version of AutoGluon that they were originally trained on.
Loading models trained in different versions of AutoGluon is not supported.
This release contains 263 commits from 25 contributors!
See the full commit change-log here: v0.5.2...v0.6.0
Special thanks to @cheungdaven, @suzhoum, @BingzhaoZhu, @liangfu, @Harry-zzh, @gidler, @yongxinw, @martinschaef,
@giswqs, @Jalagarto, @geoalgo, @lujiaying and @leloykun who were first time contributors to AutoGluon this release!
Full Contributor List (ordered by # of commits):
@shchur, @yinweisu, @zhiqiangdon, @Innixma, @FANGAreNotGnu, @canerturkmen, @sxjscience, @gradientsky, @cheungdaven,
@bryanyzhu, @suzhoum, @BingzhaoZhu, @yongxinw, @tonyhoo, @liangfu, @Harry-zzh, @Raldir, @gidler, @martinschaef,
@giswqs, @Jalagarto, @geoalgo, @lujiaying, @leloykun, @yiqings
This version supports Python versions 3.7 to 3.9. This is the last release that will support Python 3.7.
Changes
AutoMM
AutoGluon Multimodal (a.k.a AutoMM) supports three new features: 1) object detection, 2) named entity recognition, and 3) multimodal matching. In addition, the HPO backend of AutoGluon Multimodal has been upgraded to ray 2.0. It also supports fine-tuning billion-scale FLAN-T5-XL model on a single AWS g4.2x-large instance with improved parameter-efficient finetuning. Starting from 0.6, we recommend using autogluon.multimodal rather than autogluon.text or autogluon.vision and added deprecation warnings.
New features
-
Object Detection
- Add new problem_type
"object_detection"
. - Customers can run inference with pretrained object detection models and train their own model with three lines of code.
- Integrate with open-mmlab/mmdetection, which supports classic detection architectures like Faster RCNN, and more efficient and performant architectures like YOLOV3 and VFNet.
- See tutorials and examples for more detail.
- Contributors and commits: @FANGAreNotGnu, @bryanyzhu, @zhiqiangdon, @yongxinw, @sxjscience, @Harry-zzh (#2025, #2061, #2131, #2181, #2196, #2215, #2244, #2265, #2290, #2311, #2312, #2337, #2349, #2353, #2360, #2362, #2365, #2380, #2381, #2391, #2393, #2400, #2419, #2421, #2063, #2104, #2411)
- Add new problem_type
-
Named Entity Recognition
- Add new problem_type
"ner"
. - Customers can train models to extract named entities with three lines of code.
- The implementation supports any backbones in huggingface/transformer, including the recently FLAN-T5 series released by Google.
- See tutorials for more detail.
- Contributors and commits: @cheungdaven (#2183, #2232, #2220, #2282, #2295, #2301, #2337, #2346, #2361, #2372, #2394, #2412)
- Add new problem_type
-
Multimodal Matching
- Add new problem_type
"text_similarity"
,"image_similarity"
,"image_text_similarity"
. - Users can now extract semantic embeddings with pretrained models for text-text, image-image, and text-image matching problems.
- Moreover, users can further finetune these models with relevance data.
- The semantic text embedding model can also be combined with BM25 to form a hybrid indexing solution.
- Internally, AutoGluon Multimodal implements a twin-tower architecture that is flexible in the choice of backbones for each tower. It supports image backbones in TIMM, text backbones in huggingface/transformers, and also the CLIP backbone.
- See tutorials for more detail.
- Contributors and commits: @zhiqiangdon @FANGAreNotGnu @cheungdaven @suzhoum @sxjscience @bryanyzhu (#1975, #1994, #2142, #2179, #2186, #2217, #2235, #2284, #2297, #2313, #2326, #2337, #2347, #2357, #2358, #2362, #2363, #2375, #2378, #2404, #2416, #2407, #2417)
- Add new problem_type
-
Miscellaneous minor fixes. @cheungdaven @FANGAreNotGnu @geoalgo @zhiqiangdon (#2402, #2409, #2026, #2401, #2418)
Other Enhancements
- Fix the FT-Transformer implementation and support Fastformer. @BingzhaoZhu @yiqings (#1958, #2194, #2251, #2344, #2379, #2386)
- Support finetuning billion-scale FLAN-T5-XL in a single AWS g4.2x-large instance via improved parameter-efficient finetuning. See tutorial. @Raldir @sxjscience (#2032, #2108, #2285, #2336, #2352)
- Upgrade multimodal HPO to use ray 2.0 and also add new tutorial. @yinweisu @suzhoum @bryanyzhu (#2206, #2341)
- Further improvement on model distillation. Add example and tutorial. @FANGAreNotGnu @sxjscience (#1983, #2064, #2397)
- Revise the default presets of AutoMM for image classification problems. @bryanyzhu (#2351)
- Support backend=“automm” in autogluon.vision. @bryanyzhu (#2316)
- Add deprecated warning to autogluon.vision and autogluon.text and point the usage to autogluon.multimodal. @bryanyzhu @sxjscience (#2268, #2315)
- Examples about Kaggle: Feedback Prize prediction competition. We created a solution with AutoGluon Multimodal that obtained 152/1557 in the public leaderboard and 170/1557 in the private leaderboard, which is among the top 12% participants. The solution is public days before the DDL of the competition and obtained more than 3000 views. @suzhoum @MountPOTATO (#2129, #2168, #2333)
- Improve native inference speed. @zhiqiangdon (#2051, #2157, #2161, #2171)
- Other improvements, security/bug fixes. @zhiqiangdon @sxjscience @FANGAreNotGnu, @yinweisu @Innixma @tonyhoo @martinschaef @giswqs @tonyhoo (#1980, #1987, #1989, #2003, #2080, #2018, #2039, #2058, #2101, #2102, #2125, #2135, #2136, #2140, #2141, #2152, #2164, #2166, #2192, #2219, #2250, #2257, #2280, #2308, #2315, #2317, #2321, #2356, #2388, #2392, #2413, #2414, #2417, #2426, #2028, #2382, #2415, #2193, #2213, #2230)
- CI improvements. @yinweisu (#1965, #1966, #1972, #1991, #2002, #2029, #2137, #2151, #2156, #2163, #2191, #2214, #2369, #2113, #2118)
Experimental Features
- Support 11B-scale model finetuning with DeepSpeed. @Raldir (#2032)
- Enable few-shot learning with 11B-scale model. @Raldir (#2197)
- ONNX export example of hf_text model. @FANGAreNotGnu (#2149)
Tabular
New features
-
New experimental model
FT_TRANSFORMER
. @BingzhaoZhu, @Innixma (#2085, #2379, #2389, #2410)- You can access it via specifying the
FT_TRANSFORMER
key
in thehyperparameters
dictionary or viapresets="experimental_best_quality"
. - It is recommended to use GPU to train this model, but CPU training is also supported.
- If given enough training time, this model generally improves the ensemble quality.
- You can access it via specifying the
-
New experimental model compilation support via
predictor.compile_models()
. @liangfu, @Innixma (#2225, #2260, #2300)- Currently only Random Forest and Extra Trees have compilation support.
- You will need to install extra dependencies for this to work:
pip install autogluon.tabular[all,skl2onnx]
. - Compiling models dramatically speeds up inference time (~10x) when processing small batches of samples (<10000).
- Note that a known bug exists in the current implementation: Refitting models after compilation will fail
and cause a crash. To avoid this, ensure that.compile_models
is called only at the very end.
-
Added
predictor.clone(...)
method to allow perfectly cloning a predictor object to a new directory.
This is useful to preserve the state of a predictor prior to altering it
(such as prior to calling.save_space
,.distill
,.compile_models
, or.refit_full
. @Innixma (#2071) -
Added simplified
num_gpus
andnum_cpus
arguments topredictor.fit
to control total resources.
@yinweisu, @Innixma (#2263) -
Improved stability and effectiveness of HPO functionality via various refactors regarding our usage of ray.
@yinweisu, @Innixma (#1974, #1990, #2094, #2121, #2133, #2195, #2253, #2263, #2330) -
Upgraded dependency versions: XGBoost 1.7, CatBoost 1.1, Scikit-learn 1.1, Pandas 1.5, Scipy 1.9, Numpy 1.23.
@Innixma (#2373) -
Added python version compatibility check when loading a fitted TabularPredictor.
Will now error if python versions are incompatible. @Innixma (#2054) -
Added
fit_weighted_ensemble
argument topredictor.fit
. This allows the user to disable the weighted ensemble.
@Innixma (#2145)
Other Enhancements
- Improved logging clarity when using
infer_limit
. @Innixma (#2014) - Significantly improved HPO search space of XGBoost. @Innixma (#2123)
- Fixed HPO crashing when tuning Random Forest, Extra Trees, or KNN. @Innixma (#2070)
- Optimized roc_auc metric scoring speed by 7x. @Innixma (#2318, #2331)
- Fixed bug with AutoMM Tabular model crashing if not trained last. @Innixma (#2309)
- Refactored
Scorer
classes to be easier to use, plus added comprehensive unit tests for all metrics. @Innixma (#2242) - ...
v0.5.2
Version 0.5.2
v0.5.2 is a security hotfix release.
This release is non-breaking when upgrading from v0.5.0. As always, only load previously trained models using the same version of AutoGluon that they were originally trained on. Loading models trained in different versions of AutoGluon is not supported.
See the full commit change-log here: v0.5.1...v0.5.2
This version supports Python versions 3.7 to 3.9.
v0.4.3
Version 0.4.3
v0.4.3 is a security hotfix release.
This release is non-breaking when upgrading from v0.4.0. As always, only load previously trained models using the same version of AutoGluon that they were originally trained on. Loading models trained in different versions of AutoGluon is not supported.
See the full commit change-log here: v0.4.2...v0.4.3
This version supports Python versions 3.7 to 3.9.
v0.5.1
Version 0.5.1
We're happy to announce the AutoGluon 0.5 release. This release contains major optimizations and bug fixes to autogluon.multimodal and autogluon.timeseries modules, as well as inference speed improvements to autogluon.tabular.
This release is non-breaking when upgrading from v0.5.0. As always, only load previously trained models using the same version of AutoGluon that they were originally trained on. Loading models trained in different versions of AutoGluon is not supported.
This release contains 58 commits from 14 contributors!
Full Contributor List (ordered by # of commits):
- @zhiqiangdon, @yinweisu, @Innixma, @canerturkmen, @sxjscience, @bryanyzhu, @jsharpna, @gidler, @gradientsky, @Linuxdex, @muxuezi, @yiqings, @huibinshen, @FANGAreNotGnu
This version supports Python versions 3.7 to 3.9.
See the full commit change-log here: v0.5.0...v0.5.1
AutoMM
Changed to a new namespace autogluon.multimodal
(AutoMM), which is a deep learning "model zoo" of model zoos. On one hand, AutoMM can automatically train deep models for unimodal (image-only, text-only or tabular-only) problems. On the other hand, AutoMM can automatically solve multimodal (any combinations of image, text, and tabular) problems by fusing multiple deep learning models. In addition, AutoMM can be used as a base model in AutoGluon Tabular and participate in the model ensemble.
New features
-
Supported zero-shot learning with CLIP (#1922) @zhiqiangdon
- Users can directly perform zero-shot image classification with the CLIP model. Moreover, users can extract image and text embeddings with CLIP to do image-to-text or text-to-image retrieval.
-
Improved efficient finetuning
- Support “bit_fit”, “norm_fit“, “lora”, “lora_bias”, “lora_norm”. In four multilingual datasets (xnli, stsb_multi_mt, paws-x, amazon_reviews_multi), “lora_bias”, which is a combination of LoRA and BitFit, achieved the best overall performance. Compared to finetuning the whole network, “lora_bias” will only finetune <0.5% of the network parameters and can achieve comparable performance on “stsb_multi_mt” (#1780, #1809). @Raldir @zhiqiangdon
- Support finetuning the mT5-XL model that has 1.7B parameters on a single NVIDIA G4 GPU. In AutoMM, we only use the T5-encoder (1.7B parameters) like Sentence-T5. (#1933) @sxjscience
-
Added more data augmentation techniques
- Mixup for image data. (#1730) @Linuxdex
- TrivialAugment for both image and text data. (#1792) @lzcemma
- Easy text augmentations. (#1756) @lzcemma
-
Enhanced teacher-student model distillation
- Support distilling the knowledge from a unimodal/multimodal teacher model to a student model. (#1670, #1895) @zhiqiangdon
More tutorials and examples
-
Beginner tutorials of applying AutoMM to image, text, or multimodal (including tabular) data. (#1861, #1908, #1858, #1869) @bryanyzhu @sxjscience @zhiqiangdon
-
A zero-shot image classification tutorial with the CLIP model. (#1942) @bryanyzhu
-
A tutorial of using CLIP model to extract embeddings for image-text retrieval. (#1957) @bryanyzhu
-
A tutorial to introduce comprehensive AutoMM configurations (#1861). @zhiqiangdon
-
AutoMM for tabular data examples (#1752, #1893, #1903). @yiqings
-
A Kaggle notebook about how to use AutoMM to predict pet adoption: https://www.kaggle.com/code/linuxdex/use-autogluon-to-predict-pet-adoption. The model achieves the score equivalent to top 1% (20th/3537) in this kernel-only competition (test data is only available in the kernel without internet access) (#1796, #1847, #1894, #1943). @Linuxdex
v0.5.0
We're happy to announce the AutoGluon 0.5 release. This release contains major new modules autogluon.timeseries
and autogluon.multimodal
. In collaboration with the Yu Group of Statistics and EECS from UC Berkeley, we have added interpretable models (imodels) to autogluon.tabular
.
This release is non-breaking when upgrading from v0.4.2. As always, only load previously trained models using the same version of AutoGluon that they were originally trained on. Loading models trained in different versions of AutoGluon is not supported.
This release contains 91 commits from 13 contributors!
Full Contributor List (ordered by # of commits):
- @Innixma, @canerturkmen, @zhiqiangdon, @sxjscience, @yinweisu, @Linuxdex, @yiqings, @gradientsky, @csinva, @FANGAreNotGnu, @huibinshen, @Raldir, @lzcemma
The imodels integration is based on the following work,
Singh, C., Nasseri, K., Tan, Y.S., Tang, T. and Yu, B., 2021. imodels: a python package for fitting interpretable models. Journal of Open Source Software, 6(61), p.3192.
This version supports Python versions 3.7 to 3.9.
See the full commit change-log here: v0.4.1...v0.5.0
Full release notes will be available shortly.
v0.4.2
Version 0.4.2
v0.4.2 is a hotfix release to fix breaking change in protobuf.
This release is non-breaking when upgrading from v0.4.0. As always, only load previously trained models using the same version of AutoGluon that they were originally trained on. Loading models trained in different versions of AutoGluon is not supported.
See the full commit change-log here: v0.4.1...v0.4.2
This version supports Python versions 3.7 to 3.9.
v0.4.1
Version 0.4.1
We're happy to announce the AutoGluon 0.4.1 release. 0.4.1 contains minor enhancements to Tabular, Text, Image, and Multimodal modules, along with many quality of life improvements and fixes.
This release is non-breaking when upgrading from v0.4.0. As always, only load previously trained models using the same version of AutoGluon that they were originally trained on. Loading models trained in different versions of AutoGluon is not supported.
This release contains 55 commits from 10 contributors!
See the full commit change-log here: v0.4.0...v0.4.1
Special thanks to @yiqings, @leandroimail, @huibinshen who were first time contributors to AutoGluon this release!
Full Contributor List (ordered by # of commits):
- @Innixma, @zhiqiangdon, @yinweisu, @sxjscience, @yiqings, @gradientsky, @willsmithorg, @canerturkmen, @leandroimail, @huibinshen.
This version supports Python versions 3.7 to 3.9.
Changes
AutoMM
New features
-
Added
optimization.efficient_finetune
flag to support multiple efficient finetuning algorithms. (#1666) @sxjscience- Supported options:
bit_fit
: "BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models"norm_fit
: An extension of the algorithm in "Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs" and BitFit. We finetune both the parameters in the norm layers as long as the biases.
- Supported options:
-
Enabled knowledge distillation for AutoMM (#1670) @zhiqiangdon
- Distillation API for
AutoMMPredictor
reuses the.fit()
function:
from autogluon.text.automm import AutoMMPredictor teacher_predictor = AutoMMPredictor(label="label_column").fit(train_data) student_predictor = AutoMMPredictor(label="label_column").fit( train_data, hyperparameters=student_and_distiller_hparams, teacher_predictor=teacher_predictor, )
- Distillation API for
-
Option to turn on returning feature column information (#1711) @zhiqiangdon
- The feature column information is turned on for feature column distillation; for other cases it is turned off by default to reduce dataloader‘s latency.
- Added a
requires_column_info
flag in data processors and a utility function to turn this flag on or off.
-
FT-Transformer implementation for tabular data in AutoMM (#1646) @yiqings
- Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, Artem Babenko, "Revisiting Deep Learning Models for Tabular Data" 2022. (arxiv, official implementation)
-
Make CLIP support multiple images per sample (#1606) @zhiqiangdon
- Added multiple images support for CLIP. Improved data loader robustness: added missing images handling to prevent training crashes.
- Added the choice of using a zero image if an image is missing.
-
Avoid using
eos
as the sep token for CLIP. (#1710) @zhiqiangdon -
Update fusion transformer in AutoMM (#1712) @yiqings
- Support constant learning rate in
polynomial_decay
scheduler. - Update
[CLS]
token in numerical/categorical transformer.
- Support constant learning rate in
-
Added more image augmentations:
verticalflip
,colorjitter
,randomaffine
(#1719) @Linuxdex, @sxjscience -
Added prompts for the percentage of missing images during image column detection. (#1623) @zhiqiangdon
-
Support
average_precision
in AutoMM (#1697) @sxjscience -
Convert
roc_auc
/average_precision
tolog_loss
for torchmetrics (#1715) @zhiqiangdontorchmetrics.AUROC
requires both positive and negative examples are available in a mini-batch. When training a large model, the per gpu batch size is probably small, leading to an incorrectroc_auc
score. Conversion fromroc_auc
tolog_loss
improves training stablility.
-
Added
pytorch-lightning
1.6 support (#1716) @sxjscience
Checkpointing and Model Outputs Changes
-
Updated the names of top-k checkpoint average methods and support customizing model names for terminal input (#1668) @zhiqiangdon
- Following paper: https://arxiv.org/pdf/2203.05482.pdf to update top-k checkpoint average names:
union_soup
->uniform_soup
andbest_soup
->best
. - Update function names (
customize_config_names
->customize_model_names
andverify_config_names
->verify_model_names
) to make it easier to understand them. - Support customizing model names for the terminal input.
- Following paper: https://arxiv.org/pdf/2203.05482.pdf to update top-k checkpoint average names:
-
Implemented the GreedySoup algorithm proposed in paper. Added
union_soup
,greedy_soup
,best_soup
flags and changed the default value correspondingly. (#1613) @sxjscience -
Updated the
standalone
flag inautomm.predictor.save()
to save the pertained model for offline deployment (#1575) @yiqings -
Simplified checkpoint template (#1636) @zhiqiangdon
- Stopped using pytorch lightning's model checkpoint template in saving
AutoMMPredictor
's final model checkpoint. - Improved the logic of continuous training. We pass the
ckpt_path
argument to pytorch lightning's trainer only whenresume=True
.
- Stopped using pytorch lightning's model checkpoint template in saving
-
Unified AutoMM's model output format and support customizing model names (#1643) @zhiqiangdon
- Now each model's output is dictionary with the model prefix as the first level key. The format is uniform between single model and fusion model.
- Now users can customize model names by using the internal registered names (
timm_image
,hf_text
,clip
,numerical_mlp
,categorical_mlp
, andfusion_mlp
) as prefixes. This is helpful when users want to simultaneously use two models of the same type, e.g.,hf_text
. They can just use nameshf_text_0
andhf_text_1
.
-
Support
standalone
feature inTextPredictor
(#1651) @yiqings -
Fixed saving and loading tokenizers and text processors (#1656) @zhiqiangdon
- Saved pre-trained huggingface tokenizers separately from the data processors.
- This change is backwards-compatibile with checkpoints saved by verison
0.4.0
.
-
Change load from a classmethod to staticmethod to avoid incorrect usage. (#1697) @sxjscience
-
Added
AutoMMModelCheckpoint
to avoid evaluating the models to obtain the scores (#1716) @sxjscience- checkpoint will save the best_k_models into a yaml file so that it can be loaded later to determine the path to model checkpoints.
-
Extract column features from AutoMM's model outputs (#1718) @zhiqiangdon
- Add one util function to extract column features for both image and text.
- Support extracting column features for models
timm_image
,hf_text
, andclip
.
-
Make AutoMM dataloader return feature column information (#1710) @zhiqiangdon
Bug fixes
-
Fixed calling
save_pretrained_configs
inAutoMMPrediction.save(standalone=True)
when no fusion model exists (here) (#1651) @yiqings -
Fixed error raising for setting key that does not exist in the configuration (#1613) @sxjscience
-
Fixed warning message about bf16. (#1625) @sxjscience
-
Fixed the corner case of calculating the gradient accumulation step (#1633) @sxjscience
-
Fixes for top-k averaging in the multi-gpu setting (#1707) @zhiqiangdon
Tabular
-
Limited RF
max_leaf_nodes
to 15000 (previously uncapped) (#1717) @Innixma- Previously, for very large datasets RF/XT memory and disk usage would quickly become unreasonable. This ensures that at a certain point RF and XT will no longer become larger given more rows of training data. Benchmark results showed that the change is an improvement, particularly for the
high_quality
preset.
- Previously, for very large datasets RF/XT memory and disk usage would quickly become unreasonable. This ensures that at a certain point RF and XT will no longer become larger given more rows of training data. Benchmark results showed that the change is an improvement, particularly for the
-
Limit KNN to 32 CPUs to avoid OpenBLAS error (#1722) @Innixma
- Issue #1020. When training K-nearest-neighbors (KNN) models, sometimes a rare error can occur that crashes the entire process:
BLAS : Program is Terminated. Because you tried to allocate too many memory regions. Segmentation fault: 11
This error occurred when the machine had many CPU cores (>64 vCPUs) due to too many threads being created at once. By limiting to 32 cores used, the error is avoided.
-
Added
get_results
andmodel_base_kwargs
(#1618) @Innixma- Added
get_results
to searchers, useful for debugging and for future extensions to HPO functionality.
Added new way to init aBaggedEnsembleModel
that avoids having to init the base model prior to initing the bagged ensemble model.
- Added
-
Update resource logic in models (#1689) @Innixma
- Previous implementation would crash if user specified
auto
for resources, fixed in this PR. - Added
get_minimum_resources
to explicitly define minimum resource requirements within a method.
- Previous implementation would crash if user specified
-
Updated feature importance default
subsample_size
1000 -> 5000,num_shuffle_sets 3
-> 5 (#1708) @Innixma- This will improve the quality of the feature importance values by default, especially the 99% confidence bounds. The change increases the time taken by ~8x, but this is acceptable because of the numerous inference speed optimizations done since these defaults were first introduced.
-
Added notice to ensure serializable custom metrics (#1705) @Innixma
Bug fixes
v0.4.0
We're happy to announce the AutoGluon 0.4 release. 0.4 contains major enhancements to Tabular and Text modules, along with many quality of life improvements and fixes.
This release is non-breaking when upgrading from v0.3.1. As always, only load previously trained models using the same version of AutoGluon that they were originally trained on. Loading models trained in different versions of AutoGluon is not supported.
This release contains 151 commits from 14 contributors!
See the full commit change-log here: v0.3.1...v0.4.0
Special thanks to @zhiqiangdon, @willsmithorg, @DolanTheMFWizard, @truebluejason, @killerSwitch, and @Xilorole who were first time contributors to AutoGluon this release!
Full Contributor List (ordered by # of commits):
- @Innixma, @yinweisu, @gradientsky, @zhiqiangdon, @jwmueller, @willsmithorg, @sxjscience, @DolanTheMFWizard, @truebluejason, @taesup-aws, @Xilorole, @mseeger, @killerSwitch, @rschmucker
This version supports Python versions 3.7 to 3.9.
Bugs in v0.4
- #1607
pip install autogluon.text
will error on import if installed standalone due to missingautogluon.features
as a dependency. To fix:pip install autogluon.features
. This will be resolved in v0.4.1 release.
Changes
General
- AutoGluon now supports Windows OS! Both CPU and GPU are supported on Windows.
- AutoGluon now supports Python 3.9. Python 3.6 is no longer supported.
- AutoGluon has migrated from MXNet to PyTorch for all deep learning models resulting in major speedups.
- AutoGluon v0.4 Cheat Sheet: Get started faster than ever before with this handy reference page!
- New tutorials showcasing cloud training and deployment with AWS SageMaker and Lambda.
Text
AutoGluon-Text is refactored with PyTorch Lightning. It now supports backbones in huggingface/transformers. The new version has better performance, faster training time, and faster inference speed. In addition, AutoGluon-Text now supports solving multilingual problems and a new AutoMMPredictor
has been implemented for automatically building multimodal DL models.
- Better Performance
- Compared with TextPredictor in AutoGluon 0.3, TextPredictor in AutoGluon 0.4 has 72.22% win-rate in the multimodal text-tabular benchmark published in NeurIPS 2021. If we use
presets="high_quality"
, the win-rate increased to 77.8% thanks to the DeBERTa-v3 backbone. - In addition, we resubmitted our results to MachineHack: Product Sentiment Analysis, "MachineHack: Predict the Price of Books", and "Kaggle: Mercari Price Suggestion". With three lines of code, AutoGluon 0.4 is able to achieve top places in these competitions (1st, 2nd, 2nd correspondingly). The results obtained by AutoGluon 0.4 also consistently outperform the results obtained by AutoGluon 0.3.
- Compared with TextPredictor in AutoGluon 0.3, TextPredictor in AutoGluon 0.4 has 72.22% win-rate in the multimodal text-tabular benchmark published in NeurIPS 2021. If we use
- Faster Speed
- The new version has ~2.88x speedup in training and ~1.40x speedup in inference. With g4dn.12x instance, the model can achieve an additional 2.26x speedup with 4 GPUs.
- Multilingual Support
- AutoGluon-Text now supports solving multilingual problems via cross-lingual transfer (Tutorial). This is triggered by setting
presets="multilingual"
. You can now train a model on the English dataset and directly apply the model on datasets in other languages such as German, Japanese, Italian, etc.
- AutoGluon-Text now supports solving multilingual problems via cross-lingual transfer (Tutorial). This is triggered by setting
- AutoMMPredictor for Multimodal Problems
- Support an experimental AutoMMPredictor that supports fusion image backbones in timm, text backbone in huggingface/transformers, and multimodal backbones like CLIP (Tutorial). It may perform better than ensembling ImagePredictor + TextPredictor.
- Other Features
- Support continuous training from an existing checkpoint. You may just call
.fit()
again after a previous trained model has been loaded.
- Support continuous training from an existing checkpoint. You may just call
Thanks to @zhiqiangdon and @sxjscience for contributing the AutoGluon-Text refactors! (#1537, #1547, #1557, #1565, #1571, #1574, #1578, #1579, #1581, #1585, #1586)
Tabular
AutoGluon-Tabular has been majorly enhanced by numerous optimizations in 0.4. In summation, these improvements have led to a:
- ~2x training speedup in Good, High, and Best quality presets.
- ~1.3x inference speedup.
- 63% win-rate vs AutoGluon 0.3.1 (Results from AutoMLBenchmark)
- 93% win-rate vs AutoGluon 0.3.1 on datasets with >=100,000 rows of data (!!!)
Specific updates:
- Added
infer_limit
andinfer_limit_batch_size
as new fit-time constraints (Tutorial). This allows users to specify
the desired end-to-end inference latency of the final model and AutoGluon will automatically train models
to satisfy the constraint. This is extremely useful for online-inference scenarios where you need to satisfy an
end-to-end latency constraint (for example 50ms). @Innixma (#1541, #1584) - Implemented automated semi-supervised and transductive learning in TabularPredictor.
Try it out viaTabularPredictor.fit_pseudolabel(...)
! @DolanTheMFWizard (#1323, #1382) - Implemented automated feature pruning (i.e. feature selection) in TabularPredictor.
Try it out viaTabularPredictor.fit(..., feature_prune_kwargs={})
! @truebluejason (#1274, #1305) - Implemented automated model calibration to improve AutoGluon's predicted probabilities for classification problems.
This is enabled by default, and can be toggled via thecalibrate
fit argument. @DolanTheMFWizard (#1336, #1374, #1502) - Implemented parallel bag training via Ray. This results in a ~2x training speedup when bagging is enabled
compared to v0.3.1 with the same hardware due to more efficient usage of resources
for models that cannot effectively use all cores. @yinweisu (#1329, #1415, #1417, #1423) - Added adaptive early stopping logic which greatly improves the quality of models within a time budget. @Innixma (#1380)
- Added automated model calibration in quantile regression. @taesup-aws (#1388)
- Enhanced datetime feature handling. @willsmithorg (#1446)
- Added support for custom confidence levels in feature importance. @jwmueller (#1328)
- Improved neural network HPO search spaces. @jwmueller (#1346)
- Optimized one-hot encoding preprocessing. @Innixma (#1376)
- Refactored
refit_full
logic to majorly simplify user model contributions and improve multimodal support with advanced presets. @Innixma (#1567) - Added experimental TabularPredictor config helper. @gradientsky (#1491)
- New Tutorials
Tabular Models
NEW: TabularNeuralNetTorchModel (alias: 'NN_TORCH')
As part of the migration from MXNet to Torch, we have created a Torch based counterpart
to the prior MXNet tabular neural network model. This model has several major advantages, such as:
- 1.9x faster training speed
- 4.7x faster inference speed
- 51% win-rate vs MXNet Tabular NN
This model has replaced the MXNet tabular neural network model in the default hyperparameters configuration,
and is enabled by default.
Thanks to @jwmueller and @Innixma for contributing TabularNeuralNetTorchModel to AutoGluon! (#1489)
NEW: VowpalWabbitModel (alias: 'VW')
VowpalWabbit has been added as a new model in AutoGluon. VowpalWabbit is not installed by default, and must be installed separately.
VowpalWabbit is used in the hyperparameters='multimodal'
preset, and the model is a great option to use for datasets containing text features.
To install VowpalWabbit, specify it via pip install autogluon.tabular[all, vowpalwabbit]
or pip install "vowpalwabbit>=8.10,<8.11"
Thanks to @killerSwitch for contributing VowpalWabbitModel to AutoGluon! (#1422)
XGBoostModel (alias: 'XGB')
- Optimized model serialization method, which results in 5.5x faster inference speed and halved disk usage. @Innixma (#1509)
- Adaptive early stopping logic leading to 54.7% win-rate vs prior implementation. @Innixma (#1380)
- Optimized training speed with expensive metrics such as F1 by ~10x. @Innixma (#1344)
- Optimized num_cpus default to equal physical cores rather than virtual cores. @Innixma (#1467)
CatBoostModel (alias: 'CAT')
- CatBoost now incorporates callbacks which make it more stable and resilient to memory errors,
along with more advanced adaptive early stopping logic that leads to 63.2% win-rate vs prior implementation. @Innixma (#1352, #1380)
LightGBMModel (alias: 'GBM')
- Optimized training speed with exp...
v0.3.1
v0.3.1 is a hotfix release which fixes several major bugs as well as including several model quality improvements.
This release is non-breaking when upgrading from v0.3.0. As always, only load previously trained models using the same version of AutoGluon that they were originally trained on. Loading models trained in different versions of AutoGluon is not supported.
This release contains 9 commits from 4 contributors.
See the full commit change-log here: v0.3.0...v0.3.1
Thanks to the 4 contributors that contributed to the v0.3.1 release!
Special thanks to @yinweisu who is a first time contributor to AutoGluon and fixed a major bug in ImagePredictor HPO!
Full Contributor List (ordered by # of commits):
@Innixma, @gradientsky, @yinweisu, @sackoh
Changes
Tabular
- AutoGluon v0.3.1 has a 58% win-rate vs AutoGluon v0.3.0 for
best_quality
preset. - AutoGluon v0.3.1 has a 75% win-rate vs AutoGluon v0.3.0 for high and good quality presets.
- Fixed major bug introduced in v0.3.0 with models trained in refit_full causing weighted ensembles to incorrectly weight models. This severely impacted accuracy and caused worse results for high and good quality presets. @Innixma (#1293)
- Removed KNN from stacker models, resulting in stack quality improvement. @Innixma (#1294)
- Added automatic detection and optimized usage of boolean features. @Innixma (#1286)
- Improved handling of time limit in FastAI NN model to avoid edge cases where the model would use the entire time budget but fail to train. @Innixma (#1284)
- Updated XGBoost to use
-1
asn_jobs
value instead of usingos.cpu_count()
. @sackoh (#1289)
Vision
- Fixed major bug that caused HPO with time limits specified to return very poor models. @yinweisu (#1282)
General
- Minor doc updates. @gradientsky (#1288, #1290)
v0.3.0
v0.3.0 introduces multi-modal image, text, tabular support to AutoGluon. In just a few lines of code, you can train a multi-layer stack ensemble using text, image, and tabular data! To our knowledge this is the first publicly available implementation of a model that handles all 3 modalities at once. Check it out in our brand new multimodal tutorial! v0.3.0 also features a major model quality improvement for Tabular, with a 57.6% winrate vs v0.2.0 on the AutoMLBenchmark, along with an up to 10x online inference speedup due to low level numpy and pandas optimizations throughout the codebase! This inference optimization enables AutoGluon to have sub 30 millisecond end-to-end latency for real-time deployment scenarios when paired with model distillation. Finally, AutoGluon can now train PyTorch image models via integration with TIMM. Specify any TIMM model to ImagePredictor
or TabularPredictor
to train them with AutoGluon!
This release is non-breaking when upgrading from v0.2.0. As always, only load previously trained models using the same version of AutoGluon that they were originally trained on. Loading models trained in different versions of AutoGluon is not supported.
This release contains 70 commits from 10 contributors.
See the full commit change-log here: v0.2.0...v0.3.0
Thanks to the 10 contributors that contributed to the v0.3.0 release!
Special thanks to the 3 first-time contributors! @rxjx, @sallypannn, @sarahyurick
Special thanks to @talhaanwarch who opened 21 GitHub issues (!) and participated in numerous discussions during v0.3.0 development. His feedback was incredibly valuable when diagnosing issues and improving the user experience throughout AutoGluon!
Full Contributor List (ordered by # of commits):
@Innixma, @zhreshold, @jwmueller, @gradientsky, @sxjscience, @ValerioPerrone, @taesup-aws, @sallypannn, @rxjx, @sarahyurick
Major Changes
Multimodal
- Added multimodal tabular, text, image functionality! See the tutorial to get started. @Innixma, @zhreshold (#1041, #1211, #1277)
Tutorials
- Added a new custom model tutorial to showcase how to easily add any model to AutoGluon! @Innixma (#1238)
- Added a new custom metric tutorial to showcase how to add custom metrics to AutoGluon! @Innixma (#1271)
- Added FairHPO tutorial. @ValerioPerrone (#1090, #1236)
Tabular
- Overall, AutoGluon-Tabular v0.3 wins 57.6% of the time against AutoGluon-Tabular v0.2 in AutoMLBenchmark!
- Improved online inference speed by 1.5x-10x via various low level pandas and numpy optimizations. @Innixma (#1136)
- Accelerated feature preprocessing speed by 100x+ for datetime and text features. @Innixma (#1203)
- Fixed FastAI model not properly scaling regression label values, improving model quality significantly. @Innixma (#1162)
- Fixed r2 metric having the wrong sign in FastAI model, dramatically improving performance when r2 metric is specified. @Innixma (#1159)
- Updated XGBoost to 1.4, defaulted hyperparameter
tree_method='hist'
for improved performance. @Innixma (#1239) - Added
groups
parameter. Now users can specify the exact split indices in agroups
column when performing model bagging. This solution leverages sklearn's LeaveOneGroupOut cross-validator. @Innixma (#1224) - Added option to use holdout data for final ensembling weights in multi-layer stacking via a new
use_bag_holdout
argument. @Innixma (#1105) - Added neural network based quantile regression models. @taesup-aws (#1047)
- Bug fix for random forest models' out-of-fold prediction computation in quantile regression. @jwmueller, @Innixma (#1100, #1102)
- Added
predictor.features()
to get the original feature names used during training. @Innixma (#1257) - Refactored AbstractModel code to be easier to use. @Innixma (#1151, #1216, #1245, #1266)
- Refactored BaggedEnsembleModel code in preparation for distributed bagging. @gradientsky (#1078)
- Updated RAPIDS version to 21.06. @sarahyurick (#1241)
- Force dtype conversion in feature preprocessing to align with FeatureMetadata. Now users can specify the dtypes of features via FeatureMetadata rather than updating the DataFrame. @Innixma (#1212)
- Fixed various edge cases with out-of-bounds date time values. Now out-of-bounds date time values are treated as missing. @Innixma (#1182)
Vision
- Added Torch / TIMM backend support! Now AutoGluon can train any TIMM model natively, and MXNet is no longer required to train vision models. @zhreshold (#1249)
- Added regression
problem_type
support to ImagePredictor. @sallypannn (#1165) - Added GPU memory check to avoid going OOM during training. @Innixma (#1199)
- Fixed error when vision models are hyperparameter tuned with forked multiprocessing. @gradientsky (#1107)
- Fixed crash when an image is missing (both train and inference). Use TabularPredictor's Image API to get this functionality. @Innixma (#1210)
- Fixed error when the same image is in multiple rows when calling
predict_proba
. @Innixma (#1206) - Fixed invalid preset configurations. @Innixma (#1199)
- Fixed major defect causing tuning data to not be properly created if tuning data was not provided by user. @Innixma (#1168)
- Upgraded Pillow version to '>=8.3.0,<8.4.0'. @gradientsky (#1262)
Text
- Removed pyarrow as a required dependency. @Innixma (#1200)
- Fixed crash when
eval_metric='average_precision'
. @rxjx (#1092)
General
- Improved support for GPU on Windows. @Innixma (#1255)
- Added quadratic kappa evaluation metric. @sxjscience (#1104)
- Improved access method for
__version__
. @Innixma (#1122) - Upgraded pandas to 1.3. @Innixma (#1258)
- Upgraded ConfigSpace to 0.4.19. @Innixma (#1265)
- Upgraded numpy, graphviz, and dill versions. @Innixma (#1275)
- Various minor doc improvements. @jwmueller, @Innixma (#1089, #1091, #1093, #1095, #1219, #1253)
- Various minor updates and fixes. @Innixma, @zhreshold, @gradientsky (#1098, #1099, #1101, #1113, #1117, #1118, #1166, #1177, #1188, #1197, #1227, #1229, #1235, #1245, #1251)