Making it easy to add a Hugging Face-style Transformers model

We refactored jiant to make it easier to add a Transformers-style model to the library. Please see the guide to add a model for more details. We added DeBERTa V2 as part of these changes.

Breaking Changes

The simple API now uses hf_pretrained_model_name_or_path instead of model_type as an argument. hf_pretrained_model_name_or_path is used as an input to Hugging Face's Auto Classes.

Features

de5437a Merge easy_add_model feature branch (#1309)
56ceae5 Updating notebooks, removing model_type (#1270)
723786a Switch export_model to use AutoModel and AutoTokenizer (#1260)
84f2f5a Adding Acceptability judgment and SentEval tasks (#1271)
f796e5a improve robustness of the simple runscript (#1307)

Tests

4d0f6a9 Add test matrix (#1308)

Bugfixes

ee65662 Update README.md
65888b4 Benchmark script fixes (#1301)
b4b5de0 Assert spans <= max_seq_len (#1302)
5ba72f7 axg->axb fix (#1300)
235f646 MNLI diagnostic example bug (#1294)

Maintenance

4ab0c08 Bump lxml from 4.6.2 to 4.6.3 (#1304)
741ab09 Documentation + cleanup (#1282)
dbbb4e6 export_model tweak (#1277)

Tasks

e9d6c68 Adding download code for ARCT, MCTest, MCTACO, MuTual, and QuAIL (#1258)

Examples

a88956a Add edge probing notebook (#1261)

Bugfixes

d5e3b2e Prevents computing loss in multigpu when not available (#1257)
9cfe644 Truncate MCQ inputs from the start and not end (#1256)
33c4a26 Bump lxml from 4.5.1 to 4.6.2 (#1263)

Tests

9a45712 Add test for export_model (#1259)

Tasks

1ab34a4 Adding ROPES, RACE tasks (#1234)

Bugfixes

ce62495 Add simple save model test + fix (#1227)
e5fbea4 fix create examples wsc (#1247)
605d794 committing quail fix (#1249)
192d6b5 Namespace default cache dir by model_type. (#1246)

Bugfixes

838cdd2 SQuAD tokenization update (#1232)
5329c7e Winogrande Task Property (#1229)
50b0116 Further fix for encoder_only (#1242)
1f66050 Allow force_overwrite to override "done" condition (#1241)
59438ed change checkpoint save default (#1233)
e2e85c9 guids_fix (#1225)
74c6ba0 fix notebooks path (#1231)

Features

18c41fc Load only encoder weights (#1240)

Documentation

711c6c8 Introduction filename correction (#1239)
f340d04 minor typo fix (#1238)

Added Tasks

e4f1c4b Winogrande (#1203)
e7eefc6 Fever NLI task and data downloader (#1215)
cb601cf Quail (#1208)
c00360f MCTest and MCTACO (#1197)
76e2826 Mcscript Task Property (#1219)

Documentation

9892766 Add docs for adding tasks to data downloader (#1221)

Bugfixes

cb7ee4a Fix save-last behavior (#1220)

Cleanup

0cc8cbb use task_name instead of task.name (#1224)

Added Tasks

442a2b0 - piqa (#1216) (William Huang)
c535e78 - Natural Questions (MRQA), NewsQA, Quoref (#1207) (Jason Phang)
d1b14c1 - mcscript (#1152) (William Huang)
da7550d - Adding arc_easy, arc_challenge, mutual, mutual_plus (#1206) (yzpang)
f4bca4e - add arct task doc documentation (#1154) (jeswan)
b23c0f7 - arct (#1151) (William Huang)
58beb8f - anli download (#1147) (Jason Phang)

Features

0b3dff5 - Adding ability to resume run in Simple API (#1205) (Jason Phang)

Notebooks

b81254b - Fix git clone in example notebooks (#1155) (jeswan)

Bugfixes

aa4d111 - Bugfix for single-task configurator (#1143) (Jason Phang)
14fac1c - Fix colab link in README (#1142) (Jonathan Chang)
02bb070 - setup.py fix (#1141) (Jason Phang)
7d1cc29 - Adding SingleTaskConfigurator, some cleanup (#1135) (Jason Phang)

Maintenance

bump torch>=1.5.0. bump transformers==3.1.0. notebook installation switched to local pip install. (#1218) (jeswan)
b20f30a - resolve_is_lower_case fix (#1204) (Jason Phang)
5724fee - Adjust case for span prediction (#1201) (Jason Phang)
c3387a3 - nlp to datasets (#1137) (Jason Phang)
04bbb39 - update issue numbers from jiant-dev to jiant transfer (#1196) (jeswan)
392976c - Task tweaks (#1149) (Jason Phang)
82ed396 - use hidden_size (#1148) (Jason Phang)

Highlighted changes:

• jiant 2.0 is a complete rewrite of jiant, built natively on transformers and nlp/datasets
• Support for 50+ natural language understanding task, including the GLUE, SuperGLUE and XTREME benchmarks
• Support for BERT, RoBERTa, ALBERT, XLM-R, etc
• Includes data-downloading, tokenize-and-caching, training-and-evaluation code, and benchmark submission code for relevant tasks

Highlighted changes:

New Tasks

• Masked Language Modeling (for RoBERTa and ALBERT) (#1030) (@pruksmhc and @phu-pmh)
• Sentence Order Prediction (for ALBERT) (#1061) (@pruksmhc and @phu-pmh)

Minor changes and fixes

• Fixed target training data fraction bug where target training data fraction was not reflected in logging and scheduler (#1071) (@HaokunLiu)
• Fixed target train data fraction overwriting pretrain data fraction bug (#1070) (@pyeres)
• Added CONTRIBUTING.md (#1036, #1038) (@pyeres)

Dependency changes

• transformers 2.3.0 → transformers 2.6.0 (#1059) (@zphang)

Minor changes and fixes

• Fixed QAMR and QASRL tasks (#1019) (@pyeres)
• Changed tasks names using underscores to use hyphens (#1016) (@HaokunLiu)
• Fixed cola inference script (#1023) (@zphang)
• Re-ordered GPT-style inputs for consistency with GPT paper (#1031) (@HaokunLiu)
• Fixed edge probing and Senteval tasks (#1025) (@pruksmhc)

Highlighted changes:

New Tasks

• QA-SRL (#716) (@zphang)
• QAMR (#932) (@zphang)
• Abductive NLI (aNLI) (#922) (@zphang)
• SocialIQA (#924) (@pruksmhc)
• SentEval Probing (#926) (@pruksmhc)
• SciTail (#943) (@phu-pmh)
• CommonsenseQA (#942) (@HaokunLiu)
• HellaSwag (#942) (@HaokunLiu)
• Acceptability probing (#949) (@HaokunLiu)
• Adversarial NLI (#966) (@pyeres)
• Two-class MNLI variant (#976) (@sleepinyourhat)
• WinoGrande (#996) (@HaokunLiu)

New Models

• ALBERT (#990) (@HaokunLiu)

New Features

• Faster retokenization (#935) (@pruksmhc)
• Gradient accumulation option (#980) (@pyeres)
• Full/stable data-parallel multi-GPU support (#992) (@pruksmhc)

Minor changes and fixes

• Fixed bug in restoring checkpoints in multi-GPU mode (#928) (@pruksmhc)
• Fixed bugs in RoBERTa retokenization (#982) (@HaokunLiu) and ids (#959) (@njjiang)
• Fixed load_target_train_checkpoint with mixing setting (#960) (@pruksmhc)
• Fixed bug in CCG loss function that artificially reduced accuracy (#948) (@HaokunLiu)
• Fixed label parsing for QQP (#956) (@zphang)
• Updated CoLA inference script (#931) (@zphang)

Dependency changes

• PyTorch 1.0.0 → 1.1.0 (#965) (@pyeres)
• Numpy 1.14.5 → 1.15.0 (#965) (@pyeres)
• pytorch-transformers 1.2.0 → transformers 2.3.0 (#990) (@HaokunLiu)

Minor changes:

• Fix a bug that could slightly hurt performance on WIC and WSC (#916).

Highlighted changes:

• Add support for RoBERTa, XLM, and GPT-2 via pytorch_transformers 1.2.
• Add support for pip installation (and moved the body of main.py and the config directory to accomodate that change).
• Fix a bug that produced invalid micro/macro average scores during validation.

Minor changes:

• Refactor old GPT (v1) implementation to use pytorch_transformers.
• Make the code that adds git status information to logs more robust.
• Minor cleanup to data loading and to MNLI data handling logic.
• Fix a short-lived bug invalidating hypothesis-only MNLI results.
• Restore (partial) support for sequence-to-sequence tasks, though with no fully supported demonstration tasks in place yet.

Dependency changes:

• Updated requirement pytorch_transformers to 1.2.0.
• Updated requirement to NLTK 3.4.5 to avoid a potential security issue.

We expect another release within a week or two that will add support for RoBERTa (see #890), but this is a quick intermediate release now that XLNet support is stable/working.

Highlighted changes:

• Full support for XLNet and the whole-word-masking variants of BERT.
• Many small improvements to Google Cloud Platform/Kubernetes/Docker support.
• Add small but handy option to automatically delete checkpoints when a job finishes.
• max_vals is now used when computing warmup time with optimizers that use warmup.
• New auto option for tokenizer chooses an appropriate tokenizer for any given input module.
• Some internal changes to how <SOS>/<EOS>/[SEP]/[CLS] tokens are handled during task preprocessing. This will require small changes to custom task code along the lines of what is seen in #845.

Dependency changes:

• AllenNLP 0.8.4 now required
• pytorch_transformers 1.0 now required when using BERT or XLNet.

Warnings:

• Upgrading to 1.1 will break existing checkpoints for BERT-based models.

Bug fixes:

• Addresses an issue that prevented temporary checkpoints from being deleted.

The first stable release of jiant.

Highlighted changes:

• Support for the SuperGLUE v2.0 set of tasks, including all the baselines discussed in the SuperGLUE paper.
• A simpler and more standard code structure.
• Cleaner, more-readable logs.
• Simplified logic for checkpointing and evaluation, with fewer differences between pretraining and target task training.
• Fewer deprecated/unsupported modules.
• Many small bug fixes and improvements to errors and warnings.

Dependency changes:

• Upgrade to AllenNLP 0.8.4, which adds the option to use the GitHub development version of pytorch-pretrained-bert, and with it, the whole-word-masking variants of BERT.

Warnings:

• Upgrading from 0.9 to 1.0 will break most older model checkpoints and cached preprocessed data.

This release contains code to recreate part of the experiments from the paper "Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling". For the remaining experiments, see this branch.

A release with fixes to SuperGLUE baseline scripts/configurations.

The initial work-in-progress release coinciding with the launch of SuperGLUE.

Highlights:

We currently support two-phase training (pretraining and target task training) using various shared encoders, including:

• BERT
• OpenAI GPT
• Plain Transformer
• Ordered Neurons (ON-LSTM) Grammar Induction Model
• PRPN Grammar Induction Model

We also have support for SuperGLUE baselines, sentence encoder probing experiments, and STILTS-style training.

Examples

They can be found in https://github.com/nyu-mll/jiant/tree/master/config/examples