Liger-Kernel

Patch Release

Fix bug in Gemma patch function that FLCE and CE are both true by default ruh roh

What's Changed

• Bug fix for gemma: fused_linear_cross_entropy flag and cross_entropy flag are mutual exclusive by @JasonZhu1313 in https://github.com/linkedin/Liger-Kernel/pull/168
• Add gemma 7b it benchmark by @JasonZhu1313 in https://github.com/linkedin/Liger-Kernel/pull/166
• bump patch ver by @yundai424 in https://github.com/linkedin/Liger-Kernel/pull/171

Full Changelog: https://github.com/linkedin/Liger-Kernel/compare/v0.2.0...v0.2.1

Opening Thoughts 🫶

Thank You!

We'd love to take this chance to express our sincere gratefulness to the community! 2500+ ⭐ , 10+ new contributors, 50+ PRs, plus integration into Hugging Face 🤗, axolotl and LLaMA-Factory in less than one week since going open sourced is totally beyond our expectation. Being able to work together with all the cool people in the community is a bliss and we can't wait for further collaborations down the road!

Looking Ahead

We look forward to further enhancing our collaboration with the community, to work together on a lot of cool stuff -- support for more model families, squeeze out all optimization opportunities for kernels, and, why not, llama.triton? 😉

Get Involved and Stay Tuned

Please feel free to join our discord channel hosted in CUDA MODE server, and follow our repo's official account on X: https://x.com/liger_kernel !

Welcome Phi3 and Qwen2 🚀

This release ships with support for other popular models including Phi3 and Qwen2. All existing kernels in Liger repo can be leveraged to boost your training with models from these families now. Please refer to our API guide for how to use.

Even Easier API ❤️

Experimenting with different model families and tired of having if-else everywhere just to switch between kernel patching functions? You can now try out our new model-agnostic API to apply Liger kernels. Still a one-liner, but more elegant :) For example:

from liger_kernel.transformers import AutoLigerKernelForCausalLM

# This AutoModel wrapper class automatically monkey-patches the
# model with the optimized Liger kernels if the model is supported.
model = AutoLigerKernelForCausalLM.from_pretrained(...)

More Features

• Support optional bias term in FusedLinearCrossEntropy (#144)
• Mistral is now equipped with the humongous memory reduction from FusedLinearCrossEntropy now (#93)
• Gemma is now equipped with the humongous memory reduction from FusedLinearCrossEntropy now (#111)

Bug Fixes

• Fixed import error when using triton>=3.0.0 on NGC containers (#79)
• Fixed the missing offset in Gemma RMSNorm (#85) oops
• Added back missing dataclass entries in efficiency callback (#116)
• There was some confusion on which Gemma do we support, we now support all! (#125)
• Fallback to torch native linear + CrossEntropy when without label (#128)
• Match the exact dtype up and downcasting in Llama & Gemma for RMSNorm (#92)
• Address the bug that RoPE gets very slow when using dynamic sequence length (#149)

What's Changed

• Updated test tolerances for H100 by @shimizust in https://github.com/linkedin/Liger-Kernel/pull/55
• Update README.md by @lancerts in https://github.com/linkedin/Liger-Kernel/pull/58
• Update benchmark result of Medusa for batch size = 6 setup by @JasonZhu1313 in https://github.com/linkedin/Liger-Kernel/pull/59
• Add star graph by @shivam15s in https://github.com/linkedin/Liger-Kernel/pull/60
• Add monkey patch for Qwen2 models by @chiwanpark in https://github.com/linkedin/Liger-Kernel/pull/69
• Add pytest and datasets to dev dependencies by @chiwanpark in https://github.com/linkedin/Liger-Kernel/pull/68
• Fix typos by @pchng in https://github.com/linkedin/Liger-Kernel/pull/77
• Remove unused images in examples/medusa/docs/images/ by @pchng in https://github.com/linkedin/Liger-Kernel/pull/78
• chore: update cross_entropy.py by @eltociear in https://github.com/linkedin/Liger-Kernel/pull/84
• Fix incorrect import for triton 3 by @arvindsun in https://github.com/linkedin/Liger-Kernel/pull/79
• update install from source guide by @yundai424 in https://github.com/linkedin/Liger-Kernel/pull/86
• Fix Gemma RMSNorm by @davidgonmar in https://github.com/linkedin/Liger-Kernel/pull/85
• Fix example bugs by @qingquansong in https://github.com/linkedin/Liger-Kernel/pull/88
• Make tests passing on AMD GPU with 24GB ram by @helloworld1 in https://github.com/linkedin/Liger-Kernel/pull/90
• modified: README.md by @leaf-soba in https://github.com/linkedin/Liger-Kernel/pull/91
• pytest without need to dealing with PYTHONPATH by @helloworld1 in https://github.com/linkedin/Liger-Kernel/pull/95
• Update test_cross_entropy.py by @lancerts in https://github.com/linkedin/Liger-Kernel/pull/94
• Add FusedLinerCrossEntropy support for Mistral by @Tcc0403 in https://github.com/linkedin/Liger-Kernel/pull/93
• Remove duplicate images by @qingquansong in https://github.com/linkedin/Liger-Kernel/pull/107
• Add Qwen benchmarks by @shivam15s in https://github.com/linkedin/Liger-Kernel/pull/108
• Fix Mixtral typo by @Tcc0403 in https://github.com/linkedin/Liger-Kernel/pull/109
• Explicitly add dependencies in req.txt for medusa example by @JasonZhu1313 in https://github.com/linkedin/Liger-Kernel/pull/110
• Add convergence tests and trainer integration test for Qwen2 by @Tcc0403 in https://github.com/linkedin/Liger-Kernel/pull/105
• [Bug fix] Efficiency callback missing dataclass entries by @tyler-romero in https://github.com/linkedin/Liger-Kernel/pull/116
• Monkeypatch for Phi3 by @tyler-romero in https://github.com/linkedin/Liger-Kernel/pull/76
• Add FusedLinearCrossEntropy to Gemma by @Luke-Chesley in https://github.com/linkedin/Liger-Kernel/pull/111
• Makefile command for env-report by @tyler-romero in https://github.com/linkedin/Liger-Kernel/pull/114
• [WIP] Fix confusion on Gemma by @yundai424 in https://github.com/linkedin/Liger-Kernel/pull/121
• [tiny] reformat code by @tyler-romero in https://github.com/linkedin/Liger-Kernel/pull/122
• Revert "[WIP] Fix confusion on Gemma (#121)" by @yundai424 in https://github.com/linkedin/Liger-Kernel/pull/123
• Fix gemma 1 and 2 support by @yundai424 in https://github.com/linkedin/Liger-Kernel/pull/125
• Adding AutoLigerKernelForCausalLM by @shimizust in https://github.com/linkedin/Liger-Kernel/pull/115
• fallback to torch native linear+CE when without label by @yundai424 in https://github.com/linkedin/Liger-Kernel/pull/128
• Add code to save medusa heads and model by @JasonZhu1313 in https://github.com/linkedin/Liger-Kernel/pull/130
• Add FusedLinerCrossEntropy support for Phi3 by @tyler-romero in https://github.com/linkedin/Liger-Kernel/pull/103
• Add GPU CI support by @helloworld1 in https://github.com/linkedin/Liger-Kernel/pull/134
• Make GPU CI optional until it is more stable by @helloworld1 in https://github.com/linkedin/Liger-Kernel/pull/141
• Add gemma lightning example for single L40 GPU by @qingquansong in https://github.com/linkedin/Liger-Kernel/pull/120
• feat: correct casts in RMSNorm to match references by @davidgonmar in https://github.com/linkedin/Liger-Kernel/pull/92
• Bias for fused linear cross entropy by @davidgonmar in https://github.com/linkedin/Liger-Kernel/pull/144
• Rerun FLCE benchmark after bias added by @ByronHsu in https://github.com/linkedin/Liger-Kernel/pull/148
• updated sl to be non-constexpr by @AndreSlavescu in https://github.com/linkedin/Liger-Kernel/pull/149
• update readme to use absolute paths by @shaoruu in https://github.com/linkedin/Liger-Kernel/pull/157
• fix convergence test, phi3 import and update benchmark by @yundai424 in https://github.com/linkedin/Liger-Kernel/pull/155
• bump lowest HF version by @yundai424 in https://github.com/linkedin/Liger-Kernel/pull/158
• Add missing tf_keras to req.txt by @JasonZhu1313 in https://github.com/linkedin/Liger-Kernel/pull/161
• Re-enable GPU CI enforce by @helloworld1 in https://github.com/linkedin/Liger-Kernel/pull/142
• Bump package ver by @yundai424 in https://github.com/linkedin/Liger-Kernel/pull/163
• Update version in setup.py to 0.2.0 by @yundai424 in https://github.com/linkedin/Liger-Kernel/pull/164

New Contributors

• @chiwanpark made their first contribution in https://github.com/linkedin/Liger-Kernel/pull/69
• @pchng made their first contribution in https://github.com/linkedin/Liger-Kernel/pull/77
• @eltociear made their first contribution in https://github.com/linkedin/Liger-Kernel/pull/84
• @arvindsun made their first contribution in https://github.com/linkedin/Liger-Kernel/pull/79
• @davidgonmar made their first contribution in https://github.com/linkedin/Liger-Kernel/pull/85
• @leaf-soba made their first contribution in https://github.com/linkedin/Liger-Kernel/pull/91
• @Tcc0403 made their first contribution in https://github.com/linkedin/Liger-Kernel/pull/93
• @tyler-romero made their first contribution in https://github.com/linkedin/Liger-Kernel/pull/116
• @Luke-Chesley made their first contribution in https://github.com/linkedin/Liger-Kernel/pull/111
• @AndreSlavescu made their first contribution in https://github.com/linkedin/Liger-Kernel/pull/149
• @shaoruu made their first contribution in https://github.com/linkedin/Liger-Kernel/pull/157

Full Changelog: https://github.com/linkedin/Liger-Kernel/compare/v0.1.1...v0.2.0

What's Changed

• Fix unwanted scale/bias while testing and simplify _test_memory function by @shivam15s in https://github.com/linkedin/Liger-Kernel/pull/50
• Update README by @JacobHelwig in https://github.com/linkedin/Liger-Kernel/pull/44
• Added metadata for PyPI and bumped version by @shimizust in https://github.com/linkedin/Liger-Kernel/pull/52
• Replace model / data with public HF path, update readme by @JasonZhu1313 in https://github.com/linkedin/Liger-Kernel/pull/53

New Contributors

• @JacobHelwig made their first contribution in https://github.com/linkedin/Liger-Kernel/pull/44

Full Changelog: https://github.com/linkedin/Liger-Kernel/compare/v0.1.0...v0.1.1

What's Changed

• Update PR template and contribution guide by @lancerts in https://github.com/linkedin/Liger-Kernel/pull/20
• Add GeGLU and updage readme by @yundai424 in https://github.com/linkedin/Liger-Kernel/pull/3
• Added CI workflow with checkstyle job by @shimizust in https://github.com/linkedin/Liger-Kernel/pull/27
• Create bug_report.yaml and feature_request.yaml by @lancerts in https://github.com/linkedin/Liger-Kernel/pull/29
• Update feature_request.yaml and bug_report.yaml by @lancerts in https://github.com/linkedin/Liger-Kernel/pull/30
• update gif by @zain-merchant in https://github.com/linkedin/Liger-Kernel/pull/31
• Add lightning trainer and HF trainer fine-tuning example by @yundai424 in https://github.com/linkedin/Liger-Kernel/pull/17
• use correct fsdp act ckpt & redo benchmark by @ByronHsu in https://github.com/linkedin/Liger-Kernel/pull/32
• Update README.md by @ByronHsu in https://github.com/linkedin/Liger-Kernel/pull/33
• Update README.md with Kernel descriptions by @qingquansong in https://github.com/linkedin/Liger-Kernel/pull/34
• remove mfu and non used methods by @zain-merchant in https://github.com/linkedin/Liger-Kernel/pull/35
• Byhsu/readme 3 by @ByronHsu in https://github.com/linkedin/Liger-Kernel/pull/37
• Zain/singletest by @zain-merchant in https://github.com/linkedin/Liger-Kernel/pull/38
• Add deepspeed to lightning example by @yundai424 in https://github.com/linkedin/Liger-Kernel/pull/36
• Update README.md by @lancerts in https://github.com/linkedin/Liger-Kernel/pull/39
• improve rms norm code quality by @ByronHsu in https://github.com/linkedin/Liger-Kernel/pull/43
• Refactored convergence tests to be portable by @shimizust in https://github.com/linkedin/Liger-Kernel/pull/41
• Added more generic monkey patch function by @shimizust in https://github.com/linkedin/Liger-Kernel/pull/42
• Remove override dependency by @shivam15s in https://github.com/linkedin/Liger-Kernel/pull/45
• Changed pointer variable names for clarity for SwiGLU by @zain-merchant in https://github.com/linkedin/Liger-Kernel/pull/46
• Update CONTRIBUTING.md by @lancerts in https://github.com/linkedin/Liger-Kernel/pull/47
• Release version 0.1.0 by @shimizust in https://github.com/linkedin/Liger-Kernel/pull/49

New Contributors

• @shimizust made their first contribution in https://github.com/linkedin/Liger-Kernel/pull/27
• @shivam15s made their first contribution in https://github.com/linkedin/Liger-Kernel/pull/45

Full Changelog: https://github.com/linkedin/Liger-Kernel/compare/v0.0.1...v0.1.0

What's Changed

• Update Readme.md by @lancerts in https://github.com/linkedin/Liger-Kernel/pull/1
• Update Readme.md by @lancerts in https://github.com/linkedin/Liger-Kernel/pull/8
• Added requirements from licensing team including notice and contributing by @zain-merchant in https://github.com/linkedin/Liger-Kernel/pull/5
• Test GitHub PR setting by @ByronHsu in https://github.com/linkedin/Liger-Kernel/pull/12
• Update README.md with bib cite by @qingquansong in https://github.com/linkedin/Liger-Kernel/pull/13
• Create pull_request_template.md by @lancerts in https://github.com/linkedin/Liger-Kernel/pull/9
• Update README.md bib by @qingquansong in https://github.com/linkedin/Liger-Kernel/pull/15
• Update pull_request_template.md by @lancerts in https://github.com/linkedin/Liger-Kernel/pull/14
• Updated readme gif by @zain-merchant in https://github.com/linkedin/Liger-Kernel/pull/18
• make fused linear+CE default for llama by @yundai424 in https://github.com/linkedin/Liger-Kernel/pull/22
• create rms norm tensor at input.device instead of device 0 by @ByronHsu in https://github.com/linkedin/Liger-Kernel/pull/21
• Add medusa patching code and example job with memory efficient Liger Kernel by @JasonZhu1313 in https://github.com/linkedin/Liger-Kernel/pull/11
• forward compatibility with triton 3.0.0 for tanh by @yundai424 in https://github.com/linkedin/Liger-Kernel/pull/24
• ignore e203 in flake8 to resolve black conflict by @ByronHsu in https://github.com/linkedin/Liger-Kernel/pull/25
• src directory polishing by @ByronHsu in https://github.com/linkedin/Liger-Kernel/pull/23
• Polish test/ and others by @ByronHsu in https://github.com/linkedin/Liger-Kernel/pull/26

New Contributors

• @lancerts made their first contribution in https://github.com/linkedin/Liger-Kernel/pull/1
• @zain-merchant made their first contribution in https://github.com/linkedin/Liger-Kernel/pull/5
• @qingquansong made their first contribution in https://github.com/linkedin/Liger-Kernel/pull/13
• @yundai424 made their first contribution in https://github.com/linkedin/Liger-Kernel/pull/22
• @JasonZhu1313 made their first contribution in https://github.com/linkedin/Liger-Kernel/pull/11

Full Changelog: https://github.com/linkedin/Liger-Kernel/compare/0.0.2...v0.0.1