A Gecko-oriented implementation of the Encoding Standard in Rust
OTHER License
encoding_rs an implementation of the (non-JavaScript parts of) the Encoding Standard written in Rust.
The Encoding Standard defines the Web-compatible set of character encodings, which means this crate can be used to decode Web content. encoding_rs is used in Gecko starting with Firefox 56. Due to the notable overlap between the legacy encodings on the Web and the legacy encodings used on Windows, this crate may be of use for non-Web-related situations as well; see below for links to adjacent crates.
Additionally, the mem
module provides various operations for dealing with
in-RAM text (as opposed to data that's coming from or going to an IO boundary).
The mem
module is a module instead of a separate crate due to internal
implementation detail efficiencies.
Due to the Gecko use case, encoding_rs supports decoding to and encoding from UTF-16 in addition to supporting the usual Rust use case of decoding to and encoding from UTF-8. Additionally, the API has been designed to be FFI-friendly to accommodate the C++ side of Gecko.
Specifically, encoding_rs does the following:
u16
/ char16_t
).u16
/ char16_t
) into a sequence of bytes in an Encodingdocument.characterSet
.Additionally, encoding_rs::mem
does the following:
std::io
Notably, the above feature list doesn't include the capability to wrap
a std::io::Read
, decode it into UTF-8 and presenting the result via
std::io::Read
. The encoding_rs_io
crate provides that capability.
no_std
EnvironmentThe crate works in a no_std
environment. By default, the alloc
feature,
which assumes that an allocator is present is enabled. For a no-allocator
environment, the default features (i.e. alloc
) can be turned off. This
makes the part of the API that returns Vec
/String
/Cow
unavailable.
For decoding character encodings that occur in email, use the
charset
crate instead of using this
one directly. (It wraps this crate and adds UTF-7 decoding.)
For mappings to and from Windows code page identifiers, use the
codepage
crate.
This crate does not support single-byte DOS encodings that aren't required by
the Web Platform, but the oem_cp
crate does.
Normalizing text into Unicode Normalization Form C prior to encoding text into
a legacy encoding minimizes unmappable characters. Text can be normalized to
Unicode Normalization Form C using the
icu_normalizer
crate.
The exception is windows-1258, which after normalizing to Unicode Normalization
Form C requires tone marks to be decomposed in order to minimize unmappable
characters. Vietnamese tone marks can be decomposed using the
detone
crate.
TL;DR: (Apache-2.0 OR MIT) AND BSD-3-Clause
for the code and data combination.
Please see the file named COPYRIGHT.
The non-test code that isn't generated from the WHATWG data in this crate is under Apache-2.0 OR MIT. Test code is under CC0.
This crate contains code/data generated from WHATWG-supplied data. The WHATWG upstream changed its license for portions of specs incorporated into source code from CC0 to BSD-3-Clause between the initial release of this crate and the present version of this crate. The in-source licensing legends have been updated for the parts of the generated code that have changed since the upstream license change.
Generated API documentation is available online.
There is a long-form write-up about the design and internals of the crate.
An FFI layer for encoding_rs is available as a separate crate. The crate comes with a demo C++ wrapper using the C++ standard library and GSL types.
The bindings for the mem
module are in the
encoding_c_mem crate.
For the Gecko context, there's a C++ wrapper using the MFBT/XPCOM types.
There's a write-up about the C++ wrappers.
There are currently these optional cargo features:
simd-accel
Enables SIMD acceleration using the nightly-dependent portable_simd
standard
library feature.
This is an opt-in feature, because enabling this feature opts out of Rust's guarantees of future compilers compiling old code (aka. "stability story").
Currently, this has not been tested to be an improvement except for these
targets and enabling the simd-accel
feature is expected to break the build
on other targets:
If you use nightly Rust, you use targets whose first component is one of the above, and you are prepared to have to revise your configuration when updating Rust, you should enable this feature. Otherwise, please do not enable this feature.
Used by Firefox.
serde
Enables support for serializing and deserializing &'static Encoding
-typed
struct fields using Serde.
Not used by Firefox.
fast-legacy-encode
A catch-all option for enabling the fastest legacy encode options. Does not affect decode speed or UTF-8 encode speed.
At present, this option is equivalent to enabling the following options:
fast-hangul-encode
fast-hanja-encode
fast-kanji-encode
fast-gb-hanzi-encode
fast-big5-hanzi-encode
Adds 176 KB to the binary size.
Not used by Firefox.
fast-hangul-encode
Changes encoding precomposed Hangul syllables into EUC-KR from binary search over the decode-optimized tables to lookup by index making Korean plain-text encode about 4 times as fast as without this option.
Adds 20 KB to the binary size.
Does not affect decode speed.
Not used by Firefox.
fast-hanja-encode
Changes encoding of Hanja into EUC-KR from linear search over the decode-optimized table to lookup by index. Since Hanja is practically absent in modern Korean text, this option doesn't affect perfomance in the common case and mainly makes sense if you want to make your application resilient agaist denial of service by someone intentionally feeding it a lot of Hanja to encode into EUC-KR.
Adds 40 KB to the binary size.
Does not affect decode speed.
Not used by Firefox.
fast-kanji-encode
Changes encoding of Kanji into Shift_JIS, EUC-JP and ISO-2022-JP from linear
search over the decode-optimized tables to lookup by index making Japanese
plain-text encode to legacy encodings 30 to 50 times as fast as without this
option (about 2 times as fast as with less-slow-kanji-encode
).
Takes precedence over less-slow-kanji-encode
.
Adds 36 KB to the binary size (24 KB compared to less-slow-kanji-encode
).
Does not affect decode speed.
Not used by Firefox.
less-slow-kanji-encode
Makes JIS X 0208 Level 1 Kanji (the most common Kanji in Shift_JIS, EUC-JP and ISO-2022-JP) encode less slow (binary search instead of linear search) making Japanese plain-text encode to legacy encodings 14 to 23 times as fast as without this option.
Adds 12 KB to the binary size.
Does not affect decode speed.
Not used by Firefox.
fast-gb-hanzi-encode
Changes encoding of Hanzi in the CJK Unified Ideographs block into GBK and
gb18030 from linear search over a part the decode-optimized tables followed
by a binary search over another part of the decode-optimized tables to lookup
by index making Simplified Chinese plain-text encode to the legacy encodings
100 to 110 times as fast as without this option (about 2.5 times as fast as
with less-slow-gb-hanzi-encode
).
Takes precedence over less-slow-gb-hanzi-encode
.
Adds 36 KB to the binary size (24 KB compared to less-slow-gb-hanzi-encode
).
Does not affect decode speed.
Not used by Firefox.
less-slow-gb-hanzi-encode
Makes GB2312 Level 1 Hanzi (the most common Hanzi in gb18030 and GBK) encode less slow (binary search instead of linear search) making Simplified Chinese plain-text encode to the legacy encodings about 40 times as fast as without this option.
Adds 12 KB to the binary size.
Does not affect decode speed.
Not used by Firefox.
fast-big5-hanzi-encode
Changes encoding of Hanzi in the CJK Unified Ideographs block into Big5 from
linear search over a part the decode-optimized tables to lookup by index
making Traditional Chinese plain-text encode to Big5 105 to 125 times as fast
as without this option (about 3 times as fast as with
less-slow-big5-hanzi-encode
).
Takes precedence over less-slow-big5-hanzi-encode
.
Adds 40 KB to the binary size (20 KB compared to less-slow-big5-hanzi-encode
).
Does not affect decode speed.
Not used by Firefox.
less-slow-big5-hanzi-encode
Makes Big5 Level 1 Hanzi (the most common Hanzi in Big5) encode less slow (binary search instead of linear search) making Traditional Chinese plain-text encode to Big5 about 36 times as fast as without this option.
Adds 20 KB to the binary size.
Does not affect decode speed.
Not used by Firefox.
For decoding to UTF-16, the goal is to perform at least as well as Gecko's old uconv. For decoding to UTF-8, the goal is to perform at least as well as rust-encoding. These goals have been achieved.
Encoding to UTF-8 should be fast. (UTF-8 to UTF-8 encode should be equivalent
to memcpy
and UTF-16 to UTF-8 should be fast.)
Speed is a non-goal when encoding to legacy encodings. By default, encoding to legacy encodings should not be optimized for speed at the expense of code size as long as form submission and URL parsing in Gecko don't become noticeably too slow in real-world use.
In the interest of binary size, by default, encoding_rs does not have encode-specific data tables beyond 32 bits of encode-specific data for each single-byte encoding. Therefore, encoders search the decode-optimized data tables. This is a linear search in most cases. As a result, by default, encode to legacy encodings varies from slow to extremely slow relative to other libraries. Still, with realistic work loads, this seemed fast enough not to be user-visibly slow on Raspberry Pi 3 (which stood in for a phone for testing) in the Web-exposed encoder use cases.
See the cargo features above for optionally making CJK legacy encode fast.
A framework for measuring performance is available separately.
It is a goal to support the latest stable Rust, the latest nightly Rust and the version of Rust that's used for Firefox Nightly.
At this time, there is no firm commitment to support a version older than
what's required by Firefox, and there is no commitment to treat MSRV changes
as semver-breaking, because this crate depends on cfg-if
, which doesn't
appear to treat MSRV changes as semver-breaking, so it would be useless for
this crate to treat MSRV changes as semver-breaking.
As of 2024-04-04, MSRV appears to be Rust 1.36.0 for using the crate and
1.42.0 for doc tests to pass without errors about the global allocator.
With the simd-accel
feature, the MSRV is even higher.
A compatibility layer that implements the rust-encoding API on top of encoding_rs is provided as a separate crate (cannot be uploaded to crates.io). The compatibility layer was originally written with the assuption that Firefox would need it, but it is not currently used in Firefox.
To regenerate the generated code:
https://github.com/hsivonen/encoding_c
encoding_rs
directory.https://github.com/hsivonen/codepage
encoding_rs
directory.https://github.com/whatwg/encoding
encoding_rs
directory.be3337450e7df1c49dca7872153c4c4670dd8256
of the encoding
repo.f381389
was the revision of encoding
used from before the encoding
repoencoding_rs
directory as the working directory, runpython generate-encoding-data.py
.usize
instead of u8
at a time).alloc
(with lesser API surface).std::simd
unsafe
slice access by larger types than u8
/u16
to align_to
.portable_simd
nightly feature of the standard library instead of the packed_simd
crate. Only affects the simd-accel
optional nightly feature.unsafe
.rust-version
to Cargo.toml
.packed_simd
instead of packed_simd_2
again now that updates are back under the packed_simd
name. Only affects the simd-accel
optional nightly feature.build.rs
. (This removal should resolve false positives reported by some antivirus products. This may break some build configurations that have opted out of Rust's guarantees against future build breakage.)no_std
support.no_std
environment (with alloc
).simd-accel
feature.packed_simd
dependency to packed_simd_2
.cfg-if
dependency to 1.0.cfg-if
updated to edition 2018 without a semver break.Decoder::latin1_byte_compatible_up_to
return None
in moreNone
case in a reasonableconvert_str_to_utf16
.mem::convert_utf8_to_utf16_without_replacement
.mem::utf8_latin1_up_to
and mem::str_latin1_up_to
.Decoder::latin1_byte_compatible_up_to
.bincode
(dev dependency) version requirement to 1.0.simd
crate to packed_simd
.simd-accel
(README-only release).clippy::
prefix from clippy lint names.static
when defining another static
).is_single_byte()
on Encoding
.mem::decode_latin1()
and mem::encode_latin1_lossy()
.--features simd-accel
work with stable-channel compiler tois_foo_bidi()
not treat U+FEFF (ZERO WIDTH NO-BREAK SPACEis_foo_bidi()
functions report true
if the input containsconvert_utf16_to_latin1_lossy
.mem
module assert that the input is in the rangemem
module provide conversions from Latin1 and UTF-16 to UTF-8simd-accel
-specific memory corruption introduced inmem
module.#[inline(never)]
annotation that was not meant for release.mem
module to increase the performance whenmem
module.mem
replacement
a label of the replacementEncoding::for_name()
. (Encoding::for_label(foo).unwrap()
isparallel-utf8
cargo feature.&'static Encoding
.Encoder::has_pending_state()
public.simd
crate dependency to 0.2.0.7F
correctly in ISO-2022-JP.Hash
for Encoding
.InputEmpty
correct precedence over OutputFull
when encodingPartialEq
and Eq
for the CoderResult
, DecoderResult
EncoderResult
types.Encoder::encode_from_utf16
. (Due to an oversight, it lacked the fix thatEncoder::encode_from_utf8
already had.)#[must_use]
.parallel-utf8
).simd-accel
is used.Encoding
from const
to static
FOO_INIT
instances of Encoding
to allowstatic
arrays with references to Encoding
&'static Encoding
-typed array items with &'static Encoding
-typedstatics
.const
Cow
s from Rust-only non-streaming methods for encode and decode.Encoding::for_bom()
returns the length of the BOM.simd-accel
feature flag. (RequiresEncoder.encode_from_utf8_to_vec_without_replacement()
.Add Encoding.is_ascii_compatible()
.
Add Encoding::for_bom()
.
Make ==
for Encoding
use name comparison instead of pointer comparison,
because uses of the encoding constants in different crates result in
different addresses and the constant cannot be turned into statics without
breaking other things.
The initial release.