CharSetEncodings

Julia Version	Unit Tests	Coverage
Julia Latest

Architecture

This provides the basic types and mode methods for dealing with character sets, encodings, and character set encodings.

Types

Currently, there are the following types:

• CodeUnitTypes a Union of the 3 codeunit types (UInt8, UInt16, UInt32) for convenience
• CharSet a struct type, which is parameterized by the name of the character set and the type needed to represent a code point
• Encoding a struct type, parameterized by the name of the encoding

Built-in Character Sets / Character Set Encodings

• Binary For storing non-textual data as a sequence of bytes, 0-0xff

• ASCII ASCII (Unicode subset, 0-0x7f)

• Latin Latin-1 (ISO-8859-1) (Unicode subset, 0-0xff)

• UCS2 UCS-2 (Unicode subset, 0-0xd7ff, 0xe000-0xffff, BMP only, no surrogates)

• UTF32 UTF-32 (Full Unicode, 0-0xd7ff, 0xe000-0x10ffff)

• UniPlus Unvalidated Unicode (i.e. like String, can contain invalid codepoints)

• Text1 Unknown 1-byte character set

• Text2 Unknown 2-byte character set

• Text4 Unknown 4-byte character set

Built-in Encodings

• UTF8Encoding
• Native1Byte
• Native2Byte
• Native4Byte
• NativeUTF16
• Swapped4Byte
• Swapped2Byte
• SwappedUTF16
• LE2
• BE2
• LE4
• BE4
• UTF16LE
• UTF16BE
• 2Byte
• 4Byte
• UTF16

Built-in CSEs

• BinaryCSE, Text1CSE, ASCIICSE, LatinCSE
• Text2CSE, UCS2CSE

• Text4CSE, UTF32CSE

• UTF8CSE UTF32CharSet, all valid, using UTF8Encoding, conforming to the Unicode Organization's standard, i.e. no long encodings, surrogates, or invalid bytes.

• RawUTF8CSE UniPlusCharSet, not validated, using UTF8Encoding, may have invalid sequences, long encodings, encode surrogates and characters up to 0x7fffffff

• UTF16CSE UTF32CharSet, all valid, using UTF16 Encoding (native order), conforming to the Unicode standard, i.e. no out of order or isolated surrogates.

Internal Unicode subset types

• _LatinCSE Indicates has at least 1 character > 0x7f, all <= 0xff
• _UCS2CSE Indicates has at least 1 character > 0xff, all <= 0xffff
• _UTF32CSE Indicates has at least 1 non-BMP character

API

The cse function returns the character set encoding for a string type, string. Returns RawUTF8CSE as a fallback for AbstractString (i.e. same as String) The charset function returns the character set for a string type, string, character type, or character. The encoding function returns the encoding for a type or string. The codeunit function returns the code unit used for a character set encoding The cs"..." string macro creates a CharSet type with that name The enc"..." string macro creates an Encoding type with that name The @cse(cs, enc) macro creates a character set encoding with the given character set and encoding

Also Exports the helpful constant Bool flags BIG_ENDIAN and LITTLE_ENDIAN