CharSetEncodings

Julia Version	Unit Tests	Coverage
	[![][]][]
Julia Latest

Architecture

This provides the basic types and mode methods for dealing with character sets, encodings, and character set encodings.

Types

Currently, there are the following types:

• CodeUnitTypes a Union of the 3 codeunit types (UInt8, UInt16, UInt32) for convenience
• CharSet a struct type, which is parameterized by the name of the character set and the type needed to represent a code point
• Encoding a struct type, parameterized by the name of the encoding

Built-in Character Sets / Character Set Encodings

• Binary For storing non-textual data as a sequence of bytes, 0-0xff

• ASCII ASCII (Unicode subset, 0-0x7f)

• Latin Latin-1 (ISO-8859-1) (Unicode subset, 0-0xff)

• UCS2 UCS-2 (Unicode subset, 0-0xd7ff, 0xe000-0xffff, BMP only, no surrogates)

• UTF32 UTF-32 (Full Unicode, 0-0xd7ff, 0xe000-0x10ffff)

• UniPlus Unvalidated Unicode (i.e. like String, can contain invalid codepoints)

• Text1 Unknown 1-byte character set

• Text2 Unknown 2-byte character set

• Text4 Unknown 4-byte character set

Built-in Encodings

• UTF8Encoding
• Native1Byte
• Native2Byte
• Native4Byte
• NativeUTF16
• Swapped4Byte
• Swapped2Byte
• SwappedUTF16
• LE2
• BE2
• LE4
• BE4
• UTF16LE
• UTF16BE
• 2Byte
• 4Byte
• UTF16

Built-in CSEs

• BinaryCSE, Text1CSE, ASCIICSE, LatinCSE
• Text2CSE, UCS2CSE

• Text4CSE, UTF32CSE

• UTF8CSE UTF32CharSet, all valid, using UTF8Encoding, conforming to the Unicode Organization's standard, i.e. no long encodings, surrogates, or invalid bytes.

• RawUTF8CSE UniPlusCharSet, not validated, using UTF8Encoding, may have invalid sequences, long encodings, encode surrogates and characters up to 0x7fffffff

• UTF16CSE UTF32CharSet, all valid, using UTF16 Encoding (native order), conforming to the Unicode standard, i.e. no out of order or isolated surrogates.

Internal Unicode subset types

• _LatinCSE Indicates has at least 1 character > 0x7f, all <= 0xff
• _UCS2CSE Indicates has at least 1 character > 0xff, all <= 0xffff
• _UTF32CSE Indicates has at least 1 non-BMP character

API

The cse function returns the character set encoding for a string type, string. Returns RawUTF8CSE as a fallback for AbstractString (i.e. same as String) The charset function returns the character set for a string type, string, character type, or character. The encoding function returns the encoding for a type or string. The codeunit function returns the code unit used for a character set encoding The cs"..." string macro creates a CharSet type with that name The enc"..." string macro creates an Encoding type with that name The @cse(cs, enc) macro creates a character set encoding with the given character set and encoding

Also Exports the helpful constant Bool flags BIG_ENDIAN and LITTLE_ENDIAN