|Julia||Windows||Linux & MacOS||Package Evaluator||CodeCov||Coveralls|
It represents an attempt to give Julia better string handling than possible with Base
I am now trying to make sure that all of the functionality in String and Char is implemented for Str and Chr, and to start optimizing the functions (although they are already substantially faster)
Strs.jl is now a container for a number of different packages from JuliaString.org
|Package||Release||Release Date||Linux & MacOS||Windows||Description|
|ModuleInterfaceTools||Tools to create a common API for all of these packages|
|StrAPI||Common API for string/character functionality|
|CharSetEncodings||Basic types/support for Character Sets, Encodings, and Character Set Encodings|
|MurmurHash3||Pure Julia implementation of MurmurHash3|
|Format||Python/C style formatting (based on Formatting)|
|StrLiterals||Extensible string literal support|
|StrFormat||Formatting extensions for literals|
|StrTables||Low-level support for entity tables|
|HTML_Entities||HTML character sequences|
|Emoji_Entities||Emoji names (including composite ones)|
|LaTeX_Entities||Julia LaTeX character names|
|Unicode_Entities||Unicode standard character names|
|StrEntities||Entity extensions for literals|
|InternedStrings||Save space by interning strings (by @oxinabox!)|
The new package ModuleInterfaceTools is used to set up a consistent and easy to use API for most of the cooperating packages, without having to worry too much about imports, exports, using, and what functions are part of a public API, and which ones are part of the internal development API for other packages to extend.
I would very much appreciate any constructive criticism, help implementing some of the ideas, ideas on how to make it perform better, bikeshedding on names and API, etc. Also, I'd love contributions of benchmark code and/or samples for different use cases of strings, or pointers to such (such as a way to get lots of tweets, to test mixed text and emojis, for example).
The general philosophy of the architecture is as follows: have a single easy to use type that can replace
String that conforms to the recommendations of the Unicode Organization (which internally uses 4 types and is implemented currently as a Julia Union, and has O(1) indexing to characters, not just code units), as well as types to represent binary strings, raw unvalidated strings (made up of 1, 2, or 4 byte codepoints), as well as types for validated ASCII, Latin1, UCS2 (16-bit, BMP [Basic Multilingual Plane]), UTF-8, UTF-16, and UTF-32 encoded strings.
Optimizations for multi code unit encodings such as UTF-8 & UTF-16 will be moved to
packages (before splitting them out, I'll make sure that the functionality still works, only with slower generic methods, so that you only take up the extra space if you need the faster speed).
Extensions such as converting to and from non-Unicode encodings, such as Windows CP-1252 or China's official character set, GB18030, will be done in another package,julia-observer-quote-cut-paste-9_work`.
Since things like
which I originally implemented for this package have now been added tojulia-observer-quote-cut-paste-12work
(in master), I have moved support for those to a file to provide them for v0.6.2.
There is ajulia-observer-quote-cut-paste-13work
iterator, which iterates over the unsigned integer codepoints in a string
(for strings with thejulia-observer-quote-cut-paste-14work
trait, it is basically the same as thejulia-observer-quote-cut-paste-15_work`
Also in the works is using the new ability to add properties, in order to allow for different "views" of a string, no matter how it is stored internally, for example a
mystring.utf8 view, or a
mystring.utf16 view (that can use the internal cached copy if available, as an optimization).
Currently, there are the following types:
Str, which is the general, parameterized string type, and
Chr, the general, parameterized character type.
BinaryStr for storing non-textual data as a sequence of bytes.
ASCIIStr an ASCII string, composed of
ASCIIChr 1-byte codepoints
LatinStr a string using the Latin1 subset of Unicode, composed of
LatinChr 1-byte codepoints.
UCS2Str a string composed of characters (
UCS2Chrs) only in the Unicode BMP, stored as 2 byte code units (that each store a single codepoint)
UTF32Str a string with only valid Unicode characters, 0-0xd7ff, 0xe000-0x10ffff, stored as 4 byte code units.
UTF8Str a string with only valid Unicode characters, the same as
UTF32Str, however encoded using UTF-8, conforming to the Unicode Organization's standard, i.e. no long encodings, surrogates, or invalid bytes.
UTF16Str a string similar to
UTF8Str, encoded via UTF-16, also conforming to the Unicode standard, i.e. no out of order or isolated surrogates.
a text string that may contain any sequence of bytes
a text string that may contain any sequence of 16-bit words
a text string that may contain any sequence of 32-bit words
a Union type, that can be one of the following 4 types, ASCIIStr
, and 3 internal types:
'µ': (Unicode U+00b5 (category Ll: Letter, lowercase) and
'ÿ': Unicode U+00ff (category Ll: Letter, lowercase)julia-observer-quote-cut-paste-43work
'Μ': Unicode U+039c (category Lu: Letter, uppercase) and
'Ÿ': Unicode U+0178 (category Lu: Letter, uppercase)julia-observer-quote-cut-paste-45work
in the name.
For all of the built-in Str
types, there is a corresponding built-in character set encoding, i.e. BinaryCSE
There are also a number of similar built-in character sets defined (CharSet
), and character types (*Chr
function returns the character set encoding for a string type or a string.
returns the character set, and encoding
returns the encoding.
For example, cse(UTF8Str)
There is a new API that I am working on for indexing and searching, (however there is a problem on v0.7 due to the deprecation for find
being overbroad, and causing a type of type piracy, deprecating methods of types not in Base):
* find(First, ...)
* find(Last, ...)
* find(Next, ...)
* find(Prev, ...)
* index(str, pos)
replaces thisind(str, pos)
* index(Next, ...
* index(Prev, ...
Also there are more readable function names that always separate words with `julia-observer-quote-cut-paste-78_work`is
* In addition, I've added is_alphabetic
Nobody is an island, and to achieve great things, one must stand on the shoulders of giants.
I would like to thank some of those giants in particular:
* The four co-creators of Julia: [Jeff Bezanson](https://github.com/JeffBezanson),[Viral B. Shah](https://github.com/ViralBShah), [Alan Edelman](https://github.com/alanedelman), and [Stefan Karpinski](https://github.com/StefanKarpinski), without their uncompromising greediness, none of this would be possible.
* [Tom Breloff](https://github.com/tbreloff), for showing how an ecosystem could be created in Julia, i.e. "Build it, and they will come", for providing some nice code in this [PR](https://github.com/JuliaIO/Formatting.jl/pull/10) (which I shamelessly pirated in order to create [Format](https://github.com/JuliaString/Format.jl), and for good advice at JuliaCon.
* [Ismael Venegas Castelló](https://twitter.com/SalchiPapa1337) for encouraging me to [tweet](https://twitter.com/GandalfSoftware) about Julia starting at the 2015 JuliaCon, for good advice, and being a great guy in general.
* [Chris Rackaukas](https://github.com/ChrisRackauckas) simply a star in Julia now, great guy, great advice, and great blogs about stuff that's usually way over my head. Julia is incredibly lucky to have him.
* [Jacob Quinn](https://github.com/quinnj), for collaborating & discussions early on in [Strings](https://github.com/quinnj/Strings.jl) on ideas for better string support in Julia, as well as a lot of hard work on things dear to me, such as databases and importing/exporting data [SQLite](https://github.com/JuliaDatabases/SQLite.jl), [ODBC](https://github.com/JuliaDatabases/ODBC.jl), [CSV](https://github.com/JuliaData/CSV.jl), [WeakRefStrings](https://github.com/JuliaData/WeakRefStrings.jl), [DataStreams](https://github.com/JuliaData/DataStreams.jl), [Feather](https://github.com/JuliaData/Feather.jl), [JSON2](https://github.com/quinnj/JSON2.jl)
* [Milan Bouchet-Valat](https://github.com/nalimilan), for discussions on string handling and encoding in [StringEncodings](https://github.com/nalimilan/StringEncodings.jl)
* [Tim Holy](https://github.com/timholy) for the famous "Holy" Trait Trick, which I use extensively in the Str* packages, for the work along with [Matt Bauman](https://github.com/mbauman) on making Julia arrays general, extensible while still performing well, and hence very useful in my work.
* [Steven G. Johnson]() for illuminating me on how one could create a whole package in very few lines of code when I first started learning Julia, see [DecFP](https://github.com/stevengj/DecFP.jl)
* [Tony Kelman](https://github.com/tkelman), for very thorough reviews of my PRs, I learned a great deal from his (and other Julians') comments), including that I didn't have to code in C anymore to get the performance I desired.
* [Lyndon White](https://github.com/oxinabox), I've already "appropriated" :grinning: his very nice [InternedStrings](https://github.com/JuliaString/InternedStrings.jl) into this package, I'm really lucky to have gotten him to join the organization!
* [Bogumił Kamiński](https://github.com/bkamins) who has been doing a great job testing and reviewing Strs` (as well as doing the same for the string/character support in Julia Base), as well as input into the design. (Also very glad to have co-opted him to become a member of the org)
Last but not least, Julia mathematical artist (and blogger!) extraordinaire, Cormullion, creator of our wonderful logo!
Kudos to all of the other contributors to Julia and the ever expanding Julia ecosystem!
about 1 month ago