This Julia package computes various distances between strings (ASCII)

The function `compare`

returns a similarity score between two strings. The function always returns a score between 0 and 1, with a value of 0 being completely different and a value of 1 being completely similar.

```
using StringDistances
compare(Hamming(), "martha", "martha")
#> 1.0
compare(Hamming(), "martha", "marhta")
#> 0.6666666666666667
```

- Damerau-Levenshtein Distance
`DamerauLevenshtein()`

- Hamming Distance
`Hamming()`

- Jaro Distance
`Jaro()`

- Levenshtein Distance
`Levenshtein()`

Q-gram distances compare the set of all substrings of length `q`

in each string.

- QGram Distance
`Qgram(q)`

- Cosine Distance
`Cosine(q)`

- Jaccard Distance
`Jaccard(q)`

- Overlap Distance
`Overlap(q)`

- Sorensen-Dice Distance
`SorensenDice(q)`

- RatcliffObershelp Distance
`RatcliffObershelp()`

The package includes distance "modifiers", that can be applied to any distance.

Winkler boosts the similary score of strings with common prefixes. The Winkler adjustment was originally defined for the Jaro similarity score but this package defines it for any string distance.

`compare(Jaro(), "martha", "marhta") #> 0.9444444444444445 compare(Winkler(Jaro()), "martha", "marhta") #> 0.9611111111111111 compare(QGram(2), "william", "williams") #> 0.9230769230769231 compare(Winkler(QGram(2)), "william", "williams") #> 0.9538461538461539`

Modifiers from the Python library fuzzywuzzy .

- Partial returns the maximal similarity score between the shorter string and substrings of the longer string.
`compare(Levenshtein(), "New York Yankees", "Yankees") #> 0.4375 compare(Partial(Levenshtein()), "New York Yankees", "Yankees") #> 1.0`

- TokenSort adjusts for differences in word orders by reording words alphabetically.
`compare(RatcliffObershelp(), "mariners vs angels", "angels vs mariners") #> 0.44444 compare(TokenSort(RatcliffObershelp()),"mariners vs angels", "angels vs mariners") #> 1.0`

- TokenSet adjusts for differences in word orders and word numbers by comparing the intersection of two strings with each string.
`compare(Jaro(),"mariners vs angels", "los angeles angels at seattle mariners") #> 0.559904 compare(TokenSet(Jaro()),"mariners vs angels", "los angeles angels at seattle mariners") #> 0.944444`

- TokenMax combines scores using the base distance, the
`Partial`

,`TokenSort`

and`TokenSet`

modifiers, with penalty terms depending on string lengths.`compare(TokenMax(RatcliffObershelp()),"mariners vs angels", "los angeles angels at seattle mariners") #> 0.855`

## Compare vs Evaluate The function`compare`

returns a similarity score: a value of 0 means completely different and a value of 1 means completely similar.

- Partial returns the maximal similarity score between the shorter string and substrings of the longer string.

In contrast, the function `evaluate`

returns the litteral distance between two strings, with a value of 0 being completely similar.

```
compare(Levenshtein(), "New York", "New York")
#> 1.0
evaluate(Levenshtein(), "New York", "New York")
#> 0
```

As a rule of thumb,

- Standardize strings before comparing them (correct for uppercases, punctuations, whitespaces, accents, abbreviations...)
- Don't use Edit Distances if word order do not matter.
- The distance
`Tokenmax(RatcliffObershelp())`

is a good default choice.

- The stringdist Package for Approximate String Matching Mark P.J. van der Loo
- fuzzywuzzy blog post

10/22/2015

4 days ago

127 commits