Improve Finnish hyphenation - #32
Conversation
|
Sorry for the ping @laurmaedje, but any chance to get this merged and maybe into hypher 0.1.8? |
|
Do you know which of these patterns are in use in other projects? Is this also the default in LaTeX now? Also, what does "recall" mean as opposed to "precision"? I feel like I'm missing some context or definitions. |
|
Sorry, I don't know about Liang pattern sets other projects use, nor really anything about LaTeX (mostly because Typst exists 😄). I do know that these results are objectively better to me (a native Finnish speaker); this whole thing started out because I noticed Typst was hyphenating some Finnish in a bad way. I know Voikko is used by large projects like OpenOffice, etc. for Finnish, but as noted above, it's a full morphological hyphenator, which is a bit of a different beast. For our purposes here, it served as a good machine-drivable gold standard. Recall here means "of all the breaks Voikko says exist, what fraction the PR found", and precision "of all the breaks the PR proposes, what fraction Voikko agrees with". |
This PR switches
Lang::Finnishtohyph-fi-x-schoolpatterns, replacing the Saarinenhyph-firules that haven't been updated in... a while. They're older than me, originally...The school rules follow the simpler syllabification rules taught in Finnish schools and, crucially, include systematic vowel-sequence break rules (
i1o,e1a,o1a, …) that the old set omitted.This both improves accuracy for Finnish as well as shrinks the embedded trie. I originally noted this with the word "äänioikeus", which was mishyphenated as
ää-nioi-keus(whenää-ni-oi-keusis correct).The new set was validated mechanically against Voikko, which is a full morphological hyphenator for Finnish, with a 109K word list.
For this set, 7 540 native words become newly correct and 316 become newly wrong, which translates to a roughly 24:1 improvement.
The residual errors can't be resolved by Liang patterns:
The old hand-written Saarinen file had a bunch of hand-written patterns (see the comments in the file linked above) that technically help keeping clusters together in certain internationalisms (that would have been common in 1980-1990 TeXnical writing). However, for general use, they break things (499 words fixed c.f. Voikko, 1 976 words broken). So, no use adding them back.