Repository navigation
Add support for merging overlapping scripts - #268
Conversation
|
I'm taking a look at this "with help", which adds the caveat that I don't really know what I'm talking about. So I'll first check our understanding:
Anyway, to the extent I'm getting this right I have very favorable views about the PR. One thing I was wondering was whether it might be worth plumbing through an exception to |
Just reviewed that doc and yeah the cost function ends up naturally doing much of what's talked about in there.
That's right, at the moment we use a simple check to identify which scripts a font supports which is: if there is at least one codepoint unique to the script which is found in the font, then we treat the script as supported (relevant code: https://github.com/w3c/ift-encoder/blob/main/ift/config/auto_segmenter_config.cc#L248). Once a script is detected then we will do all of the usual merging optimization for it. I opted for an overly conservative 1 codepoint threshold for now, but I think we'll want to raise that up higher. That would have the effect of disabling optimization for incomplete scripts. I'll need to do a bit of research and figure out a reasonable approach to the threshold.
That makes sense, I've opened an issue for it: #269 |
…tors. Retains existing interface while remainder of the code is migrated to support multiple probability calculators per strategy.
…babilities as appropriate.
Also reworks how MergeStrategy's are constructed. Cost based now takes probability profile + added unigram/bigram constructors of probability profiles.
… per group. SegmenterConfigUtil is updated to handle the new fields. The pre-existing fields for specifying frequency data are left, but marked as deprecated. Mixing legacy and the new fields is an error.
When deciding if two scripts overlap we look at the probability weight of overlapping codepoints and ignore overlaps below a threshold. This avoids pulling together too many scripts that have small inconsequential overlaps into a single merge group.
…n merge group. When a primary script has been requested we should aim to optimize the segmentation for that script, so if the primary script is sharing a merge group with other scripts it will be moved into it's own merge group. The left over scripts in the group will be given coverage of any codepoints not covered by the primary script.
… script. Previously this defaulted to Script_latin if not explicitly set.
This time the cache is located with the probability calculator instances. Modifies probability calculator API so that segment indices can be optionally supplied + cache invalidation triggers.
48754ba to
6b0ee36
Compare
|
Couple of things that I'll need to do before this is ready:
|
|
Another data point:
This shows a cost comparison to the existing behaviour of segmenting against a single Unified CJK script vs using a multi script cost function from this PR. We see signficant accross the board cost improvements using the new multi script cost function. Still need to run a similar analysis across a larger collection of fonts. As before targetting a single primary script improves performance on that script at the cost of the others. |
|
Additional issue that I'll need to fix: table keyed segments are currently derived from the merge groups. With this change you can get multiple scripts placed into a single merge group which would result in one large table keyed segment instead of one per script. Will need to rework automatic table keyed segment generation to account for this. Will likely want to have auto config emit explicit table keyed segments if scripts have been merged into a single merge group instead of relying on the |
…t with the base segment. Previously CollectCompositeCandidateMerges() would check all composite conditions for merge with base. The vast majority of these are not competive merges but are rechecked on each iteration. This change limits candidates to only those which include base segment, which are more likely to produce cost improvements. Tested against the Google Fonts corpus. Overall showed fairly consistent speedups and cost reductions on average. For fonts with large composite condition sets drastic speedups are observed. Additionally this adds a de-duplication step to composite candidate collection which avoids rechecking equivalent merges (including skipping segment pair merges previously checked during CollectExclusiveCandidateMerges).
In the merged bigram probability computation we are computing the probability of a new segment resulting from merging two or more existing segments. Previously this was done by generating the merged segment's definition and computing the probability from scratch, which involves computing all codepoint pair probabilities. This reworks the merged probability computation: - Per segment probability calculations now cache the various intermediate sums needed to compute Kounias bounds for that segment. - Then in the merged computation the previously cached intermediate sums are used as a starting point and augmented with any missing inter-segment codepoint pairs. - Ultimately this means we need to only deal with inter segment code point pairs and can skip all intra segment code point pairs. Gives moderate speedups to probability calculations for quality levels 3 and above (those using bigrams).
…n the candidate delta. Modify ComputeCostDelta() so that all of the positive delta contributions are added up last. Then we can stop iteration early as soon as the inprogress delta exceeds to current best delta. Since the inprogress sum is done in non-deterministic order we require the inprogress sum exceed the best delta by more than the maximum amount of floating point error. This maintains overall determinism. In tests w/ bigram probability on a CJK font gave a ~14% speedup.
32f1638 to
b20b05d
Compare
Instead of using merge groups which may be the result of merging multiple input scripts, construct them from the input frequency data sets. Deprecates segmenter config 'generate_table_keyed_segments' and adds 'generate_table_keyed_segments_mode' which has a new third mode FROM_FREQ_DATA.
|
Got results back from batch testing using the unified CJK script vs multi script cost calculation on CJK fonts. It shows cost improvements for nearly all families. Cost delta distribution:
As expected there is increased segmentation time for using multi script (though a few cases got faster). Segmentation time distribution:
Both runs use the optimizations I just added, so this is a comparison only between single vs multi script. |
It now estimates how many table keyed patches will be produced by a config and configures max depth if necessary to keep the total count below a threshold.
- Move merger ReassignInitSubset() to after fallback move to init font. Since that also affects the init font definition. - Don't mix live segment codepoints with cached segment information. - In calculator_storage use unique_ptr's so we don't have to worry about resize invalidating pointer addresses. - Use post segmentation initial segment instead of pre when constructing table keyed segments. - Handle case where primary script has not intersection with font.
Previously discussed here: https://github.com/w3c/ift-encoder/blob/main/docs/segmenter.md#multiple-frequency-data-sets-and-merge-groups
Fixes: #261 and supports w3c/ift-encoder-data#8
The approach:
As an example here is a table of per script final cost after merging with and without a primary script specified on Noto Sans JP:
When JP is selected as the primary script the JP final cost is much lower then in the no primary case, however the cost of this is that the cost for the other three scripts are increased relative to the no primary case.