Implement reverse diacriticization based on GNDB dataset

[ronaldtse]

GNDB dataset: https://github.com/interscript/geonames-transliteration-data/releases

We want to perform the following steps:

1. For every "unpointed Arabic" and "transliterated Arabic" pair
2. reverse transliterate the "transliterated Arabic" into "pointed Arabic"

Then, use this data of "unpointed Arabic" to "pointed Arabic" to verify that the Arabic diacriticization models work.

Interscript now supports reverse script conversion (need to read it up).

Ping @AhMohsen46 @gilgameshjw .

[ronaldtse]

The additional challenge here is that we may or may not know the original conversion system used, so we could technically “blast” the transliterated text using all Arabic systems and find out which system it actually came from.

[AhMohsen46]

@ronaldtse , I was just in a meeting with Jair, I was thinking that maybe instead of reverse transliterating, we can point the text, and compare the results with the destination in the DB

I'll still implement the reverse transliteration anyways, cause I know we'll eventually need it, but it might take more time than the first approach

[ronaldtse]

@AhMohsen46 but who's going to point the text? I don't mind as long as get useful data for testing.

In any case we need to do reverse transliteration because one of the project goals is to detect the transliteration systems used in GNDB entries.

[AhMohsen46]

@ronaldtse the model by Jair, so, we'll be testing it anyway, and seeing if it's working as we expect

-- yes I agree, we're gonna work on the reverse transliteration