= رُبابَة RABABA the Middle-Eastern Language Diacritization Library

Middle-Eastern Language diacritization is useful for several practical business cases like text to speech or Romanization of texts or scripts.

Purpose

This repository contains everything to train a diacritization model in Python and run it in Python and Ruby.

It supports the following languages:

• Arabic

• Hebrew

Library

This library was built for the Interscript project (at GitHub).

Diacritization strategy is following several steps with at heart a deep learning model:

1. text preprocessing

2. neural networks model prediction

3. text postprocessing

This repository contains:

• lib is the Ruby library using NNet model in ONNX format.

• docs contains an application focused summary of latest (2021-06) relevant papers and solutions.

• python

  • A neural network solution for automatised diacritization based on the work of almodhfer, from which we overtook the baseline and more advanced and efficient CBHG models only. This very recent solution allows for efficient predictions on CPU’s with a reasonable sized model.

  • PyTorch to ONNX conversion of PyTorch to ONNX format

  • Strings Pre-/Post-processing, also from almodhfer

• tests and benchmarking utilities, allowing to compare with other implementations.

  • tests are taken from diacritization benchmarking

  • we have added own, realistic datasets for the problem of diacritization

• models-data directory to store models and embeddings in various formats

About the name

A Rababa is an antique string instrument.

In a similar fashion that a Rababa produces melody from a simple strings and pieces of wood, our library and diacritization gives a whole palette of colour and meanings to arabic scripts.

Try out Rababa

Rababa can be run both in Python and Ruby. Go to the directory corresponding to the language you prefer to use.

Training of Rababa models are only available in Python.

Python (3.8+)

Prerequisites

The Python version of Rababa supports Python 3.8 and later. We recommend using a version manager like asdf or pyenv.

Important

PyTorch Compatibility: Rababa works with PyTorch 1.7.0 and later. Different versions of PyTorch may handle model loading differently, especially with the weights_only parameter introduced in PyTorch 2.6+. If you encounter model loading errors mentioning weights_only, you might be using a newer PyTorch version that is incompatible with the saved models. The setup steps below have been tested and should work across PyTorch versions.

With asdf:

# Install asdf (if not already installed)
# See: https://asdf-vm.com/guide/getting-started.html

# Install Python plugin
asdf plugin add python

# Install desired Python version
asdf install python 3.9.13

# Set Python version for this project
asdf local python 3.9.13

# Verify your Python version
python --version

With pyenv:

# Install pyenv (if not already installed)
brew install pyenv

# Install desired Python version
pyenv install 3.9.13

# Set Python version for this project
pyenv local 3.9.13

# Verify your Python version
python --version

Arabic Diacritization

1. Install the required dependencies:

   cd python/arabic
   pip install --upgrade pip
   pip install torch numpy matplotlib pandas ruamel.yaml tensorboard diacritization-evaluation tqdm onnx onnxruntime pyyaml

2. Download the pre-trained model:

   mkdir -p log_dir/CA_MSA.base.cbhg/models
   curl -sSL https://github.com/secryst/rababa-models/releases/download/0.1/2000000-snapshot.pt -o log_dir/CA_MSA.base.cbhg/models/2000000-snapshot.pt

3. Run diacritization:

   # Diacritize a single text
   python diacritize.py --model_kind "cbhg" --config config/cbhg.yml --text 'قطر'
   
   # Diacritize a text file
   python diacritize.py --model_kind "cbhg" --config config/cbhg.yml --text_file path_to_input_file --diacritized_text_file path_to_output_file

Example Output

# Input:  قطر
# Output: قَطُرُ

# Input:  مرحبا كيف حالك انا بخير
# Output: مَرْحَبًا كَيْفَ حَالُك انَّا بِخَيْرٍ

Note

If you encounter model loading errors with newer versions of PyTorch (especially 2.6+), you may see messages related to the weights_only parameter. The code has been updated to handle this compatibility issue, but if you still experience problems, please refer to the Troubleshooting section at the end of this document.

Hebrew Diacritization

1. Install the required dependencies:

   cd python/hebrew
   pip install --upgrade pip
   pip install torch numpy matplotlib pandas ruamel.yaml tensorboard diacritization-evaluation tqdm onnx onnxruntime pyyaml

2. Download the pre-trained model and configuration:

   mkdir -p log_dir/base.cbhg
   curl -sSL https://github.com/secryst/rababa-models/releases/download/hebrew.0.1/bchg_len90_dec0.9766_wor_0.8877.yml -o config/model_hebrew.yml
   curl -sSL https://github.com/secryst/rababa-models/releases/download/hebrew.0.1/model_len90_dec0.9766_wor_0.8877.pt -o log_dir/base.cbhg/model.pt

3. Run diacritization:

   # Diacritize a single text
   python diacritize.py --model_kind "cbhg" --config config/model_hebrew.yml --text 'מה שלומך'
   
   # Diacritize a text file
   python diacritize.py --model_kind "cbhg" --config config/model_hebrew.yml --text_file path_to_input_file --diacritized_text_file path_to_output_file

Example Output

# Input:  מה שלומך
# Output: מָה שֶׁלוֹמְךַ

# Input:  אני אוהב לקרוא ספרים
# Output: אֲנִי אוֹהֵב לִקְרוֹא סְפָרִים

Note	The Hebrew model has similar PyTorch compatibility considerations as the Arabic model. The fixes applied ensure smooth operation across different PyTorch versions.

For more detailed instructions, please see the language-specific READMEs:

• Arabic

• Hebrew

• Ruby

Under development

We are working on the following improvements:

• Enhancing architecture and encoding

• Enhancing datasets to improve models

License and copyright

Rababa is copyright (c) 2021-2025, Ribose Inc. All rights reserved.

Rababa is licensed under the BSD-2 Clause license. See the LICENSE.adoc file for details.

Attributions

General

The Rababa team would like to express their appreciation for the open-source work of these authors and researchers:

• M. A. H. Madhfar and A. M. Qamar for their work on effective deep learning models for automatic diacritization of Arabic text

• Taha Zerrouki for the original Tashkeela dataset

The team acknowledges the contributions of these authors and researchers in the field of Arabic diacritization and recognizes the importance of their work in advancing the state of the art in this area.

Rababa does not redistribute any code or data from these attributed sources. Any redistribution of these attributed sources should be done in accordance with their respective licenses.

Rababa is not responsible for any issues that may arise from the use of these external sources. These sources are provided for reference purposes only, and their use is at the user’s own risk.

Arabic diacritization models

The neural network solution for Arabic diacritization is based on the work of M. A. H. Madhfar:

• Repository: https://github.com/almodhfer/Arabic_Diacritization

• License: MIT License

• Citation: M. A. H. Madhfar and A. M. Qamar, "Effective Deep Learning Models for Automatic Diacritization of Arabic Text," in IEEE Access, vol. 9, pp. 273-288, 2021, doi: 10.1109/ACCESS.2020.3041676.

Tashkeela dataset

The Tashkeela dataset used for training is provided under GPL v2 license:

• Original dataset by Taha Zerrouki: https://sourceforge.net/projects/tashkeela/

• Processed dataset by Hamza Abbad: https://sourceforge.net/projects/tashkeela-processed/

• License: GPL v2

Troubleshooting

PyTorch Compatibility Issues

If you encounter model loading errors related to the weights_only parameter in PyTorch, try one of the following solutions:

1. For Hebrew models:

   If you see errors when loading the Hebrew model, verify that the fix for the weights_only parameter is applied in the Hebrew config_manager.py file. The load_model function should not include the weights_only parameter:

   saved_model = (
       torch.load(model_path)
       if torch.cuda.is_available()
       else torch.load(model_path, map_location=torch.device("cpu"))
   )

2. For Arabic models:

   If you see errors when loading the Arabic model, the weights_only parameter should be explicitly set to False in the config_manager.py file:

   saved_model = torch.load(last_model_path, weights_only=False) if torch.cuda.is_available() else torch.load(last_model_path, map_location=torch.device('cpu'), weights_only=False)

3. Using a specific PyTorch version:

   If problems persist, installing a specific PyTorch version compatible with the saved model formats may help:

   pip install torch==1.13.1

4. Model conversion:

   For persistent issues, you may need to load and re-save the models with your current PyTorch version:

   # Load model with explicit weights_only=False setting
   old_model = torch.load("model.pt", weights_only=False)
   
   # Save model in a compatible format
   torch.save(old_model, "new_model.pt")