Paper Details: Downloads: 1631
Serial Number: P1121636514
Title: Arabic Full Text Diacritization using Light Layered Approach
Authors: Aya S. M. Hussein and Mohsen A. A. Rashwan and Amir F. Atiya
Abstract: Text diacritic restoration is a very vital problem for languages that use diacritics in their orthography systems. Actually, it plays an important role for improving the performance of many NLP tasks. In this paper, we handle the problem of Arabic text diacritization; such that our system diacritizes input sequence of words both morphologically and syntactically. The operation of the system is divided into three layers; each layer handles a specific problem. To evaluate the performance of the system, we used the benchmark LDC Arabic Treebank datasets used by the state of the art systems, for the sake of fair comparison. Besides, we also used an extra test set to give an indication of the real performance of the system on any totally independent data set. For morphological diacritization, we use both Hidden Markov Model and an external morphological analyser to achieve high accuracy as well as high coverage. The morphological diacritization WER achieved by the system on the benchmark test set is 3.7%. We also introduce the use of Random Forest for the syntactic diacritization and show how this simple and light classifiers is very effective such that it outperforms very powerful classifiers by achieving syntactical WER of 8.3%. Finally, we provide time analysis for each component of the proposed system
Keywords: Arabic Text Diacritization, Natural Language Processing, Machine Learning
Journal/Conference: International Journal of Artificial Intelligence and Machine Learning
Volume: 16
Issue: 1
Submission Date: 8/29/2016 12:00:00 AM
Review Date: 9/12/2016 12:00:00 AM
Publishing Date: 10/2/2016 12:00:00 AM
Article Downloads: 1631
Download:

Facebook