Abstract
The natural language processing of Arabic dialects faces a major difficulty, which is the lack of lexical resources. This problem complicates the penetration and the business of related technologies such as machine translation, speech recognition, and sentiment analysis. Current solutions frequently use lexica, which are specific to the task at hand and limited to some language variety. Modern communication platforms including social media gather people from different nations and regions. This has increased the demand for general-purpose lexica towards effective natural language processing solutions. This chapter presents a collaborative web-based platform for building a cross-dialectical, general-purpose lexicon for Arabic dialects. This solution was tested by a team of two annotators, a reviewer, and a lexicographer. The lexicon expansion rate was measured and analyzed to estimate the overhead required to reach the desired size of the lexicon. The inter-annotator reliability was analyzed using Cohen's Kappa.
TopIntroduction
Arabic is the official language of 22 countries, one of the six official languages of the United Nations and has more than 420 million speakers (Kamusella, 2017). Arabic has three varieties: classical Arabic (CA) used in Quran and old literature, modern standard Arabic (MSA) used in formal communication nowadays and dialectal Arabic (DA) which is the spoken variety (Habash, 2010). The last decade has experienced a growing interest in the natural language processing of dialectical Arabic. This growth is attributed to the wide usage of this language variety in social media (Guellil et al., 2019) and to the role of these media in the Arabic revolutions (Aouragh, 2016).
The topics treated by computational linguists for Arabic dialects can be assigned to one of four main categories (Shoufan & Alameri 2015):
- 1.
Basic language analysis such as morphological and syntactical analysis (Maamouri et al. 2006; Habash et al., 2012; Masmoudi et al., 2015; Khalifa, 2017; Zalmout, 2018).
- 2.
Building language resources such as lexica, corpora, treebanks, and wordnets (Graff et al., 2006; Cotterell and Callison-Burch, 2014; El-Beltagy, 2016; Qwaider et al., 2018).
- 3.
Semantic-level analysis and synthesis, such as machine translation and sentiment analysis (Sawaf 2010; Duwairi et al., 2014; Medhaffar et al., 2017).
- 4.
Identifying Arabic dialects, e.g., using speech recognition techniques (Kirchhoff and Vergyri 2005; El Haj et al., 2018; Ali, 2018).
Besides well-known challenges in NLP for Modern Standard Arabic (MSA) such as the lack of short vowel letters and the non-existence of capitalization, (Farghaly & Shaalan, 2009) processing Arabic varieties is challenged by the lack of language resources such as lexicons and annotated corpora. The few proposed solutions for building lexicons in the literature show two major restrictions. First, the lexicons are specific to one dialect only. This restriction is especially problematic when social media data should be processed because social networks cross the geographic borderlines and allow addressing topics that are of interest to most Arabic people regardless of their regions or countries. Thus, a relevant part of social media data can be described as multi-dialectical. Processing this data using a single-dialect language resource is not expected to provide desired performance. Second, the proposed lexicons are restricted in terms of annotation width, and some of them are even application-specific, e.g., for PoS tagging or sentiment analysis. The authors believe that the development of general-purpose lexicons is essential to serve different NLP applications. This does not only facilitate interfacing the lexicon to different tools. It also allows the community to concentrate its efforts towards building a sophisticated and comprehensive lexicon that can be used as a reference in many applications.
Key Terms in this Chapter
Platform-as-a-Service (PaaS): A cloud service that provides users with a platform to create and manage applications without worrying about scalability or administration.
Lexicon Builder: A tool that helps in collaborative building of a lexicon.
Relational Database: A structured set of data with relations between stored items of information.
Modern Standard Arabic: A formal variety of the Arabic language that is used in today’s official communications.
Arabic Dialect: A variety of the Arabic language which is mostly spoken but has no standard written format.
Lexical Annotation: The process of assigning attributes to a word which should be added to a lexicon.
Interrater Agreement: A statistical measure of the level of agreement between two or more raters.