Article Preview
TopIntroduction
The understanding of textual documents is based on the semantic meaning of each word in the natural language when studied via social networks, machine translation, information extraction, sentiment analysis, text classification, and other semantic-based natural language processing research. However, the semantics are generally unclear to a computer due to the existence of a high number of out-of-vocabulary (OOV) terms. Therefore, a semantic understanding of OOV is an obstacle to overcome in the field of natural language processing.
The China Internet Network Information Center’s (CNNIC, 2022) 49th Statistical Report on the Development Status of China’s Internet showed that the size of China’s Internet users reached 1.032 billion as of December 2021. This number was an increase of 42.96 million users from December 2020. Such a large user base provides a rich corpus for Chinese natural language processing-related research.
A Chinese sentence includes several consecutive words. To understand the semantics of Chinese, it is necessary to divide the sentence into strings of words, with each word serving as a basic unit. There is no obvious separation between words in Chinese; therefore, the wrong separation can lead to ambiguity (Blythe et al., 2012). This creates challenges in finding OOV in text (see Figure 1). As a phenomenal short text-based real-time social network, Twitter provides a rich corpus of information for natural language processing (Murthy et al., 2019). Similarly, to study Chinese OOV, Ahmed et al. (2022) noted that social media platforms provide unique opportunities for conducting social science and Web-based research. Therefore, this study chose the Twitter-like Chinese media social network, Weibo (https://www.weibo.com), as its corpus (Zhu et al., 2021). Weibo contains many Chinese colloquialisms and slang, which provides useful information when exploring OOV and semantics.
Figure 1.
Examples of ambiguity in the same Chinese sentence with different word separation
In this work, the authors seek to extract Chinese OOVs from social networks to understand the meaning of Chinese OOVs in social networks. The authors propose to use information from the context to understand OOVs. The current study was inspired by Nagy et al. (1987), in which context was used to understand the meaning of words. To utilize this approach to understanding word meaning, the authors must extract, analyze, and use relevant contextual information from the OOV. The word2vec’s continuous bag of words (CBOW) model can capture valid contextual information to calculate word similarity and understand word meanings.
Most OOV-related research uses a large corpus and named entity extractions. However, there are fewer studies on OOV of social networks and their lexical meanings. The contributions of this research include:
- •
Limited amount of relevant social network content as a corpus, which addresses the problem of sparse low-resource linguistic corpora
- •
Contextual information of OOV to quickly understand the meaning of OOV words, which facilitates semantic and natural language processing-related research.
- •
Use the content of people via OOV in social networks as a corpus, which makes its semantic features more controllable and effective for OOV meaning understanding