Advanced Applications on Bilingual Document Analysis and Processing Systems

Introduction

With the increased demand of document digitization, many automated N-lingual analysis and processing systems come into existence which processes the heterogeneous text-based and image-based documents in different application areas. The evolution of such N-lingual document categorizers and processors give birth to the need of an integrated, generic and robust offline document classification system, whose design must handle these documents in plain text, printed image, and handwritten image forms. Bilingual Document Analysis and Processing Systems (BDAPS) is a type of N-lingual processing systems (Puri and Singh, 2016) that implement multi-variant algorithms with 2-lingual heterogeneous document sets. These systems use the concepts of Natural Language Processing (NLP), artificial intelligence, machine learning, image processing and text mining fields. Today, a large area of this globe use to generate such type of documents in one or two languages, where one language is the world communication language, English, and other one is the native language. In India, many Government and other legal documents are found in English as well as Hindi (or native) like voter card, Aadhar card, driving license etc. Few Indian documents contain contents in three languages also, English, Hindi and native languages, as some Indian states follow 3+/- language formula.

Over two decades, many NLP and image processing applications have been developed with Indian monolingual and bilingual text documents. As such NLP and image processing are broad areas in which numerous bilingual processing-based applications reside. NLP based text processing systems include the solutions for those applications, which work on plain text documents, whereas image processing-based text processing systems include the solutions for those applications, which require scanned printed and handwritten documents. In general, their primary applications include text extraction, identification, discrimination and recognition tasks for both plain text and image documents (Hassan, Garg, Chaudhury, & Gopal, 2011). Additionally, they include NLP based applications like text and web page classification, Named Entity Recognition (NER), Machine Translation (MT), and Cross Lingual Information Retrieval (CLIR); and the image processing applications like physical and logical layout extraction, Optical Character Recognition (OCR), word recognition, line extraction, script identification, character segmentation, printed and handwritten lines discrimination and content identification (Marinai, 2008; Nevetha & Baskar, 2015). However, all these systems are found limited in scope for different perspectives.

Although many NLP systems exist for monolingual document classification, but no system exists for bilingual document classification. Secondly, no system exists for document image classification. Thirdly, no existing system processes both image and non-image documents together. Fourth limitation is that no bilingual English-Hindi classification system exists, which categorizes monolingual and bilingual documents both. Fifth limitation is that there exists no generic and integrated system, which works upon all types of heterogeneous plain text, scanned printed and scanned handwritten documents. Lastly, the slow progress of research development in Indian languages (Puri & Singh, 2018) becomes another limitation. Therefore, there is a need of a generic and integrated document classification system. This paper introduces a new branch of BDAPS, called Bilingual English-Hindi Document Classification System (BE-HDCS), which is a generic and integrated system to classify pure-text non-image and image documents into mutually exclusive pre-defined categories (Puri & Singh, 2019; Puri and Singh, in press). The pure-text documents contain only text with no diagrams, pictures, and photos. These documents exist in plain text, scanned printed or scanned handwritten forms.