Data Gathering, Processing, and Visualization for COVID-19

Introduction

Since December 2019, a novel coronavirus, namely Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2), which causes Coronavirus Disease-19 (COVID-19), has been spreading rapidly in the whole world. Due to its high human-to-human transmission efficiency and the severe consequence of the infection, COVID-19 has become one of the most significant health crises in modern history. World Health Organization (WHO) declared that COVID-19 is a global pandemic on March 11, 2020. According to WHO, a total of 236,599,025 individuals have been confirmed as infected cases worldwide, including 4,831,486 deaths, as of 6:49 CEST, October 8, 2021 (WHO, 2020). The highest numbers of COVID-19 cases have been reported in the United States of America (44.2M), followed by India (33.9M), Brazil (21.5M), the United Kingdom (8.05M), and Russia (7.58). Till date, the COVID-19 pandemic is ongoing despite the availability of medications and high vaccination rates in some regions, due to a series of COVID-19 variants, such as B.1.1.7 (Alpha), B.1.351 (Beta), B.1.617.2 (Delta), etc. (Rubin, 2021; Bernal et al., 2021). Fighting against the COVID-19 pandemic has become the main theme in the past two years. Research communities, public health organizations, government authorities, and industrial sectors have been involved to overcome the challenges of the pandemic, while many innovative disease control measures have been proposed by using cutting-edge technologies, such as big data analytics, artificial intelligence (AI), Internet of Things (IoT), and 5G (Rodríguez-Rodríguez et al., 2021; Siriwardhana et al., 2020).

Various state-of-the-art technologies have been applied to deal with the current urgency of the pandemic, while big data is the foundation that supports many other applications, such as epidemiological models, pandemic trend predictions, and machine learning tasks. One such noteworthy component is tracking major measurements of COVID-19 through visual mining tools, while big data processing plays a crucial role among the full cycle of the project development. Big data analytics is an automated process which uses a set of techniques or tools to access large-scale datasets for extracting useful information and insights. Such a process involves a series of customized and proprietary steps. It requires specific knowledge to handle and operate the workflow properly. Due to the nature of 4Vs (Volumes, Variety, Velocity and Veracity), a robust, reliable and fault-tolerant data processing pipeline is essential for any big data project. The sustainable automation can consolidate all tasks throughout ETL, data warehousing, and data visualization. The upstream vendor data will be ingested into a data lake, where source data is maintained and go through the data processing of cleaning, scrubbing, normalization and insight extraction. Prior to the visual mining and data analytical tasks, data should be preprocessed towards a user-friendly manner.

A massive amount of data visualizers and dashboards has been generated to track and analyze the feature of COVID-19 spreading patterns. Data can be classified into temporal measurements (i.e. infection, recovery, death counts, hospitalizations, test results over time), geographical information (i.e. disease measurements at different geo-spatial levels), and demographical data (such as disease measurements per race). However, gathering appropriate data for tracking the pandemic facets of COVID-19 is still challenging due to differences of regional reported records and the complexity of data formats. Data visualization, on the other hand, has become one of the most powerful tools for tracking data related to COVID-19, along with many web-based visualizers and dashboards. Prominent example dashboards include Coronavirus COVID-19 Global Cases by the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University (JHU), WHO Coronavirus Dashboard, Coronavirus Data in the United States by New York Times (NYT), the U.S. Centers for Disease Control and Prevention (CDC) COVID Data Tracker, etc., in which big data techniques have been widely applied. It is important to compare different regions in terms of identifying potential emerging hotspots and assessing possible effects of disease control strategies by understanding the dynamics of the pandemic.