Abstract
COVID-19 began in 2019, and by the advent of 2020, it had become widespread and adversely affected the world. In this work—Covid Live, COVID-19 data is scraped from an online website, which gives an overview of the status of the pandemic in the desired format. The authors built an application interface using a Python micro web Flask framework. The data scraping uses a multi-threading concept to reduce the program's runtime error, resulting in receiving the data quickly, and faster than existing web crawlers and scrapers. This paper focuses on dealing with storing scraped data in the desired format. It also provides options to hear the audio of the scraped data and to download the scraped data. The authors present visualizations of current trends with scraping period details and demonstrate an efficient application that does the data scraping quickly and efficiently.
TopIntroduction
The COVID-19 pandemic has infected many people (Yang et al., 2020). Furthermore, in attempts to slow down the spread of the novel virus, nations imposed complete and partial lockdowns hoping to prevent further spread of the virus (Atalan, 2020). Many lives have been affected by the pandemic. Scientists and researchers are constantly analyzing the impact and devising methods to purge the spread of the virus further and predict possible variants. Therefore, positive case patterns, environmental and biological factors, and policies are crucial for COVID-19 research (Sha et al., 2021). Therefore, there should be a database for maintaining records or spatiotemporal COVID-19 records that countries publish from virus testing after the advent of 2020. As per the public data, most data come from a few international agencies, such as WHO, GHC, or CDC. The subcommittees within the organization make the data public after collecting information and producing the dataset (COVID Data Tracker, 2021).
Also, it seems that data is the new differentiation. It is the focus of market research and business strategy. Whether one has to start a new work or shake out a new strategy for an existing business, one needs to access and analyze a large amount of data for better results. Web scraping plays a role in easing up the process, and in the present scenario, large institutions are also collaborating to provide facilities to track COVID-19 in real-time. For example, Johns Hopkins University developed a COVID-19 dashboard that is regularly updated by extracting data from around eight non-government sources and publicly providing it as a single dataset (Dong, Du, & Gardner, 2020). 1Point3Acres provides a similar service, which aims to be transparent to the public about COVID-19 cases (1point3acres Global COVID-19 Tracker & Interactive Charts, 2021).
Web scraping is the process of mining data, usually unstructured data from any number of sources, efficiently and faster and storing it as structured data for further analysis (Sirisuriya, 2015). It is effortless because it does not involve visiting the web pages to copy-paste the extensive data. Data extraction can be done from any website, anywhere, no matter how large and complex the data is. Moreover, some websites may have the type of data that cannot be copied and pasted directly. For example, it can be in the form of CSV, image, or text. Web scraping can be copying, grabbing, pasting text, or parsing HTML (Sirisuriya, 2015). Further, many tools and methods are available for web scraping (Persson, Evaluating tools and techniques for web scraping, 2019; Saurkar, Pathare, & Gode, 2018). Not only in python but also in other programming languages (Easily harvest (scrape) web pages, n.d.).
Web data stored in text or CSV format is for further analysis and operations. Web scraping shortens the process of extracting data, as it is equivalent to a human interacting with the web page. It increases the interaction simulation speed by automating the process and creates easy access to the scraped data by providing it in any format (Vargiu & Urru, 2013; Breton et al., 2015; Salerno & Boulware, 2006). Web scraping is an essential process, especially when dealing with a large quantity of data, as it leads to the quick and efficient extraction of relevant information from different sources. Covid Live does the same but with some additional features. The subcommittees within the organization make the data public after collecting information and producing the dataset (COVID Data Tracker, 2021).
Key Terms in this Chapter
Parsing: It is the analysis and processing of a sequence of symbols in any language or data structure, be it computer-based or human interpretable, that follows formal grammar rules for that language or data-structure.
Progressive Web Application: Also called as PWAs are web-delivered applications that can be downloaded on mobile devices to act like a native application without specifically and separately writing code for that mobile device operating system.
HTTP Request: HTTP Requests are requests for an action to be performed sent by the client to the server. Some examples of HTTP Requests are GET, POST, PUT, etc.
Web Scraping: Web scraping is the extraction of desired information from web pages for processing or any use.
Parse Tree: A parse tree is a set of the terminal and non-terminal symbols arranged in a hierarchical tree-like structure that symbolizes the syntactic structure of a sequence of symbols based on some context-free grammar.
Beautiful Soup: Beautiful Soup is a package in python that parses the HTML and XML documents to extract relevant information.
Thread: A thread is a sequential control flow or instructions (programmed) which is also called a lightweight process and can be managed by a scheduler independently, within an Operating System.
Multithreading: Running multiple threads concurrently is multithreading. It is within the same process and shares the process resources, but executes independently.