Web Crawlers
Web crawler refers to a program or script that systematically and automatically browses websites (Kausar et al., 2013; Lawson, 2015). With the widespread use of the internet in recent years, it has provided people with vast amounts of information, often in unstructured form (Sirisuriya, 2015), making the task of finding relevant and valuable information time-consuming. Therefore, the ability to automatically discover valuable information from the web has been developed as a response to information overload (Lu et al., 2017). Through web crawlers, it becomes possible to quickly locate interesting content in this vast and complex internet landscape without manually searching websites (Hillen, 2019).
The application of web crawlers in various fields is not uncommon (Khder, 2021). The data collected through web crawling not only saves time but also lays the foundation for a significant amount of basic data for data mining (Bar-Ilan, 2001; Thelwall, 2001). This allows for more in-depth analysis and information applications, such as market analysis, price comparison, trend analysis, and more (García-Mendoza & Juárez Gambino, 2022; Gendreau et al., 2022; Lee et al., 2023; Lu et al., 2017). Data mining has become a popular contemporary topic, and to better cope with this scenario, enhancing awareness and skills in web crawling is particularly important. Therefore, this study will use air quality indicators, Taiwan Bank exchange rates, weather forecasts, and real-time weather as data collection targets.