Big Data Management Challenges in a Meteorological Organisation

Big Data Management Challenges in a Meteorological Organisation

Lee Wilson (Meteorological Service of New Zealand, New Zealand), Tiong T. Goh (Victoria University of Wellington, New Zealand), and William Yu Chung Wang (Auckland University of Technology, New Zealand)
Copyright: © 2012 |Pages: 14
DOI: 10.4018/jea.2012040101
OnDemand PDF Download:
No Current Special Offers


Data management practices strongly impact enterprise performance, especially for e-science organisations dealing with big data. This study identifies the key challenges and issues facing information system managers in growing demand for big data operations to deliver timely meteorological products. Data was collected from in-depth interviews with five MetService information system managers, including the CIO. Secondary data sources include internal documents and relevant literatures. The study revealed the pressing and challenging big data management issues can broadly be classified as data governance, infrastructure management, and workflow management. The study identifies a gap in adopting effective workflow management system and coordinated outsourcing plan within the organisation. Although the study is limited by its sample size and generalisation, the findings are useful for other IT managers and practitioners of data-intensive organisations to examine their data management practices on the need to balance the demand for efficient scientific operations and sustainable business growth. This study recognised that although the organisation is implementing up-to-date and practical solutions to meet these challenges, effort is needed to harmonise and align these solutions with business growth strategies to sustain future growth. This study enhanced societies’ understanding to the current practices of a real world organization.
Article Preview

1. Introduction

Data management is the development, execution and supervision of plans, policies, program and practices that control, protect, deliver and enhance the value of data and information assets (DAMA, 2010). Aiken et al. (2006) define the goal of data management is to understand the current and future data needs of an enterprise and making the data effective and efficient in supporting business activities. A recent survey by Aiken et al. (2011) concurred that successfully investing in data management is a current real challenge to IT and organisations. Haug and Arlbjørn (2011) revealed that adoption of good quality data management practices is a pre-condition for the efficiency of a company and many organisations do not give adequate attention to data management issues. A recent survey by the Economist Intelligence Unit further revealed that many companies are still struggling with the most basic aspect of data management (SAS-Big Data, 2011). Recent concerns over global warming, climate change and the energy commodity market are driving a global demand for more and more sophisticated meteorological data processing and presentation technologies. The demand for weather data and products from computer forecast models is increasing dramatically (Alexander et al., 2011). Weather information is continually consumed globally by millions of presentation systems such as TV stations, websites, mobile devices, email, and fax machines. Advances in storage, processing, networking and multimedia technologies have enabled scientists and companies to create complex global models that account for more and more measurements with higher resolution, and greater forecast horizons. This resulted in many organisations facing big data management challenges that will have a great impact on organisations performance (Alexander et al., 2011; Szalay, 2011; SAS-Big Data, 2011; “Data, data everywhere,” 2010).

This paper presents a case study on the big data management challenges facing MetService. Over time the volume of data generated by MetService systems creating products for the market and provisioning their weather forecasting tools has increased exponentially. This growth in data is further fuelled by an emergence of technologies that allow MetService to develop larger and more sophisticated weather forecast models and presentation applications and to distribute larger volumes of more complex data models across a greater variety of networks and global locations. All this data needs to be efficiently and effectively ingested, catalogued, processed, formatted, replicated, and distributed. MetService is facing the challenge to remain viable as a public and commercial service. Inefficient data management practices in organisations will have a significant impact on business performance and are extremely costly (Yoon et al., 2000; Haug & Arlbjorn, 2011).

Therefore in the domain of big data management, the main research question is: what are the main challenges and issues facing information system managers in MetService and how they react to the situation. The outline of the paper is as follows:

  • Section 2 introduces to the company and provide background information

  • Section 3 reviews of the existing framework on large data management

  • Section 4 describes the research method

  • Section 5 describes the main findings learned from the case

  • Section 6 provides the recommendations

  • Section 7 provides a conclusion

Complete Article List

Search this Journal:
Volume 15: 2 Issues (2023)
Volume 14: 3 Issues (2022): 2 Released, 1 Forthcoming
Volume 13: 2 Issues (2021)
Volume 12: 2 Issues (2020)
Volume 11: 2 Issues (2019)
Volume 10: 2 Issues (2018)
Volume 9: 2 Issues (2017)
Volume 8: 2 Issues (2016)
Volume 7: 2 Issues (2015)
Volume 6: 2 Issues (2014)
Volume 5: 4 Issues (2013)
Volume 4: 4 Issues (2012)
Volume 3: 4 Issues (2011)
Volume 2: 4 Issues (2010)
Volume 1: 4 Issues (2009)
View Complete Journal Contents Listing