Optimising Cloud Operation Cost Utilising External Memory

Background

Systems designed to handle large amounts of data are prevalent in scientific computing, cloud computing, “big data” processing, HPC, and distributed/cluster computing. Databases and archival storage require guarantees of integrity that function at scale (Doorn & Rivero, 2002; Ailamaki & Papadomanolakis, 2007). Medical imaging needs precision to be effective, and precision is guaranteed with greater detail, necessitating larger datasets. In the realm of the Internet large companies such as Google, Facebook, Amazon, etc. build and maintain huge software systems, and therefore must deal with enormous amounts of data passing through them to ensure acceptable operating times for end users of their web-based products. For surveying purposes, in the Architectural, Engineering, and Construction (AEC) industries, TLS and similar methods produce point clouds that contain millions of points representing scanned 3D space encompassing gigabytes of data. Point cloud files are multidimensional and extendable, i.e. each data point may capture spatial coordinates, surface normals, colour information, etc. The multidimensional nature of each individual point beyond a three-value Cartesian coordinate raises its memory requirement, which can be a significant chunk of memory when the volume of points stored grows.