In difference to hash tables that eliminate duplicate states on-the-fly during the exploration, the process of duplicate detection can be delayed until a large set of states is available. It is very effective in external search, where it is efficiently achieved by external sorting and scanning.
Published in Chapter:
Disk-Based Search
Stefan Edelkamp (University of Dortmund, Germany) and Shahid Jabbar (University of Dortmund, Germany)
Copyright: © 2009
|Pages: 6
DOI: 10.4018/978-1-59904-849-9.ch076
Abstract
The need to deal with large data sets is at the heart of many real-world problems. In many organizations the data size has already surpassed Petabytes (1015). It is clear that to process such an enormous amount of data, the physical limitations of RAM is a major hurdle. However, the media that can hold huge data sets, i.e., hard disks, are about a 10,000 to 1,000,000 times slower to access than RAM. On the other hand, the costs for large amounts of disk space have considerably decreased. This growing disparity has led to a rising attention to the design of external memory algorithms (Sanders et al., 2003) in recent years. In a hard disk, random disk accesses are slow due to disk latency in moving the head on top of the data. But once the head is at its proper position, data can be read very rapidly. External memory algorithms exploit this fact by processing the data in the form of blocks. They are more informed about the future accesses to the data and can organize their execution to have minimum number of block accesses. Traditional graph search algorithms perform well as long as the graph can fit into the RAM. But for large graphs these algorithms are destined to fail. In the following, we will review some of the advances in the field of search algorithms designed for large graphs.