Within the web, it is possible to find an important portion of the knowledge of humankind in terms of studies and development. Preserving versions of this web content is required for future reference. File systems or databases are two main alternatives for preserving historical web content. Since databases offer a convenient way to organize and query data with respect to the file systems, they may be preferred for preserving historical web content. Thus, the goal of this chapter is to propose techniques in order to store collections of web files on a database management system. Five techniques were developed and experimentally compared in terms of performance and space. Experimental results show that the proposed techniques based on the change detection require a smaller amount of space, they are more efficient in terms of load on the database, and they have lower performance in terms of processing web file to store.
TopIntroduction
Millions of files are left at the public’s disposition each day, the minute they’re published on the Internet. The amount of information residing there greatly exceeds the one present at any library or physical storeroom in the world.
Within the web, it is possible to find contents regarding politics, society, culture and education. This last subject involves all that’s related to research papers and topics of academic courses which conform an important portion of the knowledge of humankind in terms of studies and development. This is why the emergence of the need to preserve this content for future reference seems only natural.
In this context, suppose the Professor Hernandez’s homepage1; Professor Hernandez is a member of the Computer Science Department at the Universidad Simón Bolívar. He runs his own website under the university’s platform, using it to publish all the information pertinent to every course the professor’s ever given, including a general description of the subject, as well as one web page per time he’s imparted it. Any memo, evaluation date, exam solution and student grade is published on this website. Thus, it represents the main source of studying material for the alumni in that academic period, particularly because the specialization level of the courses in which the professor is responsible, is very high and other tools of the same quality are quite scarce.
Therefore, it is desirable to preserve these contents, in order to avoid inconveniences, such as the website being down and migrations or deletions of the data. To establish the reasons why it is useful to maintain a historical record of this website, three hypothetical situations may occur. First, a drastic change of the website happens. This change contemplates the possibility of rewriting already existent files; for instance, substituting the former version of the folder containing the class examples for the course by a newer one, losing the chance to search for any data regarding an older version of the course. Second, the server is down, completely preventing every possible visitor from visualizing or downloading the contents of the website. Third, the professor decides to remove all the contents from his website, the information that once was there would become unavailable to the public.
One of the well-known alternatives when it comes to preserving web content is the Wayback Machine (Internet Archive, 1996), capable of storing captures of a website through its HTML files. However, by simply doing this, it is possible that if a change occurs, such as the deletion or substitution of a referenced image, the state of that capture becomes altered to the point that it loses its original meaning. For example, suppose that there’s a website in which exists an image of a cat, called “A.jpg”. Then, Wayback Machine is used to obtain a capture of this website and later on, that “A.jpg” is replaced by another image of the same name showing a dog. Hence, the moment the content captured by the Wayback Machine is opened; the original image will not appear there and for it will show the newer version of the “A.jpg” file, losing its meaning. Thus, it’s necessary to find a mechanism that guarantees the persistence of the collected information.
Once that the capture of the website is available, it must be stored in order to make its posterior use possible. This could be carried on mainly by selecting one of these two options: through the file system (which represents the most traditional way to proceed) or directly inside a database. Both approaches allow data organization, access and manipulation with the greatest integrity and quality possible.