Preservation Intent - Selective Web Harvesting

NLA Digital collections: Statement of Preservation Intent

Collection Area: Selective Web Harvesting

The Library's selective web harvesting program, currently consists of the PANDORA Web Archive collection which contains a selective collection of web publications and websites relating to Australia and Australians. The PANDORA Web Archive was established by the National Library in 1996 and therefore contains historic online materials harvested from 1996 to the current period. Online materials (ranging from discrete publications to complete web sites) are selected for inclusion in the collection with the purpose of providing long-term and persistent access to them.

Web Archiving intends that:

• All PANDORA digital preservation masters including all associated metadata (currently known as the preservation master, the display master and the metadata master) should be retained in perpetuity. All technical properties should be maintained to the full extent possible.
• Content, connections and context are of primary importance. How it is ultimately presented to a user is a secondary consideration.
• The original harvested copy, that is the preservation master that represents the initial and untouched collection of files gathered by the harvest robot, is of less importance than the display master which is the copy that includes the results of quality assurance and curatorial work. However, as the implications of curatorial and QA work upon harvested instances may not be known, the preservation master, (although a lesser version in terms of completeness), should be retained at least at the bit level.
• The derivative copy derived from the display master and created for display and access should be maintained only for as long as useful; a new derivative version may be generated according to future access requirements.

The Library understands that web archives are problematic in that:

• The Library has no control over the creation of the original content and consequently its format, standards or quality and can only harvest what is delivered in a single published form through a browser/server request (e.g. the original data in the publishers databases are not collected);
• Current methods for collecting and rendering are also not ideal in ensuring the complete capture of all files or retaining full functionality in the version delivered from the archive;
• We are only taking artefactual snapshots of web content.

Therefore, the NLA accepts that what is to be preserved is not a mirror representation of the web, or even a website, but a snapshot of content that was once arranged and published as a website, with only limited functionality of the original. The archival artefact is formed out of the web collection process which is inevitably a lossy process in itself. Our concern is to define and control this loss. In addition, the way in which the content is collected and displayed may mean that there is significant limitation on the presentation of the archived artefact as an authentic record of the publisher's original data or even the version of that data as originally published on the web.

Other Preservation Issues:

• The intention is long term access for all users. However, over time access to certain content may only be available on site due to technical considerations.
• The harvested web content being complex objects are contained in either a compressed package (i.e. tarball) or a container file (i.e. WARC file).  While the tarball retains the directory structure of the original harvested website, the WARC file may contain random collections of files plus metadata which are managed and located by indexes.
• The PANDORA collection, having begun in 1996 includes content collected through various methods and has acquired a legacy of inconsistency in respect to URIs, metadata and quality assurance interventions. Processes are underway to move the content to a consistent archival format (WARC) although the underlying legacy variations may not necessarily be removed in this process.
• The PANDORA collection can be broadly categorised as consisting of about 75% text, 20% images (JPEG, GIF, PNG) and 5% multimedia and style elements (Java script, CSS files, Flash etc.) including linkages. Because of the variable nature of the collected entity (the PANDORA title and its archival instances) ranging from simple documents to complex multi-file objects, there are some parts of the collection  where style elements are more important, and some parts where this is less so. Style elements are problematic from the outset since they are sometimes difficult to harvest and remain often impossible to render (e.g. Heritrix wayback sometimes does not render some older content at all). Because content is harvested through a browser type request on a server in many cases only a subset of possible style element files are delivered (those required for the browser request). Moreover, harvesters are not able to thoroughly parse complex JavaScript which may also result in the collecting process not identifying and missing many style elements (JS, XML etc).
• Contemporary browsers are fairly tolerant for accessing both current and legacy web content. However, due to the variability of this content (collected from 1996 to the present day) and factors such as being poorly formed (no standards) can mean that viewing content as it was at the time of creation can be problematic.
• The status of the visual accuracy of the harvested copy of a site has not been kept in any systematic way (although implied in QA workflow). Thus, the look of the original may only be surmised from the content collected, the context of embedded links, tags and file types and the context of technologies known to exist at the time of harvesting.
• The Library's objective is not to misrepresent the material in any way that would compromise its legal warrant to collect, preserve and make accessible the archival content. Thus particular care in retaining the integrity of the intellectual content including embedded links and domain related image material is a priority.
• This collection is currently in a state of transition in how it is stored, described and understood via technical metadata.
• Below is an anecdotal list of file types that are known problems in the archive. Primarily the problems are first and foremost in how they content is managed in the archive rather than the file format itself:
  • RealMedia files – Files in this format are a main concern given that there would be quite a bit of this content in the archive, it took a lot of work to obtain and the files are generally significant parts of the sites. The archive includes a number of sites that include .rm and .ra files. Since these files were delivered in their live environment using a metafile (.ram) to point to the actual media file hosted on a RealMedia server, the NLA required the publishers to send us the .rm or .ra files which we then hosted on an RTSP enabled server. The continued support for this server at some point broke down – without notice to the business area – and consequently those archived sites with RealMedia content delivered in this way no longer function and it is uncertain if the files have been discarded.
  • VRML – This content required a plug-in (e.g LivePictureViewer) to function which does not seem to have been incorporated into browsers like other media players. This content will not function in the archive.
  • Shockwave – Older sites with Shockwave such as Director (.dcr) files do not work. Seems to be missing necessary plug-in components.
  • Quicktime VR (Virtual Reality) does not seem to work.

Version 1.0 1 March 2013