Today marks 30 years since the invention of the World Wide Web. Fortunately, the National Library of Australia and its partners have been archiving Australian websites and preserving our history for generations to come.
Our Assistant Director-General, Collections Management, Amelia McKenzie witnessed the beginning of web archiving at the National Library. Read her fascinating insights into how the National Library began to capture the internet.
Today, we mark 30 years since the World Wide Web began and I can’t help thinking back to when the National Library began to archive the web. It’s quite an achievement.
Twenty-two years of collecting Australian websites, capturing Australian society itself at a unique moment when it found this new medium, connected with it and began to express itself through it. If the Web Archive were human it would be finishing up university or if not (can’t have everything), in its first job - all grown up!
But where did it come from - and why exactly did it start?
The National Library and other heritage collecting institutions like the state and territory libraries - and our counterparts overseas - are often called memory institutions. We have collections dating back centuries that document history, society, what happened, how people lived and thought; ‘deep memories’, according to Alex Byrne, former state librarian of NSW.
But all memories start somewhere. I was there at the very beginning of web archiving at the NLA (as were several of my colleagues), and it’s interesting to think now about the choices and decisions we made then. We had no knowledge that 20-odd years later we would be opening up access to a massive new collection - 600 TBs of rich and diverse web-based content for people to find and explore.
In 1996 the internet was still just beginning. Back then, fewer than a quarter of households owned a computer and only a tiny 4% of households even had a modem - that was how the ABS described internet connectivity back then. If you were lucky enough to have a PC with access to the internet at work, you might have had Pine email, and discovered listservs for the first time. In the NLA’s collection we welcomed CDs and floppy disks - those apparently indestructible marvels of technological progress, the solution to all our problems of storing and accessing written information in digital form. (It was a revolution that didn’t last long, fortunately, as the life of digital information stored on physical media turns out be quite short without intervention to preserve it).
From CDs and other physical formats, the web itself really challenged us. This was not something you could hold in your hand, it was ‘born’ digital; but the point was that it was published information and needed to be acquired and added to the national collection. We talked about it, put together guidelines about what to collect, brought together some like-minded people, worked with our IT colleagues and found ways of copying and archiving websites on the 1996 version of the Australian web. On the non-IT side, none of our staff was trained or had experience of this work, so it was a matter of finding people with an aptitude and an interest, teaching ourselves what was involved and just starting it. Everyone understood was that it was about taking responsibility for Australian published content, collecting it, cataloguing it and storing it for the future, and that was what the NLA was there to do. Change was in the air.
We downloaded the first two websites in October 1996, and another 30 or so by the end of the financial year in June 1997. That’s how small this operation was - it was highly experimental, and took some of our smartest people on the library and IT side to get their heads around things like data models and business processes.
As soon as we were sure enough about policy, procedures and systems, we made it a national project and invited partner institutions to use our software and to collaborate in a shared project to archive material in their jurisdictions. Today our partner colleagues are still much the same group - the mainland state and territory libraries, the War Memorial, National Gallery, AIATSIS. That group created what we originally called the PANDORA Archive, which is now at the heart of the Australian Web Archive. I’ve always felt that our web archiving partnerships, like our microfilming and digitisation collaborations, were examples of the way the NLA worked best - cooperatively and towards shared goals, starting small but providing leadership when it was needed.
To that content, over time, we added Commonwealth and state government websites, and in 2005 began archiving single annual snapshots of the entire Australian web domain (later going back to complete the collection with the addition of content back to 1996). So I’ve also had the privilege of seeing what an entire snapshot of the Australian web actually looks like, essentially big hard drives in a box, holding unimaginable quantities of data. We knew back then that we were creating an enormous future technical challenge to make it accessible one day, and in all those intervening years it was an act of faith to keep on harvesting the Australian web domain. I’m pleased that the Library colleagues who were there at the beginning, building, curating and creating this massive slice of Australian memory have the satisfaction of seeing it made accessible - the last and perhaps most important part of our remit.
To explore Australia's history on the internet, visit the Australian Web Archive.
