Blog

Following my proposed experiment in data destruction, a few kind readers tried it out and let me know what happened¹. I’ve summarised the results below, to try and see if there’s any common pattern.

Read More

Digital Preservation  Keeping Codes  Lessons Learned

Let’s start with an experiment…

Read More

Digital Preservation  Keeping Codes  Lessons Learned

I find working in digital preservation fascinating.

It’s not where I expected to end up. I started off interested in computing and science, and happened to find out about what was then a fairly young MPhys degree in Computation Physics offered by the University of York¹. I then did a Ph.D. in Computational Physics at Edinburgh University, working in statistical physics. After that, I spent my time oscillating between being a post-graduate researcher who used large-scale computational methods, and being a computational specialist who helped other scientists make use of those kinds of techniques.

I’d decided to move away from research and get a ‘normal’ industry programmer job, so when we moved to Leeds I applied for a few different positions. One of them turned out to be for the PLANETS Project, based at the British Library. I liked the place and the people, and the work sounded interesting, allowing me to expand my previous experience (not just in computation, but also the information theory that underlies statistical physics) to a new field. And Industry was spared my woolly ways.

I spent a happy few years working on the PLANETS Project and helping...

Read More

Digital Preservation  Keeping Codes  Lessons Learned

I came to work on digital preservation through the PLANETS project, and later the SCAPE project (for the first year) before moving over to web archiving. These were inspiring projects which achieved a great deal, but we were left with lessons to be learned.

Read More

Format Registry  Digital Preservation  Collaboration

Four years ago, during the 2012 IIPC General Assembly, we came together to discuss the recent and upcoming challenges to web archiving in the Future of the Web Workshop (see also this related coverage on David Rosenthal’s blog). That workshop made it clear that our tools are failing to satisfy many of these challenges:

• Database driven features
• Complex/variable URI formats
• Dynamically generated URIs
• Rich, streamed media
• Incremental display mechanisms
• Form-filling
• Multi-sourced, embedded content
• Dynamic login, user-sensitive embeds
• User agent adaptation
• Exclusions (robots.txt, user-agent, …)
• Exclusion by design (i.e. site architecture intended to inhibit crawling and indexing)
• Server-side scripts, RPCs
• HTML5 web sockets
• Mobile sites
• DRM protected content, now part of the HTML standard
• Paywalls

I wish I could stand here and tell you how much great progress we’ve made in the last four years, ticking entries off this list, but I can’t. Although we’ve made some progress, our crawl development resources have been consumed by more basic issues. We knew moving to domain crawling under Legal Deposit would bring big changes in scale, but I’d underestimated how much the dynamics of the crawl workflow would need to change.

News websites are a great example.

Read More

Web Archives

Originally published on the UK Web Archive blog on the 15th of February 2016.

Earlier this year, as part of the Big UK Domain Data for the Arts and Humanities project, we released our first ‘historical search engine’ service. We’ve publicised it at IDCC15, the 2015 IIPC GA and at the first RESAW conference, and it’s been very well received. Not only has it lead to some excellent case studies that we can use to improve our services, but other web archives have shown interest in re-using the underlying open source code. In particular, some of our Canadian colleagues have successfully launched webarchives.ca, which lets users search ten years worth of archived websites from Canadian political parties and political interest groups (see here for more details).

But we remained frustrated, for two reasons. Firstly, when we built that first service, we could not cope with the full scale of the 1996-2013 dataset, and we only managed to index the two billion resources up to 2010. Secondly, we had not yet learned how to cope with more than one or two users at a time, so we were loath to...

Read More

Data Mining  Web Archives  BUDDAH