Speaking of updating news archive sections on Xenite.Org, earlier this month I found a pretty serious issue was blocking the auto-archive function for Xenite’s Hobbit Movie News site. Xenite.Org has been reporting on Tolkien- and Lord of the Rings movie-related news longer than any other Web site but we had to change our format years ago.
Late last year I had the idea of auto-archiving the story links we were collecting from the Web (many sites do this already). I built a prototype system to prove to myself I could do it and then tried to do that for the Hobbit movie news site, which I expanded to include Guilermo del Toro news (and Tolkien news). All seemed well but I forgot I had years ago implemented some controls on the Xenite server to prevent some of the better known robots from scraping its content.
Well, that trick effectively blocked the auto-archive feature so from July to early December I managed to collect hardly any news archives at all. I had not been happy with the way I structured the archive directories anyway so I finally found time (and energy) earlier this month to yank the old archive and re-engineer the entire thing.