My last visit to the soon-to-close New Worlds got me:
Gaps in my library about sf:
Trillion Year Spree, by Brian Aldiss and David Wingrove
The Encyclopedia of Fantasy, edited by John Clute and John Grant
Tolkien and the Great War, by John Garth
Fiction that was on my “buy on sight in paperback list” anyway:
The Wanderer, by Fritz Leiber
The Assassin’s Edge, by Juliet E. McKenna
Emerald Magic, edited by Andrew Greeley
Reduced to £1.99 each:
Wandering Stars, edited by Jack Dann
Brian W. Aldiss, by Michael R. Collings
Aldiss Unbound, by Richard Mathews
Reduced to £1:
Red Dwarf, first series, episodes 4-6 (video)
After that, I did manage to resist the temptations of Forbidden Planet!
Speaking as someone who actively writes such web crawlers – it doesn’t have to be a “search engine”.
Do you have any more logs of that activity? IP address? User Agent?