Showing posts with label spiders. Show all posts
Showing posts with label spiders. Show all posts

11 September 2007

The Story of Our Lives

TechCrunch notes the rise of a new class of services:

part blogging, part genealogy and part something unique. They are focused on the very long term - getting and then keeping customers for decades, and encouraging friends and especially family members to join, too. Once they’re hooked, they’ve spent so much time building content that they are very unlikely to ever leave.

Nicely viral, obviously, but what really interests me is the potential of making all of this information freely available across the Net, rather than just locking it in silos (as I imagine much will be). Imagine an intelligent spider, searching, sifting and correlating the information: it would allow a tapestry of life to be spun across the entire planet (or at least those parts using such sites). Tricky privacy issues, of course....

27 March 2006

Searching for an Answer

I have always been fascinated by search engines. Back in March 1995, I wrote a short feature about the new Internet search engines - variously known as spiders, worms and crawlers at the time - that were just starting to come through:

As an example of the scale of the World-Wide Web (and of the task facing Web crawlers), you might take a look at Lycos (named after a spider). It can be found at the URL http://lycos.cs.cmu.edu/. At the time of writing its database knew of a massive 1.75 million URLs.

(1.75 million URLs - imagine it.)

A few months later, I got really excited by a new, even more amazing search engine:

The latest pretender to the title of top Web searcher is called Alta Vista, and comes from the computer manufacturer Digital. It can be found at http://www.altavista.digital.com/, and as usual costs nothing to use. As with all the others, it claims to be the biggest and best and promises direct access to every one of 8 billion words found in over 16 million Web pages.

(16 million pages - will the madness never end?)

My first comment on Google, in November 1998, by contrast, was surprisingly muted:

Google (home page at http://google.stanford.edu/) ranks search result pages on the basis of which pages link to them.

(Google? - it'll never catch on.)

I'd thought that my current interest in search engines was simply a continuation of this story, a historical relict, bolstered by the fact that Google's core services (not some of its mickey-mouse ones like Google Video - call that an interface? - or Google Finance - is this even finished?) really are of central importance to the way I and many people now work online.

But upon arriving at this page on the OA Librarian blog, all became clear. Indeed, the title alone explained why I am still writing about search engines in the context of the opens: "Open access is impossible without findability."

Ah. Of course.

Update: Peter Suber has pointed me to an interesting essay of his looking at the relationship between search engines and open access. Worth reading.