Searching for an Answer

It was the arrival of the first-generation search engines like Yahoo and Lycos in the mid-1990s that turned a collection of disparate online data into a usable source of information. Today, Google's pivotal role in online activity is even more pronounced.

So it's no surprise that people are working on search engines for Second Life - the thinking being that once you can find anything there, it will be even more useful as a tool. But in virtual worlds, it's not so simple:

Second Life isn't the same as the World Wide Web (at least in how its users perceive it), and probably shouldn't be treated the same way as web pages, routinely scanned by search-engine bots. I'm pretty sure that Linden Lab would prefer to that Second Life be as permeable and open as the WWW, but it's got to take a definitive step in this direction. Currently, there is no true public data in Second Life: Linden Lab owns the data comprising the world, including user avatars and objects. On the other hand, the company's Terms of Service indicate that invasions of privacy are prohibited (section 4.1). I don't understand how user-privacy even exists in a world owned by one private entity. Any shift in resident privacy-expectations Second Life is ultimately up to Linden Lab, which hasn't seemed to have decided whether Second Life is a country or an internet--whether it is a government presiding over population of residents, or a service-provider to hundreds of thousands of users.

The problem is that most people put stuff on the Web because they want others to find it: there is a conscious act of exposing stuff there. In Second Life, people (naively) assume that it's "like" real life, in the sense that virtual objects are private unless explicitly exposed. Alas, no: anything in Second Life is just data, and as such susceptible to being farmed by search bots. As the post above points out, people must now decide now much privacy needs to be built into the system. Where the dividing line should be drawn between private and public in the virtual world is not at all obvious.

Pipe Dream: Re-wiring the Net

The online world is awash with XML feeds. The great thing about XML is that you can grab it and do stuff with it very easily, because it's basically a structured text file. For example, you can feed one XML stream into another, combine them, and keep on piping them around. A bit like Unix pipes.

Hey, now that's an idea:

Pipes is a hosted service that lets you remix feeds and create new data mashups in a visual programming environment. The name of the service pays tribute to Unix pipes, which let programmers do astonishingly clever things by making it easy to chain simple utilities together on the command line.

What's particularly cool about this new service is the graphical approach, which looks a lot like programming flowcharts. The currently-available pipes are rather limited at the moment - this is still very new - but it's not hard to imagine some very rich stuff coming out of this. Bravo Yahoo. (Via GigaOM.)

Mapping the Fourth Dimension

Good to see that Yahoo is letting OpenStreetMap use its aerial imagery to speed up the process of free map creation.

Of course, we still need to get all those street names and features. GPS traces are by no means dead - think of the new housing estates and areas without imagery. But instead of cycling down every road you should be able to just pass lots of them at either end to get the names. Or just from memory.

Where does this bring us to? Well the ‘big map companies’ use expensive cars and expensive aircraft with expensive cameras and expensive GPS units to create maps. Maybe our GPS units are cheaper and less accurate, but does it matter? I think not. We now have all the pieces of the puzzle and we’re putting out great maps for Free using Free tools.

But I was even more impressed to see that OpenStreetMap has already mapped some of the future too: take a look at the entries here for March 2007.

dmoz RIP?

DMOZ - now called the Open Directory Project - just doesn't have the respect it deserves. That's partly because it's had more names than even Firefox/Firebird/Phoenix.

It started out as GnuHoo, but RMS took exception to that, and it became NewHoo - which Yahoo promptly took exception to. It managed to avoid the horrible ZURL (shouldn't that be Zurg?), before metamorphosing into the Open Directory Project, also known as dmoz (from directory.mozilla.org) to its friends.

But it's real importance is not as an open Yahoo: it was the direct inspiration for Nupedia - NewHoo, Nupedia - geddit? - which in turn gave rise to the complementary Wikipedia: need I say more?

So it's sad to hear that dmoz is fizzling. It may not serve much purpose at present, but it's had a glorious past. (Via John Battelle's Searchblog.)

5 Things I've Learned From a Year's Blogging

Today is the first anniversary of opendotdotdot. Since this fact is of little interest to anyone but myself, I thought it might be useful to extract what pearls of wisdom I can from the experience of writing 1,260 posts in that time.

1. Google is your friend

Over half the traffic that arrives on these pages comes from Google. Moreover, the absolute number of visitors directed here by the Google machine just keeps on going up. I can only presume that the more posts you make, the more Googlejuice you generate, and the more you move up the ranks. Indeed, when I take a look at some of the keywords people enter in Google before arriving here, it is gratifying to find this blog pretty highly placed, albeit for some pretty wacky terms (like "wackypedia" and "seed gestapo") as well as a few surprisingly mainstream ones (like "philip rosedale interview").

2. Yahoo and MSN are a waste of time

I'm constantly amazed at how utterly Google dominates the search engine field in terms of the visitors it provides. I had expected MSN to be way behind, but not Yahoo; in fact, I get more visitors from MSN than Yahoo, which barely registers at all. Maybe this says something about the kind of readers/searches that end up at my blog, but it also emphasises the importance of Google. Interestingly, Technorati also generates far more leads than either Yahoo or MSN.

3. People like tags

The first version of the blog had no tags, for the simple reason that Blogger did not offer the facility, and I was too lazy to do it by hand. But when the new Blogger beta came out with tags, I dutifully employed them. I'm glad I did, because people really like using them to search within the blog. In fact, like me, people are lazy: they just can't be bothered entering a search term in the blog search box, but they can summon up the energy to click on one of the tags. One consequence of this is that I intend to go back and tag all of the older posts, since it clearly is something people find useful.

4. People like weird stuff

I am resigned to the fact that I am completely unable to judge which posts will be popular or not. Sometimes I spend ages crafting some witty/profound/novel/hot post, only to have it roundly ignored by almost everybody. Equally, I've often knocked out a trivial/pointless/content-free post only to find everyone and their dog rush to admire its insights. Sigh.

5. It gets easier

In many ways, the most important lesson that I take away from a year's blogging is that the more you do, the easier it gets. This is not just because you learn to type faster, but also because I've found that blogging helps me think faster and maybe even better. It's also a direct consequence of the fact that it's such damn good fun.

Feast of the Behemoths

There's no doubt that the three giants of the online world are Microsoft, Google and Yahoo. What they get up to matters, so tracking what they're doing in terms of acquisitions, say - and who they're doing - is a fruitful activity. The problem, is keeping track. Enter this rather nice draggable timeline, which shows who did what, when and to whom. (Via John Battelle's Searchblog.)

Just What We Don't Need, Honest

One reason why work is going on to produce version 3 of the GNU GPL is that things have moved on quite a bit since version 2 came out in 1991. For example, the idea of providing software as a service across the Internet was in no one's mind at that time.

Today, of course, it's the backbone of companies like Yahoo and Google, and therein lies the problem. As I've written about elsewhere, the issue is that they use a lot of free software to provide those services, but give relatively little back to the communities that write it.

Now, in this they are (currently) quite within their rights, since they are not distributing any code based on free software, which is the trigger for making it open. But the larger issue is whether they should be distributing it anyway.

Someone who thinks they should is Fabrizio Capobianco. And he's come up with what he believes is a solution: the splendidly-named Honest Public License (HPL). As Capobianco explains:

The goal of HPL is to keep the community honest with itself. The use of the name "Honest" is ABSOLUTELY not intended to mean that GPL or any other licenses are dishonest. It is quite the opposite, actually. But some people are taking advantage of a GPL legal loophole and are defeating the spirit of the GPL. HPL is just GPL extended to cover the distribution of software as a service to the public. It does not take away any freedom (i.e. you can use it internally in your corporation), it just covers when someone distributes the code to the public (whether with a floppy or as a service). It is meant to keep people honest with their community.

I think this is a laudable attempt - laudable, but misguided. The last thing we need is another open source licence. In fact the plethora of licences is one of the banes of the free software world. Adding one more - however well intentioned - is only going to make things worse.

There are also practical objections. For example, releasing code under the HPL will discourage companies from using it; or they may use it and fail to open up their code, in which case it will be hard to discover that they are in breach.

I think a better solution is to get GNU GPL 3 right, and let companies that offer software as a service based on open source do the right thing. After all, as I suggested in my Linux Journal column, enormous amounts of goodwill can be generated by giving more than the licence requires, and such a development would be far better for the free software world than burdening it with yet another licence. (Via NewsForge.)

A Nod's as Good as a Wink

As I mentioned, I have started playing with Google Analytics for this blog. It's early days yet, but already some fascinating results have dropped out - I'll be reporting on them once the trends become slightly more significant than those based on two days' data....

But one thing just popped up that I thought I'd pass on. Some of my traffic has come from Wink, which describes itself as a social search engine. More specifically:

Wink analyzes tags and submissions from Digg, Furl, Slashdot, Yahoo MyWeb, and other services, plus user-imported tags from del.icio.us, and favorites marked at Wink, and figure[s] out which pages are most relevant.

So basically Wink aims to filter standard Web search results through the grid of social software like Digg, del.icio.us etc. It's a clever idea, although the results at the moment are a little, shall we say, jejune. But I'm grateful for the tip that Google Analytics - and one of my readers - has given me. Duly noted.

Google's Grief, Open Source's Gain?

The news that a judge has ordered Google to turn over all emails from a Gmail account, including deleted messages, has predictably sent a shiver of fear down the collective spine of the wired community, all of whom by now have Gmail accounts. Everybody can imagine themselves in a similar situation, with all their most private online thoughts suddenly revealed in this way.

The really surprising thing about this development is not that it's happened, but that anyone considers it surprising. Lawyers were bound to be tempted by the all unguarded comments lying in emails, and judges were bound to be convinced that since they existed it was legitimate to look at them for evidence of wrong-doing. And Google, ultimately, is bound to comply: after all, it's in the business of making money, not of martyrdom.

So the question is not so much What can we do to stop such court orders being made and executed? but What can we do to mitigate them?

Moving to another email provider like Yahoo or Hotmail certainly won't help. And even setting up your own SMTP server to send email won't do much good, since your ISP probably has copies of bits of your data lying around on its own servers that sooner or later will be demanded by somebody with a court order.

The only real solution seems to be to use strong encryption to make each email message unreadable except by the intended recipient (and even this is an obvious weakness).

It would, presumably, be relatively simple for Google to add this to Gmail. But even if it won't, there is also a fine open source project called Enigmail, which is an extension to the Mozilla family of email readers - Thunderbird et al. - currently nearing version 1.0. The problem is that installation is fairly involved, since you must first set up GnuPG, which provides the cryptographic engine. If the free software world could make this process easier - a click, a passphrase and you're done - Google's present grief could easily be turned into open source's opportunity.

Open Source's Best-Kept Secret

Ajax is short for Asynchronous Javascript + XML; it enables a Web page to be changed in the browser on the fly, without needing to refer back to the original server. This leads to far faster response times, and is behind many of the most interesting developments on the Web today; Gmail is perhaps the most famous example. Essentially it turns the browsers into a lightweight platform able to run small apps independently of the operating system (now where have we heard that before?).

The news of an Open Ajax project that will simplify the creation of such sites is therefore welcome. However, what is most interesting about the announcement is not the luminaries who are lining up behind it - IBM, Oracle, Red Hat and Yahoo amongst others - but the fact that it is yet another Eclipse project.

To which most people would probably say, Who? For Eclipse is open source's best-kept secret. It stands in the same relation to Microsoft's Visual Studio development tools as GNU/Linux does to Windows, and OpenOffice.org to Microsoft Office. Where these address respectively the system software and office suite sectors, Eclipse is aimed at developers. It is another example of IBM's largesse in the wake of its Damascene conversion to open source: the project was created when the company released a large dollop of code under the Eclipse Public License.

What's interesting is how Eclipse has followed a very similar trajectory to GNU/Linux: at first it was ignored by software companies, who preferred to stick with their own proprietary rivals to the Microsoft juggernaut. Later, though, they realised that divided they would certainly fall, and so united around a common open standard. The list of "Strategic Members" and "Add-in Providers" reads like a Who's Who of the world's top software companies (bar one).

This illustrates another huge - and unique - strength of open source: the fact that it represents neutral ground that even rival companies can agree to support together. The mutual benefit derived from doing so outweighs any issues of working with traditional enemies.

Even though Eclipse is relatively little known at the moment, at least in the wider world, it is not a particular bold prediction to see it as becoming the most serious rival to Microsoft's Visual Studio, and the third member of the open source trinity that also includes GNU/Linux and OpenOffice.org.

Google: Friend or Foe?

"Don't Be Evil" is the company motto: but is Google for us or against us?

I'm not talking about justifable concerns that it knows far too much about what interests us - both in terms of the searches we carry out and (if we use Gmail) the correspondence we send and receive. This is a larger issue, and relates to all the major online companies - Microsoft, Yahoo, even Amazon - that mediate and hence participate in much of our lives. What concerns me here is whether Google can be considered a friend of openness.

On the one hand, Google is quite simply the biggest open source company. Its fabled server farm consists of 10,000s/100,000s/1,000,000s (delete as applicable) of GNU/Linux boxes; this means that anyone searching with Google is a GNU/Linux user.

It has a growing list of code that it has open-sourced; it has sponsored budding hackers in its Summer of Code programme; and it keeps on acquiring key open source hackers like Guido van Rossum (inventor of Python) and Ben Goodger, (Firefox lead engineer).

On the other hand, Google's software is heavily weighted towards Microsoft Windows. Programs like Google Earth and Picasa are only available under Windows, and its latest, most ambitious foray, the Google Pack, is again only for Microsoft's operating system. This means that every time Google comes out with some really cool software, it is reinforcing Microsoft's hold on the desktop. Indeed, we are fast approaching the point where the absence of GNU/Linux versions of Google's programs are a major disincentive to adopt an open source desktop.

This dilemma is unlikely to be resolved anytime soon, since Google clearly wants to serve the largest desktop market first, while drawing on the amazing price-performance of free software for its own computing platform.

But there is another area where it has the chance to play nice with openness, one that does not require it to come down definitively on one side or the other of the operating system world.

Another Windows-only product, Google Talk, is the subject of a lawsuit alleging patent infringement. However, closer examination of the two patents concerned, Patent Number 5,425,085 - "Least cost routing device for separate connection into phone line" - and Patent Number 5,519,769 - "Method and system for updating a call rating database", suggests that one of the best ways Google could show that it is a friend of both open source and proprietary software is by defending itself vigorously in the hope that the US Patent system might start to be applied as it was originally envisioned, to promote innovation, not as an easy way of extracting money from wealthy companies.

Update 1: Google has come out with a Mac version of Google Earth. It's a start.

Update 2: There are rumours about Google working on its own desktop GNU/Linux. Frankly, I'll believe it when I see it: it's a poor fit with their current portfolio, and the margins are terrible.

Update 3: Comfortingly, these rumours have now been scotched.