0xDECAFBAD

It's all spinning wheels and self-doubt until the first pot of coffee.

More autodiscovery for bots: Finding weblogs via RSS referers

Okay, so now we've got a bit further toward our News Aggregators leaving vapor trails. Thanks, Morbus. Thanks, Dave. There's still some more work that needs to be done, though. In specific, how do I get to you via your trail? Or, to be even more specific- how to my agents get to you?

Sure, I can look in my referers now, and filter on my RSS feed to see a footprints. And for Radio users, I can pluck out the userWeblog=... param added to Radio aggregator referers. For others, like the page I set up, or Adam set up, or Jeff set up, I can try to assume that the URL leads... somewhere.

But, in thinking about this further, none of this actually fulfills my wish. I originally wanted to see these referers lead directly to a reader's blog. They don't. But, that's fine: I kind of like the idea of the thank you / hello / I'm reading your page. However, I still want to find you.

Or, rather, as I alluded to in the first paragraph, I want my agents to find you. And my agents are only semi-intelligent. So how about we set a standard by which bots can autodiscover your weblog, home page, whatever:

  1. If you don't care to put up a special page to point to as your referer when aggregating news, then point a URL somewhere, anywhere, but include a URL to your weblog in a query parameter named "userWeblog". This covers an already de facto standard set by Radio.
  2. If you do care to set up an acknowledgement page for your readers, include a LINK tag in the page's HEAD pointing to the home page of your site as the table of contents:
In this way, rock bangers cobbling together their own blog spiders can tell them where to go and for what to look. Get this meme spread, and we'll be seeing more community crawling bots very soon now.

In fact, I like this LINK tag now. I think it should be put on all of your weblog pages. Especially as I start thinking more about revisiting referers: A LINK back to your weblog home as the table of contents would allow me to maybe enrich my backlinks, especially knowing where your site root is. (If my eyes do not deceive, it looks like backlinks on Mark Pilgrim's blog do some RSS autodiscovery right now.)

Anyway, let me know what you think.

shortname=oooahe

Archived Comments

  • Josh Cooper
    I like the userWeblog idea. If all the toolkits out there would use it that would be good for us rock bangers. I agree on the LINK tag being on every page also. I'm gonna go ahead and add it to all my templates as well. Of course if a site is already using a LINK tag, then your agent could always pull the feed, then get the site link directly from that. This of course is only good for those sites that do use the LINK tag and have it on all there pages. Over all I think its a great idea enriching the weblog community.
  • Josh Cooper
    doh! I was in a dreamland when i made the 3 statement. That has nothing todo with what your trying to solve just another problem, you still have to know where the referer is coming from. first. Anyway I better go find a Red Bull and wake up some more.