http://decafbad.com/svn/trunk/feedspool/

Feeds fetched and stored on the filesystem–shortly thereafter sliced into entries, one per file. I base file and folder names on MD5 hashes of feed URLs and entry ID/content. Also considering leaving SQL out of the picture altogether, store per-user metadata on the filesystem, maybe indexed for searchability with Berkeley DB or Lucene.

Then you really have recoverability, etc.

Though it increases the risk that you’ll do something tacky and update a file without triggering an update of the meta-data, etc. but you could do the same thing regardless of the infrastructure layer…

(Probably-not-related, an interview with Hans Reiser I read recently included him saying that BLOBs are a bad thing. But I didn’t track down his reference documents….)

Rather predictably I’m opting for the RDF store approach, but that’s not to say the other approaches don’t have their merits. Jon Udell (search his blog) has done some interesting work with feed data stored as XML, basically using XPath as the query language. It works, but I would suggest that he’s tied to the XML structure too much. A year ago I might have gone for a hybrid RDF/RDBMS approach, with the core RSS material being contained in fixed tables bridged across to an RDF interface (somehow) to allow FOAF etc. But two factors put me off that idea now – the RSS 2.0 extensions that have appeared in recent weeks and SPARQL! Having a SQL-like query language that can be used against an RDF store (with XSLT-able results) really does simplify matters. Performance is likely to be an issue but I’m hoping a bit of filtering & caching will help there. I’m getting feed data into the store currently using XSLT (to RDF/XML), it’s ok but the setup I’ve got could be a lot more modular.

I think Microsoft have got things partly right in their RSS Longhorn architecture, separating out the “Sync Engine”. But in the data handling there’s heavy use of objects, which I don’t think should be needed when you can do things declaratively using SQL or SPARQL. Their object model is limited, I can guarantee that’ll turn into a big ball of mud ;-)

What I’m intrigued by is the potential for using multiple stores on different hosts. My immediate target being a fairly big aggregator/store on a remote server and a shiny-GUI client tool with more limited storage that selectively addresses the remote store. But I’m still at early-experiment stage.

Also, I’ve found that most people, if told that their feed is broken, will thank you for the note and proceed to fix it. Those who aren’t technically savvy enough to fix their feeds generally don’t have malformed ones to begin with because they use some ready-made webblogging tool — and those tend to have correct feed generation by default.

So if all your repair kit does is check and fix the feed encoding, then you should almost never see any bogosity.