I think looking at averages throws things off, considering how "clumpy" I'd guess most update frequencies are. Some thoughts:

• probably makes sense to check each feed at least once per day

• for each feed, look at the average time-of-day of its first-post-of-the-day. Actually, look at a distribution curve, and pick the time at which there's an 80% chance that the first post will have been made (if it's going to be made at all).

• check at that time; if no posting then check 12 hours later?

• if found posting at first check of day, then start that additive/multiplicative process

• regardless of the state of that latter calculation, check the next morning at that time-of-first-post prediction

Parallel idea:

• calc average time between posts like you're doing, but just over the window each day when posts are being made (e.g. 8am-11pm = 15 hrs, not 24 hrs).

This is not as polite as your interpretation, which definitely backs off very quickly. My reasoning went like this: weblog posts tend to clump, so the best indicator of an upcoming post is a new post...:

• If there aren't any new posts, lengthen the check interval by a little bit and check again later; keep lengthening the check interval up to a certain limit.
• If there is a new post, then it's likely that another new post will follow soon, so substantially decrease the poll time (down to a certain limit) and check again soon.

This approach is less biased towards decreasing server load and more biased towards detecting quick clumps of updates, which seem to be the norm. I don't know any human webloggers who have such a predictable posting pattern that they are subject to statistical analysis

Thanks for correcting me!

My thoughts on how to do it would be to break the day into blocks of time (start with hours, say), and build a histogram of how many posts fit into each block. Then, you could collapse the series of hours where nothing was posted into a big block, and split any hours where something was posted into two sections, to average out the number of posts per hour. Then, if my theory is correct, you can poll once per block of time, and would have a reasonable chance of getting a new post.

Some notable flaws with the algorithm:
1. It fails to account for whole days where there's nothing posted. This might be overcome by having your initial blocks of time be the days of the week.
2. It fails to account for the relationship (or the average time) between posts. So if someone posts a lot at 9:00 or 10:00, but never both at 9:00 and 10:00, my theory will still check the 10:00 time even if something was found at 9:00. I can't think of a way to get around this, off the top of my head.

Another way of thinking of this idea is pre-calculating a guess at the fall-off, based on previous posts.

Holy Moly: One Million Weblogs Tracked - "Technorati is currently tracking about 7,000 new weblogs per day" Linus Torvalds...