Comments on: Queue everything and delight everyone http://decafbad.com/blog/2008/07/04/queue-everything-and-delight-everyone It's all spinning wheels and self-doubt until the first pot of coffee. Fri, 19 Mar 2010 00:06:39 +0000 hourly 1 http://wordpress.org/?v=3.0-alpha By: Levy Carneiro » Blog Archive » Flickr engineers do it offline http://decafbad.com/blog/2008/07/04/queue-everything-and-delight-everyone/comment-page-1#comment-407439 Levy Carneiro » Blog Archive » Flickr engineers do it offline Tue, 16 Jun 2009 02:54:04 +0000 http://decafbad.com/blog/?p=1194#comment-407439 <p>[...] seems that using queuing systems in web apps is the new hottness . While the basic idea itself certainly isn’t new, its application to modern, large, scalable [...]</p> [...] seems that using queuing systems in web apps is the new hottness . While the basic idea itself certainly isn’t new, its application to modern, large, scalable [...]

]]> By: Chris Henry http://decafbad.com/blog/2008/07/04/queue-everything-and-delight-everyone/comment-page-1#comment-406951 Chris Henry Thu, 11 Jun 2009 20:09:47 +0000 http://decafbad.com/blog/?p=1194#comment-406951 <p>Handling load is probably one of the biggest problems facing websites today. Queueing is definitely the way to go, but like you said, sites need the type of architecture where it's easy to deploy services to different machines. Usually by the time the site is under load, it's too late...</p> Handling load is probably one of the biggest problems facing websites today. Queueing is definitely the way to go, but like you said, sites need the type of architecture where it’s easy to deploy services to different machines. Usually by the time the site is under load, it’s too late…

]]> By: jmxz http://decafbad.com/blog/2008/07/04/queue-everything-and-delight-everyone/comment-page-1#comment-388343 jmxz Fri, 13 Feb 2009 20:00:13 +0000 http://decafbad.com/blog/?p=1194#comment-388343 <p>Wow those comments make me feel old. I remember when these java queues everyone's referring to reminded me of how I had a VAX dedicated to queuing and scheduling batch jobs for a Cray.</p> Wow those comments make me feel old. I remember when these java queues everyone’s referring to reminded me of how I had a VAX dedicated to queuing and scheduling batch jobs for a Cray.

]]> By: Jay Yukes http://decafbad.com/blog/2008/07/04/queue-everything-and-delight-everyone/comment-page-1#comment-378544 Jay Yukes Sun, 23 Nov 2008 10:35:55 +0000 http://decafbad.com/blog/?p=1194#comment-378544 <p>I had to solve a similar problem. Needed the fastest possible response, so had to rule out interacting with the Database directly from the web app. Used PHP message queue Dropr to defer all DB work. It is very fast, easily over 1000 messages/second</p> I had to solve a similar problem. Needed the fastest possible response, so had to rule out interacting with the Database directly from the web app. Used PHP message queue Dropr to defer all DB work. It is very fast, easily over 1000 messages/second

]]> By: Bill Seitz http://decafbad.com/blog/2008/07/04/queue-everything-and-delight-everyone/comment-page-1#comment-363382 Bill Seitz Mon, 28 Jul 2008 21:23:06 +0000 http://decafbad.com/blog/?p=1194#comment-363382 <p>I wonder if you're still setting the bar too high for low-priority connections. I mean, microblogging <em>isn't</em> really messaging, and maybe isn't (shouldn't-be?) conversation. </p> <p>So why wouldn't 10-15min be good enough?</p> <p>What % of "messages" are <em>read</em> instantly after they hit an inbox?</p> I wonder if you’re still setting the bar too high for low-priority connections. I mean, microblogging isn’t really messaging, and maybe isn’t (shouldn’t-be?) conversation.

So why wouldn’t 10-15min be good enough?

What % of “messages” are read instantly after they hit an inbox?

]]> By: Extenuating Circumstances – links for 2008-07-15 http://decafbad.com/blog/2008/07/04/queue-everything-and-delight-everyone/comment-page-1#comment-361484 Extenuating Circumstances – links for 2008-07-15 Tue, 15 Jul 2008 10:47:51 +0000 http://decafbad.com/blog/?p=1194#comment-361484 <p>[...] 0xDECAFBAD » Queue everything and delight everyone Yes. (tags: queues asynchronous userinterface responsiveness decafbad blog) [...]</p> [...] 0xDECAFBAD » Queue everything and delight everyone Yes. (tags: queues asynchronous userinterface responsiveness decafbad blog) [...]

]]> By: People Over Process » links for 2008-07-12 http://decafbad.com/blog/2008/07/04/queue-everything-and-delight-everyone/comment-page-1#comment-361188 People Over Process » links for 2008-07-12 Sat, 12 Jul 2008 07:31:41 +0000 http://decafbad.com/blog/?p=1194#comment-361188 <p>[...] Queue everything and delight everyone “[D]o you really have to do everything all at once?” (tags: queue messaging architecture programming twitter) [...]</p> [...] Queue everything and delight everyone “[D]o you really have to do everything all at once?” (tags: queue messaging architecture programming twitter) [...]

]]> By: citric http://decafbad.com/blog/2008/07/04/queue-everything-and-delight-everyone/comment-page-1#comment-361031 citric Thu, 10 Jul 2008 21:49:19 +0000 http://decafbad.com/blog/?p=1194#comment-361031 <p>Web apps doing things while the user waits unnecessarily is an old phenomenon. I think it's often a matter of developers not wanting to (and/or being politically unable to) venture into what they consider the sysadmin's domain. Take the way-too-common case of apps that make the client wait while it does housekeeping. Why isn't this in a cron job? One reason is maybe this is KewlOSSBlogWikiPackage and it's simpler to say "just untar the package under htdocs and you're done" instead of saying "also, unpack these scripts in a non-servable area and set them up to run hourly, but not all at the same time; stagger them a little. And run them with the same UID your web server is running as". But we end up with a lot of apps that (badly) reimplement basic tools their OS ships with in the first place.</p> Web apps doing things while the user waits unnecessarily is an old phenomenon. I think it’s often a matter of developers not wanting to (and/or being politically unable to) venture into what they consider the sysadmin’s domain. Take the way-too-common case of apps that make the client wait while it does housekeeping. Why isn’t this in a cron job? One reason is maybe this is KewlOSSBlogWikiPackage and it’s simpler to say “just untar the package under htdocs and you’re done” instead of saying “also, unpack these scripts in a non-servable area and set them up to run hourly, but not all at the same time; stagger them a little. And run them with the same UID your web server is running as”. But we end up with a lot of apps that (badly) reimplement basic tools their OS ships with in the first place.

]]> By: Robert http://decafbad.com/blog/2008/07/04/queue-everything-and-delight-everyone/comment-page-1#comment-360996 Robert Thu, 10 Jul 2008 13:36:22 +0000 http://decafbad.com/blog/?p=1194#comment-360996 <p>I agree with everything you've said. Especially the last part, duplicating data in the format it will be retrieved in rather than using complicated and CPU intensive SQL queries. This is especially true for any sort of statistics or reporting. I learned this by seeing my website's statistics growing slower and slower to retrieve as more and more traffic caused the database to become larger and larger and all of a sudden those queries that ran nearly instantly, even with good indexing were taking several seconds to return.</p> I agree with everything you’ve said. Especially the last part, duplicating data in the format it will be retrieved in rather than using complicated and CPU intensive SQL queries. This is especially true for any sort of statistics or reporting. I learned this by seeing my website’s statistics growing slower and slower to retrieve as more and more traffic caused the database to become larger and larger and all of a sudden those queries that ran nearly instantly, even with good indexing were taking several seconds to return.

]]> By: alexis http://decafbad.com/blog/2008/07/04/queue-everything-and-delight-everyone/comment-page-1#comment-360987 alexis Thu, 10 Jul 2008 12:35:25 +0000 http://decafbad.com/blog/?p=1194#comment-360987 <p>er... eventually consistent social graphs anyone?</p> er… eventually consistent social graphs anyone?

]]> By: Brent Fitzgerald / Life & Research Notes / Blog Archive » Links for July 3rd through July 9th http://decafbad.com/blog/2008/07/04/queue-everything-and-delight-everyone/comment-page-1#comment-360929 Brent Fitzgerald / Life & Research Notes / Blog Archive » Links for July 3rd through July 9th Wed, 09 Jul 2008 18:02:23 +0000 http://decafbad.com/blog/?p=1194#comment-360929 <p>[...] 0xDECAFBAD » Queue everything and delight everyone interesting blog post on architectures for scalability of microblogging and messaging/publishing. argues for queue systems over sql queries, which is reasonable. consider extensibility of the queue processors as the hook for new apps/innovations. article blog microblogging scalability twitter web [...]</p> [...] 0xDECAFBAD » Queue everything and delight everyone interesting blog post on architectures for scalability of microblogging and messaging/publishing. argues for queue systems over sql queries, which is reasonable. consider extensibility of the queue processors as the hook for new apps/innovations. article blog microblogging scalability twitter web [...]

]]> By: A Couple of Caveats on Queuing - Laughing Meme http://decafbad.com/blog/2008/07/04/queue-everything-and-delight-everyone/comment-page-1#comment-360758 A Couple of Caveats on Queuing - Laughing Meme Mon, 07 Jul 2008 18:02:56 +0000 http://decafbad.com/blog/?p=1194#comment-360758 <p>[...] “Delight Everyone” post is latest greatest addition to the 17th letter of the alphabet for savior [...]</p> [...] “Delight Everyone” post is latest greatest addition to the 17th letter of the alphabet for savior [...]

]]> By: text/plain » Blog Archive » 3,737,844,653 http://decafbad.com/blog/2008/07/04/queue-everything-and-delight-everyone/comment-page-1#comment-360721 text/plain » Blog Archive » 3,737,844,653 Mon, 07 Jul 2008 07:47:59 +0000 http://decafbad.com/blog/?p=1194#comment-360721 <p>[...] L. M. Orchard writes the following in an otherwise quite sensible article on queue-based architectures for broadcast messaging systems: “Even with these delays, the system is still better at getting the word out than the original content creator would be at notifying all the others involved with an out-of-band system like IM or email.” [...]</p> [...] L. M. Orchard writes the following in an otherwise quite sensible article on queue-based architectures for broadcast messaging systems: “Even with these delays, the system is still better at getting the word out than the original content creator would be at notifying all the others involved with an out-of-band system like IM or email.” [...]

]]> By: Don Park http://decafbad.com/blog/2008/07/04/queue-everything-and-delight-everyone/comment-page-1#comment-360597 Don Park Sun, 06 Jul 2008 05:16:32 +0000 http://decafbad.com/blog/?p=1194#comment-360597 <p>Good suggestions. For social services like Twitter, I would also add one more item:</p> <p>Prioritize by Relationship</p> <p>For example, two-way Twitter relationships (mutual-follow or recent @ or direct message exchange) should be refreshed before one-way. One can go further by placing higher priority on users whom poster sent messages to or received from within past X-hours.</p> Good suggestions. For social services like Twitter, I would also add one more item:

Prioritize by Relationship

For example, two-way Twitter relationships (mutual-follow or recent @ or direct message exchange) should be refreshed before one-way. One can go further by placing higher priority on users whom poster sent messages to or received from within past X-hours.

]]> By: Infovore » links for 2008-07-05 http://decafbad.com/blog/2008/07/04/queue-everything-and-delight-everyone/comment-page-1#comment-360572 Infovore » links for 2008-07-05 Sat, 05 Jul 2008 23:30:32 +0000 http://decafbad.com/blog/?p=1194#comment-360572 <p>[...] 0xDECAFBAD » Queue everything and delight everyone “…that’s really the purpose of a web-based content creation interface—accepting something as quickly as possible to make the user happy enough to continue submitting more.” Leslie Orchard on message-queue-based design. (tags: queue messaging development software programming architecture) [...]</p> [...] 0xDECAFBAD » Queue everything and delight everyone “…that’s really the purpose of a web-based content creation interface—accepting something as quickly as possible to make the user happy enough to continue submitting more.” Leslie Orchard on message-queue-based design. (tags: queue messaging development software programming architecture) [...]

]]>