Firefox Sync server on Google App Engine
TL;DR: I built an implementation of the Firefox Sync server API for Google App Engine.
To celebrate Independence Day, I figured I might take a shot at liberating Firefox Sync from the tyranny of Mozilla's servers.
Thus, over the past few days, I've built a sync server using the 1.0 Sync API, hosted on Google App Engine.
I lied about the tyranny thing, though—I just wanted to say something clever about the holiday. In reality, with respect to Firefox Sync, Mozilla has done all of the following:
- Published the Sync API spec;
- Released the source code for the server used in-house;
- Explicitly included the option to use a custom server when setting up sync in the browser.
This means that, although Mozilla offers servers to go along with Firefox Sync, you're totally free to take your data elsewhere. Since your sync data is encrypted and practically opaque to the server, there's no direct profit for Mozilla in offering free sync hosting—not even through any clandestine data mining for devious purposes. It's just that sync makes Firefox a better browser, and somebody has to run some servers to make it work.
So, there's every incentive to make it easy for you to switch sync providers and stop freeloading on Mozilla's servers. Building a server on Google App Engine means I can freeload on Google's servers!
I kid, of course. No one's really complaining about freeloaders, and App Engine has quotas in place to head off any serious mooching—which is why I'm not telling you where to find my sync server deployed on Google App Engine, by the way.
No, I did this because:
- Firefox Sync and Google App Engine are interesting and important technologies;
- I've already done a bit of work on the PHP-based Firefox Sync server at Mozilla;
- I really wanted to take a break from PHP and spend some time with my old friend Python.
There are, of course, a number of bugs in this server. But, it seems to be working between a number of machines and browser profiles I have at home. Things are really in need of optimization, it suffers from my inexperience with App Engine, and I keep running into those aforementioned App Engine resource limits—especially when updating or deleting large numbers of items (ie. 1000's to 10000's of items).
Pull requests and issue reports on GitHub are welcome!
A next step I'd like to take with this thing is to revisit another old friend, the desktop web app server. (Also known as the desktop website.) It seems to me that it would be interesting to scale this server down to a household appliance—say, just for use by my wife and I.
I'd be especially happy if the work I'm doing for a Google-hosted app could be self-hosted at home. Seeing as the development environment for App Engine runs on my laptop, I'm willing to bet I can hack the whole shebang into a simple, special-purpose app to download and double-click on a home desktop PC for use as your sync hub.
Anyway, check it out and let me know what you think.
Archived Comments
Assuming you already had a network and a Mac Mini or something at home, it would be pretty interesting to run your own small Sync server just to keep your home computers together. Maybe even use localtunnel for when you're away.
You are a real-life hero!
@James: Yup, exactly that. Maybe even roll in some SSL and UPnP port forwarding for easier setup behind a home router. And, if there were a Windows version, it could run on that cruddy desktop back in the spare room.
@Mike: Naw... this guy is the real hero, even John Lilly agrees!
Is there already a App Id for this tool?
@Crash: Sure, there's an app ID for my instance of this sync server. But, as I said in the blog post, I'm not sharing it. At least, not until or unless I get the quota usage down to a point that I wouldn't exhaust the free hosting limits.
It's pretty easy to deploy your own sync server on AppEngine with the source, though.
@Leslie: The app ID of your instance is visible in app.yaml, so you've shared it anyway :)
@Peter hah! Right you are! But, at least you had to look at the source of the app first to figure that out :) That is mostly the point of this blog entry after all. Anyone who does that and then uses my installation anyway will probably be sad when I regularly blow away data and eventually make it invite only
Neat! TyphoonAE http://typhoonae.googlecode.com might help you to build your household appliance.
Hey Les, this is super cool! Google App Engine's Datastore API is still very shaky though. In fact, it's been so bad lately that I've been considering porting one of my heavily used apps over to something else. Thankfully, this post suggests that fixing the Datastore is their top priority: http://googleappengine.blogspot.com/2010/06/datastore-performance-growing-pains.html
Well, I actually feel more comfortable with my data being on Mozilla servers than on Google servers - even with the decreased trust I have in the Mozilla organization, I'd trust it more than Google any day! ;-)
Instead of porting GAE to your desktop, why don't you just port your app to plain Django? Django should run off your desktop with no problems.