Inviting Contributors to Open Source Webdev through Virtualization

Who am I?

I'm also a Mozillian

What do I do?

I've been a wandering webdev

Every time I wander, I find this...

Webdev is hard.

Webdev

http
apache; lighttpd; nginx; ...
languages
perl; python; php; ruby; javascript; java; erlang; ...
frameworks
rails; django; pylons; mojo; kohana; cake; symfony; ...
html5
localstorage; indexeddb; websockets; forms; canvas; offline; ...
content
css; ux; ia; design; illustration; tech writing; creative writing; ...
[no]sql
mysql; postgres; redis; mongo; couch; ...
caching
memcache; squid; nginx; zeus; ...
queues
gearman; rabbitmq; celery; amqp; ...
performance
concat; minify; coffeescript; less; sass; graphite; statsd; ...
templates
jinja2; twig; erb; jade; genshi; kid; dtl; ...
localization
l10n; i18n; rtl; charsets; gettext; verbatim; ...
security
xss; csrf; bleach; tls; csp; rfi; lfi; hijacking; spoofing; ...
metrics
webtrends; google analytics; statsd; pentaho; ...
search
xapian; lucene; sphinx; elasticsearch; ...
clustering
load balancers; server roles; ...

Why do people contribute?

INSTALL.txt is a waste of (human) time

Webdev (server side)

http
apache; lighttpd; nginx; ...
languages
perl; python; php; ruby; javascript; java; erlang; ...
frameworks
rails; django; pylons; mojo; kohana; cake; symfony; ...
html5
localstorage; indexeddb; websockets; forms; canvas; offline; ...
content
css; ux; ia; design; illustration; tech writing; creative writing; ...
[no]sql
mysql; postgres; redis; mongo; couch; ...
caching
memcache; squid; nginx; zeus; ...
queues
gearman; rabbitmq; celery; amqp; ...
performance
concat; minify; coffeescript; less; sass; graphite; statsd; ...
templates
jinja2; twig; erb; jade; genshi; kid; dtl; ...
localization
l10n; i18n; rtl; charsets; gettext; verbatim; ...
security
xss; csrf; bleach; tls; csp; rfi; lfi; hijacking; spoofing; ...
metrics
webtrends; google analytics; statsd; pentaho; ...
search
xapian; lucene; sphinx; elasticsearch; ...
clustering
load balancers; server roles; ...

Webdev (client side)

http
apache; lighttpd; nginx; ...
languages
perl; python; php; ruby; javascript; java; erlang; ...
frameworks
rails; django; pylons; mojo; kohana; cake; symfony; ...
html5
localstorage; indexeddb; websockets; forms; canvas; offline; ...
content
css; ux; ia; design; illustration; tech writing; creative writing; ...
[no]sql
mysql; postgres; redis; mongo; couch; ...
caching
memcache; squid; nginx; zeus; ...
queues
gearman; rabbitmq; celery; amqp; ...
performance
concat; minify; coffeescript; less; sass; graphite; statsd; ...
templates
jinja2; twig; erb; jade; genshi; kid; dtl; ...
localization
l10n; i18n; rtl; charsets; gettext; verbatim; ...
security
xss; csrf; bleach; tls; csp; rfi; lfi; hijacking; spoofing; ...
metrics
webtrends; google analytics; statsd; pentaho; ...
search
xapian; lucene; sphinx; elasticsearch; ...
clustering
load balancers; server roles; ...

Webdev (localization)

http
apache; lighttpd; nginx; ...
languages
perl; python; php; ruby; javascript; java; erlang; ...
frameworks
rails; django; pylons; mojo; kohana; cake; symfony; ...
html5
localstorage; indexeddb; websockets; forms; canvas; offline; ...
content
ux; ia; design; illustration; tech writing; creative writing; ...
[no]sql
mysql; postgres; redis; mongo; couch; ...
caching
memcache; squid; nginx; zeus; ...
queues
gearman; rabbitmq; celery; amqp; ...
performance
concat; minify; coffeescript; less; sass; graphite; statsd; ...
templates
jinja2; twig; erb; jade; genshi; kid; dtl; ...
localization
l10n; i18n; rtl; charsets; gettext; verbatim; ...
security
xss; csrf; bleach; tls; csp; rfi; lfi; hijacking; spoofing; ...
metrics
webtrends; google analytics; statsd; pentaho; ...
search
xapian; lucene; sphinx; elasticsearch; ...
clustering
load balancers; server roles; ...

INSTALL.sh: let's waste a machine's time, instead.

Puppet: INSTALL.txt for machines

puppetlabs.com
(Chef works, too!)

  1 #
  2 # Excerpts from puppet/manigests/mozilla_kuma.pp:
  3 #
...
 46 package {
 47 
 48     [ "man", "man-pages", "vim-enhanced", "httpd-devel", "mysql-devel",
 49         "openssl-devel", "nfs-utils", "nfs-utils-lib", "mysql-server", "php-mysql",
 50         "vixie-cron"]:
 51         ensure => installed;
 52 
 53     [ "tidy", "git", "wv", "subversion-devel", "cabextract", "html2ps",
 54         "html2text", "memcached-devel", "libxml2", "libxml2-devel", "libxslt",
 55         "libxslt-devel", "libjpeg", "libjpeg-devel", "libpng", "libpng-devel",
 56         "python26-devel", "python26-libs", "python26-distribute",
 57         "python26-mod_wsgi", "python26-jinja2", "python26-imaging",
 58         "python26-imaging-devel", "python-lxml" ]:
 59         ensure => installed, require => Exec['repo_epel'];
 60 
 61 }
...
276 exec {
...
298     "setup_mysql_wikidb":
299         command => "/usr/bin/mysql -u root wikidb < /tmp/wikidb.sql",
300         unless => "/usr/bin/mysql -uroot wikidb -B -e 'show tables' 2>&1 | grep -q 'pages'",
301         require => [
302             File["/tmp/wikidb.sql"],
303             Service["mysqld"],
304             Exec["setup_mysql_databases_and_users"]
305         ];
...
318     "kuma_sql_migrate":
319         cwd => "/vagrant", command => "/usr/bin/python2.6 ./vendor/src/schematic/schematic migrations/",
320         require => [
321             Service["mysqld"],
322             Package["python26-devel", "python26-mod_wsgi", "python26-jinja2",
323                 "python26-imaging"],
324             Exec["mysql-python-install", "lxml-install",
325                 "setup_mysql_databases_and_users",
326                 "vendor_lib_git_submodule_update"]
327         ];
328     "kuma_south_migrate":
329         cwd => "/vagrant", command => "/usr/bin/python2.6 manage.py migrate",
330         require => [
331             Exec["kuma_sql_migrate"]
332         ];
333 
...
338     "kuma_update_product_details":
339         cwd => "/vagrant", command => "/usr/bin/python2.6 ./manage.py update_product_details",
340         creates => "/home/vagrant/product_details_json/firefox_versions.json",
341         require => [
342             Exec["kuma_south_migrate"],
343             File["/home/vagrant/product_details_json"]
344         ];
345     "kuma_update_feeds":
346         cwd => "/vagrant", command => "/usr/bin/python2.6 ./manage.py update_feeds",
347         onlyif => "/usr/bin/mysql -B -uroot kuma -e'select count(*) from feeder_entry' | grep '0'",
348         require => [
349             Exec["kuma_south_migrate"]
350         ];

Vagrant: Living in boxes

vagrantup.com

 1 Vagrant::Config.run do |config|
 2 
 3     # To rebuild from mostly scratch, use this CentOS 5.6 (32 bit) image:
 4     config.vm.box = "centos-56-32"
 5     config.vm.box_url = "http://people.mozilla.com/~lorchard/centos-56-32.box"
...
13     # Once you've gotten a successful initial from-scratch build, export the
14     # box for faster destroy/up next time.
15     #  $ vagrant package
16     #  $ vagrant box add kuma kuma.box
17     config.package.name = "kuma.box"
18 
19     # Uncomment this line to use your packaged box
20     # config.vm.box = "kuma"
21 
22     # On OS X and Linux you can use an NFS mount; virtualbox shared folders are slow.
23     # see also: http://vagrantup.com/docs/nfs.html
24     config.vm.share_folder("v-root", "/vagrant", ".", :nfs => true)
...
30     # This thing can be a little hungry for memory
31     config.vm.customize do |vm|
32         vm.memory_size = 768
33     end
34 
35     # Increase vagrant's patience during hang-y CentOS bootup
36     # see: https://github.com/jedi4ever/veewee/issues/14
37     config.ssh.max_tries = 50
38     config.ssh.timeout   = 300
39 
40     # uncomment to enable VM GUI console, mainly for troubleshooting
41     #config.vm.boot_mode = :gui
42 
43     # Add to /etc/hosts: 192.168.10.50 dev-kuma.developer.mozilla.org
44     config.vm.network("192.168.10.50")
45 
46     config.vm.provision :puppet do |puppet|
47         puppet.manifests_path = "puppet/manifests"
48         puppet.manifest_file = "mozilla_kuma.pp"
49     end
50 
51 end

What about The Cloud?

 1 #!/bin/bash
 2 #
 3 # Quick bootstrap script for a Centos 5.5 linux host
 4 #
 5 # Example Rackspace Cloud usage: 
 6 #
 7 #   * Grab a copy of [rscurl](https://github.com/jsquared/rscurl).
 8 #       * You can use web-based management, but the command line can be fun.
 9 #
10 #   * Spin up a server:
11 #       rscurl.sh -a $RACKSPACE_API_KEY -u $RACKSPACE_USERNAME \
12 #           -c create-server -f 2 -i 51 -n 'kuma-dev-1'
13 #
14 #   * Note the server IP and root password reported by Rackspace.
15 #
16 #   * Wait until server is active. Check like so:
17 #       rscurl.sh -a $RACKSPACE_API_KEY -u $RACKSPACE_USERNAME -c list-servers
18 #
19 #   * Once the server is active, kick off the bootstrap:
20 #       ssh root@$HOST 'wget --no-check-certificate -O- \
21 #           https://github.com/mozilla/kuma/raw/HEAD/scripts/centos55-bootstrap.sh | bash'
22 #
23 
24 GIT_REPO_URL="git://github.com/mozilla/kuma.git"
25 
26 # Need the EPEL repo right away, for git and puppet
27 rpm -Uvh http://download.fedora.redhat.com/pub/epel/5x86_64/epel-release-5-4.noarch.rpm
28 yum install -y git puppet
29 
30 # Clone the project from github
31 git clone $GIT_REPO_URL /vagrant
32 cd /vagrant
33 
34 # I like to git push from my laptop to the dev VM, and this makes it easier
35 git config --local receive.denyCurrentBranch ignore
36 echo 'unset GIT_DIR && cd .. && git reset --hard' > .git/hooks/post-receive
37 chmod +x .git/hooks/post-receive
38 
39 # Let puppet take it from here...
40 puppet /vagrant/puppet/manifests/*.pp

What about the content?

(Not affiliated with Anonymous; I just fail at image search)

github.com/davedash/mysql-anonymous

  1 # Anonymization rules for MDN wiki_mdc_deki
  2 
  3 databases:
  4 
  5     lmo_wiki_mdc_deki:
  6 
  7         truncate:
  8             - objectcache
  9             - querycache
 10             - requestlog
 11             - requeststats
 12 
 13         tables:
 14             attachments_backup:
 15                 hash_value:
 16                     - at_user_text
 17                     - at_removed_by_text
 18             logins:
 19                 random_ip:
 20                     - login_ip_address
 21             users:
 22                 hash_value:
 23                     - user_name
 24                     - user_real_name
 25                 hash_email:
 26                     - user_email
 27                 nullify:
 28                     - user_password
 29                     - user_newpassword
 30                     - user_token
 31                     - user_external_name
 32 
 33     lmo_developer_mozilla_org_django:
 34 
 35         truncate:
 36             - auth_message
 37             - django_admin_log
 38             - django_session
 39             - threadedcomments_freethreadedcomment
 40             - threadedcomments_testmodel
 41 
 42         tables:
 43             actioncounters_actioncounterunique:
 44                 random_ip:
 45                     - ip
 46                 nullify:
 47                     - user_agent
 48                     - session_key
 49             auth_user:
 50                 hash_value:
 51                     - username
 52                 random_email:
 53                     - email
 54                 random_int:
 55                     - first_name
 56                     - last_name
 57                 nullify:
 58                     - password
 59             contentflagging_contentflag:
 60                 random_ip:
 61                     - ip
 62                 nullify:
 63                     - user_agent
 64                     - session_key
 65             user_profiles:
 66                 random_int:
 67                     - location
 68                     - homepage

landfill.addons.allizom.org/db

So someday, on my project, you'll find this...

# Install VirtualBox 4 from http://www.virtualbox.org/

# Install vagrant, see vagrantup.com
gem install vagrant

# Clone the repo from GitHub
git clone git://github.com/mozilla/kuma.git
cd kuma

# Fire up the VM and install everything,
# go take a bike ride (approx. 15-30 min)
vagrant up

# Add dev-kuma.developer.mozilla.org to /etc/hosts
echo '192.168.10.50 dev-kuma.developer.mozilla.org' >> /etc/hosts

Prior art & Examples

landfill.bugzilla.org
"Landfill is the home of test installations for Bugzilla. These are demo installations that you can use to 'try out' Bugzilla."
hack.dreamwidth.net
"The Dreamwidth code is, at present, not the easiest code in the world to install."
Others?
???

The FUTURE

To sum up...