Branching by bucket

Posted on by
Reply

I’d like to talk about a strategy for feature control which I like to call “branching by bucket”. I guess you could say it’s a type of feature toggle, although it has advantages that are so specific to an A/B framework that I think it deserves its own name.

Branching by bucket is essentially a way of utilising an A/B framework to do new feature development on a software product in a safe and manageable way. The basic concept is simple: do your new feature development in an “experimental bucket”, then use configuration to push the feature live to all customers when it’s ready.

For those not used to the concept, an A/B framework is a feature of an application (usually a website) that allows different features to be exposed to different users, most often to determine which feature is most valuable. For example, if you would like to know whether your users prefer a search box at the top or at the bottom, you could develop both features and show each of them to 50% of your userbase by using your A/B framework to randomly assign 50% of your users to each of two “buckets”, one that has the box at the bottom and the other at the top. (Decent analytics software is useful to this process, since that’s a great way to figure out which version of the search box was being clicked on more often, but it’s not essential, since there are other ways to decide on a winner.) Once you’ve discovered, say, that users prefer the search box at the top, you can easily remove the “search box at the bottom” feature, first by disabling it, and then later by actually removing the code.

Branching by bucket involves the application of this most useful feature to the task of new feature development. So you’re getting frustrated with using feature toggles to build every single new feature; it can get annoying for anything but a complete new module, largely because you risk your application becoming littered with these toggles over time, plus the building of the toggle itself can sometimes be a risky process.

Instead, why don’t you develop your feature by building it in a bucket allocated to 0% of regular users? Any good A/B framework should have the mechanism to easily force your requests into a bucket – just use that mechanism and hey presto: you are the only user of your “experimental bucket”!

Here’s a real world example using PHP and a simple key/value pair-based configuration system. It’s heavily based on an application I once worked with, and I would like to give primary credit for the design to the architect of the platform. Thanks Okan, I hope you don’t mind me using your design in this post! Unfortunately the application is no longer in use, but the concept is a great one.

Every one of the users who visits this website is given a cookie that contains a more or less randomly generated value with a more or less even distribution of digits. If you take the last few digits of this value and collapse them down to, say, a value 0-99, then create buckets of features that use this value, you can assign specific percentages of users to specific buckets.

Now, in order to develop features that are shown only to users in these buckets, you can either: 1. override configuration to, say, rearrange the modules on the page, or 1. override classes and other types of file that are used in preference of the default versions when formulating a response to users.

For example, you could override your page layout configuration key for the “search_top” bucket like so:

bucket.search_top.page_layout="search_box,header,content,footer"

and for the other bucket like so:

bucket.search_bottom.page_layout="header,content,search_box,footer"

Alternatively, if your bucket involves more complex logic than just rearranging the order of modules, you could extend an existing class to perform some specific logic:

class SearchBox_search_top extends SearchBox {
    function functionToOverride() {
        // new business logic goes in here...
    }
}

Your application will then load your new class for users inside the search_top bucket, but fall back to the original SearchBox class for anyone else. And if the _search_top class doesn’t exist, it’ll fall back to the default value again (because we don’t want to override every single class in our application!).

For testing, you can either fake a cookie value that lands your request inside the search_top bucket (annoying and difficult) or you could use something more explicit like a query string parameter to force yourself into the search_top bucket, e.g. “www.example.com?bucket=search_top”. (NB. If you need to test the bucketing mechanism itself then you are stuck with the “fake cookie” method, but this example is all about testing the new functionality in the search_top bucket, so a query parameter is sufficient.)

I’ve spent most of this post explaining the functionality of a good A/B framework rather than talking about its application specifically for feature control, but many readers should now already see how useful this could be for a developer. In short:

  1. Create a new bucket that is assigned to 0% of users.
  2. Build your new feature (via config or code overrides as described above).
  3. Test your work by forcing yourself into the bucket.
  4. When the feature is ready, it can be tested on a small percentage of users by altering a single line of bucket configuration to increase from 0% to, say, 10%.
  5. Once you are convinced that the new feature is working (and is beneficial!) you can push it live by either configuring 100% of users to see the bucket, or by rolling the overridden functionality into the default bucket (assuming that there is a default bucket for your application).

Comments welcomed!

Vagrant – recompiling kernel headers

Posted on by
Reply

As the reader may have guessed, I’ve been learning about Vagrant base boxes. In this post, I’ve been using the lucid64 box from vagrantup.com as a basis to build a default base box for my local dev environment (based on the Ubuntu Lucid Lynx release). The first step for me was to upgrade all of the packages to their latest patches, which included the kernel, via aptitude safe-upgrade. After upgrading the kernel one has to rebuild the Guest Additions, so I had to install the new kernel headers then recompile the Guest Additions kernel modules to allow them to use the new kernel. Here’s how to do everything from start to finish to give you a completely up to date base box:

In the host, set up a temporary environment:

vagrant box add lucid64 http://files.vagrantup.com/lucid64.box
mkdir vagrant-temp
cd vagrant-temp
vagrant init lucid64
vagrant up

Now open VirtualBox, click on the running VM and choose Install Guest Additions from the menu.

In the guest:

sudo aptitude update
sudo aptitude safe-upgrade
sudo mount /dev/scd0 /media/cdrom
sudo sh /media/cdrom/VBoxLinuxAdditions.run # install the Guest Additions
sudo apt-get install linux-headers-$(uname -r) # install headers
sudo /etc/init.d/vboxadd setup # recompile VirtualBox kernel modules

Then in the host:

vagrant reload # restart the VM to test that shared folders are working
vagrant halt # shut the VM down but don't delete it yet
vagrant package # produces package.box
vagrant destroy # get rid of the temporary dev environment
vagrant box remove lucid64-custom # remove the old version of the box
vagrant box add lucid64-custom package.box

Now you can use the new lucid64-custom box as your default box!

Setting up a Vagrant base box

Posted on by
Reply

This set of instructions is the complete set of steps I followed to get a CentOS 5.7 Vagrant base box up and running for development at work. I hope it helps others. Big thanks to Chris Adams for his very helpful instructions, which I basically followed to the letter to set up my base box. These instructions contain a little more detail, and in some cases are updated to reflect current recommended best practice (e.g. installing Chef via RubyGems).

  1. Open the Vagrant documentation.
  2. Download the CentOS 5.7 64bit netinstall ISO from the Bytemark mirror site.
  3. Open VirtualBox and create a new VM:
    1. RAM: 512MB
    2. Create a new VirtualBox dynamic hard disk expanding to 40GB
  4. Start your new VM, and load the netinstall ISO when the “first run” wizard asks you for an installation disk.
  5. Boot the CentOS installation process.
  6. When asked, select HTTP install:
    1. Web site name: mirror.bytemark.co.uk
    2. CentOS directory: centos/5.7/os/x86_64 (maps to http://mirror.bytemark.co.uk/centos/5.7/os/x86_64/images/)
  7. Now run through the visual installer process:
    1. timezone: Europe/London
    2. root password: whatever you want
    3. Uncheck all boxes for software installation
  8. Wait for the OS installation process to complete.
  9. Eject the virtual CentOS installation disk from your VM, and reboot the virtual machine.
  10. Once the VM has restarted, log in to the VM console as root.

  11. Set up port forwarding (on host machine):

    vboxname="CentOS 5.7 base box"
VBoxManage modifyvm "$vboxname" --natpf1 "guestssh,tcp,,2222,,22"

  12. Disable iptables:

    iptables -L
iptables -F
service iptables stop
chkconfig iptables off
  13. SSH into the box: ssh root@localhost -p2222
  14. Run visudo to give password-less sudo access to people in the wheel group.

  15. Install software on the VM, and remove unneeded software too:

    yum install curl ftp rsync sudo time wget which # useful for every case
yum install gcc bzip2 make kernel-devel-`uname -r` # these are needed for building Virtualbox Additions
yum erase wireless-tools gtk2 libX11 hicolor-icon-theme avahi freetype bitstream-vera-fonts

  16. Insert the virtual VBox CDROM using the VirtualBox menus, and install the VirtualBox Guest Additions by mounting the CDROM and running the installation script:

    mount -o ro -t iso9660 /dev/cdrom /mnt
/mnt/VBoxLinuxAdditions.run

  17. Install chef using RubyGems, since Chef no longer supports installation via RPM:

    rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-4.noarch.rpm
wget -O /etc/yum.repos.d/aegisco.repo http://rpm.aegisco.com/aegisco/el5/aegisco.repo
yum install git gcc gcc-c++ automake autoconf make
yum install ruby ruby-devel ruby-ri ruby-rdoc # installs 1.8.7.299-7.el5
yum remove ruby.i386 ruby-devel.i386 ruby-ri.i386 ruby-rdoc.i386 ruby-libs.i386
  18. Install rubygems from source.

  19. Install chef & ohai:

    gem install chef
gem install ohai

  20. Add the ‘vagrant’ admin user:

    useradd -G wheel vagrant
passwd vagrant # password = 'vagrant'
  21. comment the line saying Defaults requiretty in the visudo file.
  22. tweak the Defaults env_keep entry, to add PATH
  23. /home/vagrant/.bashrc: export PATH=$PATH:/usr/sbin:/sbin
  24. /etc/ssh/sshd_config: UseDNS no

  25. Download insecure public/private keys:

    mkdir .ssh
chmod 755 .ssh
curl -L -k http://github.com/mitchellh/vagrant/raw/master/keys/vagrant.pub > .ssh/authorized_keys
chmod 644 .ssh/authorized_keys

  26. Now back in the host machine…

    mkdir betfair-sports-centos-64
cd betfair-sports-centos-64
vagrant init
  27. Shut down the VM

  28. Package the VM:

    vagrant package --base "$vboxname"

  29. Test the VM:

    vagrant box add my_box package.box
mkdir test_environment
cd test_environment
vagrant init my_box
vagrant up
vagrant ssh