# 댄

My adventures with Python, functional programming, Korean, Test Driven Development and more

## A Basic Statistical Analysis of Castles of Burgundy Games on Yucata

Sooyeon and I have been playing the Castles of Burgundy (CoB) a lot recently. We play it in real life and also on online at Yucata. Sooyeon often beats me so I wanted to figure out what I was doing wrong by looking at some of the better players on Yucata and analyzing their games.

I'm not going to spell out how the game works here. If you are unfamiliar with the game I would recommend checking out the rule book and maybe even some BoardGameGeek reviews of the game.

The data I used consists of 1,742 games from the top 20 rated players on Yucata.de.

### Boards

There has been a lot of discussion about the power of each of the player boards and an analysis of board usage on boite-a-jeux. Board 8 was found to be so overpowered that Yucata doesn't allow it anymore. It was only used in 19 games in the data set. I'll start my analysis with the boards that have been used in the data set I have.

#### Overall board usage

Boards 5, 1 and 6 are the most popular among the top 20 players on Yucata.

#### Board win rate

The board with the highest win rate is the notorious banned board 8 followed by 5 and then 6. (Sooyeon's favorite board is 7 by the way.)

#### Points possible from regions per board

Each board has its own set of region sizes. These are the maximum theoretical bonus points you could get just by filling each region of a board (I'm not sure if it's possible to fill in every region in a game against a mildly competent opponent).

#### Runner-up points by board

Turn order in COB can change each round depending on the position of ships which players can advance. Going first lets you take tiles you want or your opponent needs before other players.

#### Overall first turns

Winners take more first turns than runners up on average.

#### First turns at the start of a phase

Board 6 has all blue spaces connected which might explain why players using it are able to go first so often (you can always play a boat if you get one whereas on the other boards you might need to expand before you can play a boat).

### Actions

Some tiles in CoB (Castles, Carpenter's Workshops, City Halls and Churches) give you the ability to take extra actions. You can also take an extra purchase action once per turn. The maximum number of actions seen was 7 but it is extremely rare (it only happened 6 times in the 114,074 turns that were made).

#### 7-action turn example

• play City Hall
• [bonus] play Church
• [bonus] take Castle
• purchase Carpenter's Workshop
• play Castle
• [bonus] play Carpenter's Workshop
• [bonus] take Carpenter's Workshop

#### Actions per turn winners vs. runners up

Winners take 1.43342472817 more actions than the runner-up in a game. It doesn't look like there is a significant action advantage for any particular board.

#### Average tiles played winners vs runners up

Winners play 0.426907452706 more tiles than runners up on average.

#### Placed tiles distribution

The most tiles that were seen played was 32 (out of 37 spaces).

### Tiles played breakdown

Note that the tile code in Yucata is different than the number on the tiles themselves.

#### Other tiles

##### 4-player tiles played by winner - times played by runner up

Unfortunately it turned out to be really difficult to find out how many points came from each action since points are recorded per turn instead of per action and points from knowledge tiles are earned the turn when the effects of the knowledge tile take place and are not recorded separately in the game data. I believe this is all happening on the server so calculating how many points were earned by each action would require essentially reimplementing the game (but it would be awesome data).

## Simple 0-Downtime Blue Green Deployments

Having worked on six e-commerce websites (half of which make millions of dollars in revenue every year) I can safely say that downtime is a sure fire way to upset the business side of any company. Time, after all, is money. I've worked with teams that have tried minimizing downtime incurred by releases in many different ways. Here are some of the extremes:

On one end of the spectrum you can avoid downtime during deployments by only deploying during maintenance windows. The downside here is pretty obvious - what if the release introduces a bug and you don't find out about it until during a peak traffic period? I've seen people throw their hands in the air and say "I guess our customers can't use functionality X and will get that error until we can deploy tomorrow morning" in shops where this was the way deployments were done. I've also had a front row seat when a site I was working on was brought down for an emergency deployment and we were inundated with customer complaints.

The other side of the spectrum I've seen tried is blue/green phoenix deployments - rebuilding each and every VM with the same software but a new version of the application. After testing is done on the new VMs you can cut over either a hardware switch or software like HAProxy so it points to the new version. Needless to say using this method takes a very long time if all you want to do is deploy a one line fix. If you aren't familiar with blue/green deployments be sure to check out Martin Fowler's article about them.

There is a Goldilocks solution to this problem which won't take down a site and won't take as long as a full blue/green phoenix deployment. That said, as with all technical solutions, it isn't without its own drawbacks and might not be right for all deployments.

Here is the ridiculously simple Flask application I'll be deploying as an example:

``````import os, time

@app.route("/")
def hello():
return "Hello 0-downtime %s World!" % os.environ.get('BLUEGREEN', 'bland')
``````

Here is the fabfile we will use:

``````import os
import sys
from StringIO import StringIO

from fabric.api import task, local, run
from fabric.operations import put
from fabric.state import env

sys.path.append('../')
from gitric.api import (  # noqa
git_seed, git_reset, allow_dirty, force_push,
init_bluegreen, swap_bluegreen
)

def prod():
env.user = 'test-deployer'
env.bluegreen_root = '/home/test-deployer/bluegreenmachine/'
env.bluegreen_ports = {'blue': '8888',
'green': '8889'}
init_bluegreen()

def deploy(commit=None):
if not commit:
commit = local('git rev-parse HEAD', capture=True)
env.repo_path = os.path.join(env.next_path, 'repo')
git_seed(env.repo_path, commit)
git_reset(env.repo_path, commit)
run('kill \$(cat %(pidfile)s) || true' % env)
run('virtualenv %(virtualenv_path)s' % env)
run('source %(virtualenv_path)s/bin/activate && '
'pip install -r %(repo_path)s/bluegreen-example/requirements.txt'
% env)
put(StringIO('proxy_pass http://127.0.0.1:%(bluegreen_port)s/;' % env),
env.nginx_conf)
run('cd %(repo_path)s/bluegreen-example && PYTHONPATH=. '
'BLUEGREEN=%(color)s %(virtualenv_path)s/bin/gunicorn -D '
'-b 0.0.0.0:%(bluegreen_port)s -p %(pidfile)s app:app'
% env)

def cutover():
swap_bluegreen()
``````

The updates in deploy should be idempotent (that is to say that you can run deploy multiple times and the result should be the same each time (except for the pids of the workers that are started)). One tricky bit here when you are harnessing git for your deployments is that you want to clean up your remote working copy. I didn't do this in the example but you can use git clean to make sure only the things in the repository end up in the working copy. I did this with Python but you can substitute any language that doesn't require a binary build step and has a way of installing isolated packages. I guess it could be done with Ruby and RVM. I also have a nodejs example in the gitric repository.

The directory structure that gets built out looks like this:

``````├── blue
│   ├── env
│   ├── etc
│   └── repo
├── green
│   ├── env
│   ├── etc
│   └── repo
├── live -> /home/test-deployer/bluegreenmachine/green
└── next -> /home/test-deployer/bluegreenmachine/blue
``````

To do the initial build-out all you need is an automator user on your remote server and an nginx host entry set up something like this:

``````server {
listen 80;
server_name server.name.here;

location / {
include /home/test-deployer/bluegreenmachine/live/etc/nginx.conf;
}
}

server {
listen 80;
server_name next.server.name.here;

location / {
include /home/test-deployer/bluegreenmachine/next/etc/nginx.conf;
}
}
``````

Then you can run

``````fab prod deploy
fab prod cutover
``````

These steps are intentionally separated so you can check the next environment before cutting over to the new release.

I cut over to a new release while running ab and continuously hitting the server with curl to see what the server was returning:

`````` % ab -c 100 -n 5000 http://my.server.here/
This is ApacheBench, Version 2.3 <\$Revision: 1528965 \$>
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking my.server.here (be patient)
Completed 500 requests
Completed 1000 requests
Completed 1500 requests
Completed 2000 requests
Completed 2500 requests
Completed 3000 requests
Completed 3500 requests
Completed 4000 requests
Completed 4500 requests
Completed 5000 requests
Finished 5000 requests

Server Software:        nginx/1.4.1
Server Hostname:        my.server.here
Server Port:            80

Document Path:          /
Document Length:        28 bytes

Concurrency Level:      100
Time taken for tests:   33.180 seconds
Complete requests:      5000
Failed requests:        2576
(Connect: 0, Receive: 0, Length: 2576, Exceptions: 0)
Total transferred:      922576 bytes
HTML transferred:       142576 bytes
Requests per second:    150.69 [#/sec] (mean)
Time per request:       663.607 [ms] (mean)
Time per request:       6.636 [ms] (mean, across all concurrent requests)

Connection Times (ms)
min  mean[+/-sd] median   max
Connect:      164  326  87.5    321    1393
Processing:   161  308 188.7    284    4045
Waiting:      161  307 186.5    284    4045
Total:        338  635 216.9    646    4409

Percentage of the requests served within a certain time (ms)
50%    646
66%    675
75%    689
80%    699
90%    723
95%    758
98%    789
99%    899
100%   4409 (longest request)
``````

My server is the tiniest VM Linode offers and I'm on the other side of the Earth from it so I'm not really concerned about the performance. I am checking that all the incoming requests were served while a release was deployed without any downtime. You can see that ab counted 2576 failing length requests - those aren't actually failures - ab counts different content from the initial response it receives as a failure and halfway through the load test I cut over to a new release:

`````` % for x in \$(seq 100); do curl -s -S http://my.server.here/ && echo; done
Hello 0-downtime blue World!
Hello 0-downtime blue World!
Hello 0-downtime blue World!
Hello 0-downtime blue World!
Hello 0-downtime blue World!
Hello 0-downtime blue World!
Hello 0-downtime blue World!
Hello 0-downtime blue World!
Hello 0-downtime blue World!
Hello 0-downtime blue World!
Hello 0-downtime blue World!
Hello 0-downtime blue World!
Hello 0-downtime blue World!
Hello 0-downtime blue World!
Hello 0-downtime blue World!
Hello 0-downtime blue World!
Hello 0-downtime blue World!
Hello 0-downtime blue World!
Hello 0-downtime blue World!
Hello 0-downtime blue World!
Hello 0-downtime blue World!
Hello 0-downtime blue World!
Hello 0-downtime green World!
Hello 0-downtime green World!
Hello 0-downtime green World!
Hello 0-downtime green World!
Hello 0-downtime green World!
Hello 0-downtime green World!
Hello 0-downtime green World!
Hello 0-downtime green World!
Hello 0-downtime green World!
Hello 0-downtime green World!
Hello 0-downtime green World!
Hello 0-downtime green World!
Hello 0-downtime green World!
Hello 0-downtime green World!
Hello 0-downtime green World!
Hello 0-downtime green World!
Hello 0-downtime green World!
Hello 0-downtime green World!
Hello 0-downtime green World!
Hello 0-downtime green World!
Hello 0-downtime green World!
Hello 0-downtime green World!
Hello 0-downtime green World!
Hello 0-downtime green World!
Hello 0-downtime green World!
Hello 0-downtime green World!
Hello 0-downtime green World!
``````

The special sauce is leveraging the reload functionality that most webservers (Apache, nginx) offer. Existing workers are told that they should not handle new requests and the new workers that are spawned proxy all traffic to the new version. Here is a trace from my server right after a cutover:

``````COMMAND    PID          USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME

nginx    13636          root    8u  IPv4 11283302      0t0  TCP *:80 (LISTEN)
nginx    29426      www-data    8u  IPv4 11283302      0t0  TCP *:80 (LISTEN)
nginx    29427      www-data    8u  IPv4 11283302      0t0  TCP *:80 (LISTEN)
nginx    29428      www-data    8u  IPv4 11283302      0t0  TCP *:80 (LISTEN)
nginx    29429      www-data    8u  IPv4 11283302      0t0  TCP *:80 (LISTEN)
nginx    29381      www-data   14u  IPv4 16961706      0t0  TCP SERVER_IP:80->PING_IP:46083 (ESTABLISHED)
nginx    29381      www-data   15u  IPv4 16961707      0t0  TCP localhost:48628->localhost:8889 (ESTABLISHED)
nginx    29429      www-data    5u  IPv4 16961753      0t0  TCP SERVER_IP:80->PING_IP:46084 (ESTABLISHED)
nginx    29429      www-data    6u  IPv4 16961754      0t0  TCP localhost:33233->localhost:8888 (ESTABLISHED)

gunicorn 29223 test-deployer    5u  IPv4 16953570      0t0  TCP *:8888 (LISTEN)
gunicorn 29340 test-deployer    5u  IPv4 16953579      0t0  TCP *:8889 (LISTEN)
gunicorn 29345 test-deployer    5u  IPv4 16953579      0t0  TCP *:8889 (LISTEN)
gunicorn 29345 test-deployer    9u  IPv4 16962807      0t0  TCP localhost:8889->localhost:48628 (ESTABLISHED)
gunicorn 29391 test-deployer    5u  IPv4 16953570      0t0  TCP *:8888 (LISTEN)
gunicorn 29391 test-deployer    9u  IPv4 16960496      0t0  TCP localhost:8888->localhost:33233 (ESTABLISHED)

root     13636  0.0  0.3  12920  3208 ?        Ss   Jun16   0:00 nginx: master process /usr/sbin/nginx
www-data 29381  0.0  0.2  12904  2104 ?        S    14:51   0:00 nginx: worker process is shutting down
www-data 29426  0.0  0.1  12920  1888 ?        S    14:52   0:00 nginx: worker process
www-data 29427  0.0  0.1  12920  1888 ?        S    14:52   0:00 nginx: worker process
www-data 29428  0.0  0.1  12920  1888 ?        S    14:52   0:00 nginx: worker process
www-data 29429  0.0  0.2  12920  2380 ?        S    14:52   0:00 nginx: worker process
``````

nginx PID 29381 (labeled "nginx: worker process is shutting down") is handling an old request to the previous release and will shut down once it is finished. A request that came in after the release is going to port 8888 (the new release). All future requests will go to the new nginx workers which forward traffic to port 8888. These are the details of how nginx handles graceful reloads but a complete understanding this isn't necessary to harness the power of this deployment method.

Using git to deploy code for languages which don't require builds like Python and Ruby shortens the time it takes to build packages and deploy. I wrote about this a few years ago. Coupling that with blue/green deployment techniques on the same server has led to a very pleasant deployment experience for me and my team for the past year and a half. Everyone takes turns deploying and as our fleet of servers grows our deployment process won't get any slower now that we use the @parallel decorator during the update phase.

It takes a tiny extra amount of planning to write code and migrations that can be deployed without bringing down a live service but with experimentation and practice you will find that it is not that much work. This video from the Disqus team is an amazing resource. You should prefix your memcache keys with a short git ref and warm up your cache before cutting over. With Postgres you can usually add new tables and even add new NULL-default columns without problems but you'll definitely want to test your migrations on a staging environment which is simulating locked rows (if you use SELECT FOR UPDATE to ensure consistency). If you use a background worker like Celery tasks might linger from previous versions so you need to handle cases where the old API is called with a default:

``````@task
def process_order(order_id, resent=None):
....
``````

If there is a scheduled process_order with the old function signature in the queue it could fail unless you give the new parameters you add default values. These are just a few of the caveats I could think of. Always test deployments and rolling back on staging when in doubt until you get the hang of it.

There are numerous reasons why you would want to deploy updates to an API or website without downtime using the blue/green deployment method:

• Customer satisfaction - living and working on the other side of the world (Korea) I find it very frustrating that services I depend on to get my work done think that "maintenance hours" are in the middle of my day just because the sun has set on their side of the world.
• You can roll back from bad releases without re-deploying - the old release is still there so you can cut back to it in case you find problems in the new release.
• Ability to fix unforeseen problems quickly - should you determine that there is a problem which isn't large enough to warrant cutting back to the old release you can still deploy a fix even while there are thousands or millions of customers using your service without interrupting them.
• You are one step closer to continuous deployment.

As I said above there are countless techniques that can be used to deploy software and they all have their trade-offs. OS-level packages can't be upgraded in isolation like virtualenv and the application can. Critics might say that this only works for language-level packages and not OS-level packages or even OS upgrades. I fully understand this point and I guess it's just a trade-off. The future looks very bright when it comes to techniques that provide even further isolation and faster deployments like docker and other similar projects. I look forward to using tools like this to make it so there is even less downtime on the projects I work on in the future. In the meantime this porridge is just right for the type of projects I'm working on.

## June 2012 update

Sooyeon and I have been very busy for the past 7 months with our brand new baby daughter Haereen. When I get a few spare seconds I still work on my personal projects but it's been pretty slow going as of late. I'm definitely not complaining since Haereen is totally worth it. Anyway, here is a summary of some of the things I've worked on in my free time.

#### dongsa.net 2.0 preview

dongsa.net is a Korean verb conjugation algorithm that explains the contractions and exceptional rules for many tenses and levels of politeness. The current version is written in Python but there was a whole rewrite of the engine into JavaScript over two years ago to make it easier to port to Android and iOS. I've had the rewrite sitting in a branch for around a year now but it's only been a month or so since I pushed up preview.dongsa.net.

• English definitions and search
• Stemmer - the new stemmer reverses the conjugation process. You can see how it works here. Basically, there is list of flattened out Korean verbs (so "춥다" is stored as "ㅊㅜㅂㄷㅏ". It goes through the flattened version of the conjugated form (say "추웠어요" -> "ㅊㅜㅇㅜㅓㅆㅇㅓㅇㅛ") that is passed in and strips one bit off at a time and looks for all verbs that match it. It requires that the verb is in the database and doesn't yet work for some forms of irregulars (ㄹ dropping verbs) but it's a start.
• I've had these changes sitting in a branch for almost a year now. There are only a few items left on the todo list before I can switch dongsa.net over to this new version. The biggest todo item is to make it look better. I am clearly not a designer. If that's your thing and you want to help out I would greatly appreciate it.

#### qc a QuickCheck implementation in Python

• Added support (in the python-3 branch) for checking annotations (a new Python 3 feature):
``````
>>> def simple_adder(a : int, b : int) -> int:
...    return a + b
...
>>> from qc import check_annotations
>>>
>>> def lying_adder(a : int, b : int) -> int:
...     return 'result: %d' % (a + b,)
...
Traceback (most recent call last):
File "", line 1, in
File "qc/__init__.py", line 137, in check_annotations
f.__name__, output, response_type, test_args))
AssertionError: Was expecting lying_adder to return <class 'int'> but got <class 'str'> with these arguments: {'a': 0, 'b': 0}
>>>
``````

## git-based fabric deploys are awesome

When I was pointed to Python Deployment Anti-Patterns by a colleague I was a little shocked to see that the way we had been deploying applications with fabric and git over the past two years (over 1500 deployments) with no problems was being called an Anti-Pattern. There are definitely many ways to deploy software applications and they all have their pros and cons. Our process is by no means perfect but the way that we use git within fabric is definitely one of the best parts of our deployment process.

In his follow-up article Hynek made the case that deploying with native packages is better. On my team we actually started out deploying packages but since developers deploy we got sick of waiting for the packages to build and upload so we switched to git-based deploys. Packages are, of course, a valid way to deploy software, but I think the criticisms leveled against fabric git-based deploys might have been against doing these deploys in a specific way. I'm writing this article to show you how we have been successful using git-based fabric deployments.

I agree with many of his points:

• "Don't use ancient system Python versions"
• "Use virtual environments"
• "Look into alternatives to Apache + mod_wsgi setups"
• "Don't run your daemons in a tmux/screen"

Upstart is my personal favorite because it is very stable and the configuration is succinct. Here's an example of a daemon that I've had running on one of my personal projects for several years with no issues:

``````start on runlevel [12345]
stop on runlevel [0]

respawn

exec sudo -u www-data PATH=path/to/app VIRTUAL_ENV=path/to/virtual_env path/to/python_server_script
``````

Why anyone would want to write a billion line init script now that upstart exists is beyond me. Perhaps they don't know about upstart. It could also be that they are stuck on CentOS or RedHat. My heart goes out to you if that's the case. I know how that feels.

Here are some of the points I disagree with:

• Configuration is not part of the application

I've seen others make this same claim and on the face of it it makes sense up to a point. On my team developers deploy so we keep templates of configurations and the differences are kept in context variables that are passed into the templates. If there is sensitive information we keep it outside of version control. Really, if you want to test changes from dev through staging and onto production why not keep the configuration as similar as possible? On projects where teams are creating very generic apps that are being deployed with many different configurations I understand the need for this but most web application developers are deploying to a very specific target (production). It makes sense to keep your development settings as close to that target as possible. For example, if staging and production have the ENCRYPT_STUFF setting set to TRUE then your development environment should have it set too. But they should all have different keys and the production setting should be kept out of version control.

• What's wrong with Fabric+git-pull?

It doesn't scale. As soon as you have more than a single deployment target, it quickly becomes a hassle to pull changes, check dependencies and restart the daemon on every single server. A new version of Django is out? Great, fetch it on every single server. A new version of psycopg2? Awesome, compile it on each of n servers.

Fabric will roll through all commands on all servers in a predictable manner one after the other. That way they can be taken out of the load balanced pool before the service is HUP'd and put them back in after it comes back. If this is done automatically with unattended package upgrades (as proposed later in the article) isn't there the possibility that all your servers become unavailable at the same time?

You should always run pip and if there is nothing to upgrade it will simply do nothing. There's no need to download all of the packages - you can have them seeded on each server before starting the upgrade.

It's hard to integrate with Puppet/Chef. It's easy to tell Puppet "on server X, keep package foo-bar up-to-date or keep it at a special version!" That's a one-liner. Try that while baby sitting git and pip.

I can't speak to integrating fabric with Puppet and Chef but it's basically a one-liner to update a remote target with fabric:

``````cd path/to/git/repo && git reset --hard [deployment-sha1] && pip install -r path/to/requirements.txt
``````

It can leave your app in an inconsistent state. Sometimes git pull fails halfway through because of network problems, or pip times out while installing dependencies because PyPI went away (I heard that happens occasionally cough). Your app at this point is – put simply – broken.

A git pull will not leave your app in an inconsistent state. If the network fails it won't change your working copy and fabric will stop the script because git will return an error. That said I don't think you should use git pull anyway since it is one more moving part that can fail during deployment and it requires that your private repository be open to the world. Since git is distributed a developer can push their repo's immutable store to the target using git push during deployment. Running git reset --hard [deployment-sha1] after the push is finished will update the working copy. Since there is a repo on the other end you'll only be sending the new objects since the last push to the target. This is why git-based deploys beat packages speed-wise. Most of our code deploys take a fraction of a second.

Even a private PyPI mirror can fail. Why not upload the packages to the target and run pip like this?

``````pip install --no-index --find-links file:///[local-path-to-packages] -r requirements.txt
``````

You could even store your packages in a git submodule and sync your submodules at the same time. (We sync submodules as well, it's only a little extra work.)

Weird race conditions can happen. Imagine you're pulling from git and at the same time, the app decides to import a module that changed profoundly since its last deployment. A plain crash is the best case scenario here.

When you install with a package you have to stop and restart the app. You need to do the same thing if you use git and fabric. With git, it takes much less time to update because only the modified files are swapped out. Packages copy whole trees of files many of which are most likely not modified between releases so the app will be down longer while this disk IO takes place.

Check out the gitric fabric module I wrote that performs git deployments in the way I've described above.

One other valid problem I've heard raised about git-based deploys is that you can end up with cruft in your working copy that sticks around like .pyc files where the original .py file is deleted and there is the chance that this file could still be imported even though the original .py was deleted. Since cloning a local git repository uses hard links you can seed your remote repository and then clone it locally on the same machine (even for slightly large projects this only takes a little extra time). Stop your server, move the old repository out of the way and move the new cloned repo where the old one was (or use a current symlink) and then restart the server.

Git-based deployments make sense for scripting languages where there isn't a compile step so the repo can be sent as-is to production (so it wouldn't make sense for a Java application). It's worth harnessing git to make deployments faster. If we only had to deploy once a month we might've settled for package-based deployments but we push often and got sick of waiting for packages to build and upload.

• Packages are, of course, a legitimate way to push out changes but the downside of deploying with packages is that it takes time to build them and upload them
• Git + fabric is suitable for deployments (my team has deployed using it over 1500 times)
• git-based deployments are lightning fast to deploy and roll back
• There is no build step
• You only have to upload objects that have changed
• Use git push, don't use git pull
• It's one less moving part that can fail during deployment
• You don't have to open your git repository to the world
• You can pre-seed git's immutable object store without affecting your running application
• You can have pip use local packages which are more reliable and you also avoid having to set up a PyPI mirror

## Why I don't use git's staging area as part of my normal workflow

Git has a lot of bells and whistles and there are a lot of different ways to achieve any given task. I've seen several workflow documents explaining how to use the staging area and git add --patch to only commit some of the changes in your working copy so you can keep nice clean logical commits. I love the idea of having a clean history and logical commits but I think there are some drawbacks to using the index as part of a normal workflow.

### The problem with the staging area

There you have it. That's why I avoid the index and frequently use git stash --patch in my git workflow.

## When Failure is the Best Option

In Python (and most sane scripting languages) when something unexpected happens an exception is raised and execution stops. Damien Katz calls this the "Get the Hell out of Dodge" error handling method in his seminal Error codes or Exceptions? Why is Reliable Software so Hard?. In his article Damien explains several different ways of handling errors. None of the options is to ignore that something went wrong. That's because ignoring problems only makes them worse. But that's exactly what PHP and MySQL do for certain classes of errors.

Here's how Python handles failure:

``````% python
>>> print a
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'a' is not defined
>>>
``````

PHP's default behavior is to just keep chugging along ignoring problems that could cause huge issues:

`````` % php 2> >(while read line; do echo -e "stderr> \$line"; done)
<?
printf("%d\n", \$a);
?>
0
stderr> PHP Notice:  Undefined variable: a in - on line 2
stderr> PHP Stack trace:
stderr> PHP   1. {main}() -:0
``````

A "notice", eh? Really? If you try to delete a record from a database using sprintf to ensure it is a decimal and accidentally pass in an undefined variable as the id PHP will happily tell the database to delete the record with the id of "0". In my opinion this deserves more than a "notice" in the logs. PHP's default error-handling behavior is a recipe for disaster.

Fortunately, if you must use PHP, there is a way to make PHP behave in a more sane manner and force every unexpected event to raise an exception (exception_error_handler from http://www.php.net/manual/en/class.errorexception.php):

`````` % php 2> >(while read line; do echo -e "stderr> \$line"; done)
<?
function exception_error_handler(\$errno, \$errstr, \$errfile, \$errline ) {
throw new ErrorException(\$errstr, 0, \$errno, \$errfile, \$errline);
}
set_error_handler("exception_error_handler");

printf("%d\n", \$a);
?>
stderr> PHP Fatal error:  Uncaught exception 'ErrorException' with message 'Undefined variable: a' in -:7
stderr> Stack trace:
stderr> #0 -(7): exception_error_handler(8, 'Undefined varia...', '-', 7, Array)
stderr> #1 {main}
stderr> thrown in - on line 7
``````

There is one huge problem with this. If you are building on an existing PHP project or have a ton of PHP code it's likely that you will see frequent breaks once you make failure the default. That's a direct result of the language designers choosing such lenient default behavior. If you are starting a new project using PHP you should get your head checked (see phpsadness.com). If you pass a psychological evaluation and you still for some reason want to build a new project using PHP you should turn on immediate failure by using the error handler mentioned above and write tests to exercise your code. You'll thank me later.

Now, let's look at default behaviors of some popular databases:

`````` % psql
# create table simple_table (col varchar(10));
CREATE TABLE
# insert into simple_table (col) values ('1234567890a');
ERROR:  value too long for type character varying(10)

% mysql
mysql> create table simple_table (col varchar(10));
Query OK, 0 rows affected (0.23 sec)

mysql> insert into simple_table (col) values ('1234567890a');
Query OK, 1 row affected, 1 warning (0.08 sec)

mysql> select * from simple_table;
+------------+
| col        |
+------------+
| 1234567890 |
+------------+
1 row in set (0.00 sec)
``````

Yup, by default MySQL just silently truncates your data. Ronald Bradford, a self-proclaimed MySQL Expert sums it up nicely: "By default, MySQL does not enforce data integrity." That should set off alarm bells in your head if you are using or considering using MySQL. The whole point of a database is to store valid data. The simple solution is to use a database that cares about your data like Postgres but if you must use MySQL you should set

``````SQL_MODE=STRICT_ALL_TABLES
``````

For more on why this is necessary see Ronald Bradford's Why SQL_MODE is Important blog post.

PHP and MySQL are widely used. Maybe it is because their default settings are so lenient that it makes it easy for beginners to pick up. No one really cares if there was an error saving a hit on your personal homepage to your database. The problem is that these settings are not conducive to writing quality software. When starting from scratch it's better to choose technologies that have smarter defaults like Python and PostgreSQL because the libraries and software written using these technologies will properly fail instead of doing unexpected things and filling your database with garbage.

PS

You can (and should in most cases) also force hard failure for bash scripts by running set -e at the top of the script. See David Pashley's Writing Robust Shell Scripts for more.

## Why cherry-picking should not be part of a normal git workflow

Disclaimer: Many readers have noted that this is a straw man argument and git was not designed to use this workflow. I wrote this blog post because I had seen people at two different companies working this way so it's only addressed to people who find themselves working this way accidentally. When you have conflicts in the cherry-picking workflow you can lose track of what has been merged and what has not been merged into particular branches. I've seen teams lose emergency hotfixes and not be aware of it because of this. Cherry-picking has its uses and I don't mean to discourage it when necessary but it shouldn't be something you need to do more frequently than merging.

### Cherry-picking Workflow

Changes are made in a maintenance branch off of the release where the bug was found. The commitid from this change is then cherry-picked into the current integration branch.

``````% git checkout -b maintenance-branch <release tag or commitid> # (if the maintenance branch doesn't yet exist)
% git checkout -t origin/maintenance-branch # (if the branch already exists)
% git commit -am "Made a bug fix" # note the commitid
% git push origin maintenance-branch
% git checkout integration-branch # (e.g. master)
% git cherry-pick <commitid>
# resolve conflicts
% git push origin integration-branch
``````

#### Problems with the cherry-picking workflow

• When a change is cherry-picked into a branch and there is a conflict a new commitid is created (meaning that there are two commitids for the same change).
• All of git's diagnostic tools which determine differences between branches stop working when there are multiple commitids for the same change.
• Because git's diagnostic tools don't work it is hard to determine if all the changes that were made in the maintenance branch made their way to the integration branch.

``````% git cherry -v <upstream> <head>  # list of commit differences between branches
% git branch --contains <commitid> # which branches contain the commitid
% git branch --no-merge            # list which branches haven't been merged in
``````

For example:

Under the cherry-pick workflow, even though a bugfix was cherry-picked into the integration branch and there was a conflict git cherry -v reports that the integration branch is missing this commit from the maintenance branch.

``````% git cherry -v maintenance-branch integration-branch
+ 33de19776f4446d92b45e1fdfb2d9c37b3a867a7 Made a bug fix
``````

### Merge Workflow

Changes are made in a maintenance branch off of the release where the bug was found (same as in the cherry-picking workflow). The maintenance branch is then merged into the current integration branch.

``````% git checkout -b maintenance-branch <commitid> # (if the branch doesn't yet exist)
% git checkout -t origin/maintenance-branch # (if the branch already exists)
% git commit -am "Made a bug fix"
% git push origin maintenance-branch
% git checkout integration-branch # (e.g. master)
% git merge origin/maintenance
# resolve conflicts
% git push origin integration-branch
``````

#### Benefits

• The history of the changes made in the maintenance branch is preserved and there are no duplicate commitids for the same changes - just an extra merge commit.
• git's diagnostic tools work.
• It is much harder to lose work because all the commits from the maintenance branch make their way from the maintenance branch to the integration branch with each merge.
• You can have git automatically verify that the changes have all made their way from the maintenance branch to the integration branch.

Confusingly enough one of the most useful tools in the merge workflow to check that the state is correct is named "cherry". It shows the commits that were made in one branch and not the other. It should show no missing changes when following the merge workflow because all the changes from the maintenance should make their way into the integration branch:

``````% git cherry -v maintenance-branch integration-branch
[nothing]
``````

#### Draws

• Same number of commands for both cherry-picking (neither is more complicated)
• You need to resolve conflicts either way (this might actually be a win for the merge workflow because there are some scenarios where having git understand the merge history will make multiple merges easier than multiple cherry-picks)

## dongsa.net Korean Verb Conjugation Android App 2.0

The iPhone port of dongsa.net has had a native interface for several months now thanks to the work of Max Christian. The native interface for Android has been ready for a while but as I built it I added way too many new features. Instead of waiting until everything was fully polished I decided to strip back the new features and release an update to get the native UI out there. For those of you that were using the built-in Korean keyboard you will need to download a new input from the Market.

If you have an Android phone you can download the app directly or get it on the Android Market.

The same Javascript conjugation engine is used for both the iPhone and Android. Only the UI code has to be maintained separately. If you are curious about how this is done you can look through the source at GitHub.

## One Way to Build a Federated Social Network Part 2

In my last post I wrote up a scheme to share structured information with friends that doesn't require a central service like Facebook or Twitter. If you didn't read that post this post will make very little sense to you. In this post I will explain how losing central control might not mean losing everything you are used to and I will revise a couple of the implementation details.

#### Some Questions

So, you might be thinking, if there is no central control can we find each other? Can we have a feature similar to Twitter's hashtags? Great questions. The answer is "definitely!" Currently, websites are completely federated and there are many services that allow you to quickly get publically shared information: search engines. The protocol will make it so you mark whatever information you want as public. If you don't want to be discovered you don't have to be. Search companies can write bots that crawl the nodes to gather up public information just like they do with websites today. Private will be the default setting so you will have to explicitly mark content as public. I think this is a huge step forward from centralized services because no middle man ever has to see your private information but we can still have the benefits of the centralized services.

#### Implementation Changes

Requiring a VPS or a host that is directly accessible via the internet is probably going to limit who can use a system like this. I'm starting to think that the client should run on your machine and connect with other clients via NAT hole punching. NAT punching is used to share information among peers on P2P networks so it is perfectly suited for this project. There would need to be a service that connects clients in this scenario. Perhaps the lookup could be based on some user UUID or public key signature. This is the point where, if you are paranoid, outsiders could potentially see who is connecting with whom. There would have to be a way for people to connect directly to one another as well so you can avoid the matching services if you are paranoid. I looked at a Google project called libjingle which implements TCP on UDP for telephony. Also, Skype's protocol was partially reverse engineered very recently. Some existing library will make this functionality possible.

Git is starting to look like a pretty bad choice for synchronizing data. Since you have to synchronize all or none of the data in a repository it makes it impossible to share only some of the data with certain peers. I'm going to replace Git with a much simpler protocol that offers more flexibility. Since it knows which friend is making a particular request the system can limit what data is shared with them based on your settings. The synchronization protocol will be very similar to the protocol that CouchDB uses to show what updates have been made to a database. This is what the CouchDB update feed spits out:

``````{"seq":12,"id":"couchid","changes":[{"rev":"1-beef2479643c2b380f99507a7767f3d5"}]}
``````

Similarly, in the new synchronization protocol after a client authenticates to another client with their key (all clients will run SSH servers) the requesting client would make an HTTP request for changes since their last successful synchronization. The response would be a list of all the ids that have changed or been added which are visible to the peer making the request:

``````f572d396fae9206628714fb2ce00f72e94f2258f
7269918432597df3ec42b62acd81643d79134cf8
...
``````

I don't want to make too grand a statement about the importance of having a decentralized replacement for services like Facebook and Twitter. I will say that I think email would have failed spectacularly if it had been centralized instead of federated and my guess is that it will be better for everyone except the investors and owners of the centralized social networks if we move to more secure distributed systems.

## One Way to Build a Federated Social Network

There are companies making millions of dollars off of your personal information in exchange for giving you a way to easily share data with your friends. Facebook, Twitter and all the rest of these networks are all centralized services. You give them your data, they keep a copy and hopefully they share the data with only the people you told them to share it with. The funny thing is that for decades we have had email which is a federated service that gives us a less structured way to share data with our friends. With email we could send pictures to our friends. With Facebook we get the power of croud-sourcing. Our friends can tag and comment on our pictures. Surely there must be a way for us to do this in a federated way without requiring that we hand our data over to a middle-man.

There have been attempts at building a Federated Social Network. Diaspora is one such attempt that drew a lot of early buzz and funding. When I saw it I thought "thank goodness someone is solving that problem". I must say that one year on it appears to me as though they are not addressing the real problem. I was thoroughly disappointed with the result of their work: a Rails-based clone of Facebook. In my opinion what is needed here is a new federated protocol that can be easily extended with new content types and that protects access to data with private keys. On top of that new clients (web, desktop, mobile, whatever) can be built.

The following is a brain dump of one way of doing this.

Every user would have their own node or share a node with a group of people that they trust on a server of their choice. A working title for this project could be "A League of Nodes" but hopefully we'll come up with something better than that.

#### Basic infrastructure

Very few systems are as efficient as Git is when it comes to synchronizing data so it will be employed for sending and receiving updates.

Data will be stored in UUID filenames, similar to the way that git stores its data in .git/objects, but we will store these objects in the working tree. The files will be either JSON strings or binary data. The one required JSON field will be type. Creation date and author can be extracted from the Git logs.

A NoSQL document store such as CouchDB or MongoDB would be used to store the files and the JSON documents. At this point if you are familiar with CouchDB and its awesome built-in synchronization capabilities you might be questioning my sanity about implementing a new synchronization protocol. The problem with CouchDB's synchronization is that if we want to share with another user they would automatically get all of our friends' data as well. (There might be a way around this, please leave me a comment if you know of a way.) When an update is received from another user the UUIDs in your database would be updated with the latest content. To prevent tomfoolery UUIDs would be prefixed with your own unique UUID for the user who made the update so people could not clobber or update existing UUIDs in your database. When an update is received it is merged into your database.

``````> db.content.find({'type': 'update'}).sort({'date': -1})
{ "_id" : ObjectId("4de3d4a4475e87b4e7ce60d1"), "type" : "update", "user" : "Dan", "body" : "Dan welcomes everyone else", "date" : "Tue May 31 2011 02:32:20 GMT+0900 (KST)" }
{ "_id" : ObjectId("4de3d3f9668d1f97b29312ad"), "type" : "update", "user" : "jane", "body" : "Jane says: here I am", "date" : "Tue May 31 2011 02:29:29 GMT+0900 (KST)" }
{ "_id" : ObjectId("4de3d3db668d1f97b29312ac"), "type" : "update", "user" : "fred", "body" : "First post from Fred", "date" : "Tue May 31 2011 02:28:59 GMT+0900 (KST)" }
``````

Your Facebook photo albums are a little more work on the client (styling and such) but not too much:

``````> db.content.find({'type': {'\$in': ['photo', 'photo-tag', 'photo-comment']}}).sort({'date': -1})
{ "_id" : ObjectId("4de3d746475e87b4e7ce60d4"), "type" : "photo-tag", "user" : "Dan", "photo" : ObjectId("4de3d6f1475e87b4e7ce60d2"), "date" : "Tue May 31 2011 02:43:34 GMT+0900 (KST)", "x" : 20, "y" : 20, "body" : "There I am!" }
{ "_id" : ObjectId("4de3d721475e87b4e7ce60d3"), "type" : "photo-comment", "user" : "Dan", "photo" : ObjectId("4de3d6f1475e87b4e7ce60d2"), "date" : "Tue May 31 2011 02:42:57 GMT+0900 (KST)", "body" : "Nice photo if I do say so myself" }
{ "_id" : ObjectId("4de3d6f1475e87b4e7ce60d2"), "type" : "photo", "user" : "Dan", "photo" : "pointer to file in GridFS", "date" : "Tue May 31 2011 02:42:09 GMT+0900 (KST)" }
``````

Another thing that is great about this system is that it can handle new content types that don't need to be imagined when the system is created. In the same way that web browsers handled unknown tags during their Cambrian Explosion unknown content types can either be ignored or a little blurb can be shown explaining that the client doesn't know how to handle it. Clients could even give users the option to view the raw JSON of an entry to see if there is any useful information therein.