As a kid, my parents told me to eat my vegetables. My pastor told me to be nice to other kids. The state of California told me I couldn’t drive until I was 16 years old. My teachers told me – and more importantly, told my parents – how I should go about learning things.
Some of these things, we accept without a second thought. Others, like education, we’ve debated for as long as we can remember. Regardless, they all have one thing in common: someone else made my decisions for me.
This constituency pattern is everywhere. Families, companies, governments, product developers, and even protest movements have leaders who make decisions for the group as a whole. The group agrees with them on a few key issues, or voted for them, or is entranced by their charisma, or just doesn’t care enough to object.
The problem is that leaders don’t always get it right. My parents were probably on safe ground with the vegetables, but my teachers didn’t have it quite so easy. The Greek government? The Titanic’s captain? Hitler? They were all disastrously wrong, and their constituents suffered as a result.
We saw this firsthand with the App Engine datastore. When we first sat down to figure out replication, the biggest tradeoff facing us was synchronous or asynchronous. Synchronous is better, but it’s also slower. The datastore was already slower than its main competitor, the relational database – NoSQL wasn’t around yet – and we worried that if we made it even slower, developers wouldn’t use it. So we went with async.
It worked, but it wasn’t ideal. Availability was good, but we knew synchronous would be better. Same with operations workload, data safety, and performance consistency. I did research, talked with users, gave talks, wrote blog posts, and did more research.
Years later, we launched a synchronous version, and it was an immediate hit. Writes were a bit slower, but developers didn’t care nearly as much as we thought. The added reliability, predictability, and guarantees were more important. It’s now the default, and we strongly recommend it.
Sadly, while leaders like us aren’t always right, the alternatives aren’t much better. Much has been made of Occupy Wall Street’s General Assembly, a new incarnation of old ideas like direct democracy and participatory economics, but the dirty secret is that constituents don’t know what’s best for them any more than their leaders do. Henry Ford’s apocryphal quote, “If I had asked my customers what they wanted, they would have said a faster horse,” is popular in business and tech because we often have the same experience. We trust users to tell us if they like something, and maybe why, but not so much what they want.
Nuclear power is a good example of constituents and leaders failing hand in hand. Since Fukushima, countries have fallen over themselves in their haste to back away from nuclear, even though energy experts and the data agree that it’s is one of the cheapest, most effective, and safest energy sources available.
Sadly, that didn’t matter. Media covered the disaster, the availability heuristic kicked in, and our brains decided nuclear power was unsafe at any speed. Politicians, happy to represent their constituents when the cost was an externality, fell into line.
OK, so no one’s great at making policy decisions for constituents, but we still need to make them. What do we do?
Scientific research is an interesting place to look. Publication and peer review may be slow and messy, but it does consistently make progress in the right direction, overcoming even change aversion and other features of our charmingly unreliable brains.
Unfortunately, science works in the wrong direction. Researchers do experiments, measure them, publish results, and only then are judged by their peers. Groups, on the other hand, need to make policy decisions up front, before they know the results. History can guide them, but even when people do learn from it, the lessons are often unclear.
Also, scientific researchers agree on the rules of the game, not least because it’s so objective. When peers compare numbers, reproduce results, and verify logical arguments, there’s little room for opinion. Group goals and metrics, on the other hand, are fuzzier.
Representative democracy is often described as the worst possible form of government, except for all the others. Free market capitalism is similar, and probably other group structures too. I like them all, but I still hate it when they make bad decisions.
Unfortunately, I don’t know how they can do any better. How about you? What can groups do about this? Is there no foolproof way for them to make decisions?
I’ve always thought that the best way to make decisions varies by size of the group. The smaller the group the more direct the decision making can be. There’s also a lot to be said for checks and balances, and a mixture of different processes for different types of decisions. Finally, I think it’s not okay to make the perfect decision every time (that desire just leads to indecision :-). It’s okay if you occasionally have to make a U-turn. Sometimes (arguably in the App Engine case you use as an example) making the “wrong” decision at first may be the only way to eventually see what’s the right one; sometimes without the detour you wouldn’t ever have reached your destination. (Examples from Python’s history at request.)
Ryan, interesting question, but you might want to do something about spam comments… via Facebook
@Guido, done. and i think you have the right idea about using different strategies for different groups, and that iterating bad (or less good) decisions is often the way to end up with good ones. i wish there was a way with better guarantees, but i probably need to accept that like most things in life, this will always be imperfect at best!
http://en.wikipedia.org/wiki/Demarchy rulez! via Google+
Ryan,
I know this isn’t the point of your post, but it is interesting reading your take on the App Engine datastore performance. I think App Engine is genius, but I have to admit that I’ve found the performance of the datastore even when dealing with small data sets to be frustrating. When somebody asks me what I think about the app engine datastore, I generally say, it is cool, but it is slow all the time. What I mean is it is slow for both small and humongous datasets.
For instance here’s a little app I wrote on App Engine: http://www.yield.io/ It deals with a tiny amount of data, but I had to go through contortions to get the performance reasonable. This is going to sound odd, but app engine makes you think about scaling even when scaling isn’t your problem, so your app will probably scale when it needs to (even if that time never comes).
Oh and there is probably another reason why the hrd has taken off… Python 2.7 support… Otherwise I wouldn’t have bothered for a simple app like http://yield.io/.
hey chris, good to hear from you!
you’re absolutely right about the datastore. it is slower than an RDBMS, partly because it forces you into patterns that will scale long term. our usual answer is, for most read only requests, you should lean on memcache and friends so that you don’t hit the datastore often, if at all. more recently we’ve also encouraged using async calls with NDB and friends.
regardless, you’re right, especially since apps shouldn’t have to pay the price if they don’t need to scale in the first place. that’s one big reason why we launched http://code.google.com/apis/sql/ . give it a shot for your next side project, i’d be curious to hear what you think!
I memcache the entire JSON output, and even store pre-rendered JSON in the datastore. I should probably do some data sub-sampling, but I was surprised I had to think about this.
I recently discovered Bryan Caplan’s book The Myth of the Rational Voter, which tackles this question on behalf of elected governments. He argues that people vote for all sorts of reasons, sometimes none at all, and they often have good intentions, but don’t actually know much about public policy, so their opinions aren’t very helpful.
The good intentions actually hurt more than they help – if people were selfish, they might disagree and at least not generate new legislation, but since they all want what’s best, and also all fall victim to the same basic flaws in thinking about policy, they often all vote for the same bad initiatives and politicians, who then win.
How should groups make decisions when we want them to be democratic, but the constituents consistently make bad choices? Ugh.
i’m reading another book that applies to this, Thomas Sowell’s A Conflict Of Visions, which describes two fundamentally different outlooks on the world, public policy, and social decision making.
on one hand, the unconstrained vision would say that groups should make decisions by finding the smartest, most competent, most rational and enlightened members – by whatever mechanism – and then have them decide.
on the other hand, the constrained vision would say that the best decisions emerge bottom up, laissez faire style, when group members are free to choose for themselves. in this case, the group should generally make fewer decisions at all, and instead try to set incentives that encourage the behavior they want.
At ideacity this year, Daniel A Bell presented a couple ways to incorporate meritocracy into traditional democracies, including a bicameral horizontal model and a more hierarchical vertical alternative. He stressed the idea of allowing experimentation at the lower levels and generalizing successful experiments, similar to how the US states are (supposed to be) policy laboratories for the federal government, and pointed to China as a good example of this vertically meritocratic, experimental model.
ABC’s Future Tense recently did an episode on this and pointed out some interesting alternatives, including noocracy (aka epistocracy), which makes voters take quizzes on governance and policy and then uses it to weight their votes. fascinating idea, if unequal and elitist. they also mentioned some of the others here, including demarchy.