Engineering bioinformatics in seconds, not hours

Cross posted on the Color blog.

10,000 Year Clock, Long Now

It was winter 2014. Pharrell had just dropped Happy, the Rosetta probe landed on a comet, President Obama was opening diplomatic relations with Cuba

…and here at Color, the bioinformatics team had a problem. Our pipeline — the data processing system that crunches raw DNA data from our lab into the variants we report to patients — was slow. 12 to 24 hours slow.

This wasn’t a problem in and of itself — bioinformatics pipelines routinely run for hours or even days — but it was a royal pain for development. We’d write new pipeline code, start it running, go home, and return the next morning to find it had crashed halfway through because we’d missed a semicolon. Argh. Or worse, since we hadn’t launched yet, our live pipeline would hit similar bugs in production R&D samples, which would delay them until we could debug, test, and deploy the fix. No good. Continue reading


Building a public research database out of spare parts

Cross posted on the Color blog.

A couple months ago, we launched a public research database with DNA, health history, and more from 50,000 of our clients. You might be surprised at how little work it took us: under four person-months total. Read on to hear how we designed and built it, went above and beyond the usual privacy safeguards, and did it all in the blink of an eye.

At first glance, Color Data may seem far from unique. ClinVar, gnomAD, TCGA, 1000 Genomes, and others all address similar goals: sharing anonymized genotype and phenotype data with academic researchers to help them advance science and knowledge. We’re in an unusual position at Color, though, in that we have a large population with both sequenced DNA and self-reported phenotype that has opted to share it with researchers. Even better, our population is a bit more diverse across ethnicity, age, health history, and other characteristics than many other research datasets. Continue reading



We lost our faithful cat Snoopy a few weeks ago, just before the new year. He’d been sick for a while, technically kidney failure and feline mast cell cancer, really just old age. The vet gave him just six weeks, but we nursed him along with steroids and fluids, and he managed a good five or six months beyond that. In the end, it was his time, but it was still tough to let him go. We miss him.

Gina got Snoopy and Charlie at the same time, barely after they were weaned. They’d lived together at the shelter, and she’d only planned to get one, but she couldn’t bear to break them up. They were as close as brothers; for all they knew, they were brothers.

Snoopy constantly groomed Charlie and looked after him, but there was always only one true love of Snoopy’s life: Gina. She was his mama. He followed her around the house, sat on her legs while she worked from home, kept her company while she gardened and cooked, and slept on top of her in bed at night. He was her fast companion, her kitten, her buddy. Continue reading


Goodbye Facebook, Goodbye Google+

I deleted all of my Facebook posts last week. I deleted my Google+ posts too. They were pretty much all posted here on my web site too, so nothing was truly lost, but I still feel a bit lighter, somehow.

Plenty of ink has been spilled on the problems with big social media and the companies behind it. There’s an entire movement of people leaving social networks for various reasons. Many of them have expressed their concerns, often quite loudly and eloquently, so I don’t really need to repeat them here. Consider yourselves lucky. Continue reading


Stop paying the ETL tax

Also on the Color blog.

I want to say one word to you. Just one word. Are you listening? … ETL.

Let me guess: that didn’t set your imagination on fire. Even in software engineering and data science, it’s not exactly a household term. Nor are the more modern terms data platform or data engineering. If you do know what they are, chances are you don’t have strong opinions. You know they’re out there, people do them, and that may be the end of it.

ETL stands for Extract, Transform, Load. It’s how you get your data from your primary OLTP database, which serves your application, into an OLAP data warehouse designed for analysis, business intelligence, and data science.

Whatever your product is, it’s hopefully a core competence for your company. It’s a key differentiator. For many of us, data science and analysis are also key differentiators. ETL, however, is not. It looks basically the same everywhere, and does basically the same thing. These are all signposts that generally point toward buying or reusing, not building from scratch. Doing this kind of thing yourself just won’t move the needle. Continue reading


Good technology

My electric toothbrush is good technology.

It has one button. The button turns it on. It vibrates for 30 seconds, buzzes, then repeats three more times. It has no other controls.

It works one way: the standard, ADA recommended way. That’s what most people want. If I want to brush longer, I restart it when it’s done. If I want to stop early, I press the button again.

It has one display, a LED battery indicator. The LED has three parts: low, medium, high. The only other way it communicates is by buzzing.

It serves a purpose. Its ultrasonic vibrations clean my teeth better than manual brushing. It’s also easier to reach every tooth surface when I’m not physically brushing.

It is not configurable. It has no screen. It is not smart. It has no WiFi or Bluetooth. It has no app. It is not a platform. All it has is one button, one LED, and one mode of operation.

My electric toothbrush is good technology. More technology should be good technology.


I don’t hang out on the internet

I use Facebook. Not a ton, but I use it. I tweet, I Instagram, I read blogs. I do much of my work on GitHub. I’m on mailing lists, IRC channels, StackOverflow. Not LinkedIn, but that’s an exception. I say all this to show that I spend plenty of time on the Internet. More than my fair share.

And yet. If I hang out with people on the Internet, I generally already know them in real life. This puts me a bit at odds with online communities like open source, the IndieWeb, and others. I participate in them now and then, but I sometimes find it hard to relate to their needs and interests. They’re online communities, and I don’t really…commune…online.

This is not remarkable. For most people, it’s actually the norm, although that’s changing quickly as the more and more of the world gets online. We’re well past the halfway point! It’s a bit unusual for nerds like me, though, since discovering the internet has long been a rite of passage for us. Continue reading


How Buildings Learn

I just finished Stewart Brand‘s How Buildings Learn, a thought provoking and deeply inspiring book. I’ve been a fan of Stewart’s for a long time, through the Whole Earth Catalog, The Well, Long Now, and de-extinction, and his writing doesn’t disappoint. I’m not particularly interested in architecture, construction, or interior design, but he manages to bypass all three and find homespun philosophy and powerful insights in topics as mundane as siding materials and gardening techniques. Highly recommended.

Here are a few favorite quotes. On why buildings change (page 238):

The three things that change a building most are markets, money, and water. If you would ensure a building’s longevity, protect it from markets and water, and feed it money, but not too much and not too little.

On as-builts (page 239):

As-builts are building plans that show in detail exactly what was built, which is always significantly different from what was in the original plans. Without accurate as-builts, says Chuck Charlton, “An electrical failure can have you wandering through the building shotgunning circuit breakers and shinnying down the chases.” … If the as-builts aren’t updated constantly, each bit of repair or remodeling, each new contractor, each change of property management makes the plans more misleading.

On state-owned property (page 163):

When the landlord is the state, as it was in communist lands, you get the ultimate in negative maintenance. All visitors to the mortally rundown buildings of Eastern European nations have tales like Brian Eno’s: “My wife and I were checking in to a hotel in Moscow. Our host showed us to our room, and began switching on the lights. As he turned on the one by the door, a great tongue of flame issued forth from a light fitting in the ceiling. He calmly switched it off again and said, ‘Don’t use that one.” Since no one owned the light, why should anyone fix it? A command economy displaces responsibility even further outside the building than a market economy does.

On getting rich quick (page 165):

“People want to get rich quick.” The other side of the coin is, Go broke quick. Real estate is the classic case of soar and collapse, of tycoons going bankrupt and taking shortsighted banks with them. Work done in haste is necessarily shoddy, a house of cards. On a go-fast schedule there is no margin for a single error, and error is inevitable. High risk, high loss.

The opposite strategy is much surer, because the errors are piecemeal and correctable. When you proceed deliberately, mistakes don’t cascade, they instruct. Low risk plus time equals high gain. This strategy treats the fundamentals of the living investment with attention and respect. The lesson of realty laced with reality is: “Get rich slow.”

On temporary fixes (page 369):

Beware: in the real world “temporary” is permanent most of the time. If the cheap trial worked, it will be left alone, no matter how funky it is. If it failed, it’s embarrassing to fix. Life rushes on to more pressing or interesting problems.

On HOAs (page 153):

…new communities seek to pre-empt any such adaptivity by repressive, fiercely enforced “covenants, conditions, and restrictions.” These are the dread “CC & Rs” that homeowners’ associations use to control such details as what colors you may paint your house, what pets (and in some cases what children) you may keep, how your lawn will look, your roof, your fence, your driveway (no campers, trucks, or car repair), your backyard (no drying laundry or unstacked firewood). Any neighbor might report you. What if you ignore or defy such rulings? The homeowners’ association can take your house or send you to jail. Joel Garreau points out that these organizations have all the powers of government—the ability to tax, to legislate, and to police—without the usual restrictions of democratic representation or being answerable to the US Constitution.

Garreau contrasts a new development such as Irvine, California, to the once-deplored original Levittowns that were created for postwar families back in 1949: The old Levittowns are now interesting to look at; people have made additions to their houses and planted their grounds with variety and imagination. Unlike these older subdivisions, Irvine has deed restrictions that forbid people from customizing their places with so much as a skylight…Owners of expensive homes in Irvine commonly volunteer stories of not realizing they had pulled into the driveway of the wrong house until their garage-door opener failed to work.

This degree of institutionalization of real estate value over use value is odious enough as an invasion of privacy, but it also prevents buildings from exercising their unique talent for getting better with time.


Pessimistic induction

One of my favorite ideas from recent memory is pessimistic induction. As usual, a quick search finds plenty of smarter people who thought of it before me and easily refuted it. Even so, it’s oddly compelling, a gem of a misconception.

Looking back at history, most of our ideas, even the best ones, have turned out to be wrong. Very few have stood the test of time. Newtonian physics, the rational economic actor, and fat vs sugar are just a few famous examples.

It’s easy to think that we’re at the final culmination of our entire historical arc of science, art, and civilization. We may have been wrong in the past, but we’ve rooted out our mistakes, corrected them, and we now have everything figured out. It’s such a common misconception that it has its own name: the end of history illusion.

That name is apt, though: it really is just an illusion. The present day may feel special, but it’s usually just like every day before it. Many things are good, some things are bad, and our currently accepted scientific ideas are almost certainly wrong, to one degree or another. This is the pessimistic induction.

It’s chillingly elegant, but it has a fatal flaw. Yes, today’s best science may likely be wrong, but right and wrong are rarely black and white. Modern physics is famously incomplete, but working physicists would still say that the standard model and string theory and holographic universe are better ideas than Newtonian physics. We may not be perfectly right about everything, or maybe even anything, but we’re probably more right than we used to be.

That’s a comforting thought. The universe may be cold and indifferent, but we can admit our flaws and still hew ever closer to understanding. Onward.


What I work on

I had a conversation with a good friend recently that crystallized something I’d always felt strongly, at a gut level, but never thought through: how I choose what to work on.

When I look for a new job, I think about project, people, compensation, role, company, commute, etc. I’ve tried focusing on different factors over time, and I’ve found that for me, project is often the most important. I’ll suffer with low pay, long train rides, or a role I’m overqualified for if I’m working on something I care about and believe in.

I prefer tools over products. Systems over tools. Protocols over systems. Problems over users. Wicked over tame. Research over application. Many of these are stereotypical engineer cliches, but they boil down to an interesting theme: I prefer to work in areas where the goals and incentives don’t change much over time.

I don’t know where I developed this tendency toward the long term, but it’s a big personal motivation. The time scales I’m thinking about are centuries and millenia, not years or decades. I could just as well replace time with generations. I’m fine with not shipping code often, or not making any progress for longer stretches, if I know the problem will still be around and my work will still apply down the road.

What does this mean? Well, scratch most products – consumer, enterprise, or other. Some of them last centuries, but not many. Scratch applications and services in general. I’m happy to do work that’s used in a product or service, but usually only if there’s an underlying problem with a longer lifespan.

The two main areas that fit are research and infrastructure. Academic departments and conferences rise and fall, but the central goal of research has stayed the same forever: pursuit of truth, knowledge, and understanding. That won’t change anytime soon.

Infrastructure, on the other hand, is worlds removed. Construction workers in hard hats on building sites don’t overlap much with tweedy professors in ivory towers. They do have one thing in common, though: their goals are consistent over time. If you want to cross a river today, you build a bridge, just like a thousand years ago. We still need roads to get from one place to another. Plumbing to carry water and sewage. Electricity and communication grids may be newer, but we’ll need energy and communication in a thousand years just like we do now.

When I look at the projects I’ve enjoyed most in my career, they fit the bill. Sharding databases and later Paxos etc: classic infrastructure. Networking: infrastructure and applied research (OpenFlow). Color Genomics: applied research. App Engine: infrastructure as a product. Even the side projects I’m looking into now fit: climate change, p-hacking, the reproducibility crisis.

Why do I care if goals change over time? I’m not sure. Some of it may be the natural human desire to leave a legacy. If I work on big, long standing problems, I’m more likely to be remembered after I die. I don’t spend much time thinking about legacy, but it could still be lurking in my subconscious.

A modern variation is “changing the world.” It’s a well worn phrase here in Startupland, but for me personally, it’s always seemed hopelessly ambitious. I have no illusions that I’m personally going to change the world in any significant way. Maybe a little, if I’m lucky, but not a lot.

Another Silicon Valley buzzword is “impact.” Everyone wants to work on something impactful. Most people use it to mean a bold new product, or a big user base, or innovating and disrupting an industry. I want to have impact, sure, but I want to do it by moving the needle on a big, important, long term problem. Growth hacking and TechCrunch coverage aren’t part of my personal equation.

Research and infrastructure aren’t unique. There are plenty of other areas where goals and incentives stay the same over time. Art, clearly. Philanthropy, education, entertainment, health care, public policy…the list goes on. I’d get restless if I was a teacher or actor or nurse and didn’t do anything new, but there’s plenty of opportunity to push on big problems in those fields on the front lines. I’d hate being a campaign manager, but I could easily do a stint as a policy wonk at a think tank.

This may not mean much to you, or even to me. After all, it was guiding my career decisions long before I thought it through and wrote it up. Still, now I know…and knowing is half the battle!

Don’t get the wrong idea, I’m still loving it at Color Genomics! I’m not going anywhere. On the contrary, we’re actively looking for good people. If you want to work on something meaningful and challenging, drop me a line!

Also, scratching my own itches is one big exception to this rule. If software is the tool of the knowledge worker, I’m lucky to be a toolsmith. I’ve written and modified lots of software over the years to solve my own problems. Some took significant time and effort, like granary and P4, and some have real user bases, like Bridgy and huffduff-video.

Even so, these tools have always felt practical, utilitarian, even a bit disposable. I don’t consider them a big part of my career or life’s work. I won’t need them forever, and they’ll all grow old and die eventually. That’s OK.


A medium is born

Artistic mediums are a small, rarefied lot. The spectrum from War and Peace to The Da Vinci Code is huge, but it’s all a single medium: books. Citizen Kane and Jackass may not have much in common, but they’re both movies. Art, music, theater, radio, games, maybe graphic novels…that’s pretty much it. We can haggle over subtypes like radio drama or journalism or stand-up comedy, but still. Thousands of years of civilization, and we can count the main artistic mediums on just two hands, with room to spare.

…except now. VR is here, and it’s a truly new medium. You and I have the awesome privilege of witnessing its birth firsthand. That doesn’t happen often, and it’s pretty damn cool.

Mediums don’t spring into existence fully formed. Music has come a long way since Gregorian chant. Modern movies may descend from Méliès and the Lumières, but they’re so different as to be almost unrecognizable.

For example, film buffs celebrate movies with extended single-take scenes like Children of Men and Birdman, but the first movies were all single takes. Early filmmakers thought cuts would confuse audiences or lose them entirely. Reality doesn’t have cuts, right? Moving cameras, close-ups, subtitles, and establishing shots all have similar origin stories. They didn’t happen automatically. Early filmmakers discovered them through slow, tedious experimentation. And for every technique that lasted, ten others were tried and discarded.

We’re already seeing the same thing happen with VR. Just to get mainstream devices out the door, developers had to find the right resolution, framerate, field of view, latency, and inter-pixel distance (aka screen door effect) to make the experience viable.

Once creators got their hands on dev kits, they faced an entirely new set of challenges. VR infamously makes some people nauseous, especially when they’re moving in the virtual world but not in the real world. Creators have tried all sorts of things to prevent this, and many are now settling on two that work: teleporting and tunnel vision. Similarly, filmmakers aim their cameras, but 360° films let viewers look anywhere, so VR filmmakers are learning to frame shots with sets and lighting instead of cameras.

Good artists borrow; great artists steal, as the saying goes, and artistic techniques are no exception. Video games have faced many of these problems for decades, and developed their own solutions. Valve’s Half Life 2 famously pioneered set direction and framing to get players with freedom of movement to look where they wanted. Likewise, user interface conventions for Nintendo Wii and Microsoft Kinect motion controllers laid the groundwork for many VR UIs.

I find all this fascinating. Technology is constantly evolving, so we often think about how new technology enables and shapes art, whether it sticks (like 35mm film) or not (like multimedia). But art also shapes technology. Set framing and teleporting in VR may be intangible, but they’re arguably still technology just like 35mm film.

Even when we do think about art shaping technology, it’s much harder to actually see it happen in real time. Evolution is an apt analogy here. When life first hits a new environment, there’s an explosion of differentiation as evolution finds all the species that can survive. Hence “Cambrian explosion.” Progress soon slows down to incremental improvement, though, as more and more niches in the ecosystem are filled in.

Artistic mediums are the same way. At the beginning, no one knows the rules – there aren’t any! – so creators try anything and everything to see what sticks. This experimentation slows down as winners emerge. Novels have been around for millenia, for example, so we don’t often see truly new structures or techniques. We may gush about unreliable narrators or experimental forms these days, but even those are rooted decades or centuries ago.

OK, so there aren’t many different artistic mediums, VR is a new one, and we get to witness its birth. So what? Should we watch for anything in particular? Should we nudge it in some specific direction? Are there unique opportunities opening up that we may not see again?

Probably not. VR has plenty of attention and press coverage, and a small army of developers and creators pushing it in every direction. We’ll be learning new things for decades to come, and each step along the way will be heavily documented. More importantly, as with all technology, we don’t have much control over how it evolves anyway.

VR isn’t even the first new medium in recent history. Art, music, theater, and the written word may predate the 20th century, but radio, television, film, graphic novels, and video games were all born in just the last 2% of recorded history. We watched creators grapple with them, documented their progress, and adopted them into the mainstream just like we will with VR.

So, my only call to action is to pay attention. There’s plenty of precedent, sure, but the birth of a wholly new artistic medium still feels unique and new to me. It’s the first time I’ve ever really seen it happen. I don’t know where creators will take VR next, but I can’t wait to find out.


Drawbridge up, drawbridge down

When populations are homogenous, people see that other people are mostly like them – ethnically, culturally, socioeconomically, etc. This boosts trust broadly, which makes everyone more open to progressive social policies and safety nets.

When populations are diverse, people are more visibly different, at least on the surface, which leads to othering, dampens trust, and leads to more protectionist, socially conservative policies.

This is an oversimplification, and happens only in our subconscious, but there may still be a nugget of truth to it. It might be one reason that Scandinavian countries have long been so open, progressive, and even socialist: their populations are extremely homogenous. North America and western Europe are historically diverse, on the other hand, now more than ever, which has coincided with populist waves of nationalism, isolationism, and xenophobia.

Stephan Shakespeare, co-founder of pollster YouGov, has a famously evocative metaphor:

We are either “drawbridge up” or “drawbridge down.” Are you someone who feels your life is being encroached upon by criminals, gypsies, spongers, asylum-seekers, Brussels bureaucrats? Do you think the bad things will all go away if we lock the doors? Or do you think it’s a big beautiful world out there, full of good people, if only we could all open our arms and embrace each other?

Whichever you feel personally, this begs the question: why do you feel that way? Homogeneity or diversity in your immediate surroundings could be one answer.

Subtle, powerful point that I hadn’t fully appreciated until now. Thanks to Jonathan Haidt.


Channeling the hacker way

Also posted on the Color Genomics blog.

We have big dreams and ambitious plans. We want to push the state of the art in health and genetics, and we need broad, crazy moonshot product ideas to get us there. How do we find those ideas?

We’ve always been inspired by 20% time at Google and 3M and hack weeks at Twitter, so we decided to do our own hack week. We invited everyone to put normal work on hold for a full week to try out new ideas, no matter how crazy or tangential. We didn’t know how many people would participate, or whether we could ship any of the results, but it was worth a shot!


Our first task was to pick a theme. We chose engagement: how can we help people continue to engage with their genetics and health over time? We dragged everyone into a few big rooms and brainstormed project ideas. No judgment, no constraints, no requirements, no analysis or pro/con lists. Just ideas. By the end of these sessions, we had over 50 candidates to explore in the week to come.

We kicked off hack week proper on Monday morning with an example project: reincarnation detection. Could we identify who you’d been in past lives? With tongue firmly in cheek, we described how to recruit teammates with good karma, work with researchers to identify genetic markers across lives, partner with Buddhist monks, and design a “Talk to the Other Side” messaging UX.

We set everyone loose, and they hit the ground running. People wrote up pitches, formed teams with great names – Pink Duck was a personal favorite – and jumped on company mailing lists to recruit team members. Many people joined two or three teams each.

We emphasized that we wanted everyone to participate, across job roles. Design mockups and prototype code were nice, but not required! We encouraged people to start broad and high level, then find a vertical slice that they could flesh out in a single week. We put up screens around the office that rotated between team Slack channels, project docs, and mockups and prototypes as they came together.

The week culminated in an all hands Demo Day where each team got a few minutes to present their project. We saw live demos, skits, theme songs, and even a custom video. We voted on yearbook-style awards like Best Dressed, Most Likely to Succeed, and Crazy like a Fox. Everyone had a blast.

After the excitement wound down and the dust settled, we surveyed the results. Over half the company participated, working on 11 projects that all made significant progress toward proofs of concept.

Hack week infused us with new ideas to help our clients understand and act on their genetic data. We look forward to implementing some of the best ideas, and we hope they’ll help our clients lead longer, healthier lives. That’s the real prize.

We’re always looking for talented engineers. Join us!


Wanted: climate change project

I’m in the market for a new side project. I’m looking for something related to climate change. I don’t know if I’ll end up farming biochar, spraying aerosols into the sky, hacking solar panel trackers, or something else entirely, but those are the kinds of thing I’m thinking about.

I’m leaning toward something technical. I’m open to coding, but ideally that won’t be my primary contribution. I’ve had a blast hacking on open source projects, and I’m sure I’ll do more, but right now I’m thinking about something different.

So far, I’ve just been reading a lot and learning as much as I can. There’s lots of great stuff going on, and I’ve found a few open communities here and there, but most substantial projects are either companies or academic research labs. Those are both great, but they’re hard to join part time and contribute a handful of hours a week.

The next step is to talk to people who know more, or know other people who do. If you know the space and have an interesting problem I might be able to help with, or if you know someone who might, please drop me a line!


College is more than a job ticket

It’s become fashionable recently to second guess college. The ROI no longer works: it’s too expensive and doesn’t guarantee you a good job. It’s elitist and out of touch with reality. Student debt is predatory and out of control. Anyway, MOOCs’ unbundling model is the future of higher ed, so we might as well get on board, right?

That all may be true, but I think it’s too narrow. There’s a corollary to “you can’t optimize what you don’t measure”: measurement can give you tunnel vision. You can collect X University grads’ incomes, divide by tuition, and compare to College Y, but that doesn’t mean you can reduce either one to a simple financial investment you optimize to get the best salary.

College is an experience. It’s one of the most critical periods of your life: when you become an adult. You learn what to eat, what to drink, when to go to sleep and when to wake up. You manage your time (or not), juggle priorities (or not), make commitments and break them. You find substances, wonderful horrible tempting substances. You make friends and significant others, some more significant and some…um…less.

Most of us do this by making mistakes. No matter how mature we were already, every one of us slept through the History 101 final and flunked, or jumped into bed with someone we knew would break our heart, or woke up in a bush with no pants and the mother of all hangovers.

You can do this without college, of course, but the real world is harsh, and college has training wheels. Advisors, RAs, dorms, cafeterias, and built in health care make for a forgiving place to learn to “adult.” Drunk bicycling is a lot less dangerous than drunk driving. Classes are the perfect practice for jobs. Hated one? Failed the midterm? Start fresh next semester, older and wiser and still in the same dorm and meal plan.

Sadly, one big flaw with these safety nets is that they’re unequal. On campus housing, student advisors, and extracurricular activities all cost money. They may be standard at expensive top tier schools, but not at smaller state schools and community colleges. Maybe we should vote for Bernie next time.

I think a lot about how to prepare my daughter for the real world. I catch her if she’s about to fall off the bed, but I also show her the edge, let her look down, and say, “See? If you fall, it’ll hurt!” Sometimes I even let her fall a bit – not far, just enough to notice.

My college gave me the same kind of real world training wheels. The degree helped me get a job, sure, and the classical education made me a better person and citizen, but I treasure the safe space it gave me to grow. College is more than a financial investment. It’s a critical transition from childhood to adulthood. Don’t give that up.


Decentralized Web Summit

I spent the last few days at the Decentralized Web Summit, a small gathering of like-minded hackers, thinkers, and activists from all over. I don’t go to many conferences, but this one was inspiring and exciting. Even the mainstream press noticed. I’ll see if I can describe why.

I spent some time in the peer-to-peer community during the first dot com boom. I hung out with the p2p-hackers and CodeCon folks, co-created a toy P2P network and contributed to others, idolized Nullsoft and Bram Cohen and anonymous remailers, and generally yearned to be free of The Man in the middle. By the time BitTorrent hit it big, P2P seemed unstoppable, even inevitable.

It wasn’t, of course, but recently there’s been a resurgence of interest. The financial crisis, the NSA spying revelations, and online power concentrated into a few big silos rallied the internet’s venerable elders, Vint Cerf and Tim Berners-Lee and Brewster Kahle and Mitchell Baker. (And Richard Stallman!) They turned over a few rocks and found the P2P cypherpunks keeping the flame alive, developing projects like IPFS and Dat and Tor and falling all over themselves to sanctify Satoshi and create the Next Big Blockchain.

The MC called us OGs and New Gs, drawn together by a common desire to redecentralize the web. We discussed the past, present, and future, compared projects and protocols – including my own little ditty on the IndieWeb (video, slides) – and debated what to do next. It was great.

My first big takeaway was that the community seems to be maturing. Just by organizing the summit, the elders showed that they were paying attention and cared. That wasn’t really true last time around. And given TimBL’s position atop the W3C and Brewster’s irrepressable energy for catalyzing action, that definitely matters.

Second, there was a lot of talk about real world problems like UX and monetization. People widely acknowledged that P2P projects still aren’t usable or accessible enough and don’t always address real user problems, which may be why they haven’t hit the mainstream yet, Skype and BitTorrent notwithstanding.

Also, despite the usual appeals to micropayments and transaction fees, most people admitted that we still don’t know how to sustainably pay for systems without centralized control. The honesty was very encouraging. The first step is admitting we have a problem!

Finally – and feel free to take a drink here – the blockchain. I’d grokked it at a high level before, but I came away from #DWebsummit with a newfound appreciation.

Yes, it’s an entirely new consensus algorithm, and as Mike Burrows famously declared, those don’t come around very often. More importantly though, it’s the first open membership consensus algorithm we’ve ever seen.

Before Bitcoin and the blockchain, all consensus algorithms required a closed group of participants. They came from the distributed systems community, from people who built and ran self-contained clusters of servers they owned and controlled. Bitcoin changed all that. Anyone can mine a bitcoin, or send one, and those transactions are just as consistent and durable as a Paxos round or ZAB broadcast.

This is pretty huge. The internet depends on centralized human organizations like ICANN, CAs, and Tier 1 networks, and they’ve worked surprisingly well for a surprisingly long time, but failures like CA compromises and BGP hijacking have exposed their inherent flaws. Projects like Certificate Transparency are laudable, but they’re really just band-aids. They can contain damage, but they can’t prevent it entirely.

Human organizations regularly do superhuman things, and they may still be the right long term approach here. We’ve just never had a real alternative before. Now we do, and that’s pretty damn cool.

Don’t worry, I’m not dropping everything to create a cryptocurrency or drum up VC funding for a blockchain startup. I have downloaded a handful of papers to read, though, and I may drop by Friday lunch at the Archive now and then. DWeb renewed my interest in the once and future decentralized web, and that’s a good thing.


Judging campaign tactics

Donald Trump.

Admit it, you have half a lather worked up already. The man is incendiary; the campaign is astounding. He’s built a big base of support with tactics that we thought were off the table entirely and should have sunk him long ago.

What changed? Do we not have the grasp on campaign tactics we thought we did? Do we need to dive back into modern electioneering, break it down, and really understand it piece by piece if we want to get the leaders we think we’re electing?

Campaigning today is a complicated business. Fundraising, optics, opposition research, social media…there’s a lot to it, all far removed from the actual task of governing and policy making. (You may still spend much of your time fundraising and campaigning once you’re in office, but that’s a different issue.)

The founding idea of democracy is that citizens choose their representatives directly. For that to work, they need a clear picture of who each candidate is, what they stand for, and how they’d govern. So as a society, we’d ideally judge campaign tactics based on how well they describe a candidate. Facebook post on Alice’s home town? OK. Deceptive attack ad that lies about Bob’s voting record? Not OK.

Many common campaign tactics fail that test. We may not have large scale voter fraud, but we do have personal attacks, smears, undercover stings, and recently the rise of the “personal fact.” Social media targeting and astroturfing may be milder, but still misleading. Swinging right (or left) for the primary and back to center for the general is so common, we expect all candidates to do it.

We’ve had a bit of success outlawing and discouraging some of these tactics. Campaign funding reform has made some progress, Citizens United notwithstanding. Candidates routinely promise not to take money from certain groups (or anyone) and not to “go negative” and attack their opponents directly. Politifact has built a business determining whether campaign statements are true or pants-on-fire false.

That’s all well and good, but bad tactics persist. If we can’t prevent them, should we take them into account when we vote? Most candidates probably believe campaign tactics are for campaigns, and wouldn’t inform how they’d govern, but I’m not convinced. We routinely see elected officials smear opponents like WikiLeaks’s Julian Assange, for example.

Many candidates rationalize bad campaign tactics by saying the ends justify the means. “I’ll do good once I’m in office, so if I step on a few toes to get there, it’s worth it.” Utilitarian ethics and philosophy are beyond the reach of this little essay, but I will say that actions speak louder than words. If “ends justify the means” appeals to a candidate during a campaign, it will appeal to them afterward too, and next time those toes might be chests, or necks.

Predicting the future is hard, even if you’re creating it. If you don’t get elected – and you probably won’t! – it’s a lot harder to justify hurting people along the way. I constantly have to remind myself of this at work. It’s so tempting to add just one more thing to a product or codebase because I’m sure I’ll need it down the road, but I often don’t, and then I’m stuck with the extra complexity for no benefit.

There’s also the character question. It’s clearly unacceptable to call Megyn Kelly a bimbo or characterize Mexicans as rapists and criminals, full stop. Even if Trump might not do that kind of thing as President, is it a sign of the policies he’d pursue? How about the more abstract notions of character, integrity, and values?

Politicians’ personal lives have only been fair game in the press for half a century, or so, at least in the US, and it took another few decades for the Clinton/Lewinsky scandal and impeachment to ask whether politicians can compartmentalize private lives and public responsibilities. We’re still figuring out how we feel about all this, campaign tactics included.

In the end, the “clear description” test might be the best measuring stick. Democracy needs voters to understand what each candidate stands for and how they’d govern. Fundraising and opposition research may make us squirm, but they don’t necessarily mislead the electorate. Smears, personal facts, and changing positions to suit polls do. We shouldn’t stand for them, and we shouldn’t vote for candidates who do.


What doesn’t kill my baby

Brooke used to have a painful head-butting habit. She’d be happily playing, then all of a sudden, she’d slam her head into my cheek…or the floor, or a toy, or anything else in range. She was just learning to use her neck, but she still cried and whimpered whenever she connected with anything solid. I cried too, when my nose or chin was in the line of fire.

We hated it every time we heard that *crack* and her shriek of pain. Our baby was hurting! We considered finding a helmet, or holding her at arm’s length, or even padding the house with bubble wrap, but they all seemed a bit overboard, even for us. Besides, she gradually strengthened her neck and stopped flailing quite so much. Life finds a way.

Brooke has decades of years of learning and life lessons ahead of her, most far harder than this one. She’ll handle most of them on her own, via trial and error. Others, she’ll learn from her friends, or from what she sees us do as role models. She may absorb a handful of things we tell her, but they’ll be few and far between.

I know we can’t change those ratios, and I’m ok with that. We still have some impact on the trial and error, though, especially while she’s still young and we control her environment. It sounds crazy, but we really could have plastered the house in bubble wrap, or popped a junior Pee Wee Football helmet on her head, and she would have avoided those painful bruises. On the other hand, she wouldn’t have learned that head butting hurts. She might still be doing it today.

I believe in the trial and error thing. I want her to make little mistakes and learn from them. Human beings are antifragile; failing and falling down usually make us stronger over the long term. It’s how the real world works. There’s plenty of ink spilled on this over the years, from Dangerism and Nurture Shock to the hygiene hypothesis and arresting a mom for letting her kid play unsupervised.

The problem is, Brooke is our baby! We hate it when she hurts. Right now, we instinctively reach out and catch her when she’s about to fall over. In a few years, we may not let her gorge herself on candy to the point of a stomachache. We may force her to study so she doesn’t get bad grades. We may even warn her if she dates the wrong boy or girl. (I expect she’ll ignore us.)

All of those pain points are life lessons. We can prevent the pain, or she can learn the lesson, but not both. So what do we do? How do we rationally let her make mistakes and get hurt when our emotions are screaming Don’t let her make herself sick! Don’t let that boy break her heart! She’ll regret it! You’ll regret it!

Honestly, I don’t know. Kahneman himself says it’s a hard problem. His advice is to identify when you’re acting from instinct or emotion, slow down, and give your conscious brain a chance to weigh in. Not easy. I can do it, but I still wonder if there are any shortcuts.

Maybe I’m overreacting. Maybe we don’t have enough impact on Brooke’s development for any of this to matter. We may not be around for most of her learning moments, and even when we are, we may not be very good at intervening. Maybe this is all just an ego-driven illusion, and I should sit back and enjoy the ride. When she’s older, I know she’ll pay way more attention to her friends and the wider world than us…but for the moment, I think we still have some impact, even if just a little.

Brooke stopped head butting things a few months ago. She sits up on her own now, and she usually stays put, but every now and then she leans back too far and falls over. Sometimes we catch her. Sometimes we don’t, and she falls and bumps her head. There’s still no rhyme or reason to our choices, and we definitely don’t have any hard-won wisdom or master plan yet. If you know of any, I’m all ears!


Thoughts on having a baby

  1. You expect your world to narrow a bit. You can’t just bounce out the door whenever you want, and besides, you still need to figure out what the hell you’re doing.

    Even so, it’s a surprise when your thoughts are suddenly consumed by poop, and poop exclusively. Smell, color, texture, volume, frequency, how it arrives, how it departs, which clever literary references describe it best.

    Friends told you this would happen, but still. You realize you’ve hit rock bottom when you start gushing at a dinner party about soft, even grains in a thick yellow paste. Don’t fight it. Embrace it. (The talking, that is. Not the poop.)

  2. Asking other parents for advice is like reading the Bible. It’s a deep well, and there is some good stuff in there, but it’s couched in language you don’t understand, and when you do piece together a few bits, they all contradict each other.
  3. The advice you hear most often isn’t really advice at all, so much as a doomsday prophecy. “Your life will change forever,” they intone vacantly. “What should I do? How can I get ready?” They laugh, “There is no ready. Forsake all hope, ye who enter parenthood.”

    This is, of course, maddening. It’s the biggest change in your life, bar none – marriage isn’t even in the same league – and you can’t prepare for it. If you’re used to bending the world to your will, this will be tough. You can’t help it. You’ll read books, buy clothes, put together a nursery, and that’s all well and good, but it won’t matter. When she arrives, it’ll still hit you like a ton of bricks.

  4. Sleep deprivation. This is the other advice/doomsday prophecy you hear constantly. Know how you greet new parents with the gentle joke, “Getting any sleep?” It’s no joke. They’re not getting any. They’re up all night, feeding and rocking and pacifiering and white noising and not sleeping. Night after night after night.
  5. Having said that, we’re pretty lucky. Brooke was extremely chill for her first month or so. She’s more awake and vocal now, but still, she’s pretty damn easy. We have no idea which god smiled on us after all the goats we sacrificed, but we won’t argue.

    (I’m totally kidding, Baal, we know it was you. The goat’s in the mail.)

  6. The cats have been great. They took a few days to warm up to Brooke, but now they love her, and they’re very gentle. In general, if you worry about pets attacking your baby or smothering them, don’t. They’re more likely to get kidnapped by the chupacabra. Instead, worry about actual dangers: falling into the pool, getting hit by a car, developing allergies and metabolic syndrome (among others) due to oversanitizing.
  7. …but you won’t get much chance to worry at all, since one of the biggest things you lose is unbroken time. Love going for a long run, playing a great game until 2am, curling up and reading a trashy book cover to cover? Too bad. Free time now comes in 5-15 minute bursts. Get used to running laundry back and forth, playing Candy Crush, and reading Facebook posts. Pro tip: try podcasts or audiobooks, they’re hands free.
  8. The worst part might be the unrelenting sense of helplessness. If you’re accustomed to controlling your environment, being productive, getting things done, hoo boy get ready. A baby will smack you upside your head. The first time they scream for an hour straight and you can’t calm them down, no matter what you do? That’s tough. The tenth time? The hundredth? Straight up demoralizing.

    Take a step back, leave her in the crib and close the door and catch your breath. She may be pissed, but she’s also healthy, vibrant, and damn but she has a pair of lungs on her. She could be the next Lady Gaga.

    Repeat after me. The baby will cry. The milk will spill. The diapers will blowout. It is as it ever was, time and time again. Amen.

  9. I’m obviously not much for helpful tips, but I will happily shill for one product: Dr. Brown’s bottles and pacifiers. They’re great.

    I don’t know what it is about babies and burping and farting, but it’s kind of a love/hate relationship. There are techniques and positions galore to avoid swallowing air, but in the end, Dr. Brown’s bottles worked for us. Brooke stopped fighting and actually relaxed while she ate. Winner winner bottle dinner!


Happy 1000th, Bridgy

Bridgy, my little IndieWeb side project, hit a milestone a couple days ago: 1000 users! Congratulations Brett Glisson, you win the prize!

1000 isn’t a big number. We’re a long way from viral marketing, growth hacking, and customer acquisition tracking, and that’s fine with me. I built Bridgy to scratch my own itch, in fine open source and IndieWeb tradition, and only launched it publicly because I thought other people might have the same itch.

Milestones are good excuses to count blessings, and Bridgy has plenty to be grateful for. Kyle Mahan has stepped up in a major way, building substantial new functionality and supporting servers and users alike with aplomb. He’s basically a co-owner at this point. You rock, Kyle! Emma Kuo, Barnaby Walters, and Kartik Prabhu have also contributed great code, and the IndieWeb community‘s support has been invaluable. Thanks everyone!

Milestones are also good excuses for navel gazing, so here are some graphs. They have pretty colors!

First, user growth. Clearly not exponential, but it is keeping up a steady clip. Also see the cumulative graph at the top.

Many people sign up for more than one silo – Facebook, Twitter, etc. – so it’s technically 1000 accounts, not users. I’d guess there are only 400-500 distinct users. It’s not always easy to tell the difference automatically, though, and 1000 is a nice round number, so I’m going with it.

Now, the stuff Bridgy actually does. Note that the second graph is log scale, since I threw in everything but the kitchen sink. Tufte would have a fit.

So much for the candy; now it’s time to eat my vegetables. What lessons have I learned so far? What has building Bridgy taught me?

Honestly, I’m not sure. I can’t think of anything particularly interesting or insightful. We need a balanced meal, though, so here are some thoughts.

First, the “scratch your own itch” thing really worked, at least in this case. I always knew what to build next, and how to prioritize, based on what I wanted for myself. Tellingly, the major features I added later – Bridgy Publish and webmentions for blogs – weren’t as strong itches for me personally, and correspondingly, each one has seen less uptake than the last. The graph ain’t lying.

Second, I want happy users, but tech support is no fun. When someone has a problem or a question, I try to find and fix (or automate away) the root cause, or at the very least update docs so the next person won’t need to ask. It’s not perfect, but I think it helps…

…or maybe it’s just that for most users, Bridgy is fire and forget. They sign up, poke at their user page, maybe skim the docs, and then never come back. They don’t need to. Comments and retweets and +1s start showing up on their web site automatically. This is my favorite kind of UI: none at all. And thank God for that, since I’m worthless at UI design.

Lastly, unit tests. I won’t go all rabid religious dogma on you, but man. The freedom to make big changes, refactor core logic, and then push that new code live, confident that it won’t break anything? It’s incredibly liberating. Not to mention that running them before deploying has saved my ass on more than one occasion.

Automated monitoring paged you at 3AM because the new release has a regression bug? Good. At least you caught it. Slept until morning because your tests caught the regression before it went live? Priceless.

Anyway. Contributors are still cranking away, and I’ve been pushing out a steady trickle of tweaks and bug fixes, but that’s slowed down recently. None of the remaining feature requests are above my itch threshold, so they may not happen anytime soon. I happily accept pull requests, but otherwise, Bridgy is basically on ops autopilot right now.

Keep it up, Bridgy! Here’s to 1000 more!