Building a public research database out of spare parts

Cross posted on the Color blog.

A couple months ago, we launched a public research database with DNA, health history, and more from 50,000 of our clients. You might be surprised at how little work it took us: under four person-months total. Read on to hear how we designed and built it, went above and beyond the usual privacy safeguards, and did it all in the blink of an eye.

At first glance, Color Data may seem far from unique. ClinVar, gnomAD, TCGA, 1000 Genomes, and others all address similar goals: sharing anonymized genotype and phenotype data with academic researchers to help them advance science and knowledge. We’re in an unusual position at Color, though, in that we have a large population with both sequenced DNA and self-reported phenotype that has opted to share it with researchers. Even better, our population is a bit more diverse across ethnicity, age, health history, and other characteristics than many other research datasets. Continue reading →

More 4 year old quotes:

“71 is weird because the 1 is like boop and the 7 is like boop boop!”

“Have you ever heard of hard socks?”
“Wow, no. What are they? Are they like shoes?”
“No no no. Ok let me explain it to you. They’re flappy but they don’t flap. You wear them when you ride a bicycle but only at night.”
“Whoa. Have you ever worn hard socks?”
“Of course not. But I really want to someday!”

Technically correct: the best kind of correct.

Venus de Milo statue.

Gina: [asks Brooke a question]

Brooke: “OK so this is not really a good question Mama.”

😂😂😂

Blockchain is heartbreaking.

As a computer scientist, I love it. It’s a fundamental breakthrough: the first open participation distributed consensus algorithm ever. That’s a big deal.

As an engineer, I can’t responsibly recommend it to anyone for anything. It’s slow, unreliable, immature, hard to use, and functionality impoverished. It’s a database that hates you.

Truly heartbreaking.

Snoopy

We lost our faithful cat Snoopy a few weeks ago, just before the new year. He’d been sick for a while, technically kidney failure and feline mast cell cancer, really just old age. The vet gave him just six weeks, but we nursed him along with steroids and fluids, and he managed a good five or six months beyond that. In the end, it was his time, but it was still tough to let him go. We miss him.

Gina got Snoopy and Charlie at the same time, barely after they were weaned. They’d lived together at the shelter, and she’d only planned to get one, but she couldn’t bear to break them up. They were as close as brothers; for all they knew, they were brothers.

Snoopy constantly groomed Charlie and looked after him, but there was always only one true love of Snoopy’s life: Gina. She was his mama. He followed her around the house, sat on her legs while she worked from home, kept her company while she gardened and cooked, and slept on top of her in bed at night. He was her fast companion, her kitten, her buddy. Continue reading →

Don’t solve a problem just because you can imagine it. Wait until there is a problem, then go after it. If you over-anticipate, you will design freedom out of the system.

- Stewart Brand

Goodbye Facebook, Goodbye Google+

I deleted all of my Facebook posts last week. I deleted my Google+ posts too. They were pretty much all posted here on my web site too, so nothing was truly lost, but I still feel a bit lighter, somehow.

Plenty of ink has been spilled on the problems with big social media and the companies behind it. There’s an entire movement of people leaving social networks for various reasons. Many of them have expressed their concerns, often quite loudly and eloquently, so I don’t really need to repeat them here. Consider yourselves lucky. Continue reading →

Stop paying the ETL tax

Also on the Color blog.

I want to say one word to you. Just one word. Are you listening? … ETL.

Let me guess: that didn’t set your imagination on fire. Even in software engineering and data science, it’s not exactly a household term. Nor are the more modern terms data platform or data engineering. If you do know what they are, chances are you don’t have strong opinions. You know they’re out there, people do them, and that may be the end of it.

ETL stands for Extract, Transform, Load. It’s how you get your data from your primary OLTP database, which serves your application, into an OLAP data warehouse designed for analysis, business intelligence, and data science.

Whatever your product is, it’s hopefully a core competence for your company. It’s a key differentiator. For many of us, data science and analysis are also key differentiators. ETL, however, is not. It looks basically the same everywhere, and does basically the same thing. These are all signposts that generally point toward buying or reusing, not building from scratch. Doing this kind of thing yourself just won’t move the needle. Continue reading →

Bridgy stats update

It’s that time of year again! No, not awards season…Bridgy stats time!

Looking at the graphs, the elephant in the room is clearly the Facebook shutdown. It was Bridgy’s second largest silo, numbering 1477 users when we were forced to disable it on August 1. Bittersweet.

On the plus side, I rearchitected in March and moved polling to a separate background service, which helped App Engine’s scheduler handle our workload much better. We now consistently run on one backend instance and one frontend instance, which brings costs down from $70-100/month to a very predictable $35-40. Winning!

Let’s go to the numbers. Since launch in December 2013: 3793 users, 1.22M webmentions sent successfully to almost 2000 unique domains, 12.8M responses handled (likes, comments etc), 42K POSSEs published, total cost back down to ~$.01/user/month (from ~$.02 a year ago).

Data, methodology, previously, previously, previously, previously.

Any sufficiently advanced financial instrument is indistinguishable from fraud.

— xthlc, on MetaFilter

Good technology

My electric toothbrush is good technology.

It has one button. The button turns it on. It vibrates for 30 seconds, buzzes, then repeats three more times. It has no other controls.

It works one way: the standard, ADA recommended way. That’s what most people want. If I want to brush longer, I restart it when it’s done. If I want to stop early, I press the button again.

It has one display, a LED battery indicator. The LED has three parts: low, medium, high. The only other way it communicates is by buzzing.

It serves a purpose. Its ultrasonic vibrations clean my teeth better than manual brushing. It’s also easier to reach every tooth surface when I’m not physically brushing.

It is not configurable. It has no screen. It is not smart. It has no WiFi or Bluetooth. It has no app. It is not a platform. All it has is one button, one LED, and one mode of operation.

My electric toothbrush is good technology. More technology should be good technology.

Truth.

😮

DOH (DNS over HTTPS) looks like a very small change from DNS over TLS, as they both use very similar formats on the wire. However, DOH treats the DNS response as a web object. It can be cached. It can be pre-fetched. Presumably it can be embedded in web pages. This creates the possibility of a browser defining its own DNS environment completely independent of the platformer that runs the browser, independent of the local service provider and even independent of the DNS as we know it…

Given that some 80% of all user platforms use Chrome as their browser these days then that places a huge amount of unique market power in the hands of the Chrome browser and its provider, Alphabet.

- Geoff Huston, Internet Economics

Car doesn’t understand podcasts, so it always thinks I’m listening to music instead, and guesses the album based on the title. This is one of the best so far.

New pictures are up!

🤔

Wave clouds are cool.