Empleo

JACE KOHLMEIER

131

What were some of the key differences between your computer science graduate studies and the accelerator? What did you find that changed your mind about industry versus academia?

The question should be: “Why the heck did I ever think grad school was right for me?” Because in retrospect, grad school, or specifically PhD studies in grad school, were wrong for me in just about every way. I had always been pretty commercial. I got my first job as a programmer when I was fifteen. I’ve always loved markets. When I was a boy, I scoured my monthly baseball card price guides with great enthusiasm. And while I do love math, my experience in grad school was fairly solitary. I found it to be mostly time spent with my head in a book, sitting in a library or literally in a windowless basement trying to prove math theorems.

It was lonely and slow and felt kind of devoid of any exciting risk. My experience in New York was just the opposite. It appealed to my commercial senses and I loved the intensity and time frame of trying to get software shipped or a product developed before the money ran out. I enjoyed the camaraderie and the teamwork, and basically just felt far more excited and alive.

So what did you do with these realizations after the incubator in New York?

What I chose to do was to go back and finish my Master’s degree at Princeton and then look for something more commercial. Living in New York had given me opportunity to meet some people in quantitative finance, which up until then I didn’t understand or even know existed. As a kid from Kansas, I had no idea that people were combining math and computer science in awesome ways and applying it to finance.

This is something that really opened my eyes. Rather than trying to work at the very depths of one particular subject, like computational complexity, quantitative finance seemed like the combination of all three of my main interests--the market, computer science, and math. The chance to take three of my loves and package them perfectly into a professionally rewarding space was a no-brainer for me.

I went back to Princeton and through on-campus recruiting, landed a job at Citadel — one of the world’s largest hedge funds.

For years when I was working in finance, I’d been fostering an idea about “high frequency education,” of using rapid feedback loops to test educational content and pedagogy.

JACE KOHLMEIER

132

Was that your first full-time job? What was that like at Citadel?

I joined Citadel as my first full time job out of college. I had a good experience there and after about 2-3 years, a few other people and I started a new business within Citadel that centered on high frequency trading. We traded a range of securities via sophisticated statistical models and algorithms. That internal group turned out to be very successful, and after 7 years at Citadel, I was able to start another trading firm with a partner.

Given that Citadel was your first full-time role after college, how did you approach learning new things in quantitative finance?

My job at Citadel was probably the first time that I really needed and wanted to learn about how to build empirical models. That was not something that I had ever really studied in school. Maybe in passing I came across a regression model in a statistics class, but basically, I was starting from scratch at Citadel. My approach — which may not have been optimal — was to start reading books. Sadly, there were not the great online resources that exist today, which is what I would now advise. I read books and I tried to pick the brains people around me that were doing the quality of work that I wanted to do. I hung off of every word that they would tell me or give me in terms of mentorship, which I sought. There’s no doubt that my most essential lessons, for both hard and soft skills, were learned from my mentors.

So how did you eventually end up turning to education?

After co-founding a trading firm, I decided to seek another challenge and at that point, I was looking for something different. Education was something that I had always been interested in and it also runs in my blood. My father was a high school teacher; my sister was a high school teacher and is now a professor of education. For years when I was working in finance, I’d been fostering an idea about “high frequency education,” of using rapid feedback loops to test educational content and pedagogy.

So that was a pet vision of mine; how we could port key ideas I had used within high frequency trading to education? As I was looking for what I wanted to do next, I explored various options. I even volunteered in a Chicago South Side school and took the exam to become certified as a teacher in Illinois. But when I came across Sal Khan’s TED talk in 2011 where he was describing a system of exercises and videos to optimize education, I was interested right away.

The coding side of it pervades all of this work. The faster you can code, the faster you can implement ideas. If you have a good sense of building systems, you can scale what started out as a research project into something operational.

JACE KOHLMEIER

133

Let’s talk about the questions that you attack at Khan Academy. What are the algorithms and problems like? How do you measure improvements to learning from Khan Academy’s platform?

Oftentimes, we know the thing that we want to measure and so we can apply statistical techniques to try to measure it very efficiently. Occasionally, we ask the user a question from a distribution that we control, but there are costs to that because the question we want to ask may not be what the individual wants to learn. So combining knowledge from information theory or graphical modelling, we can treat the answers as evidence and try to elicit the most information for a minimum cost to the user. This approach requires that you are fairly conversant with quantitative techniques.

The coding side of it pervades all of this work. The faster you can code, the faster you can implement ideas. If you have a good sense of building systems, you can scale what started out as a research project into something operational. We can work much faster if we are data scientists and engineers. We can build algorithms and models right into the product, but that obviously requires that we be competent in the product’s engineering. Of the people whom have applied for a data science role at Khan, what really stands out to you as being fundamentally core skills and what are the skills which can be learned on the job?

The hardest thing to teach on the job is a strong quantitative aptitude. Most people who apply would probably rank highly in that regard. They’ve been studying mathematics since they were 5 or 6 years old, and continuing through college, so it’s taken a long time for them to build up their knowledge base. These quantitative skills are definitely the hardest to pick up on-the-fly given the amount of time that needs to be invested, but I also don’t think that picking it up on the job is impossible.

We have developers or other quantitative scientists who may not think of themselves as being experts in machine learning, but who are clearly very technically minded and sharp. So, I don’t mean to say it’s impossible to learn quantitative aspects on the job, but simply it is the hardest thing to pick up on the fly.

Somewhat similar is coding experience, which is a productivity gauge. If you’re 30% slower at coding, you have less time to focus on other aspects and your productivity will go down. We look for fluency in coding.

As we built our team, I became increasingly skeptical of finding the perfect data science “unicorn” — someone that is world class in all of these categories.

JACE KOHLMEIER

134

The hardest thing to pick up when interviewing candidates is a person’s aptitude for experiment design (model design) and how these experimental outcomes will impact your organization. We’ve experimented with bringing people in for on-site interviews involving projects. Another initiative we have tried is to put candidates through collaborative exercises as well.

As we built our team, I became increasingly skeptical of finding the perfect data science “unicorn”— someone that is world class in all of these categories. By definition there are very few people who are world class in even one of those dimensions, and they are in extraordinarily high demand. So you really want to put together a team similar to the way a GM puts together a professional basketball or baseball team. There are fundamental skills that all players share, but the GM puts together a team of complementary members that specialize in position or area. More and more, that’s the way I think about building a data science team.

Given that the background of the ideal data scientist you’re describing implies a deep expertise within some fields, do you find that the people who are predominantly trained in these areas come from advanced degrees?

I think of team as an ensemble of specializations. I have seen a PhD experience be both a benefit or a drawback in a couple of ways. I’ve seen a PhD be a benefit when a candidate or employee has really learned to independently find their path through a nebulously defined problem, or to be able to craft experiments to get to a result that’s going to meaningfully answer pertinent questions.

For some people, it’s very clear that their PhD led them to develop that skill. On the other hand, some people’s PhD experiences left their pragmatism atrophied. At Khan Academy, there are no medals or ceremonies if we publish a beautiful research paper. What we really want to do is deliver demonstrable value to people trying to learn. A critical skill for a data scientist is knowing how one’s work fits within a team, and where your team sits within the concentric circles of an organization. In some cases that can be a skill that may have atrophied for someone who comes from a doctoral setting.

You’ve mentioned the importance of programming several times now — for aspiring data scientists who come from a strong quantitative research field, some might not

At Khan Academy, there are no medals or ceremonies if we publish a beautiful research paper. What we really want to do is deliver demonstrable value to people trying to learn. A critical skill for a data scientist is knowing how one’s work fits within a team, and where your team sits within the concentric circles of an organization.

JACE KOHLMEIER

135

have spent so much time with software engineering. What are some ways for them to increase their programming skills?

In my opinion, to be a great data scientist, you must be a great (or at least a very productive) programmer. That doesn’t mean that you have to be a savant in computer science, it just means that you have to be fluent with code and experienced in building real systems.

What I would suggest for someone who’s looking to build skills in those areas is, number one, you just have to write code and you have to write a lot of it. There will always be differences between a first year programmer, a fifth year programmer, and a tenth year programmer, at least for people who spent those years practicing the right way. The hack to get better faster is to get lots of good feedback. And the best way to get feedback is to find great developers to work with who will give you code reviews.

The great thing today — which wasn’t available in my day — is you can get involved with open source projects and get very specific feedback from great developers. This is a tremendous resource and opportunity for people who want to improve their programming skills. So write a lot of code, and make sure you’re getting code reviews from quality programmers.

On the process of implementing the machine learning algorithms — how do you learn more data science on the job?

There is not a steady rate at which you learn new techniques and employ them; it definitely comes in waves. When I made the transition into this new domain of education and internet-generated data, I went through a period of needing to learn new modeling techniques. I wasn’t familiar with probabilistic graphical models; that wasn’t something that I had used in high frequency trading.

Once I got past that initial learning curve, learning came very much in waves. There will be a very concrete and motivating need or goal. For example, we’re very focused on delivering value to the users and so any new foray into a new modeling technique is usually driven by that goal. Often we will have the necessary knowledge at hand. If not, we’ll take the time to learn what we need to know.

The great thing today — which wasn’t available in my day — is you can get involved with open source projects and get very specific feedback from great developers. This is a tremendous resource and opportunity for people who want to improve their programming skills.

JACE KOHLMEIER

136

Another idea that we often hear in our interviews is the importance of communication and how to cultivate more interpersonal skills. Either through their research or just natural personality, some people might be introverts. What advice do you have on how to manage communication when work relies on collaboration and teamwork?

That’s a great question and one that I can relate to profoundly. I would describe myself as a pretty hard-core introvert and it’s something that I have continuously had to work on in my career and continue to work on to this day. One of the greatest things that anyone ever did for me professionally was during my time at Citadel. My boss’s boss, came to me and said, “Hey, we think you have potential but there’s something that you really need to work on, and it’s your communication skills.” They put me in “communications training”, which was both useful and hilarious.

I was videotaped role-playing various business scenarios, which felt totally bizarre. I thought, “I’m a quant, this is ridiculous!” Then I watched the tape and I was appalled by looking at my body language and hearing my verbal mannerisms. I’m still working on this today. Despite how silly it seems, I totally recommend that my fellow introverts try the videotape technique. Andrew Ng recently shared a great post on how he used a similar technique to become a better teacher and presenter.

Another important development for me was partnering with someone who was very much an extrovert. That helped me in two ways. It gave me an exemplar for dealing with other people effectively. And, it taught me that it’s OK to lean on a trusted partner at times to handle the extrovert work, while I remained focused on my strengths.

So those are a couple of strategies that people can use. Number one, get yourself feedback — possibly through videotaping — and conscientiously work on your communication. Second, seek partners with extroverted tendencies that can complement your more introverted tendencies, and build extra close relationships with those people.

That’s fantastic advice. Switching gears to diving into your work, what’s a day like in the life of a data scientist at Khan?

It’s fast paced. The Khan Academy engineering team, in general, is very focused on

One of the greatest things that anyone ever did for me professionally was during my time at Citadel. My boss’s boss, came to me and said, “Hey, we think you have potential but there’s something that you really need to work on, and it’s your communication skills.” They put me in “communications training”, which was both useful and hilarious.

JACE KOHLMEIER

137

shipping and iterating quickly. We try to ship code everyday. So fitting into that, what I focus on and what my immediate team focuses on is what we call “learning gain,” and we try to take a very pragmatic approach to the work. We’re not looking to just produce research, we don’t pat ourselves on the back at the end of the day for writing a nice report or making a pretty graph. What we really want to do, which sounds grandiose, is to change the lives of our users through more effective or increased learning through Khan Academy. That is what we measure ourselves by and the questions that we ask must relate in some way to that goal, and hopefully, as directly as possible, to the main question, “What are we doing to improve learning through Khan Academy?”

The data that we deal with is almost entirely generated from user activities on the website. Occasionally, there are some complimentary external data sets, like geography, but it’s almost entirely user activity, which forms both their practice and their assessment, so to speak.

A good day is a lot of code writing because it’s the most direct way that we build value. Then, as a team lead I also need to make sure our team is in sync with the organization. I learned some hard lessons during my first couple of years at Khan on that front: A) make sure the product design is amenable to the research requirements, and B) promote ideas for doing experimental research that may not occur to others. For example, we might read something in the science of learning, like there’s strong evidence or justification for this particular style of learning, and we think it could be studied in this particular way. If we communicate this to the engineering team, they might be able to add this into the product, and we would be able to measure and build off of that.

So a good day is mostly writing code, checking in on the real time results from our A/B testing dashboard and then doing the interesting work of talking to other teams to understand how they are making their decisions, and what can we do to help them. We stay focused on the product because at the end of the day, that’s what the users touch. That’s our ultimate goal. So if we’re not changing the product and changing it in a way that delivers better outcomes to users, then we’re not doing our job.

Lets talk about the future. How do you see the foray of computational statistics into computer science? Do you see that data science will also become commoditized? How do you think data science will evolve?

When you’re doing analysis, you’re not