What makes a successful Data Scientist?

I've been lucky enough to manage somewhere in the low three digits of Data Scientists and other developers (Software Engineers, ML Engineers, Data Engineers, etc.) and work with and around another order of magnitude's worth. I've been even luckier that the majority of my team members have been fantastic developers and great people, every one with a unique set of strengths, weaknesses, preferences, and quirks. Some I've hired, others I inherited, but I've learned something from every single one. I have observed strong correlations between certain skillsets and traits and success across different industries and company cultures.

These are the lessons I've learned so far.

Lesson One: You do not want any brilliant assholes on your team. Ever.

If I'd led with "You don't want any assholes on your team", everyone would be like yeah, of course, nobody wants that, that seems obvious. However, the romanticized notion of a brilliant asshole has lingered in our cultural consciousness for far too long, and not just in the tech industry.

"Yeah, he's difficult to work with, but he really knows his stuff and gets things done." (These are usually a he; difficult brilliant women and nonbinary people tend not to get romanticized in the same way, the reasons for which are far outside the scope of this post.)

There are two problems with brilliant assholes. First, they are often not actually brilliant! They may have wowed people with initial success or bullied people into believing in their intellectual superiority, and their demeanor is designed to beat the masses away from the pedestal they've climbed onto so nobody realizes they aren't better than anyone else. This variety is going to spend a lot of time and energy maintaining their status at the expense of their colleagues, rather than doing the important but less visible work their role likely requires.

Second, even if they are brilliant (which is exceedingly rare), the negative effect they have on their team, and likely you as their manager, far exceeds any positives. Maybe they produce twice as much as your other devs (I don't believe in the 10x engineer except under very specific circumstances), but I guarantee they make the rest of your team less productive and more likely to leave the company, in addition to making you tend to the shattered trail of confidence and goodwill behind them.

The good news is that you can usually tell pretty quickly when you're dealing with someone like this – they'll never admit their mistakes, they are inflexible about methods and standards, and they love to tell you about all the other people's messes they've had to clean up because they're the only one who can. You should spot this within the first 15 minutes of a good behavioral interview – you are doing behavioral interviews, right?

Don't hire these people no matter their track record, and if you're considering a new job and can tell you'll inherit one of these, think long and hard about whether it's worth it (or you can find them a new spot outside your team).

Lesson Two: Curiosity, flexibility, and ability to learn are more important than raw technical skill.

Your behavioral interview (seriously, please tell me you're doing behavioral interviews) should include questions about when things didn't work out as expected, or when they had to implement something in a suboptimal way, because that's going to be the reality most of the time at most companies. Technical constraints, time constraints, and dependencies are all going to conspire against your grand plans for how to do something "the right way". What you don't want is a dev who's going to plow straight ahead, ignoring all these signs, and deliver something two months late that doesn't work, but totally would if all of these other things worked the way they're supposed to.

I think most of us would love to have a job where we can spend months experimenting with every new cutting-edge model and then fine-tuning the champion until it's so sharp it'll slice your way right into a keynote spot at a great conference, or at least a blog post that more than 20 people read, but be honest, when was the last time you had that freedom? I'm going to go out on a limb and say, outside of very unusual circumstances or grad school, never.

Your business problem is never just a data science problem. It exists within a web of requirements, stakeholders, dependencies, and KPIs – and it should, because the most beautifully designed algorithm is worth nothing if there's no business case, no way to deploy or connect it to other business functions, or your stakeholders can't use it the way they want to. The exchange rate for style points to dollars is not favorable.

Ideally, your devs will grok that these constraints are not external, they are as much a part of the problem you're trying to solve as the algorithm itself, but you can settle for devs who will take your word for it that these constraints are important and work within them. Most of them will eventually come to understand that the shape of the problem is a lot bigger than they might initially think.

Lesson Three: Be willing to compromise on technical ability, but not on your team culture.

I have always said that I'll take a room full of B+ programmers who are flexible, work well with others, and communicate well over a room full of A+ programmers whose so-called soft skills are unknown. (Side quest: can we come up with a less dismissive term than "soft skills"?)

That team of B+ programmers is going to outperform the room of A+ programmers right out of the gate, because technical ability is just one part of the puzzle. Notice I didn't say bad programmers; there's certainly a technical floor below which you shouldn't consider a candidate, but once that floor is met I'm much more concerned about these other skills and the ability of the group to work well with each other and with other teams.

I'd wager that a lot of those B+ programmers turn into A to A+ programmers as they thrive in a great working environment anyway, as long as you have clearly defined standards and expectations and provide learning opportunities.

Lesson Four: It doesn't matter if they know the tech stack.

Slight exception here if you're hiring for a short-term project and you need someone who can hit the ground running, but if you're building a team that you intend to keep together for any length of time, hiring for experience with specific technology is a trap I see so many companies and hiring managers fall into.

The reality is that you have no idea what technology you're going to be using a year or two from now, and you might be wrong about what you need right now. A good developer with a grasp of the important concepts in the field will learn whatever is needed, and will likely bring alternatives to the table that you weren't aware of.

Now, I stated this lesson a bit provocatively – a more accurate title would have been "It doesn't matter if they know the tech stack, as long as they are solid on the basics, good at learning, and willing to be flexible; unless you don't have time to get anyone up to speed and you don't care about future team needs", but that's a bit wordy.

As a concrete example, my current team heavily leverages PySpark. I put PySpark in my job description as a nice to have but not as a requirement, because do you know how hard it is to find a data scientist with PySpark experience? It shrinks your search space so severely that you might not be able to fill that role unless you're paying incredibly well and in a desirable location (or work remotely). Instead, I look for people who understand the concepts behind distributed computing and have a reasonable enough grasp of Python that they'll figure it out. And they all have. Quite well, in fact. I also know that if we switch to a new framework, they'll figure that one out too.

Lesson Five: Data Scientists do not exist in a vacuum.

I had more success building teams when I reoriented from looking for the best data scientists to looking for the best data scientists for my team and roadmap and overall company context. People are not fungible; you cannot swap two people into each other's roles and expect it to make no difference.

The natural extension of this philosophy that is somehow less obvious is that someone can be an excellent data scientist in one context and a terrible one in another. I'll take myself as an example. I think I do best in an environment where I can spend my time putting every one of my team members in the right positions to succeed, developing their careers, talking to stakeholders, focusing on architecture, designing ambitious roadmaps with clear outcomes and a balance of risks, and developing and deploying our products with some degree of autonomy and accountability. If you dropped me into a different role where I was expected to execute an existing waterfall project schedule, never take risks, not spend time developing my team, and deliver hands-on-keyboard code contributions, I can guarantee you'll find a lot of other people who would be way better than me in that context.

That probably sounds like I'm valuing certain ways of working above others, and I am – because that's the type of person I work well with (and type of company I work well in), and I shouldn't be building a team that I'm not going to work well with and isn't going to succeed with me at the helm. You should value your preferred ways of working over mine.

It also means that I'm not looking to hire one profile of person for every role on my team. For example, I value team members who are willing to tell me when I'm being an idiot, but things will be fine as long as I have a few brave souls on the team. I need a mix of visionaries and people who just want to be handed a problem and get their heads down working on it.

When I'm looking for a new team member, it's not about who is the best in a vacuum. It's about what makes my team, in that moment, in that context, the best it can be, and who's going to thrive in that role.

Lesson Six: Don't be afraid you're training your replacement.

I've been given advice more than once not to hire, promote someone, or give them visibility because it might threaten my own position. I could not disagree with this POV more. My goal is not to entrench myself in a role, build a moat around me, and defend my turf. My goal is to give the team and the company what they need to thrive – that literally what I am being paid for. If that means I've trained up my successor and they are ready to step into what I do as well as I have done it (or hopefully better!) then I view that as a successful conclusion to that chapter of my career. About as clean a closure as one can get. Ideally there's a new challenge waiting for me at that time, but if there's not, I'll just have to go find the next one elsewhere.

Endings are good. They clear the way for new opportunities, even if the ending wasn't on your terms. I have a tattoo inspired by the Death tarot card to remind myself of this.

Lesson Seven: What works for me may not work for you, and vice versa.

None of this is prescriptive. Except for the part about not wanting assholes on your team. I can't think of any context in which that's a net positive but you do you. Humans are pattern recognition machines, and these are the patterns I perceive in my work history. I think as long as you're thinking about this stuff, you're going to get most of it right. And if you get most of it right, you're going to do just fine. You need to make mistakes along the way; if you aren't, you probably aren't stretching yourself enough to learn and grow as much as you could be.

As I alluded to above, I'd love to hear about where you think I'm being an idiot, and what you think I left out. We all do better when we share our experiences and learn from each other – more data points for the pattern.