How Should We Handle Failure?
In tech, there's a lot of pressure to keep things running smoothly. That makes even a small mistake or a brief outage seem disastrous. When people fail at something, they can experience many different emotions: Anger, regret, or even fear. Compiler speaks with guests from enterprise IT and beyond about how we should handle failure, and how teams and individuals can benefit from processing it safely and effectively.
00:00 - Kim Huang
Hey, this is Kim. I'm one of the producers for Compiler. Before I bring in Brent and Angela, I wanted to talk to you about something a little personal. It's about failure; specifically, my biggest failure.
00:21 - Kim Huang
I've worked in digital media for a long time, so I decided to take a course in user experience (or UX). It was towards a certification, and one of the classes was Introduction to Python. Now, this class gave me quite a bit of trouble. And when I say that, I'm being pretty generous because I didn't understand a single thing that was said to me. I couldn't get the simplest commands to work. I couldn't get anything to work, and ultimately, I failed the class. And I don't fail things, not often. Fast forward to now, I've been at Red Hat for, well, more than two years now. It's full of smart people, people who do code every day, and I couldn't even get a Hello World command to work. And while I'm good at my job, I always come back to that Python class. How do I move forward from that shortcoming? How should I handle that type of failure?
01:32 - Angela Andrews
This is Compiler, an original podcast from Red Hat.
01:36 - Brent Simoneaux
We're your hosts.
01:37 - Angela Andrews
I'm Angela Andrews.
01:39 - Brent Simoneaux
And I'm Brent Simoneaux.
01:40 - Angela Andrews
We're here to break down questions from the tech industry: big, small, and sometimes strange.
01:46 - Brent Simoneaux
Each episode, we get perspectives on where things are and where they're headed.
01:51 - Angela Andrews
Today's question: How should we handle failure?
01:55 - Brent Simoneaux
Producer Kim Huang is here to get things started.
02:03 - Kim Huang
Angela, Brent, I want to talk today about failure. We're going to hear from a few people who are going to help us gauge what failure really means for people, personally and also what it means for teams when they work together. Before we get into that though, I want to know, what do you think about failure?
02:25 - Angela Andrews
Oh, well, failure happens to all of us. It's a part of life, and I've had my own personal failures. When I think back to college, I failed calculus, and that means I had to change my major. That was a personal failure for myself. I really don't know where my career would have gone had I stuck with my computer science major, but I wonder.
02:51 - Brent Simoneaux
I wonder if failure stings more when the thing you're failing at, is part of your identity or part of the person that you imagined you want to be.
03:06 - Angela Andrews
03:07 - Kim Huang
I think that that's very accurate. I read a lot of research going into the story, and what you said, Brent, reminded me of some studies that I read from this doctor from Northwestern University. His name is Dashun Wang, and he does research into success and failure. He talks about success being like a collective phenomena. And that means, in order for you to be successful, everyone has to agree, not just you. Everyone else around you has to agree that you were successful in whatever you were doing.
03:42 - Brent Simoneaux
03:43 - Angela Andrews
I never thought of it that way. I mean, collectively, what people think of you is what you are, to some degree?
03:52 - Brent Simoneaux
03:52 - Angela Andrews
If everyone thinks you're successful, then you are successful? And if everyone thinks you're a failure, they think you are a failure? That's a weird dynamic to even consider, putting that much weight. Look at me saying this out loud, putting that much weight on what other people think of you as a way to define yourself. But that's interesting. What else do you have for us? I'm very curious. You have my full attention.
04:19 - Brent Simoneaux
Yeah. Let's dig into this, Kim. So who did you talk to first?
04:24 - Kim Huang
Well, I wanted to know how people look at failure, so I spoke with someone we know pretty well. She was a Red Hatter, and she did a lot of work actually in early seasons of Command Line Heroes, dealing specifically with failure. She told me the way that people deal with failure personally translates into how they process failure on teams and how we all treat each other in the process.
04:55 - Jen Krieger
I'm Jen Krieger, and currently, I'm working at GitHub as the senior director of the Technical Program Management team, and I am just super excited to be here.
05:04 - Kim Huang
She had a lot to say about how failure was framed in her working life.
05:08 - Jen Krieger
Failure was the focus. Failure was the thing that you led from in every situation. In fact, if I even look earlier in my career, when I think about that word, every single framework, technique, all that stuff that I learned, was mainly in place to prevent teams from failing. And so there was never really a conversation so much about why we were so focused on the negative part and less on the outcome of failure, which was to learn from our mistakes. And it seems like the industry still hasn't quite gotten there yet. We still focus on this concept of failure and safety and making it safe to fail. When I actually prefer to think about it a lot differently, which is making it safe to learn, which is a way different and more positive way of looking at that word.
06:02 - Brent Simoneaux
I have a lot of things going through my head right now.
06:04 - Kim Huang
06:07 - Brent Simoneaux
Making it safe to fail is something that I hear a lot. And I have actually never heard this phrase, making it safe to learn.
06:16 - Angela Andrews
Yeah. I've never heard that phrase used before. It's always, it's safe to fail, or fail fast.
06:23 - Brent Simoneaux
06:23 - Kim Huang
Fail fast, yes.
06:26 - Angela Andrews
That's the phrase we always hear, but the safety, placing it on the positive, that's a different twist on it.
06:36 - Kim Huang
Jen sees a lot of development teams go through really hard challenges. Many of them, very inherent in the constant cycles of production. For these teams, failure denotes chaos and fear.
06:51 - Jen Krieger
I keep looking at this when I think about software development. And I think about the fact that early in my career, folks were so incredibly afraid of a production outage. That was the thing that you would get paged about, that would basically destroy your weekend, your life for a period of time.
07:11 - Kim Huang
So, you have all these forces. You have failure, you have fear, you have personal narratives, you have team failure and they’re all kind of like, to me, they're kind of like swirling around in a whirlpool of negativity. Is it just me?
07:31 - Angela Andrews
No, it's not just you. What you said had me thinking. I mean, it happens sometimes. Sometimes there's an outage, and we hear about it, talk about folks getting fired and heads will roll. And I think that's knee-jerk because someone needs someone to blame. Right? And when we think about what Jen said, we're making the team feel safe to learn. I'm sure in that postmortem, people looked at what happened, and they said, they figured out what the failure was, and they worked hard to remediate it, to make sure that it never happens again.
08:04 - Kim Huang
This is why she's focused on creating a safe environment for those teams to fail and learn at the same time.
08:11 - Jen Krieger
When I say, make it safe to learn, that's really what I mean. It's not just about, "We have deployed code to production and production is down so, therefore, we are bad because we have failed." It's more, "How can we actually deploy code to production in a way that will help us be more resilient to the failures we've experienced in the past?" That's what I mean, making it safe to learn.
08:31 - Kim Huang
Because the world is a very, very different place than it was even a few years ago, Jen says, safety– particularly psychological safety–will play an important role in failure as newer technology develops.
08:50 - Jen Krieger
People cannot learn these new technologies fast enough. They cannot adapt to the things that they're seeing grow out of these new technologies. At the end of the day, we don't always know the answers to the questions we're being asked to solve right now. And so, if we are constantly focused on this idea that failure is the thing that we need to make better for folks, we need to make it safe to fail. We need to make sure that people understand that they bring production down, they're not going to get fired. That's probably a good idea to do, but at the same time, companies that are just simply setting that bar there, that's not enough. They have to go even further than that. They have to make space for people to learn from those mistakes that they've made, and I just don't see the industry actually doing that.
09:39 - Brent Simoneaux
Oh, that is really interesting. And that's something that's... I don't know what the word is, maybe it's bothers or intrigued me about this phrase "failing fast" or making it safe to fail, the focus on failure. It's not the failing that we're striving toward. Right?
09:58 - Angela Andrews
09:59 - Kim Huang
10:00 - Brent Simoneaux
It's the learning that comes from the failing that we're striving toward. And I think that's what Jen is saying, right?
10:07 - Kim Huang
Yes. You two may recall there is a very famous story about Thomas Edison. And Jen, I think at some point, does mention in our discussion about Thomas Edison going through different iterations of the light bulb, and someone pointed out to him that he had failed a thousand times, but instead of saying that, he reframed it and said, "No, I just found a thousand ways that didn't work, and one that did."
10:35 - Brent Simoneaux
10:37 - Angela Andrews
That sounds very familiar. And when we hear that word failure, when we reframe it, when those things happen, because bad things are always going to happen, like she said, you need that psychological safety because that's where the growth happens, that's where the learning really happens in those moments. Because if nothing happens, where is the growth? What are you learning from exactly? Right?
11:02 - Kim Huang
11:05 - Brent Simoneaux
I think the other thing is that failure sometimes does have real world effects. You know?
11:13 - Kim Huang
11:13 - Brent Simoneaux
It actually does… like outages and things like that.
11:17 - Kim Huang
11:18 - Brent Simoneaux
So I'm trying to tease out the scale of failure and the effect of failure, and how you create safety and learning at all those different scales.
11:31 - Kim Huang
That's interesting because, ultimately, companies want to provide their services and their products to customers in a timely manner because they're all competing with each other for attention, for a market share. So, I think this is why Jen is very skeptical as to whether or not companies will make failure safer. Let's hear from her.
11:55 - Jen Krieger
There's oftentimes not the organizational muscle to do anything about what they are meant to learn, so it's not just learning but also acting upon those learnings. I don't see organizations giving people the emotional space to actually be able to digest what just happened. People are not given permission to feel disappointment.
12:17 - Kim Huang
For Jen, I think it's a really simple equation: a team can totally fail one day and then bounce back and put out an amazing product or do something really amazing.
12:30 - Jen Krieger
Yeah, we have technical failures. But the thing that I have seen from technical failures is, the technical teams that I've worked with over these years have always recovered. They've always found a way to fix the problem that they're having.
12:43 - Kim Huang
But one thing that has not come back easily: trust.
12:47 - Brent Simoneaux
12:47 - Jen Krieger
What has not always happened out of these events has been, the relationship damage has been fixed, the teams have figured out how to work together moving forward. Maybe a finger pointing event that might have been visible to more than what would be an acceptable limit of people, no one ever apologizes for those things, and so a lot of that is embedded in your ego and the way that you perceive others. And also what I see mainly today, is the reluctance to actually get on the phone with somebody and talk through something. We are so reliant on texting and text-based form of communication. People are really reluctant to have a hard conversation with each other, and that's got to change.
13:31 - Kim Huang
Sometimes, in workrooms, there is time for apologies. There's no time for hard conversations when you have deployments going out one after the other. But ultimately, how people process failure as individuals does matter to a team, especially in a fast-paced environment. If there's no time to be retrospective, how are people supposed to process what happened?
14:01 - Kim Huang
In tech, we focus a lot on team dynamics and team outlooks and visions and vision statements and goals. But ultimately, teams are made up of people, and how people process failure as individuals does matter to a team, especially in a fast-paced environment when you’ve got one thing going out after the other, so many different deployments, and there's no time to react or be retrospective.
14:31 - Brent Simoneaux
This is part of the culture of the team, and I think it's part of how the team interacts with each other. I've actually worked with Jen in the past, and I've actually done some of her exercises with my team itself where she sort of holds space for everyone to digest what has happened, all the good stuff, all the bad stuff, the stuff that didn't go so well. And I love what you're saying there, Kim, because I don't think we hold enough of that space for ourselves.
15:06 - Kim Huang
Okay. So, we just talked a lot about perfection and failure, and I was curious about whether or not failure had anything to do with a lack of knowledge or skill or passion.
15:19 - Brent Simoneaux
15:19 - Kim Huang
But what I ended up learning from this next person that we have on the show, was a little bit more insight into how important it is to fail.
15:33 - Dr. Erika Hamden
Hello. My name is Erika Hamden. I am a professor of astrophysics at the University of Arizona. And I spend most of my time building telescopes, which I have decided is the best job that somebody can possibly have.
15:47 - Brent Simoneaux
This is a new one for us. I don't think we had an astrophysicist on the show.
15:50 - Angela Andrews
15:53 - Kim Huang
No, we have not. And I wanted to ask if either of you had ever heard of Dr. Hamden.
15:56 - Angela Andrews
No, I have not.
15:57 - Brent Simoneaux
I have not, no.
15:58 - Kim Huang
Okay. A few years ago, Dr. Hamden gave this amazing TED talk called What it Takes to Launch a Telescope.
16:07 - Brent Simoneaux
16:08 - Kim Huang
The video has been watched online millions of times. If you speak with her, you'll learn that she's no stranger to failure.
16:16 - Dr. Erika Hamden
One of the projects I work on is called FIREBall. That's what I gave my TED talk about. And it's a balloon telescope, which is kind of a weird, interesting thing. It's a telescope that goes on a high altitude balloon, not on the ground or in space. And I've been working on it since 2008, so it's getting to be almost 15 years now.
16:36 - Kim Huang
Get ready for this ride. So, FIREBall has turned out to be a project plagued with many ups and downs (mostly downs). It has been marked with multiple failures, technical snafus, mechanical failures. There was even a bird that flew into the balloon one time. The balloon had a hole in it one time, and it crashed. There were so many different failures. And in her TED talk, Dr. Hamden talks at length about these repeated failures. And she says, success as a sure thing is one of the most crucial things to reject when you're working on a project.
17:19 - Brent Simoneaux
17:20 - Dr. Erika Hamden
You have to be prepared for that eventuality of like, your experiment worked correctly, you did everything right, but you were wrong in your initial question, because that's always a real distinct possibility.
17:32 - Kim Huang
Earlier in the episode, we talked about failure and success being considered collective phenomena. That means, if something is to be successful, then other people have to observe and agree. When you're talking about something as complex as the scientific process, that can come off as a little bit oversimplified. Dr. Hamden says, the successes of science can sometimes hide the many, many, many setbacks that people face.
18:07 - Dr. Erika Hamden
I mean, that process is a necessary part of getting to the thing that does work. So, I feel like if you're only reporting on the very final check, when all of the hard work has already been done, then you're really missing a key part of the story which is that, this is just a continuous process of messing up. That's a little bit of a public communications challenge because you don't want people to think that the mess ups mean that you're not reliable, the mess ups are actually a good thing. And reporting on them is a good thing too. That's like the transparency that–in an ideal process, that's how science should operate, where you can see what worked and what didn't for everybody's projects.
18:49 - Angela Andrews
We're always focused on that end, that win at the end.
18:54 - Kim Huang
18:55 - Angela Andrews
This is funny, but it reminds me a lot of the Michael Jordan, Kobe Bryant quotes. In my gym, we have all these Kobe quotes in there.
19:04 - Kim Huang
19:04 - Angela Andrews
And this is something that I see on a constant basis. So, they attribute "failure" to their success because without it, they wouldn't be Jordan, they wouldn't be Kobe. Again, people don't look at that, but that's where all that good stuff happens. That's how you become great. That's how the science gets notoriety. Anything. Anything good comes, there had to be some setbacks and failures in there to get you there. That's what I'm hearing.
19:38 - Kim Huang
Dr. Hamden juxtaposes failure with effort, which may be a good way to change the negative inner voice that pipes up when things don't go as planned.
19:51 - Dr. Erika Hamden
Everybody deals with failures of all sorts of kinds. And the idea that you should be ashamed of it or that you should keep it a secret, I think, is really discouraging. Because in my view, trying and failing, the important part of that is that you tried something, and that should be celebrated. You tried it, that's scary and good. And instead, people feel more shame about the failure part. And I think that just shuts off a conversation that's really important to have.
20:22 - Kim Huang
With a very unhealthy approach to failure, a person could take that and turn it into a personal narrative. But what she has done, she's framing failure in a more collaborative way. She is talking about having a positive mindset. She's not thinking about success as a binary.
20:41 - Brent Simoneaux
It's interesting because we tell stories about ourselves, and teams tell stories about themselves.
20:48 - Kim Huang
20:49 - Brent Simoneaux
But sometimes those narratives and those stories can take hold in your mind. Right?
20:53 - Kim Huang
20:54 - Brent Simoneaux
Which is I think what you're talking about, like framing, what just happened? How are you talking about that? And is that being lodged in your mind? Is that being lodged in the company's mind? Is that being lodged in your boss's mind? What is that story?
21:13 - Kim Huang
Exactly. So Dr. Hamden has thoughts on teams too, and how someone could frame failure in a much healthier, more collaborative way.
21:24 - Dr. Erika Hamden
I think it's partly just in a mindset and how you kind of talk about it with the people around you. I think that, especially in dealing with teams and working on these big projects, it's really important that blame not necessarily be part of the kind of analysis of what went wrong, that blaming people for things doesn't make them do better. It just makes them feel bad about it.
21:48 - Angela Andrews
21:49 - Kim Huang
21:51 - Brent Simoneaux
21:52 - Kim Huang
Are you trying to help people learn or are you trying to make them feel bad?
21:57 - Angela Andrews
Yes. I've been in postmortems where you kind of feel that finger going around the room, and you just don't want it to land on you. That is so much stress.
22:10 - Brent Simoneaux
I'm imagining everyone around that table, just sort of slumping down in their seats, trying to avoid eye contact.
22:16 - Angela Andrews
Yeah. That is so true.
22:19 - Kim Huang
And when you're in that kind of hot seat, you start building that personal narrative again. Am I going to suffer any negative consequences from this? Am I going to be trusted if I'm a lead? Am I going to be trusted with leading another project?
22:32 - Brent Simoneaux
22:33 - Kim Huang
Am I going to be trusted with my job? Am I going to lose my job? It's like an avalanche, kind of.
22:41 - Brent Simoneaux
22:45 - Kim Huang
So, there is a binary we've established, success and failure. That's typically what this looks like.
22:51 - Brent Simoneaux
22:52 - Kim Huang
Dr. Hamden has a different way of thinking about that binary.
22:56 - Dr. Erika Hamden
I think that the opposite of failure is inaction. So I view failure as a positive thing, that you tried something. And in my experience, most of the time when you try something, it doesn't work the first time, and that's fine. So, to me, I associate failure very strongly with just trying or the act of giving something a shot. So the opposite of that to me is not doing anything, it's to not take the leap.
23:24 - Kim Huang
So I asked Dr. Hamden about how our jobs–in STEM, outside of STEM–how are societies define success, and how reality is much different?
23:38 - Dr. Erika Hamden
I think it's usually defined as anything other than complete perfection in whatever the goal is. It's like this all encompassing thing that there's one way to get success and there's a million ways to fail, and all of the million ways are bad. And I think that that is problematic and that the million ways to fail, there's probably a few of them that are bad. You failed because you didn't try. You failed because you sabotage this thing. Those are not ideal, but there's a lot of things that like, you failed because you're working with incomplete information. You failed because you're trying something new for the first time that you don't really understand, and now you understand it a little better. You failed because you are the first person in the entire history of the world who is doing something. All of those failures are really helpful. They're super valuable. I actually think that a string of failures is necessary for success.
24:32 - Angela Andrews
So, this sounds a lot like the Thomas Edison analogy story that you told us earlier where that's where the discovery happened. He found a thousand ways to do it the wrong way to get to the right way. That's an amazing way of thinking about failure. It really turns it on its head, and it takes that sting away from it. Listening to this episode, it was actually a little emotional where I'm trying to internalize the amazing things that people are saying here.
25:07 - Brent Simoneaux
25:07 - Angela Andrews
Because sometimes, people can be very hard on themselves. So, can failure lead to discovery? One would hope, because as long as you don't give up and you keep trying and you do better and you iterate, the process gets better. You get better, you learn more. So when you say it like that, that's where the good stuff happens.
25:33 - Kim Huang
25:34 - Brent Simoneaux
I think we're hard on ourselves.
25:36 - Angela Andrews
25:37 - Brent Simoneaux
And I think we're hard on other people sometimes too.
25:40 - Kim Huang
25:42 - Brent Simoneaux
And I think one of the big things that I'm taking away from this, is the spaces that we create for ourselves, but also for our teams and also for people that we work with every single day, how do we create that space where learning can occur?
26:09 - Kim Huang
The way that we process failure needs space, first and foremost. I think that's what Jen was getting at. You need that space to process what happened. I think that has an impact on how teams work together because teams are just people, gathered together for a shared purpose or vision. That can also affect how individuals themselves learn from failure. If they feel like they're not good enough, if they don't have time and space to process failure, that all could come together in a way where they look at their work in a negative light. And while looking back is good, you can't keep looking back. We all have to move forward from our missteps. We have to not make our professional failures into stories about who we are as people.
27:09 - Kim Huang
In our working lives, we could use failure to discover things that we didn't know before, learn new things, go new places with our talent, push ourselves beyond what we thought was possible. We can also lift ourselves up and the other people around us as well whenever we run into a difficult situation. I think that's the most beautiful part about what I learned. Failing that Python class doesn't say anything about me personally, and it doesn't help me at all to create that story about myself. It just means I was doing something new. And when you're trying something new, and you fail, you can always try again.
27:55 - Angela Andrews
And that does it for this episode of Compiler.
28:01 - Brent Simoneaux
Today's episode was produced by Kim Huang and Caroline Creaghead. Victoria Lawton always sticks with us no matter how many times we mess up.
28:11 - Angela Andrews
Our audio engineer is Elisabeth Hart. Special thanks to Shawn Cole. Our theme song was composed by Mary Ancheta.
28:20 - Brent Simoneaux
Big thank you to Dr. Erika Hamden and Jen Krieger for sharing their perspectives with us.
28:26 - Angela Andrews
Our audio team includes Leigh Day, Laura Barnes, Claire Allison, Nick Burns, Aaron Williamson, Karen King, Boo Boo Howse, Rachel Ertel, Mike Compton, Ocean Matthews, and Laura Walters.
28:41 - Brent Simoneaux
If you liked the show, we would love for you to come back next time, so subscribe, leave us a review. It really does help us out.
28:50 - Angela Andrews
We'll see you soon. Thanks for listening.
28:52 - Brent Simoneaux
All right. See you next time.
Dr. Erika Hamden