Red Hat SummitRed Hat Blog
This is the fourth post in our series on open source game development. For posts one, two, and three, I sat down with Red Hat’s Michael Clayton and Jared Sprague to talk about how the second season of Command Line Heroes relates to their work on Command Line Heroes: The Game, and their own efforts as open source game developers.
In today’s post, we’re talking about failure. In episode 4, “Fail Better,” we learn how Google has systematically learned to embrace (each and every) failure as an opportunity to learn, grow, and prevent classes of similar problems from happening again. We also learn how one of the most popular video game franchises of all time may not have been so successful had the developers been successful in their first attempt at an algorithm for some rather prominent non-player characters (NPCs).
Whether we like it or not, some amount of failure is inevitable. To this end, I started this week’s discussion with Jared and Michael by asking about how an early failure may have led to a different—or even a beneficial—outcome.
Quick note: If you want to jump straight into the code or find out how to contribute to Command Line Heroes: The Game, visit our GitHub repository.
Failing game jams = better games
Dan: Today’s episode of Command Line Heroes, “Fail Better,” is all about embracing failure—or, unsurprisingly—failing better. In a sense, it's about looking beyond failure itself. Saron talks to some really well-informed subject matter experts including Google’s Jennifer Petoff, GitHub’s Jessica Rudder, and Red Hat’s own Jen Krieger. In listening to the episode, I started to wonder if, for each of you, there was a moment in time when you remember learning from failure?
Michael: The first thing that comes to mind isn't a single epic failure, but just the fact that when you're programming anything—most of the time, really—it's not working. It’s almost a constant state of “failure.” In those rare cases where you sit down and write a bunch of code and it ”just works” on the first try, those are really notable because they're such an exception to the rule.
Dan: Jared, how about yourself?
Jared:Yeah, so, one thing that comes to mind is the first game jam that Michael and I participated in. Specifically, we didn't address sound until the very end, and in turn, we received really low scores for audio. We quickly learned from this mistake and two game jams later had recruited a third team member to help us with audio. The result? Out of 2000+ entries, we ranked 14th in the Audio category for our game Pity About Earth.
Dan: Oh wow. An easy fix, I suppose!
Jared: Honestly, game jams are great for learning from failure, because you get tons of feedback right away. Post jam, you’re able to quickly see where your deficiencies are and this helps you know where to apply your focus for the next one.
Dan: Ha. That's awesome, too. It’s like you guys have already listened to the episode. And I say this because in the episode, when Saron is speaking with Jen Krieger, Jen recommends that you “break your build as early as possible” and use feedback loops to efficiently correct issues. It would seem that the very nature or structure of game jams build feedback loops into “the process,” thereby helping everyone learn and grow.
Jared: Yes, totally. And, what’s great is that in my own experience with game jams, it's almost always constructive feedback, too.
Michael: Yeah.
Jared: Most of the time.
Failing on the job = better work product
Dan: Switching gears slightly, let me ask you, in your day-to-day work experience, as members of the Red Hat Customer Portal team how is failure embraced (or not)?
Jared: So, that's kind of what I do in that a lot of what I do is related to monitoring. It's all about catching failures and outages and learning from them when they do happen.
Michael: Just understanding that failure of various types is always going to happen is important. And it's what you do with that—like catching an outage before customers find it and report it and learn from the failure with retrospectives—all of this happens on our team.
Dan: Yes, retrospectives. So Michael, in “Fail Better,” Google’s Jennifer Petoff relates how failure can lead to a whole lot of learning, but only if it’s blameless and viewed, culturally, as a learning opportunity.
Michael: Yeah, on our team there's always a root cause analysis after an outage and explanations are shared with anyone and everyone who wants to know and learn from what happened. One thing I think we do really well on this team (and this hasn't always been the case at other places I've worked) is embracing the blameless nature that you just mentioned.
Dan: The blameless thing, it doesn’t seem simple though. Like, in practice, I could imagine that it’s usually clear who may have dropped the ball, right?
Michael: Sure. It is hard to avoid blame because as you're digging for a root cause. It usually becomes clear whose fault it was… but the important thing is to not put more shame on that person than they're already feeling, and to understand we're all on a team together, and we're all trying to support each other and support the systems that we work on. It's important to figure out what happened and what we can do to prevent it from happening again in the future, and the person who was the responsible party is critical in allowing / enabling that to happen.
Failing with art = impossible
Dan: I'm curious, as you guys have developed a number of open source games, have there been moments where you were heading in a certain direction with development, but through either play testing (and feedback), or perhaps via a simple code blunder, you’ve discovered a better or more lucrative path forward?
Michael: In graphics programming, “code blunders” (as you put it) happen all the time. Unlike most fields of programming, when you make a mistake writing a pixel shader or some other visual code (this is true of physics code as well) the result usually isn’t a crash. It’s often something really baffling or hilarious or just plain fascinating.
Dan: Totally makes sense. You have an idea for what you want visually. And since you’re likely not mucking around with parts of the code that would affect a successful compile, the end result is sometimes surprising, in a good way.
Michael: It seems to be a shared experience (i.e., it’s not just me) because I see people posting stuff on social media all the time saying things like “...wow, I didn't expect that to happen” when, for example, a texture mapping equation is wrong and the texture gets really stretched out. In fact, if anyone is interested, I actually started a short video series called “shader mistakes” showing the mishaps while I was learning the OpenGL shading language (GLSL).
Dan: This reminds me a lot of being an artist or a musician. It's not so much that the code doesn't compile or that there's no result. There's actually always some results here and I could imagine that sometimes you might do something by accident or through failure that might be straight up incredible.
Michael: Yeah, totally.
Jared: I just thought of another example of what Michael is describing. In one of our games, Zorbio, where you're a sphere flying around in three dimensions, I was trying to figure out how to get trails attached to the spheres, to help see what direction you were flying in and what direction other people were flying in. As a part of this exercise, I made the trails really, really long—like - super, super long. So, you're flying around leaving this infinite trail behind you.
Michael: Like in Tron!
Jared: Yes! It was like a light cycle, only in three-dimensions. What it ended up doing was making this really cool curvy mesh in space that you could fly around and fly through where you had been before. And while I wasn’t aiming for this end result, I thought to myself: “This looks so cool... this could be a game all on its own and it would be so much fun.”
Lo and behold, this concept actually became the idea for our next game jam game, which (later) ended up becoming a game we called Tracer. It takes this exact idea (where you fly around and you leave a trail), but then you try to hit these points in space and if you hit all the points it makes a three-dimensional model; it’s basically like a three-dimensional connect the dots. Then you fly off (into the distance), and you can see what you created. Whether you call this failure or a weird accident or perhaps an unanticipated thing, it was fun to see how the experiment evolved into a completely different game.
Dan: My takeaway here is that when you’ve bridged into art, failure isn’t the right word. Speaking of art, maybe it’s more akin to what Bob Ross used to say: “...we don't make mistakes, just happy little accidents.”
Jared: Sound about right to me.
That’s all folks
Dan: Thank you both once again for sitting down to discuss how this latest episode of Command Line Heroes relates to both your day jobs and open source game development.
Speaking of open source game development, for anyone who’s been “keeping score,” today is the third and final day of All Things Open 2018. If you’re in the Raleigh area, do stop by. Michael, Jared, and I will all be in attendance. While I can’t speak for them I’m 99% sure that Michael and Jared would be more than happy to tell you more about how Open Jam 2018 shaped up and to give you a sense for where we’re at with Command Line Heroes: The Game.
And, if you’re not so much a gamer, but are interested in our podcast, I’d love to hear what you’re thinking. We’re always looking for new ideas and new territories to explore.
Until next time, thanks for joining us.
Useful links
