It's obvious why bad innovation teams fail. But great ones? Why do even they fail so often too?
My work across many companies and industries--plus the work I did in-house at corporations myself--comes back to this obnoxious and persistent question way too often.
Even beginning to solve it has taken hard work across many years. Here I share what I have learned that works in the real world.
So much has to go right in innovation. How can you ever make sure you get it all right?
A big field like innovation comes with buckets-full of success factors. Gobs of them! Enough to cause you analysis paralysis.
Even if you manage to break through that paralysis, it’ll take you hours to discuss (let alone implement) all those success factors.
I’ve tried it once, based on the insights of just a small subset of experts on the topic—Clayton Christensen, Melissa Perri, Rita Gunther McGrath, Roger Martin, Steve Blank, and Sydney Finkelstein. And even that was a major effort.
And don’t get me started on all the other sources of success factors at your disposal: Practitioner surveys, case studies, articles, books, your own experience, and on and on.
It’s a maddening game of whack-a-mole too. almost impossible to follow all the advice. And beyond that, most of it amounts to "point solutions" that only tackle part of the problem. No single fix realistically guarantees success. You need to make most or all factors work. And even then, there always seems to be one more problem.
But you can succeed anyway.
Don't stress about tactical advice. Look at innovation success as a system-level issue
Stare at all those factors that make or break teams, and patterns emerge; forces; dynamics. It’s like those “Magic Eye” 3D images that suddenly appear out of jumbled-seeming pictures:
Beyond all the individual factors, innovation success is a system-level issue, specifically a human one.
By itself, that doesn’t solve everything. We also need to know what kind of system. We need the common headline, the theme that unifies all the research findings and details into a single system.
One obvious option (among several plausible ones) simply focuses on the one thing that all innovation teams have in common. This obvious focus also has the advantage that the best innovation teams live this truth the most. So if we want to answer (and fix) why good innovation teams fail, this may be as good a start as any:
That rock-bottom commonality of innovation work? As people paid to be innovators, we cause change--system-level change.
Innovation means causing system change
Mind you, when we say that at root, “innovation causes change,” that doesn’t take away from all the different flavors of our work, from disruptive to transformational to Horizon 3 innovation, between Design Thinking and Lean Startup, and all the rest.
It’s just that change is a common factor underneath them all. Whatever form of innovation work we do, they all have in common that innovation changes organizations and our organizations' environments including users' lives, markets, and so on (1).
From where do we get that? Just think of innovation as one of three things we can “do” to organizations, exceptions aside (2):
- We can create organizations (startups).
- We can run them (operations).
- Or we can change them (that’s us, in innovation land).
It's not you. System change is meant to fail
Once we settle on “changing organizations” as the core of innovation work, it quickly becomes clear why even good innovation teams often fail:
It's not a tool problem. It's a human problem, because we try to change human systems. And that's ridiculously hard.
The world, including human organizations, operates in a web of stable systems. Change a system and you upset everything in it.
Depending on the size of your organization, there might be thousands of people whose entire job is to create a better version of your org. Your current org. Not some other org that you want to bring about.
People work super-hard to make the supply chain 5% more efficient, the marketing 10% more awesome, or the cyber security be able to fend off 20% more threats; all of it within the constraints of what the org currently is.
And it’s not just their work. It’s also the focus of their rewards. People get bonuses, promotions, and recognition by achieving formal and informal incentives geared to the current system.
And here you come to change it all? Cause people to lose their bonus, miss out on promotions, do things they maybe don’t even know to do? No wonder people fight innovators.
And that’s doubly true if you run a good team rather than a bad one. The good ones are most likely to stir up loads of change!
Yes, innovators see “how things could be so much better.” And yes, our current world sure isn’t perfect.
But let’s be honest. New efforts could also be “so much worse!” After all, what do we mean when we are “pivoting” and “iterating” than that we’re trying to fix things that didn’t work?
And even if you achieve outcomes that you consider “good,” that goodness is relative. You might easily make someone’s team less important, cost jobs, or make entire careers irrelevantly outdated. They would hardly see that as good.
We could slightly refine things then.
Technically, it may not be the "change" itself that causes resistance. It's the combination of objective impact that the change may have on people and the subjective, psychological triggers that bias people against change even when it may work out fine:
Issue like loss aversion, the endowment effect, choice overload, merely being surprised, and more all play a role. You'll need to diagnose this cocktail of trouble precisely when you are working to minimize resistance to change. But for this conversation, it'll be fine to summarize them all under the umbrella term of "change."
Overall: Resistance to innovation and change is standard and rational.
It may not make you feel any better that it’s not just you. But maybe it at least offers perspective. It truly isn’t just you.
By the way, just like it isn't just you facing the issue, it's also not just me saying so. One of my favorite related descriptions of this resistance to change comes from Marketing-focused innovator Scott Brinker. He identified what he calls
Martec's Law: Technology changes exponentially, organizations change logarithmically.
So the faster the innovation advances, the more the organization is likely to resist it. (Brinker later also offered fascinating add-on principles and ways to counter the problem. Those action steps also are somewhat similar to what I'll describe shortly.)
We've known for 500+ years that innovation will likely fail
The difficulties with making “change” work go way beyond our innovation context. People have known for at least 500 years that innovation is somewhere between hard and impossible.
For example, consider the observations of Nicollò Machiavelli. Many people consider him unscrupulous, appropriately so by today’s standards. But he was also a clear-eyed observer of effective and ineffective ways to make your dreams happen.
His take:
“And let it be noted that there is
no more delicate matter to take in hand,
nor more dangerous to conduct,
nor more doubtful in its success,
than to set up as a leader in the introduction of changes.
For [they] who innovate… will have for [their] enemies all those who
are well off under the existing order of things,
and only lukewarm supporters in those who
might be better off under the new.”
— Nicollò Machiavelli. Italian diplomat.
You can find the same recognition that change causes resistance among other writers (like Joseph Schumpeter with his concept of "creative destruction," thanks @LE for the tip), and in many other fields. For example, just search ”the establishment always wins” online and get a zillion different hits in fields ranging from entertainment to religion.
Even when change and innovation succeed, they can leave a hot mess in their trail, people realize. Taking an example from the business world, Jack Welch, attested with his usual bluntness to the stress that change causes:
“Willingness to change is a strength, even if it means plunging part of the company into total confusion for a while.”
Never mind your or my personal attitude to “Neutron Jack.” I think we can agree that he managed to make change happen when he wanted it, and that his observations were at least informed by long experience.
In short, change is hard wherever there are humans involved! Of course, we could still fine-tune what exactly goes on when we say that.
By the way, the difficulty of making change stick is true even in a world with “change leadership” and “change management,” helpful though those efforts are.
A lot of the work of change leadership and management helps once change is bound to come and we try to make it easier to absorb. But our work as innovators is upstream from there: We figure out what change is even worth creating and have to earn buy-in from anyone at all who might become future “change leadership” champions.
Change that actually succeeds has four dimensions
So what can you do, to set up your innovation work—your organizational changes—for success?
In short, you need not just your process, but also momentum, luck, and respect.
We'll talk about these aspects in two sets of two. First, there are the things you can largely control yourself; call them "internal dimensions." But second, there are others that you need to earn and can only influence at best; call them "external dimensions."
"Internal" dimensions
These dimensions are "internal" in the sense that you can control them yourself (more or less).
The first dimension won’t come as surprise:
- Process: This dimension covers good craft and functional work across DVF. Teams already focus on this daily to succeed—great tools, solid data, inspirational stories, empathy, MVPs, change management, and the like. Of course, it can always get better. And so we always look out for new tools or even entire methods like "Design Sprints" or "Product-led Growth."
But some teams believe that that's all you need; that any innovation can succeed, as long as it is Desirable, Viable, and Feasible. Not so.
It's simply not enough, and it wouldn't be even if you could do all the Design Thinking, Lean Startup, and Product work in the world.
The problems that even good teams still face just don't fit the kind of topics that standard innovation toolkits can solve. After all, we already said that the issue is systemic and human. And your own experience may lead to a similar conclusion: Any projects of yours that failed probably didn't do so because of poor user personas or incorrect MVP definition. There is more.
At minimum, there's also:
- Momentum (e.g., change leadership and goodwill): Your solutions just don't "sell themselves," not even inside your own organization. You need to build buy-in consciously and continuously. A range of others have tools and insights for doing that. For example, Scott Belsky's The Messy Middle and Douglas Ferguson's Beyond the Prototype cover innovation-specific ways of building momentum. And that's not even considering the more universal advice that organizational change professionals like my friend Edwina Pike might offer.
External dimensions
But there's more because not everything we need to succeed is under our control, nice as that would be. External factors, those utterly beyond our control, do matter!
Luckily, even though you can't control those dimensions, you can "engineer" or influence them to trend in your favor.
Most frustratingly, but also honestly, there's:
- Luck: Luck comes in many flavors: Being set up for success, timing, your situation in the organization, societal forces, the list goes on.
Even when we know we aren't fully in control of our fate, humans often try to get back to control anyway, via a roundabout path. For example, in innovation, we may try to “internalize” our lack of control by pointing out that we can still “build, test, learn, and iterate.” We do that! Us! We're in charge even if things have to change.
Sure. But don't let it blind you. In the real world, there are also things we truly and purely can't control: Sometimes, there are active blockers, the timing was wrong, or you're trying something amazing in a terrible market (all examples of not being set up for success. By the way, this also means that we need sufficient runway to play the long game). Other times, things just don’t work out. The odds just didn't favor you. Either way, both matter, so much so that it can be easier to succeed with middling work and many attempts in a massively attractive space than with a single bit of fantastic work in a crummy space.
This might not be terribly “actionable” in a direct way. But even just acknowledging it might help you make peace with things and maybe even let you move on from fighting the very idea to imagining ways to circumvent the issues you face.
And then, you need a last thing. This one will be the focus of everything on this site:
- Respect: In your current context, for the current topics, is anyone receptive to what you have to say?
This is where credibility fits in. It's not the only requirement for respect but a big one.
For the moment, bear with me in the interest of brevity, while I simply assert that it’s needed. I acknowledge that you deserve mucho more reasons and details in the future, and I promise you'll get them!
But for now, and in short:
You need Credibility. Credibility means earning enough receptiveness to your ideas from stakeholders and partners that they seriously listen to you. They must listen to you even when you propose things they consider hard or unpalatable. And that receptiveness, that willingness to consider you seriously, has to come according to their own outside-in standards, not yours.
Credibility is a big part of what gives a leader earned (not just title-based) reputation, authority, and power.
Acknowledging the importance of others' standards really matters. It appreciates that you (and I) don’t get to be the judge of our own awesomeness. Insisting that our work is awesome by standards that matter to us doesn’t mean that anyone else has to agree. Instead, we need to earn credibility in outsiders’ eyes, based on their perspective and by their standards, even though they have nothing to do with innovation.
Credibility is a big gap you can newly work to close. And it WILL help you chip away against resistance to change
You probably already got there ahead of me:
No, Credibility alone is not normally enough to overcome resistance to change!
That begs two questions: First, why bother with Credibility at all? And second, why not jump ahead to what will overcome this resistance?
First, even though Credibility is not sufficient, it’s still necessary. Just like I claim that “just” doing Desirable, Viable, and Feasible work is not enough, all of DVF are still needed for good work. Same thing here.
And second, Credibility is actually a critical aspect for the success of the innovation team’s leader’s work. Our focus here is on the leaders of innovation teams after all. And their Credibility, their reputation, their earned authority and power, are very much the currency that they primarily use to make change possible in their organizations.
Beyond that, I don’t know whether there ought to be a “third” point. In other words, I honestly don’t know what else may be needed to overcome the resistance to change. (Or, as they teach my kids in school as part of building a learner mindset, I don’t know it yet.)
Maybe we will discover that answer on the journey that this site represents. Maybe you will figure it out, in part thanks to what you discover here. Or maybe you or others will figure it out separately from all this.
But it would be a shame to throw out the baby with the bath water. You will find that even “just” achieving Credible Innovation is a fairly tall order!
A short overview of what I have found Credibility to require offers a glimpse into the task's magnitude.
How to achieve Credibility practically in Innovation
If we skip most complications for now, we can jump straight to a basic spine or framework that grounds Credible Innovation. We’ll return to it over and over again.
Successful, credible innovation work requires:
- Truly must-do purpose (3)
- Usable output
- Impressive craft
- Unpretentious team
When you achieve these three factors to a decent degree, your Credibility builds up, at least in a sane, functional organization.
Remember of course that all of these judgments are through the eyes of others. Your stakeholders and partners have to consider you to meet these criteria, by their own standards, not just you by yours!
(And yes, that is rather a short list for achieving Credibility. If the whole site is devoted to this concept, then it’s probably hard to boil it down to just one thing. And yet, you rightly deserved that I at least try. This is merely the most compact way I have found to describe it: Must-do purpose, usable output, impressive craft.)
Your turn: Is your team set up for success?
You might argue that the basics of our four principles—must-do purpose, usable outputs, impressive craft, and unpretentious team—aren’t terribly helpful without more details.
But these four actually function like fractals. Yes, you can revel in the glory of all their details. But they work at a zoomed-out view as well. In other words, they should make sense even without all the details behind them.
When it comes to your team, you should be able to answer simple questions like:
- Is your team’s purpose truly a “must-do?” (Not just “strategic” or “aligned”(4) but utterly unavoidable, in a way that even total skeptics can’t shirk without looking like fools? From the perspective of others in your org, not just your own, are you doing must-do work?)
- Is your team’s output truly “usable” by those to whom you hand it over? (Not just “done” with the official canon of innovation toolkits, but in a state where any receiving team and stakeholder can take it over, knows exactly what to do next, and care enough about it that they’d make it one of the priorities on their current strategic roadmap?)
- Is your team’s craft truly “impressive?” (Not just impressive to you but impressive to outside innovation professionals, and especially in the eyes of anyone important in your org, even to people who barely understand what you do, and without relying on the flimflam of “innovation theater?”)
- Is the group of humans that is your team truly "unpretentious?" (Not just impactful or competent, but a team that outsiders, by their own standards, enjoy working with, because you are open-eyed, skilled, and intellectually humble?)
Well, if you answered “yes” to all these, you are probably well on your way.
And if not, then that's perfectly ok; plus ... normal! Nobody ever has been the impersonation of all their ideals. Simply consider what you have achieved so far to be tour current stepping stone. And maybe one of these factors--must-do purpose, usable outputs, and impressive craft--might help you reach that next goal!
In summary:
Footnotes
(1) Ok, technically, not all "changes" are also "innovation." Innovators change organizations specifically in ways that we have never done and where we don’t know the right path forward, in situations other than turnaround or survival mode. But let’s set that aside for now. Those efforts are often run by temporary project teams (that might be called transformation offices, tiger teams, task forces or something similar) and are not staffed by innovation professionals. Or they involve the work of innovation professionals embedded in core teams (e.g., in Design, R&D, or Strategy). In that case, they face very different challenges that are legitimate too but that we'll set aside here for now, while we talk broad-brushstroke arguments.
Also, not all "innovative" work is done by people whose full-time job is innovation. Plenty of good innovation work comes from people in other departments, whether as part of their day job, special assignments, "work-on-your-own-priority" time, idea competitions, or in other ways. But they, too, have their own set of challenges deserving their own focus. Credible Innovation specifically focuses on those of us who have explicit innovation jobs and work on innovation teams. Other innovation needs to be out of scope for now, to keep scope sane.
(2) Yes, there are more contexts, ones like “turning around” or “shutting down” organizations. But they don’t relate terribly much to innovation and don't change the point that innovation work is fundamentally different than operational or startup work.
(3) Technically, you have two other paths to success, but they require that you play the odds: (a) You can have such a big and diverse portfolio of efforts that you can watch for any of them to turn out to be a success, or to succeed through spillover effect over the course of multiple iterations. This is, for example, what good financial portfolio managers may do. Or (b) you can hide out of sight of the main organization. But that works only if you have all the resources to succeed under your own control, so the first time they see the changes you created, the changes are already firmly embedded and impossible to kick out. This is, for example, what startups do to governments that didn’t even know a certain field existed until major scale-ups already exist and have become profitable and/ or critical to the wider business or societal ecosystem.
(4) Wonder why classifications like "strategic," "aligned," "prioritized" and the like are insufficient for innovation work? In operational settings, words like these indicate efforts that (hopefully) will strengthen the existing business system and status quo. But in innovation settings, they still change the current system more than they reinforce it. So the pushback will be similar to that that other innovation work gets. Also, strategies, alignment, and priorities change. Considering the time horizon of most innovation work, something that's a strategic, aligned priority today may no longer be one by the time you launch it.
Further reading
Blank, S. (2015). How to Avoid Innovation Theater: The Six Decisions To Make Before Establishing an Innovation Outpost. https://steveblank.com/2015/12/08/the-six-critical-decisions-to-make-before-establishing-an-innovation-outpost/
Blank, S. (2019). Why Companies Do “Innovation Theater” Instead of Actual Innovation. Harvard Business Review. https://hbr.org/2019/10/why-companies-do-innovation-theater-instead-of-actual-innovation
Belsky, S. (2018). The Messy Middle: Finding Your Way Through the Hardest and Most Crucial Part of Any Bold Venture. Portfolio. https://www.themessymiddle.com/
Christensen, C. M. (2016). The Innovator’s Dilemma: When New Technologies Cause Great Firms to Fail. Harvard Business Review Press. https://books.google.com/books/about/The_Innovator_s_Dilemma_When_New_Technol.html?hl=&id=dcjWsgEACAAJ
Christensen, C., & Raynor, M. (2013). The Innovator’s Solution. Harvard Business Review Press. https://books.google.com/books/about/The_Innovator_s_Solution.html?hl=&id=I5nBAgAAQBAJ
Christensen, C. (2017). Disruptive Innovation Dilemmas Persist. Leadership Excellence, 2017(09).
Ferguson, D. (2019). Beyond The Prototype: A roadmap for navigating the fuzzy area between ideas and outcomes. Voltage Control. https://beyondtheprototype.com/
Finkelstein, S. (2004). Why Smart Executives Fail. Portfolio Trade. https://books.google.com/books/about/Why_Smart_Executives_Fail.html?hl=&id=6PezDAEACAAJ
Lafley, A. G., & Martin, R. L. (2013). Playing to Win. Harvard Business Press. https://books.google.com/books/about/Playing_to_Win.html?hl=&id=a5Bcilcnjd0C
Machiavelli, N. (1992). The Prince. Translated by N. H. Thomson (1910). Dover Publications. http://books.google.com/books?id=lmDcDAAAQBAJ&hl=&source=gbs_api
MacMillan, Ian C., McGrath, Rita Gunther, van Putten, Alexander B, & Thompson, J. D. (2006). Using Real Options Discipline for Highly Uncertain Tech Investments. Research-Technology Management, 29-37.
Riel, J., & Martin, R. L. (2017). Creating Great Choices. Harvard Business School Press. https://books.google.com/books/about/Creating_Great_Choices.html?hl=&id=rAnrAQAACAAJ
McGrath, R. G., & MacMillan, I. C. (2009). Discovery-driven Growth. Harvard Business Press. https://books.google.com/books/about/Discovery_driven_Growth.html?hl=&id=aye6y6uAHuUC
McGrath, Rita Gunther, & MacMillan, I. (2014). The Origins of Discovery-Driven Planning. Harvard Business Review. https://hbr.org/2014/05/the-origins-of-discovery-driven-planning
Perri, M. (2018). Escaping the Build Trap. O’Reilly Media. https://books.google.com/books/about/Escaping_the_Build_Trap.html?hl=&id=PQ8dMQAACAAJ
Credits
Photo “Metal bridge near boat during daytime” by Giancarlo Revolledo on Unsplash
