The Challenges of Innovation in American Companies: An Executive Ethnographic Investigation

Effective, sustained innovation remains one of the great challenges facing almost every company in America. But research shows that the average firm is failing at innovation most of the time. Using an ethnographic research approach, including field research and depth interviewing, the authors of the paper report what executives revealed as the principal reasons why their companies are struggling with innovation.


Introduction
Effective, sustained innovation remains one of the great challenges facing industry.Keeley found that the average firm succeeds in innovation only four per cent of the time (2009), supporting similarly low successful new product introduction rates reported by Shilling and Hill (1998).
There have been many reasons posited for the lack of innovation success.Taking a fresh look at the literature, these problems were readily grouped into three buckets of ideas: cultural and organizational barriers to innovation; strategic and marketing-related barriers to innovation; and process type barriers to innovation ( Foss et al, 2011;Utterback,1994;Shilling and Hill, 1998;Sharma, 1999).
The purpose of the research reported here was to learn how executives presently in industry view the challenges facing their companies in the innovation arena, and then to see how these perceptions matched up to the established precepts of innovation management.Further, we wanted to see if these executives were focusing on certain innovation challenges not yet well researched by academics with the hope of guiding the future work of ourselves and others.
To accomplish this goal, our protocol combined field visits with depth interviewing.We spent time in the field with executives, spending several days observing their activities and actions at times when, by their own agreement, they would be dealing with new products or services, or in some instances, running meetings where the focus was a new business or a new venture.This research method -one of close observation and depth interviewing -is called "ethnography" (Hammersley and Atkinson, 2007).Our role was simply to observe.We then conducted structured interviews, analyzed the content, and then populated the framework developed from the literature as well as positing new elements as of yet not well researched.This combination of validation and discovery forms the basis of this paper.

Innovation Challenges: the Framework Emerging from the Literature
As noted above, as we revisited leading works in the innovation field, the literature tended to fall into three major categories of thought: cultural and organizational barriers to innovation; strategy and marketing-related barriers; and process-related barriers.
Cultural and Organizational barriers: Numerous researchers attribute failure to innovate to corporate culture.Wycoff (2004), for instance, states that innovation is not part of the culture of most established firms.This anti-innovation culture directly impacts those individuals responsible for carrying innovation forward.The idea that culture, among other factors, can impede the introduction, absorption, and application of new knowledge within a corporation has been formalized into the concept of "absorptive capacity" by a number of scholars (Cohen and Levinthal, 1990;Lane and Lubatkin, 1998;Zahra and George, 2002 ).
Many researchers have also posited that organizations are simply not designed well for innovation.Among the tenants of the innovation literature focused on organizational barriers are the notions of top-down command and hierarchical approaches (Kanter, 2006); inflexibility to respond to change as well as new market and technical opportunities (Abernathy and Wayne, 1974;Tushman and O'Reilly, 1997), and organizational rigidities for transitioning to next generation business paradigms (Leonard-Barton, 1992) .Others have argued that large corporations require an organizational dualism that supports efficiency in core business but allows innovation for new business development (Zahra, 1999;Assink 2006).However, the validity of this dualism as a solution to innovation has recently been called in question (Burcharth and Ulhoi, 2011).
Many of the organizational concepts purported to impede innovation are also associated with the siloing of organizations -either on a functional basis or separating different businesses -to achieve efficiencies in execution of immediate tasks.That siloing, however, then impedes knowledge and technology sharing between groups that is often needed for new product or service innovation (Gerstner, 2002;Foster and Kaplan, 2001).Finally, some literature suggests that many organizations are structured to impede individual autonomy and experimentation and prevent employees from engaging in innovative projects which can be viewed as potentially "career killing" (Adomzda et al 2009).

Strategic and marketing-related barriers to innovation:
The literature focused on the strategic and marketingrelated elements as barriers to innovation is also broad and compelling.There is a strong stream of literature that posits a firm's self-perception of its mission, purpose, market focus, strategic intent, and strategic choice can be self-limiting and constrain true innovation (Abell, 1980;Levitt, 1975;Abernathy and Wayne, 1974;Utterback, 1994;Rumelt, 1974;Barghai et al, 1999;Zook, 2004;Foss et al, 2011).For example, Christensen's work (1997) synthesizes industry dynamics and innovation strategy by showing how industry leaders become so strategically focused on serving the needs of core customers that they fail to pay attention and innovate to serve the needs of new entrants coming into market.Some have referred to this as "milking the cow too long" (Menchin, 2004).Furthermore, some researchers have found that companies simply fail to understand the linkage between strategic intent and the adaptive/absorptive capabilities of the firm and when to emphasize these capabilities (Foss et al 2011).
Process related barriers to innovation: The processes of innovation and their potential to impede or enhance innovation success has been a central component of the innovation literature for some time (Cooper 1990(Cooper , 1994;;Henderson and Clark, 1990;Tabrizi and Walleigh, 1997;Ulrich and Eppinger, 2000;McGrath, 2000).In many cases, the literature focuses principally on processes involving "new product/service development" while neglecting other aspects of innovation such as services or business model innovation (Heskett et al, 1997).Moreover, there is much discussion as to whether or not formalized innovation processes, such as stage-gate systems have actually helped or hindered the realization of commercially successful innovation (Keeley, forthcoming).
Finally, substantial research has also focused on the methods and processes for clarifying the "fuzzy front end" of new product and service development.Failure to adequately understand user needs, both present and future has been cited frequently as a major innovation roadblock (Dodgson, Gann and Salter, 2005;Schrage, 2000).In response to this challenge, there is a growing body of work that explores new methods of user research, product design, and test marketing (Leonard-Barton and Rayport, 1997;Norman 2002, Hammersley andAtkinson, 2007).Lead user research -and user as innovator -is also an important part of this research stream (von Hippel 1988, Thomke andvon Hippel, 2002).

Study Method
Our literature review showed two basic approaches to innovation research: the use of in-depth case studies in one or more companies, from which frameworks and methods emerged; or, the administration of questionnaires, from which data are gathered, combined, and heavily processed into "significant" results to test particular hypotheses.Both approaches are valid -and we, too, have engaged in both.However, for this study, our desire was to reengage directly with executives facing innovation problems in real-time, to gauge their reactions and decision-making against what we knew from the literature, and from this, to find the gaps.
The method we employed ethnography and depth interviewing of senior management.We sat down with executives, "off the record," to ask them about the present innovation challenges and as well as the innovative projects underway in their companies.We then spent several days within these companies observing employees and teams engaging in innovation projects.Forty-three individuals agreed to let us join them in staff meetings and project work that they deemed to be innovation-focused.Then, in follow-on sessions, we conducted semi-structured face-to-face interviews with these individuals.
Our interviewees consisted of CEOs, senior executives, heads of R&D divisions, and senior members of innovation teams.There was a good cross-section of industries including manufacturing, consumer packaged goods, technology-based companies and services firms.The number of corporations represented in the study group was 27, and the number of individuals, in total, was 43.These respondents were selected based on convenience and judgment and because we had access to such companies.
The structured interview portion of our field research was guided by the following questions posed to the interviewees: (1) Is innovation an important part of your organization's future growth strategy?(2) How satisfied are you with your organization's innovation efforts?
(3) Are there any impediments or roadblocks to innovation within your organization that you can identify?Ethnographic experts and management experts suggest that the use of depth interviews is a grounded theory approach and when conducted properly can provide deep insight into management issues that may not be uncovered through surveys or case methods (Fine, 1993;Hammersley and Atkinson, 2007).Moreover, Snow and Thomas (2007) suggest that this type of field research approach can help describe and explain particular phenomena under investigation.In this case, the concept under study was innovation and/or failure to innovative.
All interviews were recorded, with the respondent's permission.The interviews were then transcribed, analyzed and sorted into response categories.The sorting process sought to categorize responses into basic over-arching themes.Another researcher not involved in the field research independently examined the data and also sorted the data into thematic categories.The result was an inter-rater reliability level of over 95 percent.This approach is consistent with methods advocated by experts in the field ( Perreault and Leigh, 1989;Lofland and Lofland, 1995).

Results: Perceived Impediments to Innovation
All respondents stated that innovation was a crucial part of their enterprise's growth strategy.Yet, not a single respondent was satisfied with their organization's innovation efforts.As our literature search revealed such a wealth of research on different aspects of innovation, it should be no surprise the executive ethnography revealed a wide range of systemic, enterprise-wide innovation problems.Appendix A provides the profile of respondents participating in the study.(The verbatim responses provided by the respondents can be made available to anyone reading this paper by emailing the contact author).

No working definition of innovation
One of the first barriers we discovered was the absence of a shared definition of innovation within each respective company.While almost all of these publicly traded companies used the word "innovation" frequently in their annual reports, and strategic plans, not one executive could actually provide a working definition of innovation that was used throughout their organization.For example, one executive stated, "Our first problem is that we cannot even define innovation."Another said, "We have absolutely no idea what innovation really is."Finally, another revealed, "We talk about innovation but no one understands it." When we asked why this lack of a clear definition becomes a barrier to effective innovation management, most individuals stated that no clear definition created a major problem when it came to resource allocation."We say we spend money on innovation, but the reality is that 99.9% of our money goes to incremental developments.We aren't spending any money on real innovation because no one is clear on what that means.We talk a good game, but aren't putting resources into it." It also became clear that executives struggle defining innovation when it reaches beyond the confines of new product development.Hargadon's ideas ( 2003) on the evolving ecosystems of industries and the formation of new business models within those ecosystems -iTunes being preeminent example -was readily apparent in a number of the executive interviews.Some of these executives were more fearful of being displaced by a new business model than caught short on a particular technology.Other executives marveled at the ability of industry leaders such as IBM to excel in services while still producing new systems and software.Their companies might master one area of innovation but had failed to master another.

Cultural antibodies to innovation
Most executives revealed that innovation was not really a core corporate value.Most pointed to a culture as preached versus a culture as practiced mentality.One stated, "Innovation is really not a culture value here.We say it is, but it isn't."Another said, "We say we value innovation, but we really don't.We do not honor ideas or support risk-taking and everyone knows this.It is corporate schizophrenia and we tolerate it."Another stated, "This organization is not built for innovation, culturally or organizationally."Finally, another executive went a step further.He suggested that creativity and innovation are discouraged at an early age in our society."Innovation is not supported in our own families or in our schools, how do you expect to build a corporate culture of innovation when your people have has previous learning that innovation is not valued."In essence, the innovativeness of a company it dependent on its people and if those people do not believe innovation is valued, or do not view themselves as innovative, it is little no wonder may corporate America cannot innovate.Deming (1982) stated that in order to optimize the performance of any organization, fear must be removed.Clearly, the notion of fear was a pervasive theme stemming from this research."There is no reward for being a change agent here," said one executive.Several other executives were quick to point out that fear is a serious impediment to innovation in their organizations.One executive stated, "In this organization you quickly learn not to take a risk.If you do, and lose, in fact, you could lose your head."Another said, "Why would someone be innovative here?If you fail, you get fired."Finally, another executive told us, "When it comes down to it, we are risk-aversive.No one wants to own failure.And, we fail more than succeed with innovation."Yet another said, "new internal ventures are a career killer."

Personal, Career Risk and Fear
All respondents directly linked these negative associations with innovation as being an obvious, and often impenetrable barrier to new product line, process, and service innovation in their companies.

The lack of an effective idea management system
Previous research suggests that a lack of a sound idea management system is an impediment to successful innovation (Wycoff, 2004).Our ethnography confirmed this reality.For example, one executive stated, "Idea generation and idea management, that's our number one problem."Another said, "This organization is bankrupt in terms of ideas.We simply re-try what we have always done." Add executive added, "We either have too many ideas, and a complete lack of focus.Or, we have too few ideas, none of which can survive a simple vetting process."Another stated, "Brainstorming is all we use because we don't have other options."Finally, another says, "We hire consultants and they shove ideas down our throat, ideas we would never implement." Another executive noted that while his company had an extensive stage-gate system for approved ideas, and a well-defined process for managing intellectual property, ideas that did make it into Stage 1 of the gate system or have patent potential got "lost in the system."He went on to add, "I bet that the hottest new products introduced by our competitors over the past ten years were ideas that got lost here at one time or another.We have thought of everything on the market -but we just haven't acted on anything.We are own worst enemy." The definition of an "idea" goes beyond simply technology.As noted by one respondent working on consumer products, "We have no lack of new ideas.We have dozens of them.The problem is, we have no clue what these ideas are like as a business, unless they are just extensions.The business uncertainty kills the potential good growth ideas.So instead, all we do incremental innovation.We call it polishing the marbles.

Either no process, or too much process !
The participants fell into two camps: their companies had no effective process for innovation development and commercialization, or, an elaborate innovation process that actually limited bolder innovations.Of the former, opinions were strongly held.One executive stated, "There is no system here with regard to innovation."Another stated, "Our biggest problem is building in time for employees to engage in innovation."Yet, past research has indicated that a lack of time a major constraint on corporate innovation is time.Another executive responded saying, "There is absolutely no training provided for people engaged in innovation.The assumption is they will learn the game as they go along."Another executive told us, "It is known that integrated, cross-functional teams are a necessary ingredient for successful innovation but we ignore this reality.It is actually difficult to get our people to be part of such teams."Another executive stated, "Our approach is seat of the pants.Sometimes it works, sometimes it doesn't."One stated, "Our approach is unless you can prove the numbers, no way do you proceed with innovation of any kind."Another added, "If there was a good system we could buy, we would.What is done in-house is ineffective; we simply don't get any positive outcomes" Representative of the "too much bureaucracy" camp was a remark from one participant: "Our phased approval system kills off our innovations because of the way the process is designed.The VPs have to sign off at each gate.That simply delays progress; and if you get a "no", there is no chance for a quick fix and approval -everything takes another three months.Plus we have to do it the VPs way, make it only on our own assets or sell it through our sales force.That often limits our innovation so much."Another executive also stated, "This company went from no innovation process at all to a stifling gated system.We can't seem to find a healthy balance."Finally, one executive reported, "We put Stage-gate in place and thought this was the answer.But, people found ways to circumvent it.There was a belief we didn't believe in the system and so people worked their way around it".

Lack of progressive and equitable rewards to encourage innovation.
Our research discovered that in most organizations there was simply no reward to encourage innovation.In most cases there were actually disincentives.For example, one executive stated, "There is no incentive to innovative here."There are actually disincentives (it means more work and no extra compensation)."Another added, "Unfortunately, there are no rewards for innovation, particularly financial."Another executive stated, "If you do something innovative and it works, you get to keep your job.If you try something and it doesn't you'll probably lose your job.No one is going to be innovative in this environment."Finally, one executive acknowledged, "Your performance review here never includes innovation".

Lack of measurement of results from innovation efforts.
Another major impediment to innovation was a lack of measurement of results from innovation efforts.This creates a vexing situation for people trying to provide justification for expenditures on innovation.For example, one executive said, ""We do not track the results of the innovation spend."Another added, "No one can show a return on innovation dollars spent. So, it is a constant battle to get funds for projects."Just as problematic is the fact that some firms say they measure results from innovation efforts, but the measurement is short-term measurement.For example, one example stated that his firm has a mantra, "Do it quick, measure it quick.If the results are not there, the project is dead."Another executive indicated that, "We try to use the same metrics for innovation that we do for our established businesses, and that simply does not work."Finally, another executive stated, "Since we do not start with hard measures of success, it is impossible to measure results.So, unless you are politically tied in, you'll have difficulty getting money for innovation and proving the investment was worth it".

Lack of fundamental understanding of customers and markets.
Another basic theme that was revealed was that most companies lack a fundamental understanding of their customers and markets.One executive stated, "Our R&D people simply develop stuff and have no clue about whether or not the customers would want it."Another added, "We do not invest in research, really knowing our market.So, it is not surprising we have such a poor record of success with new products."Another respondent said, "Our approach is design it and get it out and hope the customer salutes the flag.It would be much easier, I think, if customer needs were the focus of the effort."Finally, another executive stated, "My organization spends no time or money on upfront research and we end up building things that no one wants"."Winging it" was not uncommon for highly innovative ideas in our sample companies.As remarked by one individual: "In our company, the only ventures that get funded are those that are, or are presented as, a senior executive's own personal idea.
There is no market validation behind it, just gut feel and doing what the boss thinks is cool."This becomes an impediment to a stream of innovation because most often, the boss's idea does not become a commercial success!

Not acting on opportunities for innovation.
On the other hand, our research indicated that some companies do have a good understanding of their customers and markets but are unable or unwilling to act on this understanding.For example, one executive stated, "We spent a lot of time talking with customers but then do nothing with the information."Another executive acknowledged, "Our customers will actually ask us for things and we ignore them."Still another executive told us, "I believe a good source of innovation is customer complaints.We get them but we see the complaints as irritants to us and not as opportunities to innovation."Finally, another executive stated, "We've hired consultants in the past who have provided all kinds of opportunities.But, we meet to discuss the findings, and nothing ever resulted from it".

Focusing on the big hit.
Many participants spoke about the dichotomy between the focus on the one big hit versus a fixation on incremental development.For example, one executive, stated, "We focus almost exclusively on the next big thing.We talk in terms of huge addressable markets and screen out the rest of the ideas.Our funnel is very narrow and our people realize that unless they have a big project they should shut up and not bother presenting their ideas."The laddering technique we used on this comment revealed why precisely "trying to hit a home run every time" becomes an impediment to innovation."The home run rarely works.We then lose our appetite for innovation for three years because we have lost so much money."In contrast, another executive said, "Our company never goes for the home run or big hit.Our approach is simple product proliferation.We just add more SKUs but that is not innovation.Our sales team is always coming back reporting that clients need a new product variation, and, of course, we create it.No one stops to determine if this is the right thing to do, what the costs are and what the pay-off will be.We simply end up managing more products.This, in turn, consumes all our time.There is nothing left for true innovation."

Conclusions and Implications for Research
This study was an ethnographic investigation of the meaningfulness of principles from innovation management research for individuals working in mature corporations.The sample size, both in terms of the number of companies and the number of participants, makes this work somewhat exploratory so there is no pretense of statistical validity or reliability here.But close observation and discussion is not a numbers game.Important insights can be gained by working intensely with limited panels -as long as the research participants reflect a target suitable for the research (Hammersley and Atkinson, 2007).In this study, we believe the intensity and panel size were suitable for the purpose.
Much of our work supported that findings of past research: the difficulty of clarifying the "fuzzy front end" of new product and service development; the helpfulness of a systematic process for managing projects through the development pipeline; the utility of multifunctional teams; and the fundamental importance of executive sponsorship for innovations that carry the firm into new markets and new applications.
The multifunctional team area was particularly interesting.All the companies we interviewed viewed constructing innovation teams for workflow improvement as well as new products and services as requiring people from different affected functions.We can only surmise that there has been so much written -and written well --on the beneficial aspects of multifunctional teams, over such an extended period of time, that forming such teams in now accepted practice for any sort of substantial product or service innovation (Cooper and Kleinschmidt, 1987;Katzenback and Smith, 1993.)Our work did, however, reveal gaps in the existing research, and that is perhaps its greatest value.The basic definition of the different types of innovation, and how to measure effectiveness for each type of innovative effort, still remains a mystery to senior management.The one exception in our study was a company that rigorously adopted and developed differentiated processes across all of its divisions for Horizon 1 (incremental product line extensions) versus Horizon 2 (adjacent new product line development) versus Horizon 3 (disruptive R&D) efforts.Most executives were as interested in thinking about how new technologies affected the market strategies and business models of their respective corporations.This means that a useful classification scheme for innovation must not only be based on a continuum of incremental to disruptive, but multidimensional in terms of categorizing the market, financial, and organization/business process change that accompany the technology component of a particular innovation.
The management of new ideas -be they new product concepts, new service concepts, or new business model concepts -also emerged as a major gap in terms of being much needed but not well understood by our participants (Teece, 2000;Van de Ven, A. H., & Engleman, 2004)."Knowledge management" as applied to R&D also remains vague, at best (Armbrecht et.al, 2001, McDonough et. al. 2008).No company -even two leading information technology companies in our sample that develop and market commercial knowledge management software -was able to express a structure for classifying, categorizing, storing, and nurturing innovative ideas on an enterprise level.
A systematic study of the appropriateness and efficacy of business processes for innovation in mature companies is also much needed.Much of the accepted wisdom on managing the new product development process comes from the original research by Cooper (1990), and advanced by McGrath (2000).During our interviews, all companies with sales over $500 million annually had adopted a formalized "stage gate system," often initially implemented by a consultancy.These business processes had been adapted over time -some simplified, and others, growing more complex over time.Interesting, however, with but several exceptions, all executives specifically questioned the efficacy of their management processes for innovation that was not incremental in nature.A number of executives indicated that their stage-gate processes de-energized innovation within their companies because of all the "checklists and paperwork" associated with the methodology.Even Cooper (2008) himself, in recent writings, has posited a simplified stage-gate for more venturesome innovation efforts.Understanding the balance of control in the face of uncertainty for different types of innovation would appear to be a worthy objective for future research.Is a structured control system always needed for new product, service, or technology development?If so, how much structure, report outs, and executive gates are appropriate, and associated with successful outcomes on a normative basis?
Our executive ethnography also showed that senior management is thinking broadly about innovation, perhaps more so than the innovation management academe.For our interviewees, rarely was the attention solely on a technology "bit or byte," but rather, on the implication of the technological innovation "for the business."The executives with whom we spoke also viewed "business model" as much more than pure financials.Instead, the consensus definition of the business model was how their company participated across the value chain of its commercial offerings -range from raw materials supply, to conversion and assembly, to route to market, with revenue and margin implications all along the way.Our research also revealed that he higher the rank of the interviewee, the greater the concern and sometimes, the confusion, for understanding the business model implications of innovation."How we make money doing what we do" in a company becomes the established context for conceiving and evaluating new product, processes, and services ideas.Any idea that challenges or requires change to that paradigm tends to die.While business model concepts and cases have been published, normative research that connects a taxonomy of business models with different types of innovation has yet to appear.Such research might help break through the greatest barrier to innovation: how company leaders view the next generation of their businesses, and the investment requirements in technology, marketing, and new organizational capabilities needed to achieve that goal.