ummary. Some observers are beginning to question whether gen AI will produce enough value to exceed its costs. It can, but extracting economic value from gen AI requires several different types of…more
Generative AI (gen AI) is all the rage, and with good reason. With it, AI crosses the frontier into knowledge work, generating content that could previously only be created by creative human beings. This opens the possibility of much greater productivity and performance across the knowledge economy, but it raises many issues about how best to make such work productive and effective. Namely, because knowledge workers typically have substantial autonomy and variability in their roles, introducing gen AI can be a tricky proposition.
Determining when — and whether — these tools are really of value requires considerable effort. Some observers are beginning to question whether gen AI will produce enough value to exceed its costs. We feel that it can, but that extracting economic value from gen AI requires several different types of disciplined capabilities. Unfortunately, many organizations lack such disciplines.
Here’s are the disciplines companies need to have — or build — before becoming successful with gen AI, and what they need to know about them.
Behavioral Change
Gen AI requires people to change their behavior. They need to learn whether to use gen AI tech at all during various stages of the content creation process, to use it at the right times and for the right purposes, to accompany it with their own contributions, and to confirm that the resulting output is of merit. The needed behavior changes are likely to differ by individual jobs and people. In call centers, organizations need to determine the appropriate sequence of human and machine interactions with customers. Whereas lawyers need to decide whether they should they employ it for brainstorming, generation of first drafts of briefs or contracts, refining of existing drafts, or some other purpose.
There are, however, two behaviors that will likely be common across roles and applications.
First, given the possibility of hallucinations (really just bad predictions by these statistical models), it’s important for humans to review the output of gen AI models. That may not be a natural inclination, however. In one study of knowledge work creation by MIT researchers, 68% of participants chose not to edit the output of a language model.
Second, users of gen AI need to add value to the generated content. Given that gen AI is trained on existing online content, it’s unlikely that outputs will be truly novel. That may be fine for some applications, but content creators need to realize when it’s important to go beyond well-established ideas and formats.
Organizations need to think hard about their standard processes, too. While there are some guidelines about the role of gen AI, the nature of needed interventions is generally job- and incumbent-specific. Most studies have found, for example, that entry-level employees benefit more from gen AI than experienced ones. Companies may need to create new tasks, business processes, and career paths — each of which effectively involve designing work for both a human and a machine. The most effective way to introduce gen AI is personalized to individuals, which many organizations will not have the discipline and time to pursue.
Controlled Experimentation
A key aspect of gen AI discipline is determining whether there is value to employing the technology in any specific business domain. Leaders can’t take for granted that gen AI tools will always improve the quality of output or boost productivity; it’s likely to work for some tasks and applications but not others. Therefore, the only reliable way to determine the level of value is to design a controlled experiment in which some people are using gen AI tools and some people are not, and their productivity or effectiveness is measured and compared. Companies may also want to experiment with using gen AI in different ways — for example, comparing use of it as a solo content generator vs. a “co-pilot” situation of ongoing collaboration.
While the design and analysis of experiments is not very complex — many data scientists can do that work — most organizations don’t do it on a regular basis. It is largely done only by academics and vendors now. The existing studies about gen AI’s impact on productivity have generally found positive results, but they vary considerably by the job type and attributes of the individuals in the experiment.
Companies need to run their own experiments, and ideally use their own people to do it so that they build the capability on an ongoing basis. The skills needed to do so are experimental design — how many experimental groups with how many people in each — and experiment data analysis to determine if the results are statistically significant across groups. There is software that can lead organizations through the processes of design and analysis.
Measurement of Business Value
Related to experimentation is the discipline of measuring different forms of business value. Individual-level productivity is the easiest form of value to measure with gen AI adoption and, according to some analysts, the quickest return on investment. However, even that metric is not often measured rigorously, whether in an experiment or after production deployment. It is also likely to be matched quickly by competitors.
Measurement of business value from new initiatives or strategies enabled by gen AI is often more complex, as these changes will involve multiple components in addition to new technology. However, some organizations have created new products and services involving gen AI, such as the law firms A&O Shearman and Wilson Sonsini (both related to contract review), or the pharmaceutical firm Sanofi, which is using gen AI and other AI tools to speed time to market for new drugs. These benefits can usually be measured eventually in terms of additional revenues and profits.
If companies want to succeed with gen AI, they need to have clear practices about how and when they measure business value for every use case, and be in the habit of tracking these metrics over time. At some point we may be able to take gen AI value for granted, but not yet.
Data Management
Data management is a discipline as well, and most organizations had challenges with it even for the structured, numerical data that fueled analytics and analytical machine learning. Gen AI, however, learns from and creates unstructured data such as text and images. Most organizations don’t have well-defined processes for collecting, storing, and curating this type of content. In companies that are serious about gen AI, some work environments and/or customer environments will need to be augmented to capture new data.
Data curation will mean evaluating content to assess its importance, its uniqueness, its currency, and other relevant attributes. This may require third parties to perform the curation tasks, but ideally content providers could be trained to do it themselves.
For example, the electronic health records company Epic, working with Microsoft’s Nuance business, has added the gen AI–enhanced ability to capture and summarize clinical notes. For health care providers, this may mean augmenting the examining or the operating room to seamlessly capture this new information. For other organizations, it may also mean new partnerships to gather data that was previously discarded or unavailable.
Human Capital Development
Organizations wanting to get the full benefit of gen AI need to commit to and develop their people. This starts with a commitment to using AI to augment employee capabilities, rather than using it to replace people and reduce headcount. Otherwise it will be difficult to persuade any of them to engage with the technology to improve productivity and operations.
But that commitment is just the beginning. There is a substantial amount of learning that’s necessary for employees to use gen AI successfully. Among the skills and knowledge that employees need to master are the fundamentals of how gen AI works, prompt engineering, when and how to confirm facts, how to create the highest-quality content, and how to integrate the technology into their jobs.
Systems Thinking
One of us (Sviokla) studied the keys to success of self-made billionaires, the vast majority of whom (80%) made their fortunes in highly competitive, mature markets (e.g., coffee, water, pizza). One key to their approach was to create a superior system that crafted an interlocking set of advantages into a competitive moat, and resulting higher economic returns.
We expect that systematic gen AI experimentation will eventually lead to superior models of work in many industries. For example, in sales organizations with high turnover such as high-ticket sales (think insurance, pharmaceuticals, commercial real estate, etc.), will a combination of gen AI–powered avatars in highly personalized sales presentations and simulations of client sales calls radically change sales approaches and the time to become productive? If so, the relationships among a firm’s gen AI capabilities, sales talent and incentives, and service goals may radically change its sales model.
More broadly, we believe many vital value-adding activities will yield competitive advantage if organizations undertake fundamental business systems redesign. We are only 18 months into reinventing generative work, and the AI we use today is the worst AI we will ever have.
How Can Organizations Pick the Best Gen AI Projects?
Assuming companies can develop the capabilities above, how can they make use of them? Here’s where to start.
1. Fund the responsible rebels.
Every organization has practical rebels who are longing to make things better, to do new things, and to break the status quo. The heart of operational excellence is standardization, the heart of innovation is productive variance. Many organizations are over-focused on standardization, but growth-oriented executives know how to fund those who can drive innovations for value while not creating chaos.
There are several useful ways to organize the funding. One way is to have a small innovation fund with a fixed amount of funding, and project teams must present to a senior group of operating executives to get funding. One of those leaders on the review panel must be a sponsor, to ensure political commitment to the idea. Over time, if successful, the funds can be replenished by the surplus created by the innovation. Another approach is to fund projects incrementally as they move through stage gates. Eventually, the rebels must show economic value in order to receive additional funding.
2. Choose projects that are practical, quick wins, and politically aligned.
Scientists who research extending animal lifespans often study mayflies; females live two hours, males two days at most. They rarely study sea turtles, whose lifespans can stretch from 50 years to more than a century. Likewise, pick gen AI projects that have a fast cycle time to value such as customer service, inside sales, software development, etc. Also, pick a project where the outcome variable is clearly defined — margin, sales yield, turnover, etc. At the project level applicants must show value within the fiscal year, preferably within two quarters.
Politically, pick projects where the cost of the project and the benefit both sit in the same organizational unit. If you pick something where the cost is in marketing, but the benefit ends up in the goals and objectives of customer service, the project may receive public support. However, few executives prioritize costs for their units while the benefit sits in another executive’s budget.
3. Link to the identity of the firm.
Lastly, think through how the project relates to the identity and key goals of the organization. For example, if you are working at Intuit, whose mission is “to power prosperity around the world,” even the most mundane gen AI experiment should tie directly to that identity. In Intuit’s case, its Intuit Assist offering, which employs gen AI to provide personalized and intelligent advice to customers across all of its software products, easily qualifies.
Get Serious
A core pillar of the practice of management is to create and apply practical knowledge about customers and activities faster than the competition. It’s clear that generative AI can become a powerful tool for mastering this objective. Organizations that master the disciplines we’ve described can bring about more customization, greater complexity, higher quality, faster throughput, superior labor productivity, and improved capital leverage. Those that only undertake minor tinkering with the technology are condemned to achieving only minor outcomes.
