From Succeeding with AI by Veljko Krunic

When running an AI team or project there are important pitfalls to be avoided. This article discusses some of the most common ones.

Take 37% off Succeeding with AI by entering slkrunic into the discount code box at checkout at manning.com.

Pitfalls to Avoid

When running an AI team, there are some popular pitfalls that you should avoid. Some of the most important ones are as follows:

  • Not communicating with the organizational actors who own the React part of the Sense/Analyze/React loop. Or, even worse, not working with them at all until your AI project is well on its way.
  • Transplanting use cases (and metrics) from other projects or organizations.
  • Running fashionable AI projects which are likely to grab headlines.
  • Believing that you can buy a tool, any tool, which will give you a sustainable advantage.
  • Hoping that throwing random analysis at your data will produce results.
  • Selecting which project to run based on a “gut feeling” instead of the results of analysis.

This article discusses each one of these pitfalls in more detail.

Failing to build a relationship with the business team

When using AI as a decision support system, it’s never enough to deliver a good analysis; you need to execute well on the specific business actions recommended by an AI-powered analysis. This means that executive attention must home in on the link between the analytical result and the business action. This section highlights why the AI team must build good relationships with the department of your organization which takes business actions, based on your AI analysis.

TIP  Analytics is like a speed gauge in a car. If the speedometer tells the driver that you’re going too fast, then the driver must reduce the speed. Who is the equivalent of the driver in your project/organization?

Analytical results can be misinterpreted by a non-specialist. A classic problem is that non-specialists don’t understand the limits of the analysis and when the assumptions basic to the analysis are violated. An example of that problem was shown in section 3.3.1, when the research question and business question were misaligned.

I’ve personally witnessed several organizations hand off an analysis report to separate business teams. These business teams proceeded to take business actions without the input of data scientists. This is always a mistake.

TIP  Analytical expertise should always be represented in any group which is discussing how to react on analytical results. You need to ensure that the business team understands your analytical results and prescriptions fully and correctly. The analysis result must be valid in the light of the intended business actions.

If you’re the leader of an analytical project, your job isn’t done when you deliver the results. Your job is done when the analyst’s prescription is successfully implemented. You must build good working relationships with the departments that implement the analysis. If the analysis has prescribed a particular business action, don’t underestimate the need for you to help with and follow through with its execution.

Using transplants

People are often tempted to copy what worked for the people and organizations that surround them. As a result, you see what I call transplant projects. Here, an enterprise decides to form an AI team, and embarks on some AI project they’ve heard other organizations similar to theirs performed. This section explains why transplants are a bad idea.

Examples of transplant projects abound. Some examples are projects like “let’s have our own recommendation engine” or “let’s do sentiment analysis of customer feedback.” Sometimes these projects make sense in the context of the business, but all too often they’re vanilla use cases that you heard about from someone else and didn’t analyze in the context of your own business.

NOTE  For some reason, people have more common sense when they’re thinking about real transplants as opposed to business transplants. You’d never get a kidney transplant because it worked well for your neighbor. Why should you behave differently in your business?

Instead of blindly adapting a project that worked well for someone else, consider it to be one of many possible AI projects.

Trying moonshots without the rockets

Many of the world’s largest technology companies have made fortunes based on the use of data. In the core, companies such as Google, Microsoft, and Baidu are heavily dependent on AI for their success. They’ve significant research capabilities and have a vested interest in ensuring that they won’t miss the train of important AI advancements. This section explains why your organization shouldn’t blindly follow those companies.

Imagine that you are a CEO

Suppose you’re running a company which is making $30 billion a year and you’re in a business which is associated with AI. Let’s go a step further and assume that there’s a one percent chance that someone in the next ten years might invent something approaching a strong, human-level AI –Artificial General Intelligence (AGI) [9]. If the search for AGI fails, there will still be an autonomous vehicle [1] as the condolence prize. Finally, you know that your competitors are investing heavily into an AI.

Wouldn’t you invest substantial money into AI and hire accomplished researchers to help you advance the frontiers of AI knowledge? Or would you rather take a risk? A risk that your error will be taught in every business school for many years to come?

Although the previous logic applies to businesses such as Google, Baidu, or Microsoft, there’s an unfortunate tendency for many enterprises to emulate these companies without understanding the rationale behind their actions. Yes, the biggest players make significant money with their AI efforts. They also invest a lot in AI research. Before you start emulating their AI research efforts, ask yourself, “Are you in the same business?”

If your company was to invent something important for strong AI/AGI [9], would you know how to monetize it? Suppose you’re a large brick and mortar retailer. Could you take full advantage of that discovery? Probably not – the retailer’s business is different than Google’s.

Almost certainly, your company would better benefit from AI technology when used to solve your own concrete business problems. This means that instead of teams populated by the smartest researchers and processes oriented toward the acquisition of new AI knowledge, your organization needs people who know how to make money in your business domain with existing AI technologies.

Don’t emulate organizations richer than yours without first understanding how you’d exploit success. For most organizations, the road to success isn’t found in advancing the frontiers of AI knowledge, but in knowing how to tie AI results into their business. You need a data science team focused on applications, not research. That doesn’t mean you shouldn’t hire bright PhDs, but that the leadership of your AI teams must primarily be experts in applying AI to the task of making money.

It is about using advanced tools to look at the sea of data

Another common pitfall is the belief that you can buy an AI or big data tool which makes it trivial to look at your data, find insights, and then monetize the insights found. Some organizations adopting AI might even take the attitude that the main focus of early AI efforts should be on finding the right tools. This section explains why this is a pitfall to avoid.

TIP  If monetization is trivial, explaining how it happens is as well. Ask vendors detailed questions until you understand the finest points of how to apply an out-of-the box tool all the way from the point of purchase to the endpoint where your organization made profit as a result of use of that tool.

In most business verticals, it isn’t trivial at all to monetize AI. And although many tools can help you get there, it’s unlikely that these tools can solve monetization problems for you. Even if there are tools that let you monetize by installing and running them, what you’re dealing with is a commoditized use case. Heck, someone already has a product that does it!

TIP  The early focus for your business should be on finding AI projects that provide a concrete business value. Tools are enablers of those projects.

A salesman might advise you to “Build a large data lake and unleash your data scientists on it; there has to be something in all that data.” You might even have been given an example of the unexpected insights that only analytics on a big dataset can provide, but those situations are rare and unpredictable. Don’t count on the tooth fairy.  Don’t start with the Analyze part of the Sense/Analyze/React loop. In our framework, always start with the React part.

WARNING  It’s always possible that there might be something special lurking deep within your data. With the proper analysis, it might give you some unexpected business idea which you can implement and with which you can make a ton of money. Although possible, this is certainly not guaranteed, isn’t predictable, and there’s a big question of is it likely enough to justify it as a main strategy you should adapt. Worse, the lucrative bluefin tuna you hope to catch might turn out instead be a slimy monster of the deep. Instead of going on a fishing expedition, organize your early AI projects for predictable success.

Using your gut feeling instead of CLUE

Often a decision about running an AI project is made in haphazard way, as little more than a technical idea that excites the team. Running AI project primarily because you want experience with the underlying technology is the tech equivalent of buying a sports car. This section explains why “following your gut” may result in poor business results.

Video analysis of the behavior of retail customers

There were two proposed approaches: one based on predicting sales trends and another one based on video recognition of customers’ behavior.

If I put my data scientist hat on, I’d have to admit that video recognition of customer behavior is a more technically interesting project for me. That project would excite a lot of the technology teams today. It uses cutting edge AI video recognition abilities, whereas sales prediction may make do with older time-series-based analytics methods.

Sometimes that technical allure is all it takes for a team to decide to build a prototype, and the data scientist in me certainly understands this urge, but this is a classic “gut” or “Oh, shiny!” approach to project selection.

When you’re talking with management, you may learn that your business isn’t comfortable with the legal and public relations ramifications of doing video analysis of customers’ behavior. Your effort is unlikely to be adopted even if it’s technically successful. This doesn’t take long to learn when you bother to talk with business leaders about your proposal before building a prototype.

Also, even if you somehow persuade management to allow you to continue building your AI prototype, you failed to define a business metric for measuring success. Now you have created problems in managing your project. Suppose your project is in progress and has achieved some initial success. How would you know if it’s good enough to release? How precisely does it need to recognize customer behavior? Can it make recognition mistakes and under which circumstances? Which mistakes are most damaging?

Be extremely skeptical about counting on intuition to select which AI project to run first. Section 3.2 showed you the steps necessary to correctly determine which is the best project to run. When selecting a first project, there are too many moving parts to consider for gut feeling to provide the right answer. You need to verify that the project is actionable, technically possible, and business valuable. You need to know its cost, as well as the outline and difficulty of the proposed technical solution. It’s highly unlikely that you can assess all those attributes of a proposed AI project by thinking about the problem for a minute or two.

TIP  Above all, be on the lookout for any situation in which, during a large company meeting, everyone immediately exclaims, “This looks like a great idea!” Such a social situation isn’t exactly conducive to encouraging people to perform the careful analysis needed to disprove the group consensus. In short, beware of groupthink.

But we’re using an MVP approach!

Some teams are Agile and/or use Lean Startup [7] methodologies for developing their software projects. In a Lean Startup methodology, the team is encouraged to dice projects into small chunks of work that can be presented to the customer for feedback. This chunk of work is called the Minimum Viable Product (MVP). Part of the Lean Startup methodology is that if you find that your MVP isn’t what the customer wants, you can then try something else—the so-called pivot.

Some will argue that because you’re building an MVP, you should quickly select some initial AI idea, show it to the customer, and see what the customer says. Don’t do that!

Using MVP has many advantages with real-world products, and CLUE can combine well with a Lean Startup strategy, but MVPs taken alone aren’t solving the same problems that CLUE is addressing; for example:

  1. If you chose an MVP based on a gut feeling, you still started a project before knowing if the business is willing and able to adopt your analytical solutions.
  2. Although MVP can show you that you’re on the wrong track faster, you’re still on the wrong track, right from the beginning.
  3. If your gut feeling was to think about the analysis you can do (versus starting with the React part of the Sense/Analyze/React loop), you’re still playing analysis roulette. You’re hoping and praying that the analysis you do yields a result that your business can somehow implement.

MVPs aren’t valid excuses to promote a gut feeling approach to selecting and running an AI project. MVPs reduce the cost of finding out that you’re on the wrong track, but this is reducing the price of failure. The ability to pivot isn’t an excuse to run projects haphazardly, hoping that with enough random AI ideas you’ll stumble upon something that happened to be actionable.

The CLUE approach can be integrated and is compatible with MVPs and Lean Startup. The C part of CLUE analysis is about selecting a first AI project and such a project can be MVP—an MVP which is based on analysis, not gut feeling.

The biggest cause of failure of AI projects today might be technical, but even among technically successful projects, there are far too many that aren’t even used by the businesses that paid for those. Those AI projects shouldn’t have been started at all and were usually started because a gut feeling about their value was wrong.

That’s all for now.

If you want to learn more about the book, check it out on our browser-based liveBook reader here and see this slide deck.