Take 25% off Managing Machine Learning Projects by entering fccthompson into the discount code box at checkout at manning.com.

Machine Learning (ML) projects aren’t like other software projects: you need a different approach to succeed

For the first time ML practitioners have tools that allow them to solve really valuable problems on a commercially viable time scale – we can do it and make money. Additionally the available tools are so good that we can live up to the expectations of performance that the customers have if we do it right. So, the *whats* are pretty sorted out, but there is almost no advice as to *how* to go about doing this. The sudden surge of interest in ML and the sudden change in ML’s capability with the development of practical deep networks (Lecun et al. 2015) and other advanced methods (Carpenter et al. 2017) have meant that there are a lot of new opportunities for projects like this. At the same time there are claims that 85% of ML projects fail (Gartner 2018, Kwaśniak 2019), and there are increasing concerns that ML is being applied unethically and harmfully (Bender et al. 2021).

This book provides an overview of the practices and structures that the ML community have evolved over recent years to deliver ML projects more reliably while managing the possibility of creating harm. It provides a roadmap and instruction set for people who are managing a small team that is trying to deliver for a commercial customer. The book also covers all stages of the project lifecycle, starting with identifying the project, understanding it and your customer, and setting up to deliver it. All the work to manage the data, build models and evaluate them, create interfaces, and deliver to production is described. That’s not all though! Even with a tested application in production, there’s more to be done, and the book describes how the model should be governed and looked after in life and the typical problems that crop up when you are charged with doing that.

So, the book attempts to provide a set of tools and approaches and a process that focuses the work to be done the activities that experience has shown will help the project succeed. At the same time a strand of work that explores, exposes, and provides mitigations for the ethical issues that beset ML projects and exposes them to failure.

The idea behind the program of work laid out in the book is that doing machine learning projects is a bit like going on a roller coaster ride. The brightly painted roller coaster is what everyone focuses on, but to ride it only takes 3 minutes. To do the ride you have to get everyone in the car, drive for an hour, park, walk to the ticket office, get tickets, queue for the ride. The point is to have the fun you have to do a lot of preparation. After the ride what then – well then you get to the real point of the ride. You get to sit with your kids and eat ice-cream and talk about how good it was and what you are going to do next and why. The point is that if the before and after parts of the process aren’t good then the fun bit doesn’t happen or gets wasted.

The book focuses heavily on the preparation required to use ML, the work necessary to use the results of ML, and guardrails to prevent ML going wrong. After all, if you fall off the rollercoaster then it would have been better if you had stayed in bed at home that morning.

The book is largely non-technical; it’s aiming to help people understand what needs to be done, and what the problems are, and not so much the detail of how it’s delivered. If you have some idea of what SQL is, and some basic maths then that will help, but even if you don’t know or don’t care about these things then the book should be largely accessible to you. On the other hand, it’s expected that most readers will have a deep knowledge of ML and Data Science and will be reading this because they are interested in the softer skills and project practices that can help them apply their AI magic.

Why am I writing this book?

I decided to write this book because there aren’t many learning materials that directly deal with managing the process of implementing systems using ML technology. This is something that has turned out to be very hard, and creates challenges of cost, repeatability, reproducibility as well as ethics.

Just as for software projects, ML projects have to understand and accommodate the challenges of working in a domain, whether that’s a business selling bikes, or oncology, or epidemiology. But in addition to these problems of understanding, ML projects are complex because they are handling and manipulating complex data resources, and sophisticated models as well as the code to orchestrate them.

• ML systems are dependent on data, in particular the structure and quality of the data assets that they use to create the models employed in the resulting system, and modern data assets are huge and wildly complex. Practices and processes for understanding and handling data assets that are complex, noisy, large scale and riddled with personal and confidential data are required by teams that are going to deliver. The data needs to be understood and handled at the system level and at the statistical or value levels. We need to engineer it and understand what it means.
• ML projects create and use models. The properties of the models that are created need to be measured and understood by the team and this understanding must inform the design process of the system into which the models are embedded. We need to make the models, but also, we need to evaluate them (technically and in a business context) and we need to manage their lifecycle.
• Machine Learning systems have to be developed to align with scientific, stakeholder and societal requirements (Wixom 2020). Both business and broad ethical considerations must be woven into the process of developing an ML system.

Figure 1 shows how these three concerns can be usefully represented as a Venn diagram. This diagram is helpful because it can be used to map out the work and responsibilities in an ML project.

Figure 1. Drivers of complexity in an ML project: The Domain, The Data and The Models.

The extra challenges that an ML project brings are one thing, but in addition to addressing these, there are tasks that should be done to make sure that a timely, efficient and high-quality outcome is delivered. In this book four needs are identified that have informed the work that is specified.

• First, there is a need is to identify risk and opportunity in the project as quickly and as practically possible, and the knowledge that creating the ability to understand project risk in a machine learning delivery project requires work and time.
• Second is to enable the team to react and adapt to problems fast; teams need to cope with unexpected problems and be able to change course as user needs become clearer during the project. Being able to pivot to deal with unexpected model performance problems is critical.
• Third is to tie the customer into the process; building engagement and sponsorship but also eliciting feedback and information that can really make a project useful and effective for a business.
• Fourth, the teams building ML systems think that they are delivering a system, but they also must deliver the documentation and records that are needed to understand, use, run and maintain the system.

In summary, ML projects are hard to handle, and ML models are approximate, hard to interpret and hard to develop. They don’t give “right answers” of the time, and they are robust and appropriate for some applications and not others. There is more uncertainty and risk in an ML project than in normal software development. Also, ML systems are heavily dependent on large scale data resources. Data is collected by people with agendas, whether they know that they have those agendas or not, and so it’s riddled with bias and the way that humans interact with ML systems can create loops and spirals of behaviour that the original designers find… surprising! Handling large data resources reliably and efficiently is problematic and can be challenging for teams used to running software projects.

To tackle these issues a different and tailored approach to using ML is required. Failing to approach ML projects in the right way risks failure or worse—creating something that visits harm on someone else. Not only is this an impossible position for a professional to get into, but there are now laws with nasty penalties, especially in China (Lu 2022) and the EU (Guamond 2022) for people who do so. Of course, following the processes described in this book won’t guarantee that your project will succeed, and it won’t prevent you from constructing a system that’s harmful either. But the steps that are laid out here will help, as will an understanding of how they should be strung together and delivered.

The objective is that using this book will allow you to figure out what needs to be done at each step of a significant ML project and give you some support in doing it. Hopefully it will also provide you with a guide to how much time will be needed and a way of justifying the activities and expense to project sponsors, as well as finding a way to accommodate necessary adaptation and iteration.

Who is this book for?

My idea is that the more people who can create value with ML the better, and the fewer people who create ethical problems or damage their fellow humans with ML the better as well. So, I’ve tried to make the book accessible to people who are fairly non-technical. Hopefully, most of it should be useful to any interested reader, but to get the most out of it you will likely have a background in Software Project Management, ML, or Data Science.

The ideal reader for this book is someone who is being asked to lead an ML project or team for the first time, or someone who aspires to doing that.

• For ML practioners who know far more detail and depth about some of the technical steps outlined in the book, the book provides guidance about interfacing to the business.  How to handle setting up and driving the project forward, and making sure it’s fit for purpose when it’s delivered.
• For team leaders from traditional software disciplines, or for professional project managers, the book should allow them to understand the different processes and activities that an ML project requires.
• For business stakeholders the book will provide information about what needs to be done in each step helping them to bridge the gap between their budget holders (who rightly demand results) and the team (who rightly want the time to deliver).

You can learn more about the book here.