Machine Learning and Data Science Project Management From an Agile Perspective: Methods and Challenges

Introduction

Contemporary developments in cutting - edge technologies, along with the advances in Artificial Intelligence (AI), have paved the way for various integrated and sophisticated systems. The proliferation of AI systems has enabled new business models and brought opportunities for organizations. The successful implementations of AI can have a great impact on the activities and competitiveness of an organization. On the other hand, there is also an overestimation of their benefits, opportunities, and return on investment of AI projects. How to realize AI as a long - term and stable solution is difficult, and defining the required procedures is not clear. To that aim, Bughin et al. (2017) indicate the factors for transforming an organization towards AI as: a good and reasonable business case, “the adaptation of business processes to the capabilities of AI”, effective and efficient AI environment, techniques and tools, solid data ecosystem, and organizational culture. Ransbotham et al. (2017) report the challenges in the adoption of AI systems as follows:

• •

  Difficulties in the acquisition of AI - related skills and knowledge,

• •

  “Competing of AI projects with other projects in the company”,

• •

  Safety and security aspects of systems using AI,

• •

  Cultural and organizational barriers in the AI adoption,

• •

  Limited technological capabilities,

• •

  Lack of leadership and management support for AI initiatives,

• •

  Unclear business cases for AI projects.

In this context, Machine Learning (ML), which has a close relationship with the Data Science (DS) discipline, can be viewed as one of the most popular subfields of AI. However, recent studies reveal the facts about the failure of ML and DS projects, inadequate return on investment, and unsatisfactory results. It is reported that organizations employing AI solutions with big data are at a high risk of failure and may fall short of project objectives (Kelly & Kaskade, 2013). There may be, of course, various domain / process / technology - specific causes when we view this issue from the perspectives of different disciplines, such as DS, Computer Science (CS), and Software Engineering (SE). Ponsard et al. (2017) and Saltz et al. (2019) indicate that the majority of ML or DS research studies focuses on technical aspects rather than project management issues. More than 80% of data scientist state that an explicit process model is not followed, and they note that more systematic methods would improve their performances. Therefore, one major factor has been adopting or tailoring a project management method suitable for the idiosyncratic requirements of ML and DS applications (Saltz, 2015; Cao, 2017; Nalchigar et al., 2021). Studies refer to various issues, however, there is significantly less research, which explores the DS and ML project management from an agile perspective. The objective of this chapter is, therefore, to present and discuss the findings from the literature according to the solutions, background of the ML, DS, and Agile Project Management (APM) knowledge domains.