Open source communities have had and continue to have a major influence on the evolution of the Internet. By their nature, such communities involve people with diverse coding cultures and skills. Automation has consequently been of major interest to open source software developers for a long time, and many open source tools have been developed to address code variability and sustainability challenges. This chapter discusses why open source communities must automate and the challenges they will face. Solutions and current examples of automation in open source projects are provided as a guide to what is achievable. OpenShift, OpenStack, and OPNFV communities are used to illustrate different approaches and best practices. Two recently initiated automation initiatives are detailed: “Cross Community Continuous Integration” (XCI) and “Cross Testing” (Xtesting). Finally, some recommendations are provided for new projects as a guide to ease adoption of appropriate tools and methods.
TopIntroduction
Open Source communities have had a major influence on the evolution of the Internet with many projects initiated and developed in academia. Many key software projects are actually still distributed under associated permissive software licenses such as MIT license and BSD license with minimal restrictions about how the software can be redistributed.
Since mid-2000 the Open Source model has been widely adopted by many companies as a basis for commercial R&D activities. As a consequence the number of Open Source projects and contributors has increased dramatically, as shown in Figure 1. Many industry consortiums and foundations have also been created to support the increasing number of Open Source communities. Well-known foundations include the Linux Foundation, the Apache Foundation, OW2, and the Eclipse foundation.
This chapter deals with the global need for automation of most of the Open Source projects, which are now one of the catalysts of the Internet evolution (Web, relational and NOSQL databases, Big Data, Cloud, Artificial Intelligence and Machine Learning, etc.). It details the needs, the stakes, the tools, and a set of recommendations.
This chapter does not focus on the Open Source projects or communities working specifically on automation solutions especially for providers. For example, ETSI NFV (Gray & Nadeau, 2016) defined a full NFV Management and Orchestration stack (MANO) (NFV, 2018) aiming to allow the deployment of traditional complex provider’ functions. Some early code implementations more or less close to the specification have been published such as open Source Mano (Dahmen-Lhuissier, n.d.), OpenBaton (Open Baton, n.d.), or Open Networking Automation Platform (ONAP) (ONAP, n.d.). The description of these solutions is out of scope of this chapter.
Figure 1. The number of Linux Foundation Open Source projects has rapidly increased since mid-2000 (Linux, n.d.)
TopWhy Automate?: “L’Enfer C’Est Les Autres”
Changing the R&D approaches to rely heavily on Open Source code has introduced many challenges for developers. Firstly developers come from different companies with vastly different product technologies, software development standards, and code quality requirements.
“L’enfer c’est les autres” is a quote from “No Exit” (Sartre, 1955). An interpretation is that collaborative work is difficult, in part because others are needed to create your own identity. In order to magnify your own work however there is always the temptation to attribute the responsibility of the problems to “the others”. This precisely captures a dilemma within Open Source communities but also hints at the possibility of a way out. By making collectively agreed rules for code that are checked automatically for instance, a developer will more readily accept refactoring his/her code. Feedback that is automatically given based on predefined rules, even though the rules are defined by consensus and programmed by humans, is a priori “fair” and is hence a key benefit gained from Open Source collaboration.
The number of contributors working on a particular Open Source project differs dramatically, from 2 to 10000, depending on the popularity of the community, as illustrated in Figure 2.
Figure 2. Total contributions and contributions (April 2017 - April 2018) for a variety of Open Source projects (note that the y-axis scale is logarithmic)
The explosion of Open Source projects has subsequently led to increasing dependency on other Open Source projects and hence ever increasing complex dependencies need to be managed. Open Source project dependencies usually arise from integrating libraries or plugins from other projects. Several communities have established a cadence of six months for major releases. This fast pace of evolution means that continuous code refactoring, is usually needed to maintain compatibility and feature parity. Table 1 shows the rate of change of some Open Source projects and illustrates the great variability of source code that has to be dealt with.