Abstract
Open source communities have had and continue to have a major influence on the evolution of the Internet. By their nature, such communities involve people with diverse coding cultures and skills. Automation has consequently been of major interest to open source software developers for a long time, and many open source tools have been developed to address code variability and sustainability challenges. This chapter discusses why open source communities must automate and the challenges they will face. Solutions and current examples of automation in open source projects are provided as a guide to what is achievable. OpenShift, OpenStack, and OPNFV communities are used to illustrate different approaches and best practices. Two recently initiated automation initiatives are detailed: “Cross Community Continuous Integration” (XCI) and “Cross Testing” (Xtesting). Finally, some recommendations are provided for new projects as a guide to ease adoption of appropriate tools and methods.
TopIntroduction
Open Source communities have had a major influence on the evolution of the Internet with many projects initiated and developed in academia. Many key software projects are actually still distributed under associated permissive software licenses such as MIT license and BSD license with minimal restrictions about how the software can be redistributed.
Since mid-2000 the Open Source model has been widely adopted by many companies as a basis for commercial R&D activities. As a consequence the number of Open Source projects and contributors has increased dramatically, as shown in Figure 1. Many industry consortiums and foundations have also been created to support the increasing number of Open Source communities. Well-known foundations include the Linux Foundation, the Apache Foundation, OW2, and the Eclipse foundation.
This chapter deals with the global need for automation of most of the Open Source projects, which are now one of the catalysts of the Internet evolution (Web, relational and NOSQL databases, Big Data, Cloud, Artificial Intelligence and Machine Learning, etc.). It details the needs, the stakes, the tools, and a set of recommendations.
This chapter does not focus on the Open Source projects or communities working specifically on automation solutions especially for providers. For example, ETSI NFV (Gray & Nadeau, 2016) defined a full NFV Management and Orchestration stack (MANO) (NFV, 2018) aiming to allow the deployment of traditional complex provider’ functions. Some early code implementations more or less close to the specification have been published such as open Source Mano (Dahmen-Lhuissier, n.d.), OpenBaton (Open Baton, n.d.), or Open Networking Automation Platform (ONAP) (ONAP, n.d.). The description of these solutions is out of scope of this chapter.
Figure 1. The number of Linux Foundation Open Source projects has rapidly increased since mid-2000 (Linux, n.d.)
TopWhy Automate?: “L’Enfer C’Est Les Autres”
Changing the R&D approaches to rely heavily on Open Source code has introduced many challenges for developers. Firstly developers come from different companies with vastly different product technologies, software development standards, and code quality requirements.
“L’enfer c’est les autres” is a quote from “No Exit” (Sartre, 1955). An interpretation is that collaborative work is difficult, in part because others are needed to create your own identity. In order to magnify your own work however there is always the temptation to attribute the responsibility of the problems to “the others”. This precisely captures a dilemma within Open Source communities but also hints at the possibility of a way out. By making collectively agreed rules for code that are checked automatically for instance, a developer will more readily accept refactoring his/her code. Feedback that is automatically given based on predefined rules, even though the rules are defined by consensus and programmed by humans, is a priori “fair” and is hence a key benefit gained from Open Source collaboration.
The number of contributors working on a particular Open Source project differs dramatically, from 2 to 10000, depending on the popularity of the community, as illustrated in Figure 2.
Figure 2. Total contributions and contributions (April 2017 - April 2018) for a variety of Open Source projects (note that the y-axis scale is logarithmic)
Key Terms in this Chapter
Cloud Native: Cloud native is a term used to qualify applications designed and implemented in and for the cloud. It goes beyond the virtualization of applications by integrating cloud concepts as defined in the cloud computing manifesto. It usually requires to fully redesign applications in order to split atomic functions into isolated virtual resources (micro-services).
Continuous Integration: In software engineering, continuous integration (CI) is the practice of merging all developer code contributions to a shared mainline several times a day. The term was introduced in 1991 by Grady Booch, known for his work on oriented object programming concepts.
Open Source: The open source model is a decentralized software-development model that encourages open collaboration and imposes the use of an open source license for the source code. Open source licenses must guarantee that the source code, blueprint or design can be used, modified and/or shared under defined terms and conditions. Open source is sometimes considered as a more industry-oriented rephrasing of the free software model. The free software foundation created in 1985 considers the freedom as the cornerstone of the model. The open source initiative officially announced in 1998 puts the efficiency as the key point of the collaborative model.
Virtualization: In computing, virtualization refers to the act of creating virtual resources, including virtual computer hardware platforms, storage devices, and computer network resources. Virtualization is as old as computer science but became very popular with the hardware improvement in particular in chipset industry and the development of Linux. The different virtualization technologies are heavily used by the cloud computing industry. Virtualization is also a key driver for the development of the web technologies especially on distributed storage, big data, machine learning, and artificial intelligence.
Test-Driven Development: Test-driven development (TDD) is a software development approach consisting in writing the tests prior to the code. TDD ensures that the source code is thoroughly unit tested and leads to modularized, flexible and extensible code. It aims to simplify and clarify the software design as only the code necessary to pass tests has to be written.
Gating: In software engineering, the gating consists in different check points triggered by an automation system, potentially in parallel, before merging any change to the reference source code. Gating aims to ensure the good quality of the code by running test suites and linting operations. It is part of the CI.
Continuous Delivery: The continuous delivery (CD) concept, defined by Martin Fowler in 2013, is a software development discipline where software can be promoted from release candidate to production environment automatically. It has been largely enriched since 2010 thanks to the development and the adoption of cloud and virtualization technologies. CD is usually associated with the devops approach for the methodology and the “on demand” aspect for the business.
Linux Foundation: Founded in 2000, The Linux Foundation is dedicated to building sustainable ecosystems around open source projects to accelerate technology development and industry adoption. It is among the largest open source non-profit organization. It works to promote, protect, and advance Linux and collaborative development through different and unparalleled initiatives such as the Cloud Native Computing Foundation, OPENAPI initiative, OPEN CONTAINER initiative, OPNFV, or ONAP projects.