Quantitative Productivity Analysis of a Domain-Specific Modeling Language

Introduction

Model-driven engineering (MDE) helps address the problems of designing, implementing, and integrating applications (Hästbacka, 2011)(Lukman, 2010)(Schmidt, 2006)(Hailpern, 2006)(Atkinson, 2003)(Kent, 2002). MDE is increasingly used in domains involving modeling software components, developing embedded software systems, and configuring quality-of-service (QoS) policies. Key benefits of MDE include (1) raising the level of abstraction to alleviate accidental complexities of low-level and heterogeneous software platforms, (2) more effectively expressing designer intent for concepts in a domain, and (3) enforcing domain-specific development constraints. Many documented benefits of MDE are qualitative, e.g., use of domain-specific entities and associations that are familiar to domain experts, and visual programming interfaces where developers can manipulate icons representing domain-specific entities to simplify development. There is a lack of documented quantitative benefits for domain-specific modeling languages (DSMLs), however, that show how developers are more productive using MDE tools and how development using DSMLs yields fewer bugs.

Conventional techniques for quantifying the benefits of MDE in general (e.g., comparing user-perceived usefulness of measurements for development complexity (Abrahao and Poels, 2007, 2009)) and DSMLs in particular (e.g., comparing elapsed development time for a domain expert with and without the use of the DSML (Loyall, Ye, Shapiro, Neema, Mahadevan, Abdelwahed, Koets, & Varner, 2004)) involve labor-intensive and time-consuming experiments. For example, control and experimental groups of developers may be tasked to complete a development activity during which metrics are collected (e.g., number of defects, time required to complete various tasks). These metrics also often require the analysis of domain experts, who may be unavailable in many production systems.

Even though DSML developers are typically responsible for showing productivity gains, they often lack the resources to demonstrate the quantitative benefits of their tools. One way to address this issue is via productivity analysis, which is a lightweight approach to quantitatively evaluating DSMLs that measures how productive developers are, and quantitatively exploring factors that influence productivity (Boehm, 1987) (Premraj, Shepperd, Kitchenham, & Forselius, 2005). This chapter applies quantitative productivity measurement using a case study of the Distributed QoS Modeling Language (DQML), which is a DSML for designing valid QoS policy configurations and transforming the configurations into correct-by-construction implementations. Our productivity analysis of DQML shows significant productivity gains compared with common alternatives, such as manual development using third-generation programming languages. While this chapter leverages DQML as a case study, in general the productivity gains and analysis presented are representative of DSMLs’ ability to reduce accidental complexity and increase reusability.

The remainder of this chapter includes the following objectives: highlighting related work; presenting an overview of DQML; outlining the DQML case study as a basis for DSMLs; describing productivity analysis for DSMLs leveraging the DQML case study; and presenting concluding remarks and lessons learned.