Current Trends and Challenges
Enterprise distributed real-time and embedded (DRE) systems, such as large-scale traffic management systems, supervisory control and data analysis (SCADA) systems, and shipboard computing environments, are becoming increasingly ubiquitous. As these systems grow in scale and complexity they are becoming ultra-large-scale cyber-physical systems (Institute, 2006), which exhibit the following characteristics:
- •
Requirements for simultaneously satisfying competing and conflicting QoS properties, such as low latency, high reliability, and fault tolerance, in addition to meeting their functional requirements.
- •
Heterogeneous in both their operating environment (e.g., target architecture and hardware resources) and technologies (e.g., programming language and middleware).
- •
Developed as monolithic, vertically integrated stovepipes, which are brittle, and hard to implement, maintain, and evolve.
These and other related characteristics not only increase enterprise DRE system complexity, but also complicate their development, certification, and sustainment, resulting in elongated lifecycles characterized by expensive project runs and missed delivery deadlines (Mann, 1996).
Due to increased complexity and size of enterprise DRE systems and their lifecycles, it is important to validate enterprise DRE system QoS properties early in their lifecycle, rather than waiting until system integration (i.e., late in the lifecycle), when they are expensive and time-consuming to fix. System execution modeling (SEM) tools (Smith & Williams) are one method for enabling DRE system developers and testers to validate QoS properties during early phases of the lifecycle. In particular, SEM tools provide enterprise DRE system developers and testers with the following capabilities:
- •
Rapidly model behavior and workload of the distributed system being developed, independent of its programming language or target environment, e.g., the underlying networks, operating system(s), and middleware platform(s).
- •
Synthesize a customized test system from models, including representative source code for the behavior and workload models and project/workspace files necessary to build the test system in its target environment.
- •
Execute the synthesized test system on a representative target environment testbed to produce realistic empirical results at scale.
- •
Analyze the test system's QoS in the context of domain-specific constraints, e.g. as scalability or end-to-end response time of synthesized test applications, to identify performance anti-patterns (Smith & Williams), which are common system design mistakes that degrade end-to-end QoS.
By using SEM tools to validate QoS properties throughout the lifecycle, DRE system testers can locate and resolve QoS bottlenecks in a timely and cost-effective manner.
Although SEM tools enable early validation of QoS properties, enterprise DRE system testers may lack resources needed to support testing efforts. For example, to support early integration testing of an enterprise DRE system, testers need hardware resources that may not be readily available until later in the lifecycle. Likewise, both software and hardware resources may change throughout the lifecycle. The number of test that must execute also often exceeds the operational capacity of resources available in-house for testing (Porter, Yilmaz, Memon, Schmidt, & and B. Natarajan, 2007). Enterprise DRE system testers therefore need improved methods to support early integration testing of enterprise DRE system applications.