In line with the advancement of hardware technology and increasing consumer demands for new functionalities and innovations, software applications grew tremendously in term of size over the last decade. This sudden increase in size has a profound impact as far as testing is concerned. Here, more and more unwanted interactions among software systems components, hardware, and operating system are to be expected, rendering increased possibility of faults. To address this issue, many useful interaction-based testing techniques (termed t-way strategies) have been developed in the literature. As an effort to promote awareness and encourage its usage, this chapter surveys the current state-of-the-art and reviews the state-of-practices in the field. In particular, unlike earlier work, this chapter also highlights the different possible adoptions of t-way strategies including uniform interaction, variable strength interaction, and input-output-based relation, that is, to help test engineers make informed decision on the actual use of t-way strategies.
Top2. T-Way Testing Fundamentals
In general, t-way testing can be abstracted to a covering array. Throughout out this chapter, the symbols: p, v, and t are used to refer to number of parameters (or factor), values (or levels) and interaction strength for the covering array respectively. Earlier works suggested two definitions for describing the covering array (Cohen, 2004). The first definition is based on whether or not the numbers of values for each parameter are equal. If the number of values is equal (i.e. uniformly distributed), then the test suite is called Coverage Array (CA). Now, if the number of values in non-uniform, then the test suite is called Mixed Coverage Array (MCA). Unlike earlier work in the literature, this chapter unifies the CA and MCA notations in order to include that of variable strength covering array (VCA), and input output based covering array (IOR).
Normally, the CA takes parameters of N, t, p, and v respectively (i.e. CA(N,t, v p)). For example, CA (9, 2, 2 4) represents a test suite consisting of 9x4 arrays (i.e. the rows represent the size of test cases (N), and the column represents the parameter (p)). Here, the test suite also covers pairwise interaction for a system with 4 2 valued parameter (see Figure 1). When the CA is the most optimal result, the covering array can be rewritten as CAN (N,t, v p).
Alternatively, MCA takes parameters of N, t, and Configuration (C). In this case, C captures the parameters and values of each configuration in the following format: v1p1 v2p2, ….. vnpn indicating that there are p1 parameters with v1 values, p2 parameters with v2 values, and so on. For example, MCA (8,3,2311) indicates the test size of 8 which covers 3-way interaction. Here, the configuration takes 4 parameters: 3 2 valued parameter and 1 1 valued parameter (see Figure 2).
In the case of VCA, the parameter consists of N, t, C, and Set (S) (i.e. VCA (N,t,C,S)). Similar to MCA, N,t, and C carry the same meaning. Set S consists of a multi-set of disjoint covering array with strength larger t. For instance (see Figure 3), VCA (12, 2, 3222, (Zamli, et al., 2011)) indicates the test size of 12 for pairwise interaction (with 2 3 valued parameter and 2 2 valued parameter) and 3-way interaction (with 1 3 valued parameter and 2 2 valued parameter).