In this section, we discuss multiple comparisons and ranking and selection procedures. These procedures are useful when we have “few” alternatives to evaluate, say up to approximately 100 alternatives or when one selects say the top 15 alternatives that have been identified through an optimization procedure such as OptQuest, or through the use of a metamodel of the simulation model.
Two of the major factors to consider in the application of these methodologies are 1) whether one is dealing with a terminating or a steady state simulation, and 2) which performance measure(s) is (are) being considered.
Teminating vs. Steady State Simulation
A terminating simulation is employed when there is a natural event that specifies the end of the simulation (Kelton et al., 2010). Often simulation models of service systems (e.g., the closing time of a restaurant might be 12 midnight) or of projects (e.g., the simulation model is terminated at the time that the project is finished as either a success or a failure) are terminating in nature.
A steady state simulation is typically used when dealing with a manufacturing/logistics system (such as a factory or a supply chain), for which one is interested in the long run performance of the system. Even for a factory which operates on a single shift, since the system remains the same from the end of one shift to the beginning of the next, a steady state analysis will usually be appropriate.
Analyzing a steady state simulation is typically more difficult than analyzing a terminating simulation since with a steady state simulation one must be concerned with how the effects of the initial state of the system bias the system output and also with the run length of the simulation. Typically, one will use either the approach of truncated replications (Kelton et. al, 2010, p. 324) or batching in a single run (Kelton et. al, 2010, p.325) for a steady state analysis. With truncated replications, one obtains a single sample value of the relevant performance measure from each replication of the simulation, following an appropriate warm up period for the replication.
With batching, a single replication of the simulation model is separated into sequential batches, again following an appropriate warm up period. One sample output is obtained from each batch. Of course, one must be sure to make the batches are large enough to make the correlation between the batches “small.”
With either approach (truncated replications or batching) an appropriate warm-up period (during which output data from the model is not used) must be estimated. The warm up period can differ from one alternative system to another. This warm-up period is determined through “eyeballing” a plot over time of an output from the model such as “work in process,” or through the use of a more sophisticated method such as the one due to Welch (1983).
The advantage of truncated replications as compared to batching is that one can easily obtain independent samples; however, this is at the cost of having to waste the warm-up period output for each replication. In general, if computational effort is not a concern, truncated replications is recommended.