In this chapter, the system defined by a subcontractor, which processes orders generated from multiple customers, is cast into the dynamic single-machine sequencing framework. The subcontractor is modeled as a production/inventory facility that consists of a single, failure-free server and an input buffer with infinite capacity. Jobs arrive continuously to the system at random intervals and have stochastic processing times and due dates. A setup is required when switching from one job type to another. The behaviour of the system, when operating under various rescheduling policies and job dispatching schemes, is investigated using discrete-event simulation in a series of experiments. A reinforcement learning-based approach is proposed for dynamic dispatching rule selection under the completely reactive rescheduling policy. The reinforcement learning controller is applied to two simulation cases with the aim of generating state-dispatching rule mappings that minimize mean tardiness and mean lateness.
TopIntroduction
Outsourcing can be defined as the practice of contracting out a business function, e.g. a manufacturing process, to an external provider. In many practical situations, this external provider offers its services to multiple parties. In the case where all customers are of equal importance, orders from different customers ‘compete’ for the same resource and, the subcontractor has the liberty to sequence orders at will in an attempt to enhance or optimize some locally specified performance indicator. However, the local sequencing decisions made by the external provider have the potential to influence the efficiency of the supply chain as a whole, especially if the subcontracted functions are part of a bottleneck stage in the supply chain. Therefore, the investigation of the subcontractor’s policies and methods for determining the sequencing of incoming orders is a non-trivial research topic, even if studied in isolation, as it constitutes a building block in the development of a comprehensive understanding of the mechanics of the supply chain with outsourcing options.
In this chapter, the system defined by a subcontractor which processes orders generated from multiple customers is cast into the dynamic single-machine sequencing framework. The subcontractor is modeled as a production/inventory facility that consists of a single, failure-free server and an input buffer with infinite capacity. There is no finished goods buffer; once an order has completed its processing it exits the system immediately and therefore the server is never blocked. The server can process orders/jobs of multiple types and switching from one job to another incurs a sequence-dependent idle time for setup. Jobs arrive dynamically to the system at random intervals and have stochastic processing times and due dates.
In this setting, generating complete schedules in advance is not possible since new information arrives continuously to the system and therefore, scheduling can be done following either one of two different policies: event-driven rescheduling and time-driven rescheduling. In the event-driven rescheduling context short-term schedules are constructed and subjected to revisions in time points where specific system-related events occur, i.e. the scheduler/controller is invoked in response to an unforeseen event that alters the system state. In time-driven rescheduling the current schedule is modified periodically, that is, the time interval between two successive rescheduling epochs is fixed. Completely reactive scheduling is a special case of event-driven scheduling where the controller is completely myopic, i.e. it does not construct a schedule at all but simply selects the next job to be processed.
The sequence of the orders is determined with the use of dispatching rules. A dispatching rule is a heuristic which simply assigns a priority index to each one of the queueing jobs. The job with the maximum priority is selected to be processed next. The performance of a specific dispatching rule varies analogously to the objective function considered and the configuration as well the scheduling policy of the system that it is applied to.
For different instances of the problem a distinct rule is bound to be the optimal among the set of all rules when the performance of the system is measured in the long-run by a single criterion. However due to the stochasticity of the system there will be transients where the load and due date allowance conditions will deviate significantly from the mean and therefore it would be preferable to schedule jobs with another dispatching rule. To this end, the authors develop and propose a Reinforcement Learning (RL) based controller that shifts dynamically from one dispatching rule to another according to the current system state. In order to analyze the performance of the production/inventory system discrete-event simulation is utilized.
The numerical results presented in this chapter are divided into two parts. In the first part, the behavior of the system when operating under different rescheduling policies and dispatching rules is studied in a hypothetical scenario. Six rescheduling policies and seven dispatching rules are considered. The objective measures of interest are mean tardiness/lateness and reschedule rate. In the second part of this investigation we consider only completely reactive scheduling and apply the Reinforcement Learning-based controller with the goal of minimizing mean tardiness and mean lateness separately. The performance of the proposed controller is tested in the same simulation case as before in addition to a variant of it with lower order arrival rates.