Multi-Agent Reinforcement Learning for Value Co-Creation of Collaborative Transportation Management (CTM)

Introduction

Collaborative Transportation Management (CTM) is an emerging model of collaboration in the transportation area (Tyan, Wang, & Du, 2003). Previous researchers developed CTM based on the main objective to increase the overall performance by using information sharing and resources sharing in their collaboration mechanisms. Several researchers such as Browning & White (2000), Sutherland (2003), Esper and Williams (2003) state the needs to incorporate CTM into logistics to avoid logistics bottlenecks, reduce inefficiency, reduce transportation costs, and provide mutual benefits for all collaborative parties.

CTM is developed among shipper, receiver, and carrier. In this research actors in CTM represented as agents. An agent is viewed as individual heterogeneity that represents agents’ decision rules that consists of beliefs, capabilities, choices, and commitments in the environment (Gilbert, 2008). Collaboration processes play an important role in the interactions among agents to understand the joint creation of value. Vargo and Lusch (2004) state that in a collaboration process, value is determined and created by each agent. According to Grönroos (2008), value co-creation only occurs when service providers extend their roles not only in providing services but also in influencing their services in accordance with the customers’ value.

Many researchers in CTM focus on the optimal solution for the agents’ collaborations that are carried out through mathematical modellings, computer simulations, and empirical researches (Okdinawati, Simatupang, & Sunitiyoso, 2015). However, the optimal solution in CTM does not answer how the interactions among agents’ collaborations leads to value co-creation and increase the agility and visibility if each agent’s collaboration has different objectives and different perspectives.

Multi-agent system is used in this research to model interaction among agents and learning process from their experience in synchronizing and coordinating collaboration processes, in order to share the benefit of CTM among them. Multi-agent system for CTM is formulated in vertical collaboration among shippers, receivers, and carriers to optimize the benefits for all agents through value co-creation. The information sharing mechanism is used as a foundation to make the effective decisions in this collaboration. Decision making is performed jointly by agents and is implemented in the process by one or more agents involved.

The benefit for CTM is associated with the value captured by all agents in collaboration processes. There are three benefits that are measured, namely, transportation cost, visibility, and agility. Transportation cost is a monetary measurement of operation expenses that must be spent by a carrier. Visibility is the available data that can be used to make decisions and determine strategies by all agents. Agility represents how fast all agents in CTM can respond and adapt to the changes in the environment.

The purpose of this paper is to increase understanding of value co-creation in CTM and analyze how value is co-created among agents in CTM. It provides knowledge by giving empirical evidence what kinds of activities and collaboration processes constitute value co-creation processes and how agents can learn in a collaboration process. The empirical findings present evidence on the interaction among agents, the relationship among agents, and the role of each agent in value co-creation processes.

Agent-based model in this research is adopted to model the effects of interactions among agents in CTM, where value is co-created, as well as models the roles and actions among agents in a collaboration stage in order to create value. Four scenarios are used in this research based on empirical research and based on an ideal condition. In order to model learning process of all agents in CTM, the model-free reinforcement learning method is employed.

The rest of the paper is organized as follows. First, the related literatures are briefly reviewed. Next, the problem situation of the empirical research is explained. Then the simulation process is presented. Afterwards, the simulation results and their discussion that illustrate this research are presented. The last, conclusions, and recommendations for future research conclude this paper