Awareness Based Recommendation: Passively Interactive Learning System

Interactive Reinforcement Learning With Human

A long term goal of interactive learning system is to incorporate human to solve complex tasks. Reinforcement learning is the Standard behavior learning method for among robot, animal and human. In interactive reinforcement learning, there are two roles, a learner and a trainer. The input of a reinforcement learner as a learning goal is called a reward, and the output of the learner as a learning result is called a policy. For example, as training a dog by a human trainer, Peterson (2000, 2001) showed that clicker training is an easy way to shape new behaviors. When a dog performs a new behavior to learn, the trainer clicks the clicker as a positive reward. Pryor (2006) remarks that clicker training is a method for training an animal that uses positive reinforcement in conjunction with a clicker to mark the behavior being reinforced under behavior modification principles.

In current researches of interactive reinforcement learning, there are two approaches to support a learner by giving feedback as, whether a learning goal (reward based), or a learning result (policy based). The former approach is clicker training for the robot, in that a human trainer gives a learning goal to the robot learner. In field of robot learning, Kaplan et al. (2002) showed that interactive reinforcement learning method in that reward function denoting goal is given interactively has worked to establish the communication between a human and the pet robot AIBO. The main feature of this method is the interactive reward function setup which was fixed and build-in function in the main feature of previous reinforcement learning methods. So the user can sophisticate reinforcement learner’s behavior sequences incrementally.

Ng et al. (1999) and Konidaris & Barto (2006) showed that reward shaping is the theoretical framework of such interactive reinforcement learning methods. Shaping is to accelerate the learning of complex behavior sequences. It guides learning to the main goal by adding shaping reward functions as subgoals. Previous reward shaping methods have three assumptions on reward functions as following:

• • Main goal is given or known for the designer;

• • Marthi (2007) remarks that subgoals are assumed as shaping rewards those are generated by potential function to the main goal;

• • Ng et al. (1999) showed that shaping rewards are policy invariant, it means not affecting the optimal policy of the main goal.

However, these assumptions will not be true on interactive reinforcement learning with a non-expert end-user. Main reason is discussed by Griffith et al. (2013) that human feedback signals may be inconsistent with the optimal policy. It is not easy to keep these assumptions while the end-user gives rewards for the reinforcement learning agent. It is that the reward function may not be fixed for the learner if an end-user changes his/her mind or his/her preference. However, most of previous reinforcement learning methods assume that the reward function is fixed and the optimal solution is unique, so they will be useless in interactive reinforcement learning with an end-user.

To avoid this problem, the latter approaches are that a human trainer provides a sample of learning result to the robot learner. For robot learning with human, inverse reinforcement learning proposed by Ng & Russell (2000) is the method that after the human provides demonstrations of an optimal policy, the reward function for the demonstrations is generated to learn the optimal policy. Another approach is called policy shaping proposed by Griffith et al. (2013). Instead of requiring demonstrations, it allows a human trainer to simply critique the learner’s behavior (“that was right/wrong”). Thus the human’s feedback is a label on the optimality of actions of each state.

To introduce our approach, we organize reinforcement learning methods. Table 1 shows the characteristics on interactive reinforcement learning. In reinforcement learning, an optimal solution is decided by the reward function and the optimality criteria. In standard reinforcement learning, an optimal solution is fixed since both the reward function and the optimality criteria are fixed. On the other hand, in interactive reinforcement learning, an optimal solution may change according to the interactive reward function. Furthermore, in interactive reinforcement learning with human, various optimal solutions will occur since the optimality criteria depend on human's preference.

Then the objective of this research is to recommend preferable solutions of each user. The main problem is “how to guide to estimate the user’s preference?” Our solution consists of two ideas. One is to prepare various solutions by every-visit-optimality proposed by Satoh & Yamaguchi (2006), another is the coarse to fine recommendation strategy proposed by Yamaguchi, Nishimura & Sato (2011). Our approach considers a human as a novice trainer. First, the novice trainer inputs initial learning goals, then the learning system generates and suggests the candidates of the optimal leaning result to the novice trainer in order to make clear his/her final learning goals.