Single SNN Architecture for Classical and Operant Conditioning using Reinforcement Learning

Introduction

Two of the most fundamental learning mechanisms known to exist in nature are classical conditioning (CC) and operant conditioning (OC). CC consists in strengthening the association between an unconditional stimulus (US), which automatically triggers a response, and a conditional stimulus (CS), which does not. When the CS is followed by the US systematically enough, the CS ends up triggering the response even when the US is not presented (Pavlov, 1927). OC consists in strengthening the association between a response and a reinforcement or a punishment (Skinner, 1938). If the association is between the response and the reinforcement, the frequency of the response increases. However, if the association is between the response and the punishment, the frequency of the response decreases. Generally, this sequence of a response followed by a reinforcement or a punishment will only be systematic within a given context. For example, an experimental design could be set within which a rat will only receive food when pressing a lever if a green light was presented first. In this situation, the behavior of the rat at the beginning of the experiment would be exploratory. However, once in a while, the rat will press the lever while the green light is presented and therefore, food will be given to the animal.

Separate spiking neural network (SNN) architectures were recently proposed as very low resource demanding implementations of CC and OC in robotic controllers/brains (Cyr et al., 2015; Dumesnil et al., 2016; Dumesnil et al., 2016, “Robotic”). SNNs use time stamping instead of rate coding to represent individual neural firings (Gerstner & Kistler, 2002), which makes SNNs naturally suited for CC and OC representation. Indeed, in order to implement CC and OC, it is necessary to detect delays between stimuli, responses, reinforcements and punishments. Neuronal spikes thus appear to be a good information transmission method for extracting those delays. The architectures presented in (Dumesnil et al., 2016) were simulated in very large scale hardware description language (VHDL) using an adapted version of the synapto-dendritic kernel adapting neuron (SKAN) model (Afshar et al., 2014). The latter allows implementing the delay extraction process with very few hardware resources (Afshar et al., 2014).