Evaluating Scalability of Neural Configurations in Combined Classifier and Attention Models

Neural Network Configurations and Recognition Algorithms

Many types of neural network classifiers and attention models can be found in the literature. Fortunately most can be described using a standard notation. Thus before we review begin with a review of classifiers, let’s define a standard notation. Let vector Y represent the activity of a set of labeled nodes that may be called output neurons or classes in different literatures and individually written as

They are considered supervised if the nodes can be labeled for example: Y₁ represents “dog”, Y₂ represents “cat”, and so on. Vector X represents sensory neurons or nodes that sample the environment and represent the input space to be recognized. These nodes represent input features and written as X=(X_1,X_2,X_3,... X_N)^T. The input features can be sensors that detect edges, lines, frequencies, kernel features, and so on. Output neurons are tuned to specific patterns so let’s expand the notation further. Assume neuron Y₁ is most-optimally tuned to an input pattern “A”. We will label the neuron with this pattern and write it as neuron Y_A. We describe its optimal input pattern as X_A where pattern “A” is represented by the feature sensory nodes with values X_A=(X_1A,X_2A,X_3A,... X_NA)^T.

Next, lets distinguish between supervised and unsupervised methods and phases in recognition. Unsupervised algorithms learn patterns without label constraints on Y and may find efficient representations (e.g. Olshausen & Fields, 1996; Hyvärinen et al., 2009). Although the unsupervised methods perform essential roles in dimensionality reduction and efficient coding, recognition is ultimately a process of associating labels with patterns. Without labeled associations the brain cannot interact with the world, e.g.: find food, mates, and hazards. Thus we initially focus on single-layer supervised models here. Future work will include comparisons of methods that include hidden layers and mixtures of supervised and unsupervised methods.

To distinguish between supervised recognition models, let’s define two phases in algorithm function: during recognition and learning. Recognition is when an algorithm finds values for outputs Y without modifying connection weights. Learning is when an algorithm modifies its connection weights (regardless of whether it calculates Y in the process). The most crucial aspect to understanding the difference between the models is the underlying neural configuration and when the model displays dynamics. Figure 1 compares basic neural connections between inputs and outputs.

Figure 1.

Comparison of Configurations. Using feedforward methods, information flows from inputs to output nodes (top). After feedforward processing lateral connections may connect between outputs or back to inputs of a different node (middle). Generative methods use Auto-Associative connections with symmetrical feedforward and feedback connections: each node projects back to its own inputs (bottom).

It is important to keep in mind the distinction of phases in algorithm function otherwise these configurations can be confusing. For example even though feedforward methods are feedforward during recognition, to implement their learning algorithms such as backprop (Rumelhart & McClelland, 1986) or a delta rule (Rosenblatt, 1958), auto-associative feedforward-feedback connections and dynamics are required. Thus feedforward methods are feedforward during recognition but feedforward-feedback during learning. Conversely, methods that are feedforward-feedback during recognition may use simple feedforward configurations during learning (Achler, 2012). Now we are ready to evaluate recognition algorithms.