A Selective Sparse Coding Model with Embedded Attention Mechanism

A Selective Sparse Coding Model with Embedded Attention Mechanism

Qingyong Li (Beijing Jiaotong University, China), Zhiping Shi (Chinese Academy of Sciences, China) and Zhongzhi Shi (Chinese Academy of Sciences, China)
Copyright: © 2009 |Pages: 13
DOI: 10.4018/978-1-60566-170-4.ch005
OnDemand PDF Download:


Sparse coding theory demonstrates that the neurons in the primary visual cortex form a sparse representation of natural scenes in the viewpoint of statistics, but a typical scene contains many different patterns (corresponding to neurons in cortex) competing for neural representation because of the limited processing capacity of the visual system. We propose an attention-guided sparse coding model. This model includes two modules: the non-uniform sampling module simulating the process of retina and a data-driven attention module based on the response saliency. Our experiment results show that the model notably decreases the number of coefficients which may be activated, and retains the main vision information at the same time. It provides a way to improve the coding efficiency for sparse coding model and to achieve good performance in both population sparseness and lifetime sparseness.
Chapter Preview


Understanding and modeling the functions of the neurons and neural systems are one of the primary goals of cognitive informatics (CI) [Wang 2002, 2007; Wang and Kinsner 2006]. The computational capabilities and limitations of neurons, and the environment in which the organism lives are two fundamental components driving the evolution and development of such systems. The researchers have broadly investigated them.

The utilization of environmental constraints is most clearly evident in sensory systems, where it has long been assumed that neurons are adapted to the signals to which they are exposed [Simoncelli 2001]. Because not all signals are equally like each other, it is natural to assume that perceptual systems should be able to best process those signals that occur most frequently. Thus, it is the statistical properties of the environment that are relevant for sensory process of vision perception [Field 1987; Simoncelli 2003].

Efficient coding hypothesis [Barlow 1961] provides a quantitative relationship between environmental statistics and neural processing. Barlow for the first time hypothesized that the role of early sensory neurons was to remove statistical redundancy in the sensory input. Then, Olshausen and Field put forward a model, called sparse coding, which made the variables (equivalence of neurons stimulated by the same stimulus in the neurobiology) be activated (i.e., significantly non-zero) only rarely [Olshausen 1996]. This model is named SC here. Vinje’s results validated the sparse properties of neural responses under natural stimuli conditions [Vinje 2000]. Afterwards, Bell brought forward another sparse coding model based on statistical independence (called SCI) and obtained the same results as Olshausen and Field’s model [Bell 1997]. More recent studies can be seen in survey [Simoncelli 2003]. (Figure 1 and Figure 2)

Figure 1.

The diagram of the model

Figure 2.

Basis functions randomly selected from the set. (a) the original basis functions produced by sparse coding model; (b) the corresponding binary basis functions with distinct excitatory subregion labeled with white

However, Willmore and Tolhurst [Willmore 2001] argued that there were two different ways for 'sparseness': population sparseness and lifetime sparseness. Population sparseness describes codes in which few neurons are active at any time and it is utilized in Olshausen and Field’s sparse coding model [Olshausen 1996]; while lifetime sparseness describes codes in which each neuron's lifetime response distribution has high kurtosis, which is the main contribution in Bell’s sparse coding model [Bell 1997]. In addition, it is proved that lifetime sparseness was uncorrelated with population sparseness. Just as Figure 3 (a) shows the number of variables, which have large values produced by sparse coding model and are possible to be activated, is relatively large compared with the computation capacity of neurons. Though, the kurtosis of every response coefficient is also high. So, how to reduce both population sparseness and lifetime sparseness at the same time to retain the important information as much as possible is a valuable problem in practice.

Figure 3.

Response coefficients for input stimulus. a)Coefficients produced by sparse coding model; b) Response saliency value for every simple cell; c) Coefficients produced by AGSC-P model; d) Coefficients produced by AGSC-T model

Complete Chapter List

Search this Book:
Editorial Advisory Board
Table of Contents
Yingxu Wang
Chapter 1
Yingxu Wang
Cognitive Informatics (CI) is a transdisciplinary enquiry of the internal information processing mechanisms and processes of the brain and natural... Sample PDF
The Theoretical Framework of Cognitive Informatics
Chapter 2
Withold Kinsner
This chapter provides a review of Shannon and other entropy measures in evaluating the quality of materials used in perception, cognition, and... Sample PDF
Is Entropy Suitable to Characterize Data and Signals for Cognitive Informatics?
Chapter 3
Ismael Rodríguez, Manuel Núñez, Fernando Rubio
Finite State Machines (FSM) are formalisms that have been used for decades to describe the behavior of systems. They can also provide an intelligent... Sample PDF
Cognitive Processes by using Finite State Machines
Chapter 4
Yingxu Wang
An interactive motivation-attitude theory is developed based on the Layered Reference Model of the Brain (LRMB) and the Object-Attribute-Relation... Sample PDF
On the Cognitive Processes of Human Perception with Emotions, Motivations, and Attitudes
Chapter 5
Qingyong Li, Zhiping Shi, Zhongzhi Shi
Sparse coding theory demonstrates that the neurons in the primary visual cortex form a sparse representation of natural scenes in the viewpoint of... Sample PDF
A Selective Sparse Coding Model with Embedded Attention Mechanism
Chapter 6
Yingxu Wang
Theoretical research is predominately an inductive process, while applied research is mainly a deductive process. Both inference processes are based... Sample PDF
The Cognitive Processes of Formal Inferences
Chapter 7
Douglas Griffith, Frank L. Greitzer
The purpose of this article is to re-address the vision of human-computer symbiosis as originally expressed by J.C.R. Licklider nearly a... Sample PDF
Neo-Symbiosis: The Next Stage in the Evolution of Human Information Interaction
Chapter 8
Ray E. Jennings
Although linguistics may treat languages as a syntactic and/or semantic entity that regulates both language production and comprehension, this... Sample PDF
Language, Logic, and the Brain
Chapter 9
Yingxu Wang, Guenther Ruhe
Decision making is one of the basic cognitive processes of human behaviors by which a preferred option or a course of actions is chosen from among a... Sample PDF
The Cognitive Process of Decision Making
Chapter 10
Tiansi Dong
This chapter proposes a commonsense understanding of distance and orientation knowledge between extended objects, and presents a formal... Sample PDF
A Commonsense Approach to Representing Spatial Knowledge Between Extended Objects
Chapter 11
Natalia López, Manuel Núñez, Fernando L. Pelayo
In this chapter we present the formal language, stochastic process algebra (STOPA), to specify cognitive systems. In addition to the usual... Sample PDF
A Formal Specification of the Memorization Process
Chapter 12
Yingxu Wang
Autonomic computing (AC) is an intelligent computing approach that autonomously carries out robotic and interactive applications based on goal- and... Sample PDF
Theoretical Foundations of Autonomic Computing
Chapter 13
Witold Kinsner
Numerous attempts are being made to develop machines that could act not only autonomously, but also in an increasingly intelligent and cognitive... Sample PDF
Towards Cognitive Machines: Multiscale Measures and Analysis
Chapter 14
Amar Ramdane-Cherif
Cognitive approach through the neural network (NN) paradigm is a critical discipline that will help bring about autonomic computing (AC). NN-related... Sample PDF
Towards Autonomic Computing: Adaptive Neural Network for Trajectory Planning
Chapter 15
Lee Flax
We give an approach to cognitive modelling, which allows for richer expression than the one based simply on the firing of sets of neurons. The... Sample PDF
Cognitive Modelling Applied to Aspects of Schizophrenia and Autonomic Computing
Chapter 16
Yan Zhao, Yiyu Yao
Classification is one of the main tasks in machine learning, data mining, and pattern recognition. Compared with the extensively studied automation... Sample PDF
Interactive Classification Using a Granule Network
Chapter 17
Mehdi Najjar, André Mayers
Encouraging results of last years in the field of knowledge representation within virtual learning environments confirms that artificial... Sample PDF
A Cognitive Computational Knowledge Representation Theory
Chapter 18
Du Zhang
A crucial component of an intelligent system is its knowledge base that contains knowledge about a problem domain. Knowledge base development... Sample PDF
A Fixpoint Semantics for Rule-Base Anomalies
Chapter 19
Christine W. Chan
This chapter presents a method for ontology construction and its application in developing ontology in the domain of natural gas pipeline... Sample PDF
Development of an Ontology for an Industrial Domain
Chapter 20
Václav Rajlich, Shaochun Xu
This article explores the non-monotonic nature of the programmer learning that takes place during incremental program development. It uses a... Sample PDF
Constructivist Learning During Software Development
Chapter 21
Witold Kinsner
Many scientific chapters treat the diversity of fractal dimensions as mere variations on either the same theme or a single definition. There is a... Sample PDF
A Unified Approach to Fractal Dimensions
Chapter 22
Du Zhang, Witold Kinsner, Jeffrey Tsai, Yingxu Wang, Philip Sheu, Taehyung Wang
The 2005 IEEE International Conference on Cognitive Informatics (ICCI’05) was held during August 8th to 10th 2005 on the campus of University of... Sample PDF
Cognitive Informatics: Four Years in Practice
Chapter 23
Yiyu Yao, Zhongzhi Shi, Yingxu Wang, Witold Kinsner, Yixin Zhong, Guoyin Wang
Cognitive informatics (CI) is a cutting-edge and multidisciplinary research area that tackles the fundamental problems shared by modern informatics... Sample PDF
Toward Cognitive Informatics and Cognitive Computers: A Report on IEEE ICCI'06
About the Contributors