Methodological Research for Modular Neural Networks Based on “an Expert With Other Capabilities”

Introduction

In the system of nature and human society, the operation mode of physical and virtual worlds is dominated by some basic rules and patterns. For instance, the macroscopic world exists and evolves in a harmonious and orderly manner due to the internal effect of Newton's laws for the classical mechanics. The diversity and unity of the biology world have been determined by the principle of genetics in the life sciences and the determinism declared by Laplace has been terminated by the uncertainty in the microscopic world, etc.

Another important principle called “divide and conquer” has been confirmed by some discoveries in the brain-related sciences. Specifically, some researchers in neuroscience have found in human brain that there exist sparse connections between different neuronal groups where neurons are often densely connected and meanwhile that different response patterns are produced by the neuronal groups for different perceptual and cognitive tasks (Edelman, 1987; Fodor, 1983; Kandel, Schwartz, & Jessell, 2000). These two phenomena are respectively called the structural and functional modularity. These modularity evidences suggesting that domain-specific modules are required by the specific tasks and that a variety of modules can be coordinated for more complex tasks have reflected the principle of “divide and conquer” and have motivated the development of modular neural network (MNN) (Farooq, 2000). Therefore, it can be deemed that the principle of “divide and conquer” has given birth to MNN.

With the social development and the progress in science and technology, an interesting phenomenon has been widely witnessed in various fields, such as talent cultivation, corporate development, and especially in biology. For instance, in the microscopic aspect, it has been found that a gene called Mesp1, the main controlling factor for the development of cardiovascular, can activate the transcription factors of the heart, guide the cardiac mesoderm formation, and prevent the stem cells from differentiating into other cell types. In addition, it plays an important role in the growth of the blood and skeletal muscle (Chan et al., 2013). In the macroscopic aspect, Rapamycin, which was previously a medicine for the treatment of immune rejection after transplantation, not only helps to prolong the life of yeast, worms, fruit flies and other invertebrates, but also can be used in the targeted therapy of tumor (Harrison et al., 2009; Ma & Meng, 2010). In addition, this substance is even considered as the key factor in the evolution history of diverse species. Inspired by these discoveries, Wang et al. (2008) first proposed a principle called “an expert with other capabilities”, based on which a subnet training method in MNNs is then proposed by Wang and Wang (2012). However, few experiments have been carried out to test its effectiveness due to space constraint. Therefore, this paper is devoted to further describe this method and to testify its performance with more delicate experiments. In addition, the distance measure used in Wang and Wang (2012) is replaced by a new one, which proves theoretically and empirically more effective in general cases.

The paper is arranged as follows. The principles respectively initiating the research in MNNs and motivating the work of the authors are briefly introduced. Next, the recent advances in MNNs, reflecting the active research on MNNs, are introduced. Then, the MNN is illustrated in detail and a classical subnet training method in MNNs is introduced. Next, the principle of “an expert with other capabilities” is described in depth and then a new subnet training method as well as a specific algorithm are proposed. In addition, a relative distance measure is advanced and its superiority is theoretically proved. Then the authors present the performance of the proposed subnet learning method in the experiments associated with the approximation problem and prediction problem in dynamic systems. The superiority of the relative distance measure is also illustrated with the experiment results. Finally, this paper is concluded with some directions for future research.