GNG [60] is an unsupervised incremental self-organising network independent of the topology of the input distribution or space. It uses a growth mechanism inher- ited from the Growth Cell Structure [59] together with the Competitive Hebbian Learning (CHL) rule [118] to construct a network of the input date set. In some cases the probability distribution of the input data set is discrete and is given by the characteristic function ξw : Rq → {0, 1} with ξw defined by
ξw = 1 if ξ ∈ W 0 if ξ ∈ Wc (3.11)
CHAPTER 3. OBJECT REPRESENTATION WITH SELF-ORGANISING NEURAL NETWORKS
In the network ξw represents the random input signal generated from the set W ⊆
Rq and Wc is the complement of W ∈ Rq. The growing process starts with two nodes, and new nodes are incrementally inserted until a predefined conditioned is satisfied, such as the maximum number of nodes or available time. During the learning process local error measures are gathered to determine where to insert new nodes. New nodes are inserted near the node with the highest accumulated error and new connections between the winner node and its topological neigh- bours are created.
The GNG algorithm consists of the following:
• A set A of cluster centres known as nodes. Each node c ∈ N has its associ- ated reference vector {xc}Nc=1 ∈ Rq. The reference vectors indicate the nodes’
position or receptive field centre in the input distribution. In our examples, the input probability distribution is a discrete distribution (Figures 5.3, 5.7) and a mixture of Gaussians probability density function representing skin colour (Figure 4.4). The nodes move towards the input distribution by adapting their position to the input’s geometry using a winner take all mapping. Gen- erating ξw input signals from the random vector W , we want to find a map-
ping G : Rq −→ RAand its inverse F : RA−→ Rqsuch that ∀c = 1, ..., | N |,
f (x) = EW |g(W ){W |g(W ) = x}, ∀x ∈ {xc}Nc=1⊆ Rq (3.12)
g(W ) = arg min
ν∈{xc}Nc=1
kW − xνk (3.13)
where E is the distance operator of the data points from the random vector W projecting onto f (x), g(W ) is the projection operator, {xc}Nc=1 ⊆ Rqare the
reference vectors of the network and xν is the winner node. Equations (3.12)
and (3.13) show that while the forward mapping G is approximated as a pro- jection operator, the reverse mapping F is nonparametric and depends on
CHAPTER 3. OBJECT REPRESENTATION WITH SELF-ORGANISING NEURAL NETWORKS
the unknown latent variable x. In order to compute f (x) the GNG algorithm evaluates (3.12) and (3.13) in an iterative manner. q and A denote the dimen- sionality of the input space and the reduced latent topology. In this work, current experiments include topologies of a line which is the contour of the object (A = 1) and triangular grid which is the topology preserving graph (A = 2). Figure 3.2 shows an example of a 1D GNG network.
Figure 3.2: Every sample point w on the target space is defined as the best matching of all nodes x projecting within a topological neighbourhood of w. For example, the best matching node denoted by the largest arrow, moves towards the sample point while its topological neighbors adjust their position.
Figure 3.3 shows an example of a 2D GNG network with its associated Voronoi diagram in 2D discrete distribution.
• Local accumulated error measurements and insertion of nodes. Each node c ∈ N with its associated reference vector {xc}Nc=1 ∈ Rqhas an error variable
Exc which is updated at every iteration according to:
∆Exν = kξw − xνk
2
CHAPTER 3. OBJECT REPRESENTATION WITH SELF-ORGANISING NEURAL NETWORKS
Figure 3.3: A random signal ξw on the discrete input distribution and the best
matching within the topological neighbourhood of {xc}Nc=1 ⊆ Rq. In this example,
the green node is the winner node of the network among its direct topological neighbours (orange and yellow nodes). The orange node is the second nearest node to the random signal ξw.
The local accumulated error is a statistical measure and is used for the inser- tion and the distribution of new nodes. Nodes with larger errors will cover greater area of the input probability distribution, since their distance from the generated signal is updated by the squared distance. Knowing where the error is large, if the number of the associated reference vectors belonging to the input space is an integer multiple of a parameter λ, a new node xris in-
serted halfway between the node with the largest local accumulated error xq
and its neighbour xf.
xr=
xq+ xf
2 (3.15)
All connections are updated and local errors are decreased by:
CHAPTER 3. OBJECT REPRESENTATION WITH SELF-ORGANISING NEURAL NETWORKS
∆Exf = −αExf (3.17)
A global decrease according to:
∆Exc = −βExc (3.18)
is performed to all local errors by a constant β. This is important since new errors will gain greater influence in the network resulting in a better repre- sentation of the topology.
• A set C of edges (connections) between pair of nodes. These connections are not weighted and its purpose is to define the topological structure. The edges are determined using the competitive hebbian learning method. The updating rule of the algorithm is expressed as:
∆xν = x(ξw − xν) (3.19)
∆xc= n(ξw− xc), ∀c ∈ N (3.20)
where x and nrepresent the constant learning rates for the winner node xν
and its topological neighbours xc. An edge aging scheme is used to remove con-
nections that are invalid due to the activation of the node during the adapta- tion process. Thus, the network topology is modified by removing edges not being refreshed by a time interval αmax and subsequently by removing the
nodes connected to these edges.
The learning process of the GNG in the form of a pseudo code is summarised in Algorithm 1. The analytical steps of the algorithm are discussed in Appendix A.2.
Algorithm 1The GNG algorithm
Input: input vectors xc
CHAPTER 3. OBJECT REPRESENTATION WITH SELF-ORGANISING NEURAL NETWORKS
1. Initialise two vector prototypes A = {c1, c2} at random positions {xc1, xc2},
and the connection set C, C ⊂ AxA to an empty set C = ∅
2. while the current number of prototypes ≤ to the maximum number of pro- totypes do
(a) for every input signal ξw do
• Determine the winner prototype xνand the second nearest xυ(xν, xυ ∈
A)by Equation 3.7
• Add the squared distance between the input vector ξwand the win-
ner xν to a local accumulated error variable (Equation 3.14)
• Adjust xν position and its topological neighbours by Equations 3.19
and 3.20
• Update connections between prototypes • Remove any dead nodes
i. if the current number of prototypes is an integer multiple of a parameter λ then
– Add a new prototype by Equation 3.15
– Update the connections between the prototypes
– Decrease local errors by Equations 3.16 and 3.17 ii. end if
(b) end for
3. Decrease the error for all prototypes by Equation 3.18 4. end while
The current problems with the GNG are the dead nodes, when a non-stationary input distribution occurs, and keeping nodes correspondence between successive frames. The second is very important in cases where subtle changes to the shape of the objects occur. Examples can be found in morphology where shape differences can suggest a connection or not between normal structures [39]. Thus, instead of adopting a global approach where all the nodes need to re-adapt their position,
CHAPTER 3. OBJECT REPRESENTATION WITH SELF-ORGANISING NEURAL NETWORKS
local adaptations of the nodes would be more appropriate since they will maintain the correspondences without violating the network. We tackle the above problems by proposing Active-GNG in Chapter 5.