Sujeto Activo

A possible approach for detecting the presence of well defined states in the space of solutions is the so-called whitening procedure, developed in the context of sparse random graphs.

The name for this method origins in a different CSP, the coloring problem, where one has to assign a color, chosen from a family of q colors, to each of the nodes of a given graph, with a constraint that prohibits equally colored adjacent nodes. The problem gets interesting when the graph is colorable in many diffent ways and a rich landscape of geometrical structures can be found in the space of solutions of the problem [72].

The whitening procedure is defined as a “reversed” coloring process: once a legal coloring is produced, one looks for the nodes whose color can be changed without violating any constraint, assigning them with the white color (which denotes a “free” state: these nodes can now be variably assigned a color which is most convenient for freeing more variables). The procedure continues as a cascade of white assignments, until one possibly finds the “core” node

assignments that cannot be whitened: in this way one can highlight those nodes that are essential in the legal coloring initially produced.

A slightly more complicated version of this procedure, closely related to message-passing algorithms (as BP and TAP), is defined as the directional coloring process: in this case the white state can be assigned to the cavity messages (instead of the variable states, making it harder for the process to spread into the entire factor graph). When the directional whitening can no longer proceed, an extremal directional whitening configuration is found. The variables which cannot be assigned a white color are defined to be “frozen”, while the others are said to be “unfrozen” (see also section 1.2).

At this point we can have different possible scenarios:

• If the model is below the dynamical phase transition, at α < αD, then only one trivial whitening is possible: all the graph becomes white. • After the transition, at αD < α < αC, there is an exponentially large

number of extremal directional whitenings. Their number exp(NΣ(α)) is in one to one correspondence with the number of states (well separated clusters of solutions, detectable in a 1RSB analysis) in the phase space of the problem. When different legal coloring assignments end up in the same extremal directional whitening, it means that the two solutions belong to the same cluster.

• above αC, typically no legal coloring can be produced, thus the whitening process is ill-defined.

It can be proven that the local equilibrium condition for the extremality of a directional whitening is equivalent to the BP or the TAP equations in this context.

Thus, we can try to apply a similar procedure also in the Perceptron model: in this case a white state will be assigned to a synapse whose value can be chosen arbitrarily without making any classification errors. The directional whitening, moreover, can be studied in the following way:

1. A solution is found with one of the effective heuristics described in section 3.2.

2. The BP algorithm is initialized in correspondence of the solution (by running it to convergence in the presence of strong external fields, on the variable nodes, in the direction of the solution).

3. The external fields are removed.

4. The BP algorithm is run again and convergence is reached.

Now, there are two possible scenarios for point 4: the BP marginals can either remain fixed (as set by the external fields), or they can be driven away, reaching the closest BP fixed point. In the first case the solution is completely frozen and represents a point-like state (with q1 = 1). In the second case one ends up in the BP fixed point relative to the state the solution belongs to (with q1 <1).

Unfortunately, due to the fully-connected nature of the problem, this ap- proach is not able to give a better characterization of the space of solutions in the Perceptron. In fact, we couldn’t find any states except the point-like ones (when the solver ends up in an isolated solution, with q1 = 1 as predicted in [17]), and the so-called Replica Symmetric BP fixed point, which is reached from all the unblocked solutions (solutions which are found not to be isolated at O (1) distances) or any other random initialization of the BP messages. It is important to stress that the point-like states appear to have a vanishing basin of attraction for BP (or TAP), since every small perturbation of the messages in the initialization (point 2. above) allows the message-passing algorithms to flow away towards the RS point.

In order to explain this trivial behavior we can consider the factor graph associated to a given Perceptron instance: since we are trying to study the directional whitening process, we can simplify the graph, excluding those factor nodes that correspond to patterns with a stability ∆µ_{= W · ξ}µ _strictly higher than 1 (when N is odd the stabilities can only take odd integer values {..., −3, −1, 1, 3, ...}), i.e. patterns that are robust to any one flip in the synapses. In fact, if we imagine the BP algorithm, at convergence these factors are already sending white (flat) messages to all the variables.

Therefore, consider the simplified factor graph where only the unstable patterns are present: we can connect all the variables with the factor nodes with two kind of lines, dashed if a flip in the value of the variable will increase the stability (in this case the pattern is sending back a white message), and

.

Fig. 3.6 Factors constraining the value of the synapses in the simplified factor graph associated to the Perceptron problem, where only the patterns with a stability at the threshold (∆µ= 1) are considered. The dashed lines link the synapses with the patterns that would gain in stability from a flip in the value of the synapses. The solid lines, instead, link the variables with the patterns whose stability would decrease, violating the constraint.

solid if a flip decreases the stability, implicating an error in the classification of that pattern (see figure 3.6). These solid lines are thus representing the constraints actually “felt” by each synapse.

If we are in a situation where the solution is blocked (i.e. isolated), it means that all the synapses receive at least one solid line from the unstable patterns. If we consider a directional whitening process starting from one of these solid lines, we can see that the messages sent to the other synapses by the associated pattern wouldn’t become dashed and be “freed”, since the stability ∆µ _{would be utterly decreased by a wrong assignment for the variable} we started the whitening from. Therefore the directional whitening process would stop immediately: this kind of solution is completely frozen.

On the other hand, we have the case where one of the synapses only receives dashed lines from the unstable patterns, implying that it is unblocked, since a flip would still produce a configuration with stable, correctly classified patterns. If the messages sent by this variable to the unstable patterns are set to white in a directional whitening process, automatically all the stabilities are increased to 3, and the rest of the variables are released from their constraints, i.e. all the solid lines become dashed. Therefore the whitening spreads to the entire graph.

An exemplary case is represented by the teacher, in a teacher-student learning scenario: as observed above this solution is typically isolated. The whitening procedure described above, using the BP algorithm, finds a point-like state in correspondence of this configuration. However, if the training set is selected in a way that all the patterns which have a stability lower than 3 (for the teacher) are discarded, i.e. if we consider a situation in which the teacher is unblocked and receives only dashed lines, the whitening procedure ends up in the RS fixed point.

3.7

Random walk on a connected cluster of