To present this algorithm we consider the generic equality constrained optimization problem,
minx F (x) (5.24)
s.t. H(x) = 0.
The AAO optimization problem, (5.23), can easily be written in this standard form, with x now representing the composite vector of design variables, targets, state variables, and slack variables, and F (x) and H(x) being defined appropriately.
The AAO trust-region algorithm decouples the constrained optimization into two parts, first a normal step, s⊥k, that decreases the constraint violation, and second, a tangential step, skk, that decreases the objective function value while maintaining some of the constraint violation reduction from the normal step.3 The trust-region update step is accordingly the sum of the normal and tangential steps, sk = s⊥k + skk. The trust region step is accepted if the pair (F (xk+ sk), kH(xk+ sk)k) is Pareto dominant compared with all previous iterates xk, and rejected otherwise. We will refer to the fully linear surrogate model at an iteration k for F (x) as ¯Fk(x), and the fully linear surrogates for H(x) as ¯Hk(x).
The first step of the algorithm is to find a normal step that reduces the constraint violation. To do this we compute a step minimizing the 2-norm of the constraint violation
3The terms normal and tangential do not imply that the steps are orthogonal [31, Page 658].
using the surrogate models,
We require that the normal step satisfies a condition slightly stricter than the fraction of Cauchy decrease condition, therefore the solution to Eq. 5.25 must satisfy,
k ¯Hk(xk)k22− k ¯Hk(xk+ s⊥k)k22 ≥ κamm−Hk ¯Hk(xk)k22− k ¯Hk(xk+ s⊥∗k )k22
(5.26)
for a constant κamm−H ∈ (0, 1], where s⊥∗k is the exact solution to Eq. 5.25. An immediate consequence of Eq. 5.26 is the fraction of Cauchy decrease condition,
k ¯Hk(xk)k22− k ¯Hk(xk+ s⊥k)k22 ≥ (5.27)
After computing the normal step to reduce the constraint violation we seek a tangential step that reduces the objective function value without giving up all of the progress achieved towards feasibility. To find the tangential step, skk, we solve a trust-region problem that is based on the Celis-Dennis-Tapia trust-region algorithm [21]. We require at the complete step, normal plus tangential, that the fraction of Cauchy decrease condition for the constraint violation is still satisfied. Therefore, we define an additional constant, η ∈ (0, 1), in order to require k ¯Hk(xk+ s⊥k + skk)k22 ≤ k ¯Hk(xk)k22− η k ¯Hk(xk)k22− k ¯Hk(xk+ s⊥k)k22. The constant, η, can be considered the weighting placed on whether the optimizer focuses on maintaining the constraint violation reduction, η ∼ 1, or reducing the objective function, η ∼ 0; however, regardless of the value chosen, the algorithm will converge to a feasible point that minimizes
the objective function. This tangential step trust-region subproblem is now,
An alternative tangential step trust region subproblem, which is easier to compute, is based on the Gauss-Newton approximation. The Gauss-Newton normal step using the sur-rogate models of the constraints, ˜s⊥k, is computed as
min
The value of the Gauss-Newton step is that the fraction of Cauchy decrease is the same for it as for (5.25) when the trust region is small. The fraction of Cauchy decrease requirement for (5.29) is
This slightly weaker fraction of Cauchy decrease condition is significantly less computation-ally expensive than the fraction of Cauchy decrease requirement in Eq. 5.27. The alternative
tangential step calculation is,
min
skk
F¯k(xk+ s⊥k + skk) (5.32)
s.t. bk(s⊥k + skk) ≤ bk(0) − η[bk(0) − bk(s⊥k)]
ks⊥k + skkk ≤ ∆k.
For whichever tangential step trust region problem is used, we require that the tangential step optimization be solved such that,
F¯k(xk+ s⊥k) − ¯Fk(xk+ s⊥k + skk) ≥ κamm−F h ¯Fk(xk+ s⊥k) − ¯Fk(xk+ s⊥k + sk∗k )i
, (5.33)
where sk∗k is the exact solution to either Eq. 5.32 or Eq. 5.28, and κamm−F is a constant in (2/(2 +p1 − η2), 1).
The step acceptance and trust-region update criterion for this algorithm are a filter based on [40], and enhancements from [39, 41]. The filter uses ideas from multiobjective optimization and considers reducing the objective F (x) and constraint violation kH(x)k as two goals of the optimization. The filter stores the Pareto front of pairs (F (xk), kH(xk)k), and a trust-region step xk+ sk is acceptable to the filter if it is not dominated by a previous iterate, i.e., it will appear on the filter Pareto front. The filter algorithm also requires a sufficient decrease condition so filter points cannot be arbitrarily close. Therefore a point xk+ sk is non-dominated and acceptable for the filter if for all xk visited (xˆk, ˆk = 0, . . . , k),
F (xk+ sk) ≤ F (xˆk) − β ˆ∆2k or kH(xk+ sk)k ≤ kH(xˆk)k − β ˆ∆2kkH(xˆk)k, (5.34)
∀ˆk ≤ k,
for a constant β ∈ (0,12) and where ˆ∆k = min{∆k, 1}. We typically initiate the filter with the point (−∞, kH(x0)k) in lieu of (F (x0), kH(x0)k) to ensure the constraint violation remains bounded. The points acceptable to the filter are shown graphically in Figure 5-3. If a point is acceptable to the filter, it is added to the filter and any previous points in the filter that
are then dominated are removed.
Figure 5-3: Graphic of points acceptable to the filter.
The trust region updates are all based on the whether or not a trial point is acceptable to the filter. If the trial point xk+skis acceptable to the filter, then the design vector is updated, xk+1 = xk+ sk, but if it is not acceptable the step, sk, is rejected. Similarly, if the step is accepted the size of the trust region is increased so that ∆k+1 = min{γ1∆k, ∆max} and if the step is rejected the size of the trust region is decreased so that ∆k+1 = γ0∆k with γ1 > 1 and γ0 ∈ (0, 1). ¯F (x) and ¯H(x) are then updated to be fully linear on kx − xk+1k ≤ ∆k+1. We require that the surrogate models have the correct value at xk+1, i.e. H(xk+1) = ¯H(xk+1) and F (xk+1) = ¯Fk+1(xk+1). The surrogate models for each constraint are all created concurrently.
Moreover, if no designs have been evaluated in the vicinity of the current trust region, then construction of a fully linear model requires n + 1 designs to be evaluated. All of these designs may also be evaluated concurrently.
To establish optimality of xk and terminate the algorithm, it is necessary to verify that both the constraint violation and the projection of the objective function gradient onto the nullspace of the constraint Jacobian are sufficiently small. In the general gradient-free case, the second condition cannot be verified directly. Therefore, we require three criteria to terminate the algorithm, (1) kH(xk)k ≤ , (2) k ¯Pk∇ ¯Fk(xk)k ≤ , and (3) ∆k ≤ , where
P¯k = I − ∇ ¯Hk(xk)> ∇ ¯Hk(xk)∇ ¯Hk(xk)>−1
∇ ¯Hk(xk). (5.35)
Using fully linear surrogate models, conditions (2) and (3) are equivalent to the projection of
the objective function gradient onto the nullspace of the constraint Jacobian being sufficiently small [38]. The complete trust-region algorithm is summarized as Algorithm 5.2.