The following zero-order methods are described briefly - random search, random walk, simulated annealing and differential evolution.
Random Search
The random search involves randomly selecting and evaluating a large number of candidate vectors 𝑥 from the design space. The 𝑥 with the smallest 𝑓 𝑥 and satisfying all constraints is chosen as the optimum. This approach requires a large number of randomly selected candidates in order to obtain a precise solution; hence this method is not practicable and should be avoided at all cost.
Random Walk
the current design only if it has no constraint violations and also has a better objective value. If the current design has constraints violations, the new design is considered better if the value of its constraints violation is lower than the current design, even if it has a worse objective value [48].
Simulated Annealing
Simulated annealing algorithm is analogous to the annealing process in materials science. The annealing process involves heating up a solid to a maximum temperature where the particles of the solid randomly move in the liquid phase, followed by slow cooling of the heated bath. The recombination of the molecules to form the solid is based on the difference in energy level of both molecules and the temperature of the bath [48]. In the simulated annealing algorithm, a random perturbation is made to an existing design 𝑥 to create a new design which is evaluated. If the new design is better than the previous, it replaces it and the algorithm restarts. If the previous design is better than the new one, the new design may be accepted depending on a probability calculated using its 'virtual energy' and 'virtual temperature'. The 'virtual energy' is the difference between the performance of both designs, while the 'virtual temperature' is a number that decreases with iteration steps and reduces the probability of accepting a worse design [48]. The acceptance of a worse design in the early stages of a simulated annealing algorithm helps to expand the search area. This is beneficial for global optimisation and could potentially prevent the algorithm from getting trapped in a local optimum.
Evolutionary Algorithms
Evolutionary algorithms are based on Darwinian evolution, whereby populations of individuals evolve over a search space and adapt to the environment by the use of different mechanisms such as mutation, crossover and selection. Survival or a chance to reproduce in the population is based on the fitness of individuals [48]. In evolutionary algorithms, the entire population is evaluated at each iteration in contrast to random walk and simulated annealing where one individual is evaluated in each iteration. Also, the design vector 𝑥 is called the individual, while the objective function is replaced by the fitness of an individual. Different evolutionary strategies can be used to evolve individuals in a population. Two popular methods will be discussed here in detail - Genetic Algorithms (GA) and Differential Evolution (DE).
To quickly understand genetic algorithms, let‟s take a hypothetical situation where a farmer decides to increase milk production from cows by interbreeding cows and keeping records of milk production from
initial range of cow breeds from first generation. The downside to this is that the cows become too genetically similar from one generation to the next. From a genetic algorithm point of view, this can be resolved using a process called mutation, where genes are randomly altered to reduce similarity in parents and off springs. The breeding process is the exchange of genes (crossover) between parents to produce children. Differential evolution is similar to genetic algorithm except that it allows new children to have gene that is weighted from both parents‟ genes.
Genetic Algorithms
A genetic algorithm is a heuristic search technique this is inspired by Charles Darwin‟s theory of natural evolution [104]. Genetic algorithms are considered as a global search technique based on the mechanisms of natural selection and use of the survival of the fittest approach to obtain the optimum individual. At the start of the algorithm, a set of individuals called a population is randomly generated. An individual is characterised by a set of variables referred to as genes. A string of genes form a chromosome. This is depicted in Figure 4.20. For example, an individual could be a compressor design to be optimised with genes such as impeller back sweep angle, impeller hub radius, etc. A fitness function is defined and used to evaluate each individual in the population and the fittest individuals are selected and used to create new individuals for the next generation. A fitness function may compare candidates by efficiency and decide which is fitter. The creation of new individuals is achieved by crossover and mutation. A genetic algorithm process is depicted in Figure 4.21. Crossover is the reproduction process where a crossover point is randomly selected and the genes after the crossover point are swapped between two mating individuals. A crossover process is depicted in Figure 4.23. A mutation is a random tweak in the chromosome which promotes diversity in the population of individuals. A mutation process is depicted in Figure 4.24 where three genes are mutated in an individual. The algorithm stops when a stopping criterion is satisfied, for example there is no further improvement in the population or a pre-defined number of generations are reached. In a genetic algorithm, each population has a
crossover process is illustrated in Figure 4.23 where the crossover point can vary from generation to generation. The last stage is mutation of the genes (bit-strings) as shown in Figure 4.24. The purpose of mutation is to prevent the algorithm from being trapped in a local minimum. It also prevents the population of chromosomes from becoming too similar to one another and stalling the evolution.
Figure 4.20 – Representation of gene, chromosome and population. Source [104]
Figure 4.22 - Bitwise representation of parameters. Source [55]
Figure 4.23 - Cross-over operation in genetic algorithms. Source [55]
Figure 4.24 - Mutation operation in a genetic algorithm. Source [55]
Differential Evolution
The differential evolution (DE) method developed by Price and Storn [56] is similar to genetic algorithms except that it does not require the objective function to be continuous. Also, the method does not require the representation of the parameters with binary strings. In a single
where 𝐹 is a user defined constant with values ranging from 0 to 1. Another vector 𝑧 is obtained by a recombination operator involving 𝑥 𝑡 and 𝑦 𝑡 as show in Equation 4.36.
𝑧𝑖 =
𝑦𝑖 𝑖𝑓 𝑟𝑖 ≤ 𝐶
𝑥𝑖 𝑖𝑓 𝑟𝑖 > 𝐶 𝑖 = 1 … … … 𝑛 (4.36)
where 𝑟𝑖 is a uniformly distributed random variable 0 ≤ 𝑟𝑖 < 1 and 𝐶 is a user defined
constant with values ranging from 0 to 1 [48]. The final step of the differential evolution algorithm is the selection process and the minimization of the objective function 𝑓 𝑥 𝑡 using
Equation 2.37.
𝑥 𝑡+1 =
𝑧 𝑖𝑓 𝑓 𝑧 ≤ 𝑓 𝑥 𝑡
𝑥 𝑡 𝑖𝑓 𝑓 𝑧 > 𝑓 𝑥 𝑡 (4.37)
This final step is a selection process which is a simple replacement of the original vector 𝑥 𝑡 with
the new vector 𝑧 if the objective function decreases by such an action. In differential evolution, constraints are enforced such that the highest priority is given to achieve a feasible design, followed by improvement in the design while maintaining feasibility [48]. One problem that can occur in differential evolution, when a small population size is used, is stagnation - a situation whereby the population stops evolving.