This benchmark intensively works with arrays/lists. The implementations were taken from The Computer Language Benchmarks Game [68]. The algorithm itself was initially published in [69]; fannkuch is an abbreviation for the German word Pfannkuchen, or pancakes, in analogy to flipping pancakes. It works as follows:
1. Take a permutation of h1, ..., ni, for example: h4, 2, 1, 5, 3i.
2. Take the first element, here 4, and reverse the order of the first 4 elements:
h5, 1, 2, 4, 3i.
3. Repeat this until the first element is a 1, so flipping won’t change anything more: h3, 4, 2, 1, 5i, h2, 4, 3, 1, 5i, h4, 2, 3, 1, 5i, h1, 3, 2, 4, 5i.
4. Count the number of flips, here 5.
5. Do this for all n! permutations, and record the maximum number of flips needed for any permutation.
In table 9.4 and figure 9.7 you can see the time consumed by fannkuch algorithm for n = 10.
Chapter 9. Benchmarks
0 2 4 6 8 10 12
PyPy-C PyPy-C backendopt PyPy-C -O6 PyPy-C backendopt -O6 PyPy-JVM C C -O6
Time [s]
Figure 9.7: Fannkuch Benchmark
Analysis of the Results
What we can see in the results is that PyPy-C variant without any additional optimizations is at least 5 times slower than pure C without optimizations. With GCC-level optimizations, the C code generated by PyPy is as fast as handwritten unoptimized C.
9.3.3
Repeated Memory Allocation
In section 9.2.2, we have investigated the memory consumption of a program that cyclically allocates and releases three types of memory blocks.
In this section, we investigate the execution speed of the same program. The time consumed by the 10000 reallocations is in table 9.5 and figure 9.8.
Analysis of the Results
The handwritten C version with GCC optimizations enabled is much faster than the rest. Without the GCC-level optimizations, the result is about 3 times slower. The handwritten C with Boehm GC is about two times slower than un- optimized C. The PyPy-C runs with GCC-level optimizations are about the same speed as unoptimized handwritten C. The backendopt switch without GCC-level optimizations have only moderate impact; however, the GCC-level optimizations help PyPy-C a lot.
Performance of Java byte-code version can be significantly improved if the initial heap size is 200 MiB; that probably enables the GC to run in more efficient "batch" mode.
The version that runs on the top of Python interpreter is not much slower than the compiled counterparts. It is due to the fact that memory management
Program Consumed Time [s]
PyPy-C 8.37
PyPy-C backendopt 7.43
PyPy-C -O6 3.59
PyPy-C backendopt -O6 3.60
PyPy-JVM 7.90 PyPy-JVM -Xms200M 3.40 C 3.12 C -O6 1.08 C, BoehmGC 6.08 CPython 11.04
Table 9.5: Time Consumed for Allocation Cycles
as well as the way in which we initialize the array (by the built-in range() function) is efficiently implemented in the interpreter itself, i.e., it is actually written in C.
9.4
Conclusion
We have proven that the C code generated by PyPy from the RPython source code can be generally compared with handwritten C code in both memory con- sumption and computational performance. However, we admit that the code generated by PyPy is usually bit slower and consumes more memory.
The memory consumption of the programs that build small data structures depends also on the size of the standard C library implementation. For programs with larger data structures the overall memory consumption is significantly af- fected by the way how the memory is managed: manually, by reference-counting GC, or by mark-and-sweep GC.
Boehm GC brings some overhead that is in our opinion bearable. The ref- erence counting GC works without significant memory overhead, i.e., it has the same efficiency as the manual management.
The computational performance depends on the tool-level optimizations used. The optimizations that are provided by the C compiler (GCC) can significantly improve the performance; the benefit for C code generated by PyPy is more sig- nificant than in the case of handwritten C. We have seen that GCC-optimized code from PyPy is about the same speed as handwritten C without such opti- mizations.
Chapter 9. Benchmarks
0 2 4 6 8 10 12
PyPy-C PyPy-C backendopt PyPy-C -O6 PyPy-C backendopt -O6 PyPy-JVM PyPy-JVM -Xms200M C C -O6 C, BoehmGC CPython Time [s]
Figure 9.8: Time Consumed for Allocation Cycles
Another information about the performance of the code generated by PyPy can be found in [65].
Chapter 10
Case Studies
Let us apply our approach to two examples. The first one is inspired by a real- world application; however, it is simplified in order to demonstrate formal-based testing with LTL formulae.
The second example is a real application that can be immediately used by users, or can be incorporated as a module into a larger software. It was im- plemented as a part of the project "Methods of development and verification of component-based applications using natural language specifications"1 sponsored by Grant Agency of the Czech Republic (GAČR).
10.1
Program for Logging Events
This example is a model of software embeddable into a class of devices called NVR (network video recorder). NVR is an embedded computer system that manages IP cameras over a computer network.