We have evaluated10 the extracted Haskell program from the Coq component on real historical data. The data consists of the ACT state elections for electing the representatives of the Legislative Assembly which is available online [1]. Figure 4.19 illustrates the results of executing the extracted Haskell program for ACT STV on three electoral districts of the Legislative Assembly elections held in 2012 and 2008.
electoral ballots vacancies candidates time (sec) year
Molonglo 88266 7 40 1395 2008
Brindabella 63562 5 20 205 2012
Ginninderra 66076 5 28 289 2012
Figure 4.19: ACT Legislative Assembly 2008 and 2012
ACT Legislative Assembly has three electoral districts, namely Brindabella, Gin- ninderra, and Molonglo. Molonglo is the biggest electoral district of the Legislative elections in Australia both in terms of the number of competing candidates and the ballots cast. Most of the districts of the Legislative Assembly election across Aus- tralia have a number of ballots cast which is close to Brindabella and Ginninderra. Also, those districts are mainly for electing one winner while all of the ACT elec- toral districts have five or seven winners. This means that our software computes the winner faster if there is only one candidate to win the election. When there is only one candidate to elect, the elect transition applies just once and the transfer-elected transition never applies as the election terminates by electing one candidate. As a result, costly computation is avoided which reduces time consumed for deciding the winner.
In fact, Molonglo district for the election held in 2008 had the largest participat- ing candidates among other instances of the Legislative Assembly held in the same district in other years. Therefore, we have included this one instead of the smaller ones, such as Molonglo 2012 for a better representation of our software capacity. As you see in Figure 4.19, the extracted Haskell program for ACT STV takes only about 22 minutes to compute the winners of the biggest Legislative Assembly district in Australia. This stands in sharp contrast to the folklore that theorem proving means are good only for verification but not for efficient computation.
We must note that we have intentionally not presented any data on how the program performs on randomly generated ballots. Randomly generated data are uniformly distributed so that no candidate has a meaningful margin of victory over any other. Consequently, in an instance of computation with such data, the elect tran- sition almost never applies. Instead, candidates are removed so that the remaining continuing candidates are declared elected by an application of hopeful-win transi- tion. On the other hand, elect is one of the costliest transitions because computing the transfer value that each ballot in the surplus votes of an elected draws on Coq library functions for handling rational arithmetic. The problem with this arithmetic is that all of its functions are not tail recursive. Especially, the part of the arithmetic for managing multiplication is significantly time-consuming. As a result, using them slows the computation down. Moreover, elect transition requires calling nestedmap function, also for updating the transfer value of each ballot in the surplus, which is not tail recursive. However, when the elect transition does not apply, and instead the elimination transition occurs, only the addition function is called to sum 1/1 numbers and there is no need to use nested calls to functions likemap.
In light of the above paragraph, we cannot discuss the complexity of the whole extracted program based on randomly produced data. This is partly because of the randomly generated data is biased and therefore unreliable for analysis of our programs and also we lack enough divergent real election data. The only available data is the Legislative Assembly elections of ACT whose variations does not add any meaningful difference to what we have presented in Figure 4.19. Moreover, recall that we use the proofs-as-programs paradigm. Therefore the structure and the content of proofs also plays a significant role in the overall analysis of the program complexity. Unfortunately, use of dependent types in proof contents considerably complicates evaluation of the extracted program’s complexity in a rigorous way. Since our work promotes formal reasoning as opposed to informal pen-and-paper arguments, we therefore believe that the complexity side of our programs should be left for another formal future endeavour. In Subsection 8.1.2, we discus the possibilities of improving the performance of the extracted Haskell programs.
Remark 4.5.1 We shall note that the purpose behind our experimental result is and has never been examining whether or not any possible error has occurred in the past historical elections. We claimed that software produced from our framework is capable for deployment in real elections. The experiments given above substantiate the claim which was an obligation on our side. If a reader is interested in conducting examinations for “unveiling possible errors” in
§4.5 Extraction of Certifying Programs 91
some past real elections whose data is accessible to them, they can simply visit the GitHub repository hosting our source code. The README file in the repository clearly instructs how to obtain executable Haskell programs which can be subsequently used for the experiments.
Chapter5
The Generic STV Verifier
This chapter and the subsequent one together discuss the second standalone compo- nent of the framework. The component is an environment constructed for modular formalisation, verification and synthesis of tools forverifying computationcarried out by executions of any implementation of an STV algorithm. We call tools produced from this componentverifiers.
The current chapter elaborates on the following parts of the second component’s story of synthesising formally verified verifiers.
1. We elaborate on the problem that the component is addressing.
2. We then present our approach and solution in an informal mathematical lan- guage. The intention for a mathematical description is to provide the reader with a clear picture of what a verifier technically resembles. This is impor- tant as the reader has to wait until they reach Chapter 6 to see the formalised version of the verifier in HOL4.
In a nutshell, at this stage, we use our analysis of the generic STV given in Sub- sections 2.1.2 and 2.1.3 to formalise the generic STV as a machine for verifying computation carried out with an STV algorithm. From this newly formulated abstract machine, we devise a generic notion of verifier of computation with the generic STV. We then proceed to define what an instantiation of a machine is and use it to mathematically pin down what a specific verifier of computation for a particular STV algorithm is.
3. We then present the architecture of the framework’s component to provide the reader with a holistic image of the structure of the component and the functionality of each sub-component of the structure. This also gives a roadmap of what is happening throughout the current chapter and the subsequent two chapters.
4. Having provided a vision of how the trilogy is unravelling, we continue the chapter by formally specifying in the theorem prover HOL4 a generic notion for the informally defined verifier in item 2 above.
5. We also explain how we implement a decision procedure inside HOL4 which is the computational counterpart of the formally defined generic verifier.
6. We then show in HOL4 how the specification and the implementation of the generic verifier match. We therefore establish a guarantee that the implemen- tation is a proven correct computational realisation of its specification.
Chapter 6 details instantiations of the generic verifier and synthesis of the instanti- ated generic verifier to obtain machine executable specific verifiers of computation carried out based on various STV algorithms.
5.1
Generic STV for Verifying Computation
We are motivated to tackle the following problem. How can one verify that any execution of any implementationP, whose source code may be secret, of an arbitrary known STV algorithmAcorrectly computes the end result according toA?
Before presenting our approach, we explore some possible alternatives based on which one may hope for resolving the problem. We shall weigh the alternatives against the four general framework requirements that we discussed in the intro- duction of the thesis, namely correctness, verifiability of tallying, transparency and practicality.
Full static verification approach. Verifying that an implementation always returns the correct result on any input establishes a significantly high degree of guarantee in the correctness of the program and subsequently promotes its trustworthiness. However, there is a serious generic disadvantage with this approach for verifying vote-counting programs. This disadvantage exists regardless of the fact that most often the source code of the programs are kept secret, and therefore not available for full static verifi- cation, due to commercial in confidence excuses.
The problem with this approach is impracticality. It is not practical to conduct full static verification for all programs implementing different STV algorithms. Full static verification of one single implementation in a programming language alone is a Herculean task because it requires formalisation and establishing some verification for the underlying semantics of the programming language used for implementing the vote counting program. This latter by itself is a significant resource taking a process which becomes close to impossible if one intends repeating the process for verifying other programs written in other programming languages.
Also, the implementations even if happening in one same programming language can considerably vary in technicalities of the encoding such as the choice of data structure for recording data which in turn affects the mechanism of encoding the algorithm as well. This adds to the challenge of formalisation and verification of the programming language used for encoding the implementations.
Finally, all of the above has to be repeated simply by varying the STV algorithm used for counting votes. Putting the pieces together one instantly realises that despite the initial charm, the full static verification method is entirely a waste of resources.
§5.1 Generic STV for Verifying Computation 95
Recomputing the tallying. Instead of the impractical full static verification of the source code of an implementation P, one may think verifying the correctness ofinstances of computationwith P byrecomputingthe end result using a second implementationP0 of an STV algorithm A. In this approach, one implements another programP0 and executes it on the same input election parameters and the input recorded votes. Then one examines if the returned value of P0 is the same as P’s used for counting votes in the first place. If P has computed the same winners as the implementation P0 in which one trusts, then the instance of computation with P has happened correctly. Otherwise, one rejects the output of this particular execution ofP as erroneous.
Although there is more than one way of exercising the recomputation approach, however, all of them suffer from the absence of two properties, namely transparency and verifiability of theprocessof tallying withP:
• The approach lacks transparency of the process through which P outputs the end result regardless of the correctness or incorrectness of the computation performed. With recomputing the tallying, one at best can come to attest to the correctness of the output of P which is the list of winners. However, the approach methodologically fails to provide transparency in howthe programP has computed the, considering the best scenario, correct result.
• Also, the process of tallying is unverifiable because one can only examine the end result of the computation. There is no access to how the program P has computed the outcome so that one inspects the process as well. Therefore, at best, the end result of the tallying may be verifiable depending on how one practices the recomputation approach, but the process of computation withP remains unverifiable. Also, in the case when programPandP0output different winners for one same election input parameters, there is no way to know where the error in computation with the programP resides.
Due to lack of transparency and unverifiability of the tallying process, the recom- putation approach cannot handle non-deterministic nature of STV algorithms. STV algorithms in some corner cases may use randomness to decide on how to break a tie between two equally weak candidates for eliminating one. Consequently, in two different executions of the program implementing an STV scheme, in such a situ- ation, different votes may be transferred to continuing candidates. Hence, the set of election winners may vary. The recomputation approach inherently is unable to verify the correctness of the end result of such instances of computation.
To begin addressing the challenge in our way, assume P is an implementation of an STV algorithmA. We reasonably demand each execution of P on a given input
x consisting of votes recorded in the election to output the winners y and evidence
ω as a claim for the correctness of the execution. Having evidence available for
each execution of P facilitates checking the correctness of both the process and the end result of the computation carried out independently of the (source code of) P. Therefore the original problem boils down to how one can verify the evidence of any
We verify an instance of computation by examining the evidence produced upon performing the computation. Inspecting evidence of computation for correctness happens by a decision procedure which we callverifier. A verifier accepts as input evidence generated for an instance of computation and checks the evidence for the correctness. If the verifier decides that the evidence is authentic, it certifies the cor- rectness of the computation carried out to produce the evidence. In case, the verifier identifies an error in the evidence, it signals invalidity of the evidence and thereof the instance of computation performed. Figure 5.1 schematically illustrates our approach for solving the problem of verifying the computation with a programP.
ProgramP
output evidence
input output winners
verifier
valid invalid
Figure 5.1: Our Approach in a Picture
There are still difficulties to overcome in order to achieve a satisfactory solution to the problem. The evidence as such must be enough informative to provide trans- parency of tallying and also allow voters, or at least a large pool of scrutineers, to verify themselves that the tally is authentically processed as per instructions of the counting scheme. On the other hand, we wish to verify the correctness of computa- tion with various STV algorithms and not simply one specific STV scheme. Also, we want to verify any implementation of an STV algorithm instead of merely a particular one. We consequently face two challenges:
1. We should choose an abstract data type whose values formally represent evi- dence of computation with any STV algorithm, and implement a data structure that encapsulates the abstract data type. Note that the evidence must be pro- duced for an instance of computation for a variety of STV algorithms. Therefore one challenge is deciding on a structure that is enough general for transparently accommodating data of evidence for the whole STV family without leaving any necessary information out. Also, data has to be represented according to a (for- mal) format. Consequently, another question regards the choice of format for representing the evidence.
2. Suppose we succeed in item 1 above and find out how to record data given in evidence. For validating or rejecting the correctness of evidence by a verifier, we need more than knowing how to store the data. We need an operational semantics to manipulate data for inspecting the evidence by the verifier. On
§5.1 Generic STV for Verifying Computation 97
one hand, the semantics must be flexible enough to accommodate variations existing in the counting mechanism of STV algorithms so that we can produce verifiers specifically operating for validation of evidence output by their associ- ated scheme. On the other hand, it must facilitate designing and developing the framework modularly while offering automation of the synthesising process of verifiers.
The real voyage of discovery consists not in seeking new landscapes, but in having new eyes1. Indeed, we have already come to see a perfect solution for the item 1 above. Recall that in Subsection 2.1.2, we discovered an underlying data structure for the STV family. As we explained there, an STV algorithm has a quota, some initial number of vacancies, some competing candidates. More importantly, we discussed that every instance of computation with an STV algorithm proceeds throughdiscrete states of computation. The discrete states consist of all the necessary and sufficient information to know in order to transparently observe the tallying process. The computation states also provide a chance for verifiability of the process and outcome of the tallying.
It should not then come as a surprise that we already know how to answer chal- lenges in the item 2 above as well. Remember that in Subsection 2.1.3, we detailed a general algorithmic pattern that identifies the computational commonalities among various members of the STV family. As we elaborated in the subsection, this univer- sal algorithmic content lays the foundation of a generic vote-counting mechanism. It can be used as the semantics of an abstract machine which realises the core algo- rithmic characteristics of the STV family. Also, it is capable of being easily extended to capture variations existing among different STV algorithms throughinstantiations
into the machine.
From the above realisation, the generic STV as a machine reemerges. However this time, it appears in disguise of a generic machine for verifying computation with the STV family. Using the machine, we formally realise the semantics of an STV algo- rithmAthrough an instantiation into the generic STV machine. From this discussion, we then obtain a thorough formulation of evidence for an instance of computation withAand also mathematically specify what a verifier for computation withAis.
Definition 5.1 (Machine States) Suppose C ∈Sis the list of competing candidates in an election. A non-final state n f s is a 7-tuple construct(ba,ta,p,bl1,bl2,e,h)where ba ∈ Ba,
ta ∈ Ta, p ∈ P, and bl1, bl2, e, h ∈List (C). Moreover, a final state f s is an element in
List (C). The collection of non-fnal and final states comprise all of the possible computation states which we denote byS.
The forms of action explained in Subsection 2.1.3 comprise the transition labels of the generic STV.
Definition 5.2 (Transition Labels) The set of transition labels T comprises the elements