Returning to the original research questions introduced in Section 1.6, the main thesis question RQ 1 led to a range of subsidiary questions RQ 2 to RQ 5. This section explores the answers to each of these subsidiary questions in turn, leading to an answer to the main question.
RQ 2 What interventions can change the environment for members of the development team to achieve good security, considering cost-efficiency, motivational factors, choice of tools, supporting processes, culture, awareness, training and skills? Chapter 4 identified that effective interventions are those that conform to Active Developer Model: that developers must drive the security improvements themselves. Most of the effective such interventions are known by Security Specialists as ‘Assurance Techniques’, and of these, five are particularly cost-effective: Threat Assessment, Configuration Review, Automatic Static Analysis, Code Review and Penetration Testing. A further change to development process, Stakeholder Negotiation, also contributes largely to improved security; and two interventions, Incentivisation Session and On-the- Job Training, are effective in helping change the environment (Section 4.14).
RQ 3 To what extent, and how, does a perceived need for security and privacy lead to security-enhancing activities and interactions in an Android development team and result in better software security?
A survey of successful Android developers (Chapter 5) found a high level of reported security need for the app development, but less use of practical security assurance techniques such as the five discussed above, with less than 50% regularly using them; though of the third to a half of such developers working in teams nearly 80% used them regularly.
The use of such techniques was in proportion to the perceived need, as was the involvement of security specialists (though less than a quarter had this), and the frequency of app security updates. We found little relationship, however, between these factors and the density of security defects in the resulting apps; we believe this reflects the inability of the current generation of binary analysis tools to analyse libraries effectively and separately from the main app code. Surprisingly, we did find more Cryptographic API issues in apps whose development team worked with security specialists or champions, probably since the specialists and champions correctly enforce much more Cryptography use.
Reportedly, app security improvements have been largely driven by developers themselves; GDPR has had a minor impact (Section 5.5).
RQ 4 What security outcomes did the ‘Developer Security Essentials’ package have, and what aspects contributed most to those outcomes?
In the first year’s trials of the Developer Security Essentials package (Chapter 7), two of the three organisations’ teams achieved substantial improvements both in the security of their products and in their development process with respect to security. In particular, after a year or more, the first had identified that the choice of security improvements to make was a commercial, not a technical, question and had moved to a risk-based decision process; the second had incorporated security throughout their development training and
Chapter 9: Discussion and Conclusion
process. The third company, a much larger organisation with a dysfunctional relationship between developers and security specialists, found that the interventions contributed to security awareness, but made no objectively detectable improvements as a result.
The main aspects that made the package effective were the extent to which it increased awareness rather than simply directed the teams involved. The effective teams were those who were able to do the following (Section 7.7.2.2):
Apply Team Activities to Security: The workshop effectiveness came from discussions
between participants rather than information from the intervener.
Focus on Key Assurance Techniques: Specifically, Threat Assessment, Automated
Static Analysis, and Configuration Review provide a large benefit for relatively low cost.
Address Organisational Issues: Nearly half the Motivators supporting security
improvement, and Blockers discouraging it are ascribable to organisational issues, so addressing these can make a large contribution.
RQ 5 Which aspects of the ‘Developer Security Essentials’ intervention are effective at improving security when used independently by teams from a variety of cultures and different types of organisation, and why?
A second round of trials with 8 further organisations showed that the Developer Security Essentials intervention led to security process improvements with all the groups who used it (Chapter 8), especially in the use of Threat Assessment to help participants focus their security effort on the appropriate threats; and Stakeholder Negotiation to encourage Product Management to allocate resource to threat mitigation. All three workshops were effective at helping improving security; developers proved surprisingly adept even at risk assessment and creating positive representations of security improvements.
The intervention had least impact with very security-expert companies and when led by security specialists, though this may reflect the lack of scope for improvement. They had most impact where the workshops were facilitated by managers or in a ‘hands-off’ manner, probably because these arrangements empower the development team most.
9.2.1 Main Research Question
The previous questions were all derived from considering the original research question: RQ 1 What is needed to make a cost-effective and widely applicable intervention to
help UK software development teams achieve better software security?
From the above findings, we conclude that what is needed in any such intervention are the following properties:
• It sensitises a development team to the importance of security and privacy issues in developers, and enables them to make their own choices as to the best ways to address them.
• It helps the developers understand that every project has unique requirements related to security and privacy; and teaches them Threat Assessment approaches to establish and assess those requirements.
• It introduces them to the four further assurance techniques that are most effective in removing security issues during development: Configuration Review, Automatic Static Analysis, Code Review and Penetration Testing.
• It supports the developers in what we have termed ‘Stakeholder Negotiation’: finding ways establish the business value, or lack of it, for addressing security
requirements; and hence supporting good business decisions about where to spend effort and money on security and privacy improvements.
From the experimental work described in this thesis, we can add the observation that the following two techniques provide a good basis for such an intervention:
1. Workshops, in which the developers carry out structured discussions.
2. Regular On-the-job Training sessions afterward to act as a ‘nudging’ reminder.