Software Process Improvement in Small and Medium Software Enterprises:
A Systematic Review
Francisco J. Pino°,*, Felix Garcia*, Mario Piattini*
° IDIS Research Group
Electronic and Telecommunications Engineering Faculty University of Cauca
Street 5 # 4 – 70 Popayán, Colombia.
* ALARCOS Research Group
Information Systems and Technologies Department UCLM–Soluziona Research and Development Institute
University of Castilla–La Mancha
Paseo de la Universidad, 4 – 13071 Ciudad Real, Spain.
Felix.Garcia, [email protected] Abstract
Small and medium enterprises –SMEs– are a very important cog in the gears of the world economy. The software industry in most countries is composed of an industrial scheme, made up mainly of small and medium software organizations – SMSEs. To strengthen this type of organizations, efficient Software Engineering practices are needed– practices which have been adapted to their size and type of business. Over the last two decades, the Software Engineering community has expressed special interest in software process improvement (SPI) in an effort to increase the software product quality and the software development productivity. There is a widespread tendency, though, to make a point of stressing that the success of SPI is only possible for large companies. In this article, a systematic review of the literature on the SPI efforts carried out in SMSEs is presented. The objective is to establish research works that have been carried out for SPI focused in SMSEs, and have a real study case to, in order to know a profile about SPI in SMSEs.
1. Introduction
Small and medium enterprises –SMEs– are a very important cog in the gears of the world economy.
Another very significant economic activity nowadays, which has emerged, grown and become strong over the last two decades is the software industry, a major economic force throughout a lot the nations of the world. The software industry in most countries has an industrial backcloth, made up mainly of small and medium software organizations – SMSEs– which favour the growth of national economies. According to (Fayad et al., 2000) most software development companies are small (fewer than 50 employees). They develop significant products and for the construction of these, the firms need efficient Software Engineering practices that are suitable for their particular size and type of business.
From the very beginning of the nineties onwards, the Software Engineering community (industry and researchers) has expressed special interest in Software Process Improvement – SPI. This is evidenced by the growing number of articles that deal with the topic of SPI, something that is seen in the analysis of the trends in publications on the subject of SPI, presented in (Hall et al., 2002).
Worthy of note, too, is the great number of international initiatives related to SPI, such as the models CMM, CMMI, ISO/IEC 15504:2004, SPICE (ISO/IEC 15504:1998), ISO/IEC 12207:2004 and ISO 9001:2000, among others.
Nevertheless, in educational and industrial environments there is a widespread tendency for them to go out of their way to emphasise that the success of SPI is in most cases possible for large companies that have enough resources to tackle this type of practices. This perception is based on the fact that often SPI programmes are just not viable for SMSEs, because of their organizational structure and the cost that is involved. The viewpoint also takes into account the reality that the models of improvement created by international organizations have not been considered for this type of companies, but rather for large-scale ones (Saiedian et al., 1997). Currently, (SEI, 2006) and (ISO_WG_24, 2006) show like Software Engineering Institute – SEI– and Organization International for Standardization – ISO–, respectively, begin to make efforts in order to that their standards of software processes improvement can be applied to SMSEs. Lots of authors agree that the special characteristics of small companies mean that process improvement programmes must be applied in a way that is particular to them and visibly different from how this is done in the large organizations. This is not as simple as just regarding these programmes as scaled down versions of those applied in big companies (Richardson, 2001).
In this article we present a systematic review of the literature which deals with SPI efforts carried out in SMSEs. The objective is to know what has been carried out and achieved as regards software process improvement in this type of companies. The paper proceeds as follows. In section 2 the systematic review itself is presented. Section 3 presents the results obtained and a discussion of these. Finally, the conclusions and future work are outlined.
2. Systematic review in Software Engineering
The systematic review of the literature allows the identifying, evaluating, interpreting and synthesizing of all the existing and relevant research in a subject that is of particular interest. This type of reviews is carried out in a rigorous and impartial way; thus they are of great value scientifically. The main motivation for undertaking a systematic review is to increase the possibility of detecting more real results in the subject of interest than can be detected by reviews which have a more a more limited scope.
Barbara Kitchenham in (Kitchenham, 2004) presents a general guideline for undertaking systematic reviews in the context of software engineering research. Before this proposal there were no guidelines or methods in Software Engineering for performing any exhaustive studies. A systematic review involves several separate activities which are grouped in three main phases.
Planning the Review, whose associated stages are Identification of the Need for a Review and Development of a Review Protocol.
Conducting of the Review, which is composed of the Identification of Research, Selection of Primary Studies, Assessment of the Quality of the Study, Data Extraction and Monitoring, and Data Synthesis.
Publication of the results of the review.
Conducting systematic reviews is not a simple task, being required a great effort. The review must be planned before execution and the whole process must be documented, including, intermediary results. In an effort to aid the planning and performing of systematic reviews, a review protocol template has been developed in (Biolchini et al., 2005). Its objective is to serve as a guideline to
3. Systematic Review of SPI in SMSEs
To carry out the systematic review of SPI in SMSEs, the protocol template presented by Biolchini in (Biolchini et al., 2005), was followed. All template of the systematic review on SPI in SMSEs can be seen in the appendix A. The review protocol template consists of five general stages:
question formulation, source selection, study selection, information extraction and results summarization. In this section we set out an overview of the first four stages.
3.1 Question formularization
The research question is: What approaches concerning SPI have focused on SMSEs and at the same time present a real study case? The list of keywords used to find an answer to the research question was: software, process, improvement, SPI, small, enterprises, organizations, companies, team, firms, SME, settings, CMM, CMMI, 15504, SPICE, 12207 y 9001.
The results expected at the end of the systematic review were, amongst others, to find out what the improved processes and the key factors for processes improvement were, what improvement strategies had been carried out, and what the most frequently-used models for the process improvement were. The population group that was observed were publications concerning SPI in SMEs, as found in sources selected for the review.
3.2 Sources selection
Taking as a starting point the list of keywords seen above and making combinations with logical connectors “AND” and “OR”, two basic search strings were obtained (see table 1). When the time came to carry out the searches, the basic search strings were adapted to each one of the search engines of the respective sources.
Basic search strings
1 "software process improvement" AND (small AND(enterprises OR organizations OR companies OR team OR firms OR settings ))
2 (small AND (enterprises OR organizations OR companies OR team OR firms OR settings)) AND (CMM OR CMMI OR 15504 OR SPICE OR 9001 OR 12207)
Table 1 – Basic search strings.
With the first string it is possible to find studies about SPI and SMEs related to any methods of software engineering (e. g. six-simga, agile/XP). With the second string it is possible to find studies about SMEs related to specific methods of software process improvement.
The planned list of sources for the performing of the systematic review is:
Science@Direct, on the subject of Computer Science,
Wiley InterScience, on the subject of Computer Science,
IEEE Digital Library,
ACM Digital Library and,
As grey literature, the special report “Proceedings of the First International Research Workshop for Process Improvement in Small Settings” from SEI was reviewed.
We chose this workshop because a overview about research from the world-wide community addressing the unique issues of process improvement in small settings (including small teams, small projects, small organizations, and small businesses) is presented.
3.3 Studies selection
We propose that the criterion for the carrying out of the systematic review is based on an iterative and incremental procedure. The term “iterative” indicates the repetition of one or more activities;
the term “incremental” indicates that implementation proceeds from an initial subset of the sources to more and more complete subsets, until the whole review is addressed. It is iterative, because the execution (search, information extraction, and visualization of results) of the systematic review is done completely on one search source first, and then on the others in turn. Too, it is incremental in the sense that the document (which is the product) of the systematic review grows and evolves with each iteration until it becomes the definite one.
In this way, many of the advantages that these types of development methods offer were obtained, for example:
Implementation starts from a source, reducing the complexity and scope of each execution.
Each cycle is short and provides feedback for the next cycle.
The increments can be carried out independently and in parallel.
The complexity of each increment is easier to administer.
The procedure for the gathering of the primary studies is illustrated in the activity flow diagram of figure 1.
Figure 1 – Procedure for execution of the systematic review, for obtaining the primary studies and synthesizing their information.
The inclusion criterion for what to include as regards studies took as its basis an analysis of the title, the abstract, and the key words from the articles obtained in the search. These served to determine if they were related to SPI and if they were focused on, and were applied to, SMSEs.
Another focus of analysis was how the key words improvement and small were treated in the total content of each article, in order to identify if a given study presents a real case study. The goal was
to decide if a given article should be chosen as a relevant study in the context of the systematic review (in other words whether it was a potential candidate for primary study).
From the analysis conducted in the first iteration of the relevant studies, we ascertained that they had different levels of treatment for the SPI and SMSEs:
A first level, where the study consists in the application of a process improvement effort that focuses on small and medium-size organizations (there is a study case applied on SMSEs, see all studies selected in appendix B).
A second level, where the study deals with SPI initiatives for SMSEs which nevertheless have never been applied in such companies (there is not a study case applied on SMSEs, for instance (Brodman et al., 1994)).
A third level, in which the study reports on SPI initiatives where some SMSEs are involved, but improvement in these companies was not the main objective of those initiatives (for instance see the study presented in (El Emam et al., 2000))
To determine which articles, in the context of the systematic review, were important enough to be considered as primary studies, the exclusion criterion was to exclude any of those which dealt with SPI and SMSEs at levels two and three.
After applying the procedure for the gathering of primary studies, 743 studies were found. 400 of these were non-repeated ones and from these, 45 primary studies were obtained. See table 2.
Table 2 – Distribution of studies by source. January 1996 – March 2006.
In this systematic review potential references inside primary studies not were searched.
3.4 Information extraction
As can be observed in figure A, once the primary studies were chosen, the extraction of the relevant information for the systematic review was carried out. The inclusion criterion for the information coming from the primary studies consisted in whatever gave an indication of what the improvement strategy used was like, along with information regarding the processes that had improved, the key factors in the success of the improvement, and the models used for that same improvement of the processes. Alongside the above information, the most important ideas of the study are registered.
The information from the primary publications was stored on a chart where the data extraction format was structured in the following columns: name of the study, magazine, date, authors, methodology followed, results, objectives of the study, country where the study was carried out, number of SMSEs involved, number of employees of the SMSEs involved, information related to the improvement models, the improvement proposal, key factors in the success of a SPI program, the improvement processes, and the subjective results of the study. See table 3.
Data extraction from the study
Name Practical Software Process Improvement-The IMPACT Project.
Journal/Conference 13th Australian Software Engineering Conference (ASWEC'01).
Date August de 2001.
Authors Louise Scott, Ross Jeffery, Lucila Carvalho, John D'Ambra, Philip Rutherford.
Objective results of the study Methodology Study case
Outcomes
This paper presents a framework for SPI that is designed to utilise a range of
improvement technologies and supports continuous and highly focused improvement over many projects, thus producing timely, cost-effective and tangible improvements for SMEs.
The effectiveness of the framework is illustrated in this paper with its application in a small, Sydney-based, web-development company.
Country Australia
No. SMEs 1
No. Employees 20
Models/Standards IMPACT, ISO 12207 Improvement
proposal Use an improvement framework in order to guide the programme SPI.
Success factors Guide the improvement programme. Have got freedom in choosing the improvement technology.
Improvement
processes Projects documentation, processes planning, processes estimation and requirements elicitation
Subjective results of the study
The framework is cyclic – advocating an "understand – improve – apply – measure" approach which can be applied incrementally across many projects. On each cycle a different focus for improvement is chosen so that each iteration need not require a large expenditure on the part of the company and the improvements are available to all subsequent projects. By implementing many small iterations of the improvement cycle results are achieved quickly (within a few months) and are measurable in the projects in which improvement has been applied. The framework is implemented using existing and proven technologies, frameworks and standards that can be purchased (at greater or lesser expense depending on the choice of the company) commercially.
Accordingly, our framework for software process improvement, as shown in Figure 1, differentiates between the project level and the process level. At the project level, many projects are carried out according to good project
management practice (e.g. a "Plan–Do–Check–Act" cycle). This is shown as the small cycle in Figure 1 which we call the project cycle. At the process level, experience and understanding from many projects are used to understand and improve the generic process model which is then used to guide future projects. This is the large cycle shown in Figure 1 which we call the process cycle. Note that these cycles interact closely – the process cycle drives the execution of the projects and the projects drive improvements to the process. The next paragraphs will expand more on each of these cycles and how they influence software process improvement.
Table 3 – Data extraction format.
4. Results and discussion
With the information extracted from the studies, a statistical analysis was conducted, whose purpose was to show the relevant findings of the systematic review that had been carried out. These results (presented in this section) give answer to original research question: What approaches concerning SPI have focused and applied on SMSEs?, in aspects of publications trend, involved companies, models used, processes measurement, improvement proposals, improved processes and success factors.
4.1 Publications trend
We ought to note, in first place, there is an increasing interest on the part of the Software Engineering community in tackling the whole area of SPI and SMSEs. This is, indicated by the increasing number of studies that deal with the subject, according to the analysis of the trend in publications that was carried out (see figure 2).
Figure 2 – SPI publications trend in SMSEs.
As can be seen in Figure 3, the six countries, reporting most improvement efforts in SMSEs are, in descending order, Australia, Ireland, Finland, Denmark, Brazil and USA. The reinforcement of the software industry involves the real commitment of different actors in society, such as the state, the academic sphere and the companies.
The Software Engineering Australia – SEA – is an industry-led body with the charter of improving the quality and reliability of software in Australia. SEA is supported by the Commonwealth through the Department of Communications, Information Technology, and the Arts. The SEA provided funding for the Software Quality Institute – SQI– to deliver a process improvement program to 22 small software development firms. SQI has recognized the need for approaches that can generate incremental improvements to demonstrate business benefits, particularly for smaller developers.
The European Union has encouraged initiatives like ESSI (European Software and System Initiative) which have promoted different projects for strengthening SPI in SMEs, such as SPIRE (Software Process Improvement in Regions of Europe) (SPIRE, 1993), TOPS (Toward Organized Software Processes in SMEs) (Esprit_Project, 1999), among others.
In Brazil, the government subsidized the implementation of the PBQP–Software (Productivity and Quality Software Program) (Bedini et al., 2005) and the “mps Br” project has been developed (melhoria do processo de software brasileiro) (Weber et al., 2005), both of which strive for the same objective.
Several of the reported studies come within the framework of strategies encouraged by governmental organisms. Together with all the above considerations, company commitment is vital, along with a firm belief on its part to the effect that starting an SPI project will give it competitive advantages.
Figure 3 – Distribution of improvement reports according to country.
4.2 Involved companies
The average number of SMSEs involved per study is 2.97. This means that in each one of the studies an average of three companies are used. We should bear in mind, however, that 53% of the studies involved only one company.
Currently, SPI efforts are being applied to small and micro-companies too (see figure 4). In this figure sample which the 47% of the involved companies in the improvement efforts were small (fewer than 50 employees) and 33% were micro-companies (around 1 and 9 employees), which gives us the sum of the 80% of all reported companies in which a SPI effort was carried out. Only 20% is reported in medium size companies (around 50 and 249 employees). However, it is important to underline that in most cases the improvement programs in micro and small software companies did not lead to the ISO or SEI certifications.
Figure 4 – Number of employees in the companies where an improvement effort project was conducted
4.3 Models used
Inside the Software Engineering community there is a growing concern that standards SEI and ISO is not applicable easily to small firms because it requires a huge investment (money, time, and resources) (Saiedian et al., 1997). In table 4 the standards used by each study are presented. The SMSEs adapt and use these models to start their improvement efforts (figure 5), however. The most widely-used SEI model is CMM, with a 25 % share. The SEI models have been used in 51% of the reported improvement efforts. The most extensively– used ISO models are the 15504 (SPICE) and the 9001, with 9% percent each. The ISO/IEC 15504 model has the two–fold nature of being an evaluation method, as well as a process model. The ISO models have been used in 31% of the improvement efforts carried out. 18% of the improvement efforts reported in the studies do not use either a SEI or an ISO model.
Table – 4. Standards used by each study.
Figure 5 – Models used for the improvement
On the other hand, it is important to highlight that in an improvement program different types of models are involved. We have, amongst others, the model that guides the improvement, the process assessment method and the process reference model that is to be followed. The adaptation of the IDEAL model is the way that is most widely used to guide the improvement. The study shown in (Casey et al., 2004) is a good example of this. IMPACT (Scott et al., 2001) and MESOPYME (Calvo-Manzano et al., 2002) models are also used, along with others. The MESOPYME, for instance, focuses on reducing the time and effort in the implementation of the SPI, based on the concept of action packages. This type of models was used in 9 studies, which is 20% of the studies;
this represents a low percentage, because an improvement model is the guide needed to articulate all the activities related to the improvement, and of course all the other models involved. In figure 6 the assessment methods and process reference models used in the study are shown.
As can be seen in figure 6, the assessment method that is used most in the SMSEs is the ISO/IEC 15504, in its 1998 and 2004 versions. On the other hand, from the SEI assessments, the CBA-IPI proposals, or less intensive assessments like the C class or micro-appraisal, follow in order of importance. We could add that the 15504:2004 standards is beginning to be applied on a wide scale, due mainly to the fact that it is a model easy to understand, flexible, and suitable to fit to the needs of SMSEs. In some studies, the use of assessment models developed particularly for small companies and based on the standard ISO 15504 is shown, as for example in MARES (Wangenheim et al., 2006) and RAPID (Rout et al., 2000).
Concerning process reference models, the one used most is the CMM of the SEI. In a CMM environment, the SMSEs set out to achieve level 2 when an improvement process begins. It is important to underline that from the 122 companies involved in the primary studies, only 2 medium-size companies of 70 and 150 employees achieved the formal assessment as CMM-SW level 2. It can be concluded that for the small and micro companies the SEI models do not lead to a certification. The motivation for this type of companies should therefore be to improve their processes, as to make them more effective and efficient units, more than to gain the certification itself,. Following these processes should be a means and not an improvement goal. On the other hand, 30% of the improvement efforts that used CMM also used ISO 9001. ISO 9001 helps to provide better support to the CMM initiative, to improve the areas of the organization that are not directly related to the development of the software. It should be remembered that in many countries the standard ISO 9001 is a national standard.
4.4 Processes measurement
From the 45 studies that were analyzed, 20 have used the SEI models, 18 have used ISO models and only 2 have used models for the definition of measurement programmes. This ratifies the assumption that the SMSEs’ efforts are still insufficient to establish formal software measurement processes. Improvements introduced by the SPI are measured through informal and non-objective processes based on the employees’ perception and not through formal measurement programmes, such as Goal Question Metric (GQM) or Practical Software Measurement (PSM). The definition of these measurement models would help in order to make an evaluation more objective.
4.5 Improvement proposals
The improvement proposals suggested to guide the improvement effort are diverse. In the following chart the improvement proposals found in this study are set out.
Improvement proposals Target
18%
Establishment software processes. Use of electronic process guides and experience repository – EPG-ER. Adaptation and use of practices RUP, XP, SCRUM, among others. Informal self- assessment for software developers (web tool).
The software process 9% Prioritize the SPI efforts (by means of DAIIPS method, Software Process Matrix, Express
Process Appraisal or Framework of key decisions of business strategy and product development)
The SPI programme.
4% Evaluation of a SPI programme (with low-rigour, or to define and use a metrics programme) 13% To guide the SPI efforts (Pr2imer method, Framework IMPACT, MESOPYME method/model,
using improvement patterns, following an assessment process, neuronal network)
45% Adaptation and use of SPI models/methods (PSP, TSP, CMM, CMMI, IDEAL, ISO 15504:2004, SPICE, ISO 9001)
11%
Definition and use of test framework. Lead process improvement experiment. Use knowledge management for SPI. To acquire ready-to-use technical infrastructure. Improve the relation and cooperation with clients.
Table 5 – Improvement proposals.
The improvement proposals are centred in two targets: one the software process and other one the SPI programme. From the table above, it can be observed that what most improvement proposals (71%) have concentrated on is the guidance of an improvement project and the priority of improvement implementation, along with the use of existing improvement models, adjusting them to needs, and the evaluation of the improvements introduced for the SPI program. Alongside this greater bulk of proposals, there are others (18%) which focus on the definition, evaluation, and support of the software processes.
The improvement approaches can be generated at an organizational or technical level. At an organizational level, the improvement approaches are cheaper, but the improvements can not be seen at in the short term. At a technical level, the improvement approaches are more expensive but the improvements are seen in the short term. It should be emphasized that there are studies (11%) that in the quest to support actions of process improvement, propose the supplying of technical tools and facilities that support the activities related to the processes. This is termed technical
4.6 Improved processes
From the primary studies, we searched for the information that reported the improved processes in an explicit way. All studies expressed in natural language some type of process improvement. This was stored in the data extraction forms described in section 3.4. After that, this information, related to the improvement and expressed in natural language, was analyzed and adapted, the aim being to locate it in a process model. This allowed us to have a vision of where the improvement efforts in micro, small, and medium size companies had focused. The process model used to express the improvements was the ISO/IEC 12207:2004 (ISO_12207, 2004). In figure 7 the frequency of the improved processes is presented.
Figure 7 – Processes improvement by improvement efforts in SMSEs
Improvement efforts, as is clear from Figure 7, aim to improve processes like project management,
4.7 Success factors
Some factors of success in SPI efforts that can be extracted from the studies analyzed are:
Before commencing SPI, ensure the organisation is stable enough. Initiate the improvement as soon as possible with a simple structure of the SPI model. Guide the improvement program.
Follow a systematic and coherent improvement initiative. Improve processes following an incremental approach that allows a continuous and ongoing adoption of improvement practices.
Prioritize the improvement points in an SPI program. Combine different SPI approaches. Have concrete procedures of the SPI process.
Provide fast return on investment. Propose improvements that have a maximum effect in as short a time as possible. Get optimum value out of resources, stretching them as far as they will go. Join together with other companies so as to both finance and share specialised resources that are involved in improvement. Improve processes without resources set apart for that end. Look for external financial aid.
By means of personnel training, involve the greatest number of employees possible and have as few interruptions as possible in the SPI programme. Involve everyone in the company. Train the employees in the SPI programme.
Minimize resistance to change, through organizational awareness of how much the improvement will benefit the company. Implement the improvement according to the company requirements.
Track and supervise the SPI program. Evaluation of the efficiency of an SPI programme.
Frequent assessment of processes. Rapid assessment of software processes.
The systematic use of metrics. Tailoring and carrying out measurement activities. Measurement and recording of these activities.
Establish the infrastructure that supports the communication between different actors involved in the improvement. Efficient communication.
Managerial commitment. Commitment to an ongoing quest for quality on the part of all organization employees.
Expert consultantship when beginning the SPI programme.
Tackle the problem of improvement from the technical perspective.
5. Conclusions and future work
Systematic reviews require an effort which is considerably more intensive than that needed in the conventional reviews of literature. The formality with which the systematic review is carried out allows us to validate any results reported in the systematic review which are, in turn, both supported and guaranteed by the review protocol.
In this article a systematic review of the literature on SPI efforts carried out in SMSEs has been presented, with a real case study provided by way of illustration. All of this has meant we have been able to give a comprehensive and thorough view of the current state of affairs in this area.
The first observation from the study carried out is that, in the last 5 years, there is an ever-increasing interest in the Software Engineering community in tackling SPI in SMSEs. If the Software Industry is to be strengthened, there needs to be a commitment on the part of the state, the academic world and companies themselves. Even though at the global level there is a general consensus that SEI and ISO models can be applied only with difficulty in small companies, these same companies do indeed adapt and use these models when undertaking their improvement efforts. Both small and micro companies are undertaking SPI efforts nowadays, but for these companies SEI or ISO models do not lead to their receiving a certification. To follow these models, then, should be a means towards, and not an end of, improvement.
Another relevant fact ascertained is that the improvement models used most in the SMSEs are CMM as process model, ISO 15504 as evaluation method and IDEAL as model for guiding the improvement. Nevertheless there are not studies where the use of the three models applied on a small company is reported.
The vast bulk of improvement proposals major on guiding the improvement project, prioritizing SPI implementations, using existing improvement models and fitting them to SMSEs needs, as well as on evaluation of improvement brought about by an SPI programme. Improvements introduced by SPI efforts are, however, measured through informal and non-objective processes, based on the employees’ perception and not through formal measurement processes. There is a smaller percentage of other improvement proposals which have as focal point the definition, assessment and
Furthermore, improvement efforts aim to improve processes like project management, documentation, requirement change management, process establishment, configuration management and requirement elicitation. These processes are related with the support, management and improvement of process.
Two lines of work will be undertaken, with the results of this systematic review used as a starting point. In the first place, a search using the same review protocol will be carried out in new information sources, like Springer, all SEI reports, among others. Each new source will be reviewed in a new iteration with the procedure for execution of the systematic review suggested. Thus the review would include a greater number of works, since many of these are not published in the sources looked at in this study, for example (Niazi et al., 2006). Secondly, studies which deal with SPI initiatives for SMSEs but do not have a real case study, will also be analyzed. This kind of study had been excluded from the present work, as they did not come within its scope.
Acknowledgments
This work has been funded by the following projects: “Mejora Basada en Evidencia de la Capacidad en Actividades de Software – MECENAS” (PBI06-0024) granted by “Junta de Comunidades de Castilla-La Mancha” and “Process Improvement for Promoting Iberoamerican Software Small and Medium Enterprises Competitiveness – COMPETISOFT” (506PI287) financed by CYTED.
References
Appendix A.
The protocol for this systematic review is presented below. To carry out the systematic review of SPI in SMSEs, the protocol template presented by Biolchini in was followed.
1. Question formularization 1.1 Question focus
Se pretende a través de la presente revisión sistemática identificar reportes de experiencias e iniciativas en ingeniería del software relacionados con la mejora de procesos software en pequeñas y medianas empresas desarrolladoras de software.
1.2 Question quality and amplitude
- Problem: Los modelos de mejora de organizaciones como el SEI e ISO están estructurados para ser aplicables a empresas grandes. Difícilmente pueden ser aplicados a empresas pequeñas debido a que un proyecto de mejora supone gran inversión en dinero, tiempo y recursos, además de la alta complejidad de las recomendaciones y que el retorno de la inversión se produce a largo plazo
Appendix B. List of selected studies
In this section the selected primary studies in the systematic review are presented.
List of primary studies selected in the systematic review 1
Experiences on establishing software processes in small companies. Information and Software Technology, In Press, Corrected Proof, Available online 15 February 2006, Christiane Gresse von Wangenheim, Sérgio Weber, Jean Carlo Rossa Hauck and Gisele Trentin
2
Process improvement for small firms: An evaluation of the RAPID assessment-based method. Information and Software Technology, In Press, Corrected Proof, Available online 20 December 2005, Aileen Cater-Steel, Mark Toleman and Terry Rout.
3
Understanding the use of an electronic process guide. Information and Software Technology, Volume 44, Issue 10, 1 July 2002, Pages 601-616. Louise Scott, Lucila Carvalho, Ross Jeffery, John D'Ambra and Ulrike Becker-Kornstaedt.
4 The use and effects of an electronic process guide and experience repository: a longitudinal study. Information and Software Technology, In Press, Corrected Proof, Available online 19 July 2005, Felicia Kurniawati and Ross Jeffery
5 Packaging software process improvement issues: a method and a case study. Software:
Practice and Experience. Volume 34, Issue 14, Date: 25 November 2004, Pages: 1311-1344.
Tony Gorschek, Claes Wohlin.
6
Towards software excellence - informal self-assessment for software developers. Software Process: Improvement and Practice. Volume 8, Issue 3, Date: July/September 2003, Pages:
157-168. K. Daily, D. Dresner.
7 Software process matrix: a small company SPI model. Software Process: Improvement and Practice. Volume 6, Issue 3, Date: September 2001, Pages: 157-165. Ita Richardson.
8
Product management challenge to software change process: preliminary results from three SMEs experiment. Software Process: Improvement and Practice. Volume 3, Issue 3, Date: September 1997, Pages: 165-175. Tapani Kilpi
9
Software process improvement in very small enterprises: does it pay off?. Software Process: Improvement and Practice. Volume 4, Issue 4, Date: December 1998, Pages: 209- 226. Karlheinz Kautz
10
A pattern approach to software inspection process improvement. Software Process:
Improvement and Practice. Volume 10, Issue 4, Date: October/December 2005, Pages: 455- 465. Lasse Harjumaa.
11
Managing the complexity of SPI in small companies. Software Process: Improvement and Practice. Volume 5, Issue 1, Date: March 2000, Pages: 45-54. Romana Vajde Horvat, Ivan Rozman, József Györkös
12
A minimal test practice framework for emerging software organizations. Software Testing, Verification and Reliability. Volume 15, Issue 3, Date: September 2005, Pages: 145- 166. Daniel Karlström, Per Runeson, Sara Nordén
13 SPICE for small organisations. Software Process: Improvement and Practice. Volume 9, Issue 1, Date: January/March 2004, Pages: 23-31 Angela Tuffley, Bill Grove, Gary McNair 14 A process framework for small projects. Software Process: Improvement and Practice.
Volume 6, Issue 2, Date: June 2001, Pages: 67-83. Hareton K. N. Leung, Terence C. F. Yuen
Volume 5, Issue 4, Date: December 2000, Pages: 243-250. J. Batista, A. Dias De Figueiredo
17
An evaluation of CMMI process areas for small- to medium-sized software development organisations. Software Process: Improvement and Practice. Volume 10, Issue 2, Date:
April/June 2005, Pages: 189-201. F. G. Wilkie, D. McFall, F. McCaffery
18
A practical application of the IDEAL model. Software Process: Improvement and Practice.
Volume 9, Issue 3, Date: July/September 2004, Pages: 123-132. Valentine Casey, Ita Richardson.
19
Identification of Software Process Knowledge Management. Software Process:
Improvement and Practice. Volume 7, Issue 2, Date: June 2002, Pages: 47-55. Bridget Meehan, Ita Richardson
20
Helping Small Companies Assess Software Processes. Found in: IEEE Software. By Christiane Gresse von Wangenheim, Alessandra Anacleto, Clenio F. Salviano. Publication Date: January 2006. pp. 91-98
21 Low-rigour, Rapid Software Process Assessments for Small Software Development Firms. Found in: 2004 Australian Software Engineering Conference (ASWEC'04). By Aileen P. Cater-Steel. Publication Date: April 2004. pp. 368.
22 Practical Software Process Improvement-The IMPACT Project. Found in: 13th Australian Software Engineering Conference (ASWEC'01). By Louise Scott, Ross Jeffery, Lucila Carvalho, John D'Ambra, Philip Rutherford. Publication Date: August 2001 . pp. 0182
23
Tackling the Complexity of Requirements Engineering Process Improvement by Partitioning the Improvement Task. Found in: 2005 Australian Software Engineering Conference (ASWEC'05). By Uolevi Nikula, Jorma Sajaniemi. Publication Date: March 2005. pp. 48-57
24 Making Sense of Measurement for Small Organizations. Found in: IEEE Software. By Karlheinz Kautz. Publication Date: March 1999. pp. 14-20
25
Software Process Improvement in Small Organizations: A Case Study. Found in: IEEE Software. By Kathleen Coleman Dangle, Patricia Larsen, Michele Shaw, Marvin V.
Zelkowitz. Publication Date: November 2005. pp. 68-75
26 Adopting the SW-CMM in a Small IT Organization. Found in: IEEE Software. By Felipe Guerrero, Yadran Eterovic. Publication Date: July 2004. pp. 29-35
27
An Experience in Combining Flexibility and Control in a Small Company´s Software Product Development Process. Found in: 2003 International Symposium on Empirical Software Engineering (ISESE'03). By Kristian Rautiainen, Lauri Vuornos, Casper Lassenius.
Publication Date: September 2003
28 Process Improvement for Small Organizations. Found in: Computer. By Declan P. Kelly, Bill Culleton. Publication Date: October 1999. pp. 41-47
29
An Experience: A Small Software Company Attempting to Improve its Process. Found in: Software Technology and Engineering Practice. By Sergio Otoya, Narciso Cerpa . Publication Date: August 1999. pp. 153
30 Process Improvement Towards ISO 9001 Certification in a Small Software
Organization. Found in: 20th International Conference on Software Engineering (ICSE'98).
By Elif Demirors, Onur Demirors, Ouz Dikenelli, Billur Keskin. Publication Date: April 1998
31 Key Decisions in Strategic New Product Development for Small Software Product Businesses. Found in: 29th Euromicro Conference (EUROMICRO'03). By Jarno Vahaniitty.
Publication Date: September 2003. pp. 375.
32 Software Process Improvement in Web Time. Found in: IEEE Software. By Karl Wiegers.
Publication Date: July 1999. pp. 78-86
33 A software assessment process for small software enterprises. Found in: 23rd
EUROMICRO Conference '97 New Frontiers of Information Technology. By P. Grunbacher.
Publication Date: September 1997. pp. 123
34 A Tool to Define Electronic Process Guides in Small Software Companies. Found in:
2004 Australian Software Engineering Conference (ASWEC'04). By Torgeir Dingsoyr, Nils Brede Moe.
Publication Date: April 2004. pp. 350
35 Software Process Improvement Planning with Neural Networks. Found in: 24 th.
EUROMICRO Conference Volume 2 (EUROMICRO'98). By Volkmar H. Haase. Publication Date: August 1998. pp. 20808
36 Improving the Customer-Supplier Relation in IT Development. Found in: 33rd Hawaii International Conference on System Sciences-Volume 4. By Gro Bjerknes, Lars Mathiassen.
Publication Date: January 2000. pp. 4044
37
A Process-Centred Experience Repository for a Small Software Organisation. Found in:
Ninth Asia-Pacific Software Engineering Conference (APSEC'02). By Louise Scott, Lucila Carvalho, Ross Jeffery. Publication Date: December 2002. pp. 603
38
The Long-term Effects of an EPG/ER in a Small Software Organisation. Found in: 2004 Australian Software Engineering Conference (ASWEC'04). By Felicia Kurniawati, Ross Jeffery. Publication Date: April 2004. pp. 128
39
Applying and adjusting a software process improvement model in practice: the use of the IDEAL model in a small software enterprise. Karlheinz Kautz, Henrik Westergaard Hansen, Kim Thaysen. June 2000. Proceedings of the 22nd international conference on Software engineering. Publisher: ACM Press.
40
An Experience on Implementing the CMMI in a Small Organization Using the Team Software Process. Proceedings of the First International Research Workshop for Process Improvement in Small Settings. January 2006. Miguel A. Serrano, Carlos Montes de Oca, and Karina Cedillo. p.p. 81-92
41
MoProSoft®: A Software Process Model for Small Enterprises. Proceedings of the First International Research Workshop for Process Improvement in Small Settings. January 2006.
Hanna Oktaba. p.p. 93-101
42
The Capability Maturity Model (SW and Integrated) Tailored in Small Indigenous Software Industries. Proceedings of the First International Research Workshop for Process Improvement in Small Settings. January 2006. Rosario Blowers and Ita Richardson. p.p. 175- 181
43
A Software Process Improvement Solution for Small and Medium-Size Enterprises.
Proceedings of the First International Research Workshop for Process Improvement in Small Settings. January 2006. Jose A. Calvo-Manzano, Gonzalo Cuevas Agustín, Iván García Pacheco, Tomás San Feliu Gilabert, and Ariel Serrano. p.p. 182-192
44
The Implementation of SW-CMM Level 2–A Successful Case of a Brazilian Company’s Organizational Competence and Commitment. Proceedings of the First International Research Workshop for Process Improvement in Small Settings. January 2006. Ana Patrícia Silveira Viotti, Lilian Corrêa Fontana, and Gabriela Elisa da Cunha. p.p. 228-232
45
Two Case Studies in Implementing Model Based Process Improvement in Small Organizations. Proceedings of the First International Research Workshop for Process Improvement in Small Settings. January 2006. Mary Anne Herndon and Sandra Salars. p.p.
245-250
Table A.1 – Primary studies selected in the systematic review
Biographies
Francisco J. Pino is lecturer at the Electronic and Telecommunications Engineering Faculty at the University of Cauca, in Popayán (Colombia). His research interest is software processes improvement in small companies. He currently is student of PhD in Computer Science in the University of Castilla–La Mancha (UCLM), in Spain. Contact him at Universidad del Cauca, Street 5 No. 4 – 70 – Popayán, Colombia; [email protected].
Félix García is lecturer at the University of Castilla–La Mancha (UCLM). His research interests include business process management, software processes, software measurement and agile methods. He holds the MSc and PhD degrees in Computer Science from the UCLM, and is member of the Alarcos Research Group of that University, specialized in Information Systems, Databases and Software Engineering. Contact him at Escuela Superior de Informática, Paseo de la Universidad 4, 13071–Ciudad Real, Spain; [email protected].
Mario Piattini is full professor at the UCLM. His research interests include software quality, metrics and maintenance. He holds the PhD degree in Computer Science from the Technical University of Madrid, and leads the Alarcos Research Group. He is CISA and CISM by ISACA.
He leads the Joint SOLUZIONA–UCLM Software Research and Development Center. He is member of ACM and IEEE Computer Society. Contact him at Escuela Superior de Informática, Paseo de la Universidad 4, 13071–Ciudad Real, Spain; [email protected].
