REVISOR #1.ID_revisionR1Reviewercomments

(1)

REVISOR #1.

ID_revision R1 Reviewer

comments

I am not sure that the guidelines for systematic reviews have been applied sufficiently rigorously. For example, although the research question does have various components (e.g. see Appendix A's statement of population, intervention etc.) it still seems that the research question lacks

specific focus. As Kitchenham notes in her technical report:

"The most important activity during protocol [development]

is to formulate the research question" (p. 5)

Answer We completely agree with the importance to provide a specific focus for the research question. In order to clarify the research question the description of the population, intervention and outcomes has been improved. The objective reported in the abstract has been also clarified. This explanation has been applied on the whole article.

Changes in

the paper. Pg 1. Lines 52-56

Its objective is to establish research works that have been carried out for SPI focused on SMSEs, and also to have a real study case, in order to present a profile of SPI in SMSEs.

Its objective is to analyse the existing approaches about SPI in SMSEs which include a case study in industry, discussing the significant issues related to this knowledge area, and providing an up-to-date state of art in which to position innovative research activities.

Pg 3. Lines 7 – 9

Our objective is to discover what has already been carried out and achieved as regards software process improvement in this type of companies.

Our objective is to analyse the existing approaches about SPI in SMSEs which include a case study in industry, discussing the significant issues related to this knowledge area, and providing an up-to-date state of art in which to position innovative research activities.

Pg. 4. Lines 33-35

The population group that was observed were publications concerning SPI in SMEs, as found in sources selected for the review.

The observed population group was the set of proposals concerning SPI area that has been applied inside of SMSEs and which have been published in the sources list.

Pg 22. Lines 36-37

– Intervention: To observe the application of process improvement strategies in SMSEs.

Intervention: To analyse/study the different proposals about software process improvement which have been applied on SMSEs.

Pg. 22. Lines 45-46.

Outcomes measure: Number of improvement initiatives found.

Outcome: To present an up-to-date state of art about SPI proposals applied in SMSEs in which to position innovative research activities and to discuss the significant issues related to this knowledge area.

Pg 22. Lines 47-49

- Population: Publications related to SPI in SMEs, which appear in the selected sources for the systematic review.

Population: Research proposals in the software processes improvement area which have been applied inside of SMSEs and which have been published in the sources list.

Research question clarified in Pg. 4, Lines 19-21; Pg. 22 Line 29-32

Question: What approaches concerning SPI have focused on SMSEs and at the same time present a real study case?

Question: What are the existing approaches concerning SPI focused in SMSEs which report a case study?

This paragraph has been re-written in Pg. 9, Lines 33-39.

(2)

What approaches concerning SPI have focused on and been applied to SMSEs in the aspects of publication trends, companies involved, models used, process measurement, improvement proposals, improved processes and success factors?

These relevant findings is based on the discussion of significant aspects related to this knowledge area such as publication trends, size of companies involved, models used, process measurement, improvement proposals, improved processes and success factors.

comments I also think that there are various aspects of systematic reviews (e.g. sensitivity analysis, publication bias,

application of quality criteria) that have not been clearly discussed in the manuscript and this gives the impression that they were not performed during the review itself.

Answer The scope has been clarified and the limitations of the systematic review have been included in the new section. .

The quality criteria has been included, to see Revision R5 and T3.

Changes in the paper

New section in Pg. 18 Lines 26.

3.7 Limitations

The systematic review presented in this paper has some limits to be taken into account for understanding at which extent the results can be generalized and how they can be used:

– The review has been performed in the context of a PhD thesis. So, as the researchers group is reduced (a researcher and two supervisors), the scope of the revision had to be well defined and therefore for the first execution of the systematic review procedure the secondary searches were not taken into account.

– Publication bias was not analysed as all the primary studies reported only positive results.

– Sensitivity analysis was not applied, given the heterogeneous nature of primary studies results which made difficult the quantitative comparison and so not viable to carry out meta-analysis. Additionally, there are not standardized/universally accepted criteria to report results in the field of SPI.

A continuación se enumeran las principales limitaciones de esta revisión sistemática:

Las limitaciones propias de una revisión sistemática realizada en el contexto de una PhD Theses. Ya que como el grupo investigador es reducido (a single researcher and two supervisors) hay que limitar muy bien el alcance de la revisión. Debido a esto para la primera ejecución del procedure for execution of the systematic review no se han tenido en cuenta las search secundaries.

Las publications bias ya que todos los estudios primarios presentan únicamente resultados positivos.

El sensitivity analysis por la imposibilidad de realizar meta-analysis, ya que debido a la heterogeneidad de las propuestas y de los resultados presentados en los estudios primarios es muy complicada la comparación de los análisis. Además que no hay criterios

uniformemente aceptados para reportar los resultados en la mejora de procesos.

comments

1. The title of the manuscript, and many other places in the manuscript, refer to "small and medium" software

enterprises. The basic search strings given in Table 1 only search on "small" enterprises, organizations,companies, teams etc. Section 3.1 refers to "small" but not to

"medium". Also, on p.6 of the manuscript, there is a

discussion of the key word "small" but no discussion of the keyword "medium". Section 4.2 refers to "micro-companies"!

Answer In order to clarify this point a brief description of the Planning Review phase (according to Biolchini, 2005) has been included. As a matter of fact, in the activity “test the protocol execution on one source” we initially considered the word “Medium”, as small and medium are part of the acronym SME. However, when the search of this first iteration was performed by using “small” and “medium” words, it was observed that the set of publications was a subset of the returned publications by using only the word

“small”. Because of this, only the work “small” was included in the definitive search

(3)

strings.

The word “micro-companies” has been deleted as it does not affect to the article contents and to provide more clarity in the use of terminology according to the scope of the systematic review.

Changes in

the paper To see Review S4.

Pg. 11. Lines 47 – 59.

Currently, SPI efforts are also being applied to small and micro-companies (see figure 4).

In this figure sample 47% of the companies involved in the improvement efforts were small (fewer than 50 employees) and 33% were micro-companies (between 1 and 9 employees), which gives us the sum of 80% of all the reported companies in which an SPI effort was carried out. Only 20% is reported in medium sized companies (between 50 and 249 employees). However, it is important to underline that in most cases the improvement programmes in micro and small software companies did not lead to the ISO or SEI certifications.

Currently, SPI efforts are also being applied to small companies (see figure 4). In this figure sample 47% of the companies involved in the improvement efforts were companies with a number of employees between 10 and 49 and 33% were companies with fewer than 10 employees, which gives us the sum of 80% of all the reported companies in which an SPI effort was carried out. Only 20% is reported in medium sized companies (between 50 and 249 employees). However, it is important to underline that in most cases the improvement programmes in small software companies did not lead to the ISO or SEI certifications.

Pg. 14. Lines 26 -28

It can be concluded that for the small and micro companies the SEI models do not lead to a certification.

It can be concluded that for the small software companies the SEI models do not lead to a certification.

Pg. 16. Lines 15 - 18

This allowed us to have a vision of what the improvement efforts in micro, small, and medium size companies had focused on.

This allowed us to have a vision of what the improvement efforts in small and medium size companies had focused on.

Pg. 18. Lines 55 - 60

Both small and micro companies are undertaking SPI efforts nowadays, but for these companies SEI or ISO models do not lead to their receiving certification.

The small companies are undertaking SPI efforts nowadays, but for these companies SEI or ISO models do not lead to their receiving certification.

comments 2. The abstract states: "... and also have a real study case..." (final sentence of abstract). This is somewhat confusing because it suggests that the manuscript will provide a real case study when in fact what I think the authors mean is that all of the publications selected in the systematic review should refer to real cases in industry. So some clarification on what is meant by "real study case"

would be helpful.

Answer This paragraph has been clarified.

Changes in

the paper To see the following changes in the paper: Pg 1. Lines 52-56, Pg 3. Lines 7-9 from revision R1.

comments 3. Following on from point 2, there is a further issue: *if*

all of the publications selected in the systematic review are intended to report on real cases (e.g. as case studies,

(4)

or surveys or experiments) then how is this reflected in the basic search strings of Table 1? I *think* that the authors are using their inclusion criteria to include publications that report empirical studies of real cases; in other words, the search strings of Table 1 are not intended to search on real case studies. It would help if the authors could

clarify these issues in the manuscript.

Answer We thank for the suggestion. We’ve included a paragraph to clarify this issue.

Changes in

the paper New paragraph in Pg. 5. Line 4.

As the basic search strings of Table 1 were not intended to search on specific results with empirical studies in the software industry (e.g. case studies), according to our research question we used the inclusion and exclusion criteria in order to select only the

publications which report case studies. Therefore, the "quality threshold for selection of studies" is defined as the studies which present an "evidence obtained from case series".

New paragraph in Pg. 23. Line 45.

Since search strings are not intended to search empirical studies in the software industry, we defined the inclusion and exclusion criteria in order to include publications that report case studies.

comments 4. It is not clear to me how the manuscript defines "small"

(or "medium" for that matter). Page 2 refers to Fayad et al.'s 'definition' of small being companies with fewer than 50 employees, however Figure 4 reports on companies with up to 249 employees. What's the definition of small and medium?

Presumably a definition of "small" and "medium" should form part of the inclusion or exclusion criteria? For example, of the publications that have been selected through searching,

*include* those publications that refer to companies who have less than 50 employees?

Answer In order to clarify how the manuscript defines “small” and “medium” companies, definitions of small and medium-size companies has been included.

Changes in

the paper Include paragraph in Pg. 4 Line 35.

In the context of the present work and according to (EU, 2005) small enterprises are companies with fewer than 50 employees and medium enterprises are companies which have between 50 and 249 employees.

In References page

EU, 2005. The new SME definition. User guide and model declaration. Available on:

http://ec.europa.eu/enterprise/enterprise_policy/sme_definition/sme_user_guide.pdf.

Accessed: April, 2007.

comments 5. I am not sure that the research question (as stated in section 3.1 and Appendix A) clearly relates to the results reported in section 4. Section 4 reports on: publication trends (figure 2), countries from which SMSEs have

undertaken SPI ( figure 3), number of employees (figure 4), standards (Table 4) and various models (Figure 5 and 6), improvement proposals (Table 5), etc. . The research

question is: what approaches concerning SPI have focused on SMSEs and at the same time present a real study case? (taken from section 3.1). For this question, Figure 4, Table 4 and Figures 5 - 6 seem relevant.

Answer According to this comment we have eliminated the figure 3 whose results are less related to the research question. The figure 5 has been also deleted as their results can be extracted from table 4.

However, we think that the information provided by tables 4, 5 and figure 4, 6 (standards

(5)

used by each study, assessment and reference models, improvement proposals and size of companies) is related with significant issues to discuss to inside this knowledge area in order to providing an up-to-date framework in which to position new research activities.

From the perspective of found primary studies we also consider that the trend of

publications shown in Figure 2 provides useful information to justify the growing interest of the research community on this area.

In order to clarify this aspect the paragraph has been re-written.

Changes in

the paper Pg. 9. Lines 29-39

With the information extracted from the studies, a statistical analysis was conducted whose purpose was to show the relevant findings of the systematic review that had been carried out. These results (presented in this section) give an answer to the original research question: What approaches concerning SPI have focused on and been applied to SMSEs and at the same time include a study case? Based on aspects such as publication trends, companies involved, models used, process measurement, improvement proposals, improved processes and success factors.

In this systematic review a research question, which is of interest in the industry applied SPI research area, has been formulated. The aim is to identify new research activities to do in the field of SPI through the current state of art. The overview of the state of art presented in this paper comes from the information extracted of the 45 primary studies found as a result of the systematic review. With the information extracted from the studies, a statistical analysis was conducted whose purpose was to show the relevant findings of the systematic review that had been carried out. These relevant findings is based on the discussion of significant aspects related to this knowledge area such as publication trends, size of companies involved, models used, process measurement, improvement proposals, improved processes and success factors.

comments 6. Related to point 5, the research question is phrased differently in different parts of the manuscript. For

example, section 4 (p. 9) refers to the "original question":

"What approaches concerning SPI have focused on and been applied to SMSEs in the aspects of publication trends, companies..." which is differently phrased to the question in section 3.1 and to the question in Appendix A. In some respects this is a minor point, but it does increase confusion over what exactly is being investigated in the systematic review.

Answer The research question statement is now homogeneous in the whole article.

Changes in

the paper To see the following changes in the paper: Research question clarified in Pg. 4, Lines 19-21; Pg. 22 Line 29-32 and Pg. 9, Lines 33-39 from R1.

comments

7. Section 3.4 refers to "Figure A": should this be Table 3?

Answer Fixed.

Changes in

the paper Pg 8. Line 7.

Figure A by Table 3 ID_revision R10

Reviewer comments

8. I am not sure that presenting the figures (which essentially plot only two dimensions of data) in three dimensions adds anything.

Answer The figures has been changed to two dimension versions.

(6)

comments

9. I am not sure what the benefit is in reporting each company in Table 4. Perhaps it would be just as effective to just report the total number of companies for each Standard.

I also note that the label in Table 4 refes to "Standars"

rather than "Standards"?

Answer The table 4 has been changed to address this comment.

Pg. 12. Lines 46-56

comments

10. Table 5 is some what confusing. What does the percentage column refer to? Percentage of publications? Percentage of companies? How have you extracted the improvement proposals from the publications? How did you define the 6 different categories of improvement proposals? On this item, Table 3 simply refers to "Use an improvement framework in order to guide the programme SPI."

Answer The percentage columns refers to percentage of publications. To clarify this point the label % has been added. On the other hand, a new and short paragraph has been included to explain how the categories presented in table 4 were defined.

The label % of studies in the first column of Table 4 has been included. Pg. 15 Lines 13-14.

New paragraph in Pg. 15. Line 33.

The improvement proposals categories (from IP1 to IP6) shown in table 4 were defined through a deep analysis and later synthesis of the SPI proposals extracted from every primary study. First of all, every publication was completely analysed and then a summary of the main ideas and contributions was extracted. The next step was to group the SPI contributions which were related to a more general concept in different

categories, resulting eight categories in total. For example, the improvement proposals “to use an improvement framework in order to guide the programme SPI” (see table 3) and

“to follow the concept of action packages in order to carry out the implementation of the SPI in SMEs” (Calvo-Manzano et al., 2002) can be grouped under the category IP4 “to guide improvement efforts”.

comments

a) review the conduct of the systematic review, taking account of Kitchenham's guidelines and advice (e.g. "lessons learned") from other published systematic reviews;

Answer The Kitchenham’s guidelines and the works followed to carry out the current systematic review have been explicitly stated in this new version. Additionally, the context under which the review was conducted is explained

Changes in

the paper Pg. 4 Lines 7-9.

To carry out the systematic review of SPI in SMSEs, the protocol template presented by Biolchini in (Biolchini et al., 2005), was followed.

The systematic review which follows has been carried out in the context of a PhD Thesis whose aim is to provide small companies in Latin America with a process improvement model tailored to their special characteristics and which is part of the project

COMPETISOFT (Oktaba et al., 2007). For a successful application of the method the guidelines presented in (Kitchenham, 2004), the protocol template presented in (Biolchini et al., 2005) and (Biolchini et al., 2007) were followed and other similar systematic reviews were considered (Jørgensen et al., 2007; Hannay et al., 2007)

Paragraph in Pg.22 Line 8.

To carry out the systematic review of SPI in SMSEs the protocol template presented by Biolchini was followed.

To carry out the systematic review of SPI in SMSEs the protocol template presented in (Biolchini et al., 2005), the Kitchenham's guidelines for systematic reviews presented in (Kitchenham, 2004) and some advices from other published systematic reviews were

(7)

followed.

In References page

Kitchenham, B., 2004. Procedures for performing systematic reviews, Keele University and ICT Australia Ltd. Pp: 1-28.

Biolchini, J., P. Gomes, A. Cruz, T. Uchôa and G. Travassos, 2007. Scientific research ontology to support systematic review in software engineering. Advanced Engineering Informatics Vol. 21(2) April pp. 133-151.

Hannay, J. E., D. I. K. Sjøberg and T. Dybå, 2007. A Systematic Review of Theory Use in Software Engineering Experiments. IEEE Transaction on Software Engineering Vol.

33(2) February pp. 87-107.

Jørgensen, M. and M. J. Shepperd, 2007. A Systematic Review of Software Development Cost Estimation Studies. IEEE Transaction on Software Engineering Vol. 33(1) January pp. 33-53.

(8)

REVISOR # 2.

ID_revision S1 Reviewer

comments

- page 5, line 7 talks about the "planned list", but no rationale for selection is given.

Answer - The rationale for selection of sources list is given.

Changes in

the paper Include paragraph in Pg. 5. Line 21.

The sources have been identified according to expert opinion in the work area which is the scope of the systematic review. Inside these sources there are very important journals in which the research area is widely treated, as such as: Information and Software Technology, Software: Practice and Experience, Software Process: Improvement and Practice, IEEE Software, Software Technology and Engineering Practice, Computer, among others.

comments

- page 19, line 36 talks about the identification of new sources being identified

Answer The paragraph has been clarified.

Changes in

the paper Pg. 19. Lines 35-40.

Using as a starting point the results of this systematic review, new information sources will be reviewed in new iterations following the procedure for execution of the systematic review suggested.

Using as a starting point the protocol of this systematic review, new sources (e. g.

Springer or conferences related to SPI) will be reviewed in future iterations following the procedure for execution of the systematic review presented in this work.

comments

- page 22, line 58 states that the source selection criteria is "electronic journals which can be access via the Internet from the library digital service of the University of

Castilla-La Mancha

Answer The paragraph was deleted. A new paragraph has been written.

Pg. 22. Lines 58-62.

Electronic journals which can be accessed via Internet from the Library digital service of the University of Castilla-La Mancha (http://www.biblioteca.uclm.es).

The sources have been identified according to expert opinion in the work area which is the scope of the systematic review.

Pg. 23. Lines 58-62. The paragraph was deleted.

The sources have been identified according to expert opinion in the work area which is the scope of the systematic review.

comments

There is no real discussion of why the specific keywords were used, nor the specific logical combinations used in the search strings. At the very least, some sort of pre-scan to determine the utility and effectiveness of both parts of this could have been considered and described. The specific focus on SPI and no use of common synonyms could easily have prejudiced the quality and utility of the work.

Answer A new paragraph has been included to explain how the evaluation of the protocol was performed.

Changes in

the paper Include this paragraph in Pg. 24. Line 24.

The protocol evaluation of this systematic review was performed through:

− Protocol excecution testing on one source in the first iteration of the review. During this iteration the initially planned protocol was evaluated and improved for the achievement of the review objectives, giving as a result the protocol presented in this work.

− Periodical meetings with two experts in the researched field in order to validate the protocol and to track the review execution to assure it is being carried outA través de

(9)

reuniones periódicas con dos expertos en área investigada para validar el protocolo y hacer seguimiento a la ejecución de la revisión para garantizar que se realice

adecuadamente in a suitable way.

− El resto no me convence demasiado. Lo hablamos:

− Los elementos más dinámicos en un principio del protocolo son las palabras clave y las cadenas de búsqueda. Por ejemplo, con respecto a las palabras claves se tuvo en cuenta inicialmente la palabra “medium”, así pues al hacer la búsqueda con las palabras claves small y médium ésta generaba menos resultados los cuales estaban contenidos en la búsqueda realizada con la palabra small. También se realizó una prueba considerando reemplazar la cadena de búsqueda “software process

improvement” by “software process assessment”, se pudo observar que con la nueva cadena de búsqueda se obtenían los mismos estudios primarios que con la inicial, sin reportar ningún nuevo estudio primario.

comments - the references in the manuscript to government support for SPI in SMEs: the authors ascribe cause and effect, whereas it is not clear to me whether this is just self-fulfilling;

Answer These references have been eliminated in the paper, as they do not provide an added value to the main contribution of the paper.

Changes in

the paper Pg. 10 Lines 27-62

The main software improvement initiatives in these countries have been promoted by means of the real commitment of governmental programmes, as for instance:

− Software Engineering Australia – SEA – is an industry-led body with the charter of improving the quality and reliability of software in Australia. SEA is supported by the Commonwealth through the Department of Communications, Information Technology, and the Arts. SEA provided funding for the Software Quality Institute – SQI– to deliver a process improvement programme to 22 small software development firms. SQI has recognized the need for approaches that can generate incremental improvements to demonstrate business benefits, particularly for smaller developers.

− Countries as Ireland, Finland and Sweden have participated in the European Software and System Initiative (ESSI) of the European Union the which have promoted different projects for strengthening SPI in SMEs, such as SPIRE (Software Process Improvement in Regions of Europe) (SPIRE, 1993) and TOPS (Toward Organized Software Processes in SMEs) (Esprit_Project, 1999), among others.

− In Brazil, the government has subsidized the implementation of the PBQP–Software (Productivity and Quality Software Programme) (Bedini et al., 2005) and the

“MPS.BR” project has been developed (melhoria do processo de software brasileiro) (Weber et al., 2005), both of which strive towards the same objective.

Pg. 11. Lines 7-13. The word “Too, ” has been deleted.

comments

- the assumption by the authors that the literature is specifically representative of real life, in respect of the proportionality of different kinds of SPI approach - I strongly doubt that this is the case, and in any case it is surely impossible to prove or establish without a huge amount of additional work.

Answer We completely agree that such statement is strong and huge quantity of additional work would be necessary to assure it. So, a new paragraph has been included to explain that the presented results actually provide an overview of the field based on selected primary studies. The whole article has been reviewed to clarify this assumption.

Changes in

the paper See changes in paper from R7.

The section 2 in Pg. 2 Lines 16-60 has been deleted.

(10)

comments

- much of page 3 seems to be low-value-add about the methodology, and could perhaps be assigned to an appendix;

Answer The section 2 has been deleted, as its initial objective was to provide a brief presentation of systematic reviews, but without included specific contributions on this topic.

Changes in

the paper The section 2 in Pg. 3 Lines 16-60 has been deleted.

comments

lines 45 & 46 of page 9 notes that there is "an increasing interest on ... the whole area of SPI and SMSEs", whereas we are dealing with very small amounts of data (or evidence from the study). This looks like an overstatement;

Answer This point has been fixed in the whole article by avoiding strong statements based on overgeneralization of results.

Changes in

the paper Pg 9. Line 46. The word whole has been deleted.

comments

- line 24 of page 10 notes the six countries which report most improvement efforts in SMSEs, and then goes on to note the specific support from governments in those same

countries. Cynically, I wonder if what is really being observed here is a requirement to publish results being a standard requirement of such funding;

Answer The paragrpahs related with “government support” have been deleted in this new version.

Changes in

the paper See changes in the paper from S5.

comments

- line 9 of page 11 refers to government "organisms", which I think should be "organisations".

Answer Fixed, Changes in

the paper Pg. 11. Line 9, “organisms” by “organizations” has been changed.

comments

- the first paragraph of page 11 has some interesting

assertions such as "company commitment is vital" without any citations or references. These look inappropriate without such references (whilst they may well be correct, they just look like assertions of the authors, and thus out of place);

Answer A suitable reference has been included.

In references page.

Dyba, T., 2005. An Empirical Investigation of the Key Factors for Success in Software Process Improvement. IEEE Transactions on Software Engineering Vol. 31(5) May pp.

410-424.

comments

Figure 3 on page 11 doesn't seem to add much value; if I read it correctly then 2% simply means one reference found?

Answer Yes., it does The Figure 3 has been deleted.

Changes in

As can be seen in Figure 3, the six countries which report most improvement efforts in SMSEs are, in descending order, Australia, Ireland, Finland, Denmark, Brazil and USA.

The countries which report most improvement efforts in SMSEs are, in descending order, Australia, Ireland, Finland, Denmark, Brazil, USA, Sweden and Mexico.

(11)

comments

- Section 4.3 starting on page 12 and going through to page 14 just made me think: "so what?". Not enough effort has been put into drawing useful conclusions - so perhaps there aren't any?

Answer Changes in the paper

comments

- line 28 of page 14 asserts "The motivation for this type of companies should therefore be to improve their processes, in order to make them effective and efficient units, rather than to gain the certification itself". Generally, I find that the authors do understand the differences between certification and non-certification use of standards (a common failing in articles submitted to SQJ), but I can't see where the authors conclude "therefore" in the quoted statement. Is this editorialising, or pulled from a specific report that they studied in more detail?

Answer The paragraph has been clarified.

Pg. 14. Line 28.

The motivation for this type of companies should therefore be to improve their processes, in order to make them more effective and efficient units, rather than to gain the

certification itself.

Together with the above considerations, we consider that the motivation for this type of companies should therefore be to improve their processes, in order to make them more effective and efficient units, rather than to gain the certification itself (Paulk, 1998).

In references page

Paulk, M. C., 1998. Using the Software CMM in Small Organizations. Proc. Joint 16th Pacific Northwest Software Quality Conf. and 8th Int’l Conf. Software Quality, 350-360.

comments

- lines 34 & 35 of page 14 state that "ISO 9001 helps to provide better support to the CMM initiative and to improve areas of the organisation that are not directly related to the development of the software". The authors clearly mean that compliance with ISO 9001 should have a broadly

beneficial effect on organisations beyond the scope of coverage of the CMM (so the shorthand they have used is poor), but such an assertion is beyond the scope of the study/article submitted for publication, and looks like editorialising. It could be argued that, without specific guidance applicable to software development, ISO 9001 on it's own has little relevance to software development

organisations, but arguing that is outside the scope of the article and this review;

Answer Changes in

the paper Lo hablamos ID_revision S16

Reviewer comments

- lines 37 & 38 point out that ISO 9001 also exists as harmonised national standards, but so what? By pointing this out, the authors further call into question the scope of their keyword search!

Answer The sentence has been deleted.

Changes in Pg. 14. Lines 37-38. The sentence has been delete.

(12)

the paper It should be remembered that in many countries the ISO 9001 standard is a national standard.

comments

- line 48 of page 14 uses the word "ratifies". This is surely far too strong, and should at best be "tends to support";

Answer Fixed.

Changes in

the paper Pg. 14. Line 48.

This ratifies the assumption that the SMSEs’ efforts are still insufficient to establish formal software measurement processes.

This tends to support the assumption that the SMSEs’ efforts are still insufficient to establish formal software measurement processes.

comments

- the final assertions made at the bottom of page 14 look far to strong & sweeping to come out of a sample size of 2!

Answer These strong assertions has been deleted Changes in

the paper , constructed from measurement frameworks such as Goal Question Metric (GQM) or Practical Software Measurement (PSM). The definition of these measurement models would help to make an evaluation more objective.

comments

- the final paragraph of page 15 looks like editorialising - there are no references or citations to back it up;

Answer This paragraph includes our a personal opinion of the authors from the obtained results, We have clarified it.

Changes in

From the analysis of the extracted primary studies, we it can be concluded that the improvement approaches can be generated at an organizational or a technical level.

comments - lines 40 to 43 of page 18 talk about the "current state of affairs" as if this is a universal statement, whereas it actually only applies to the literature that has been

studied (see earlier remarks about the representativeness of the literature searched);

Answer The paragraph writing has been refined in order to avoid strong generalization.

Changes in

In this article a systematic review of the literature on SPI efforts carried out in SMSEs has been presented, which has allowed us to obtain a comprehensive and thorough view of the current state of affairs in this area.

In this article a systematic review of the literature on SPI efforts carried out in SMSEs has been presented, which has allowed us to obtain a view about SPI in SMSEs through a representative set of related works obtained as a result of the systematic review.

comments

- lines 47 to 48 of page 18 state an assertion that there is an "ever-increasing interest in tackling SPI in SMSEs" - forgetting to say that it is the literature search that indicates this (see earlier remarks about the

representativeness of the literature searched);

Answer A new statement at the beginning of the paragraph has been included to tackle this aspect.

Changes in

From the obtained results in the current review, the first observation is that, in the last 5 years, the Software Engineering community has shown an ever-increasing interest in tackling SPI in SMSEs.

ID_revision S22

(13)

Reviewer comments

- lines 50 & 51 of page 18 talk about the "real commitment of governmental programmes" - this looks like

editorialising;

Answer The sentence has been deleted.

comments - line 59 of page 18 uses the word "then", but the

associated argument is not presented - decisions about this are surely the remit of SMSEs and the commercial and other pressures they face concerning certification, not for the authors to editorialise about!

Answer This paragraph has been deleted Changes in

the paper Paragraph deleted Pg. 18. Lines 59-61.

Following these models, should then be a means towards, and not an end of, improvement.

comments - line 9 of page 19 uses the words "where the use of the three models" whereas I think it should read: "where the use of all three models";

Answer Fixed.

Changes in

(14)

REVISOR #3.

ID_revision T1 Reviewer

comments

The authors emphasise the importance small and medium sized software enterprises; claiming that these companies are the mainstay of the software industry. However, this claim is not substantiated.

Answer A reference has been included to support such assertion.

Changes in

the paper In References page.

Fayad, M. E., M. Laitinen and R. P. Ward, 2000. Software Engineering in the Small.

Communications of the ACM Vol. 43(3) March pp. 115-118.

comments

A weakness of this paper is the assumption that the results of a Systematic Literature Review are generalisable (i.e.

apply to the whole population). It might help the reader if we are given a general idea as to how many SMSEs there are (roughly), so that the 45 selected 'different' studies are placed in context with the population. The conclusions discuss finding in terms of what is happening with SMSE's and SPI - whereas the results are in fact attempting to summarise the literature on SMSE's and SPI.

Answer This aspect has been deeply tackled in the new version of the paper. It has been improved the writing of the article in order to avoid strong generalization according to the scope of the review.

Changes in

the paper To see Review S6, S8, S20, S21.

comments A further concern is the lack of attention to the quality of the publications included in the systematic review. The authors quote Kitchenham's SLR guidelines (2004), but have not followed her recommendation that all papers are

subjected to a quality assessment. Although the authors state that all papers have already been peer reviewed, it is likely that they are of variable quality. For example are they all based on randomised trials? Are the sample sizes all the same (we know they are not from the analysis)? How did the authors explain their data collection/analysis? I would imagine that the SQJ would find this of particular importance.

Answer A paragraph about the quality of the publications has been included.

Changes in

the paper To see Review R5. New paragraph in Pg. 5. Line 4.

New paragraph Pg. 9. Line 24. TENEMOS QUE HABLAR PARA ACLARAR ESTA TRADUCCION. El texto quedaba ambiguo.

Regarding the quality of the included publications in the systematic review it can be stated that primary studies are observational ones with design type case series, by means of which the formulated improvement proposals were applied on SMEs. From these design type case series, 82% are post-test and 18% are pre-test post-test. In order to Con respecto a la calidad de las publicaciones incluidas en la revisión sistemática se puede decir que los estudios primarios son observational studies with design type case series, mediante el cual las SMEs fueron intervenidas con la propuesta de mejora formulada. De estos design type case series 82% son post-test and 18 % son pre-test post-test. Para present the results, qualitative analysis has been applied. However, the 27% of the studies present quantitative analysis of some of results. Besides, the 42% of the studies report obtained evidence from 2 or more SMEs.presentar los resultados se utiliza el análisis cualitativo, sin embargo el 27% de los estudios presentan análisis cuantitativo de algún resultado. Además, el 42% de los estudios reportan evidencia obtenida desde 2 ó más SMEs.

(15)

comments

Would like a definition of SME earlier in the paper (what size is a 'small' company; what size is a medium company, what is the size of a 'micro' company?) - sizes given on page 11, section 4.2.

Answer Definitions have been included.

To see Review R6.

comments

Some grammar/definitions need checking, e.g. the word 'this' is often applied to a plural. Also, does the acronym SMSE really stand for Small and medium software 'organisations' [as given in the abstract], and not 'enterprises'?)

Answer Fixed.

comments The abstract might be misleading when it states that the systematic literature review is a real study case, (which suggests primary research); the systematic literature review aggregates findings of several studies to plot the

landscape/give an overview and patterns of

behaviour/practice as reflected in the published work in the area of interest.

Changes in

the paper To see change in the paper Pg 1. Lines 52-56, Pg 3. Lines 7-9 from revision R1.

comments

Section 3: 3.1 why isn't ''medium' one of the key words? - Also the research question might be slightly less cumbersome if you use the word 'empirical' rather than real study case?

- If empirical is what is meant by 'study case' of course!

Changes in

the paper To see revision R3 and S4.

To see change in the paper Pg 1. Lines 52-56, Pg 3. Lines 7-9 from revision R1.

comments

3.2 Workshop selection is unclear in final paragraph.

Changes in

the paper Pg. 5 Lines 21-25.

we chose this workshop because an overview of research from the world-wide

community which addresses the unique issues of process improvement in small settings (including small teams, small projects, small organizations, and small businesses) is presented.

we chose this workshop because is a specialised event organised by SEI on which significant proposals of process improvement in small settings (small teams, small projects, small organizations, and small businesses) are presented. . So, a significant number of the works presented in this source are more likely to be included in the primary studies list than by considering other non specialized workshops on this topic.

comments 3.3 What about secondary searches, did authors find any papers referenced in primary studies that were worth

following up? - It looks as if this is covered in the flow

(16)

chart under "more discovered studies" perhaps? Later, at end of page 7, authors do state they do not do a follow up search. Perhaps explain why.

Answer We have explained it in the paper. In order to manage the searches, we propose that from the analysis of primary studies a list with the new potential primary studies must be generated (secondary searches). Once the search is finished and the information has been obtained from selected sources, the former studies of the list must be subjected to a new procedure, which has been adapted but is similar to former one, called “procedure for gathering information”. A new version of the flow chart where this aspect is clarified has been included.

No se realizó secondary searches, sin embargo la tendremos en cuenta como trabajo furturo.

Changes in

the paper New paragraph in Pg. 19. Line36.

Using as a starting point the results of this systematic review, new information sources will be reviewed in new iterations following the procedure for execution of the systematic review suggested.

Using as a starting point the protocol of this systematic review, secondary searches from primary studies and new studies sources (e. g. Springer or conferences related to SPI) will be reviewed in new iterations following the procedure for execution of the systematic review suggested.

New paragraph in Pg. 7. Line 54

In this systematic review potential references inside primary studies were not searched.

The reason is that the intention was to manage better the searches by means of firstly to evaluate the defined sources and then in a future iteration to carry out secondary searches from the identified primary studies.

comments

To give more confidence in the paper selection, it would help if the authors confirmed whether their protocol was validated, and if so how was it validated and by whom? What were the plans for disagreements between researchers? How many researchers took part in the selection? Did they perform any inter-rater reliability tests to validate inclusion/exclusion decisions?

Answer As the protocol was validated a new paragraph to explain this has been included.

Additionally, the context under which the review was conducted is explained Changes in

the paper To see Revision S4 and Revision R13.

comments

Figure 1: This 'flow chart' works well as a means of

representing the complex, iterative review selection process and I found this a main contribution of the paper. (Although perhaps using black as the background doesn't help the presentation). Other researchers, unfamiliar with conducting SLRs could follow this flow chart that on preliminary

inspection looks accurate (I haven't tested it out fully).

It may not be complete however, as it is lacking a 'quality' check. (The assumption that the work has been peer-reviewed and therefore doesn't need quality assessment is weak, especially as these authors have included the grey literature in their searches). The flow chart is also lacking a validation or inter-rater reliability process.

Answer The flow chart has been improved with quality check and inter-rater reliability process. As an inter-rater reliability process compares abstraction results between two abstractors to assess the comparability of findings, this process has been established to evaluate primary studies and extracted dates to in order to maintain consistency among the a review coordinator.???

(17)

Changes in

the paper Figure 1. Pg. 6. Line 8-44.

comments

Another strength of this paper is the concise reporting of the extraction process as shown in Table 2. This approach helps other researchers/practitioners replicate the study.

However, the category 'proceedings' is fairly vague.

Answer It has been clarified in table 2 that the proceedings are from the SEI workshop.

Pg. 7 Line 48.

comments

Section 3.3: page 7: second bullet point: The second level studies: do the authors mean a theoretical study, not based on direct/empirical observation?

Answer Fixed.

Changes in

the paper Pg. 7 Line 48.

A second level, where the study deals with SPI initiatives for SMSEs which nevertheless have never been applied in such companies, i.e (is a theoretical study not based on direct/empirical observation, there is no study case applied to SMSEs, for instance (Brodman et al., 1994)).

comments

Section 4: Interpretation of results: The authors need take care in how they present their results: For example, in Figure 2, they say there is a growing interest in SPI as shown in the increase in publications - however, the increase in publications in this area might be due to the fact that more papers are being published in many areas of software engineering?

Answer The interpretation of results has been reinforced by describing that also different standards, projects and proposals have appeared in relation with this topic.

Changes in

the paper Pg. 9. Line 47-51.

This is indicated by the increasing number of studies that deal with the subject, according to the analysis of the trend in publications that was carried out (see Figure 2).

This is indicated by the appearance of a great number of standards and proposals related to SPI for SMEs, and also the increasing number of studies that deal with the subject, according to the analysis of the trend in publications that was carried out (see Figure 2).

comments 4.2 Companies involved. The searches picked up that micro- companies are also applying SPI. However, as the term micro wasn't in the original search, it is possible that not all micro companies that have published case studies are

represented here. This is not noted.

Answer In order to avoid confusion, it has been decided only to include the terms “small”

and “médium” companies Changes in

the paper To see Revision R3.

comments The conclusion: If the results are based on following the review protocol, we need to know that the protocol has been validated in a 'systematic' way.

Answer The protocol was systematically validated.

Changes in

the paper To see Revision S4 and Revision R13.

ID_revision T17

(18)

Reviewer comments

The conclusion that the systematic review of the literature on SPI efforts carried out in SMSEs allows authors to obtain a 'comprehensive' and thorough view of the current state of affairs in this area is erroneous. The systematic literature review is only comprehensive if every SME involved with SPI has published their work, and the SLR has picked up on all the published work. The reporting of results is based solely on the published work, which may be a small percentage of the population.

Answer The word comprehensive has been replaced by representative to explain better the focus.

Changes in

In this article a systematic review of the literature on SPI efforts carried out in SMSEs has been presented, which has allowed us to obtain a view about SPI in SMSEs through a representative set of related works obtained as a result of the systematic review.

comments All results should be in terms of what the literature is saying - that may or may not apply to the whole

population. There is no note of publication 'bias' - as companies who 'fail' at SPI are less likely to publish their work?

Answer The scope has been clarified and the limitations related to “publications bias” of the systematic review have been included in the new section.

Changes in

the paper To see Revision R2.

comments

Page 18: "Following these models, should then be a means towards, and not an end of, improvement". The fact that companies receive certification (or not) does not suggest an 'end' to improvement. It is a continuous process.

Answer This paragraph has been deleted Changes in

the paper Paragraph deleted Pg. 18. Lines 59-61.

comments

Page 19: Not sure of these categorisations (top of page), e.g. CMM process model, ISO.. assessment method; IDEAL model guides improvement (CMM also guides improvement doesn't it?)

Answer We have followed the SWEBOK 2004 categories, according to which CMM is classified (knowledge area 9) as a “process assessment model”. IDEAL is included into “Process Implementation and Change ” category, . We have included the reference to SWEBOK to clarify this point.

Nosotros hemos tomado como base las categorías propuesta por el SWEBOK 2004.

En el área de conocimiento 9, clasifica a CMM como un Process assessment models, al cual muchos autores llaman hoy día “process referente model” or “process model”. IDEAL se incluye dentro de los “Process Implementation and Change” el cual también es conocido como Process Improvement Guide or Model.

Changes in

the paper Incluir los cambios necesarios