REVISOR #1.ID_revisionR1Reviewercomments

(1)

REVISOR #1.

ID_revision R1 Reviewer

comments

I am not sure that the guidelines for systematic reviews have been applied sufficiently rigorously. For example, although the research question does have various components (e.g. see Appendix A's statement of population, intervention etc.) it still seems that the research question lacks

specific focus. As Kitchenham notes in her technical report:

"The most important activity during protocol [development]

is to formulate the research question" (p. 5)

Answer We completely agree with the importance to provide a specific focus for the research question. In order to clarify the research question the description of the population, intervention and outcomes has been improved. The objective reported in the abstract has been also clarified. This explanation has been applied on the whole article.

Changes in

the paper. Pg 22. Lines 47-49

- Population: Publications related to SPI in SMEs, which appear in the selected sources for the systematic review.

Population: Research proposals in the software processes improvement area which have been applied inside of SMSEs and which have been published in the sources list.

Pg. 4. Lines 33-35

The population group that was observed were publications concerning SPI in SMEs, as found in sources selected for the review.

The observed population group was the set of proposals concerning SPI area that has been applied inside of SMSEs and which have been published in the sources list.

Pg 22. Lines 36-37

– Intervention: To observe the application of process improvement strategies in SMSEs.

Intervention: To analyse/study the different proposals about software process improvement which hasve been applied on SMSEs.

Pg. 22. Lines 45-46.

Outcomes measure: Number of improvement initiatives found.

Outcome: To Ppresent an up-to-date framework overview about SPI focused proposals and applied in SMSEs in which to position new research activities and to , discussing the significant issues related to this knowledge area.

Pg 1. Lines 52-56

Its objective is to establish research works that have been carried out for SPI focused on SMSEs, and also to have a real study case, in order to present a profile of SPI in SMSEs.

Its objective is to analysis analyse the existing approaches about SPI in SMSEs which include a case study in industry, discussing the significant issues related to this knowledge area, and providing an up-to-date framework in which to position new research activities.

Pg 3. Lines 7 – 9

Our objective is to discover what has already been carried out and achieved as regards software process improvement in this type of companies.

Our objective is to analysis analyse the existing approaches about SPI in SMSEs which include a case study in industry, discussing the significant issues related to this

knowledge area, and providing an up-to-date framework in which to position new research activities.

Research question clarified in Pg. 4, Lines 19-21; Pg. 22 Line 29-32

Question: What approaches concerning SPI have focused on SMSEs and at the same time present a real study case?

Question: What are the main existing approaches concerning SPI focused in SMSEs which report a case study?

(2)

Research question clarified in Pg. 9, Lines 33-39

What approaches concerning SPI have focused on and been applied to SMSEs in the aspects of publication trends, companies involved, models used, process measurement, improvement proposals, improved processes and success factors?

What are the main existing approaches concerning SPI focused in SMSEs which report a case study?

comments

I also think that there are various aspects of systematic reviews (e.g. sensitivity analysis, publication bias,

application of quality criteria) that have not been clearly discussed in the manuscript and this gives the impression that they were not performed during the review itself.

Answer The scope has been clarified and the limitations of the systematic review have been included in the new section .Se incluirán las limitaciones del trabajo realizado Changes in

the paper New section in Pg. 18 Lines 26.

3.7 Limitations

comments

1. The title of the manuscript, and many other places in the manuscript, refer to "small and medium" software

enterprises. The basic search strings given in Table 1 only search on "small" enterprises, organizations,companies, teams etc. Section 3.1 refers to "small" but not to

"medium". Also, on p.6 of the manuscript, there is a

discussion of the key word "small" but no discussion of the keyword "medium". Section 4.2 refers to "micro-companies"!

Answer In order to clarify this point a brief description of the Planning Review phase (according to Biolchini, 2005) has been included. As a matter of fact, in the activity “test the protocol execution on one source” we initially considered the word “Medium”, as small and medium are part of the acronym SME. However, when the search of this first iteration was performed by using “small” and “medium” words, it was observed that the set of publications was a subset of the returned publications by using only the word

“small”. Because of this, only the work “small” was included in the definitive search strings.

The word “micro-companies” has been deleted as it does not affect to the article contents and to provide more clarity in the use of terminology according to the scope of the systematic review.

Changes in

the paper To see Review S4.

Pg. 11. Lines 47 – 59.

Currently, SPI efforts are also being applied to small and micro-companies (see figure 4).

In this figure sample 47% of the companies involved in the improvement efforts were small (fewer than 50 employees) and 33% were micro-companies (between 1 and 9 employees), which gives us the sum of 80% of all the reported companies in which an SPI effort was carried out. Only 20% is reported in medium sized companies (between 50 and 249 employees). However, it is important to underline that in most cases the improvement programmes in micro and small software companies did not lead to the ISO or SEI certifications.

Currently, SPI efforts are also being applied to small companies (see figure 4). In this figure sample 47% of the companies involved in the improvement efforts were companies with a number of employees between 10 and 49 and 33% were companies with fewer than 10 employees, which gives us the sum of 80% of all the reported companies in which an SPI effort was carried out. Only 20% is reported in medium sized companies (between 50 and 249 employees). However, it is important to underline that in most cases the improvement programmes in small software companies did not lead to the ISO or SEI

(3)

certifications.

Pg. 14. Lines 26 -28

It can be concluded that for the small and micro companies the SEI models do not lead to a certification.

It can be concluded that for the small software companies the SEI models do not lead to a certification.

Pg. 16. Lines 15 - 18

This allowed us to have a vision of what the improvement efforts in micro, small, and medium size companies had focused on.

This allowed us to have a vision of what the improvement efforts in small and medium size companies had focused on.

Pg. 18. Lines 55 - 60

Both small and micro companies are undertaking SPI efforts nowadays, but for these companies SEI or ISO models do not lead to their receiving certification.

The small companies are undertaking SPI efforts nowadays, but for these companies SEI or ISO models do not lead to their receiving certification.

comments

2. The abstract states: "... and also have a real study case..." (final sentence of abstract). This is somewhat confusing because it suggests that the manuscript will provide a real case study when in fact what I think the authors mean is that all of the publications selected in the systematic review should refer to real cases in industry. So some clarification on what is meant by "real study case"

would be helpful.

Answer This paragraph has been clarified.

Changes in

the paper To see the following changes in the paper: Pg 1. Lines 52-56, Pg 3. Lines 7-9 from revision R1.

comments

3. Following on from point 2, there is a further issue: *if*

all of the publications selected in the systematic review are intended to report on real cases (e.g. as case studies, or surveys or experiments) then how is this reflected in the basic search strings of Table 1? I *think* that the authors are using their inclusion criteria to include publications that report empirical studies of real cases; in other words, the search strings of Table 1 are not intended to search on real case studies. It would help if the authors could

clarify these issues in the manuscript.

Answer We thank for the suggestion. We’ve included a paragraph to clarify this issue.

Changes in

the paper New paragraph in Pg. 5. Line 4.

As the basic search strings of Table 1 were not intended to search on specific results with empirical studies in the software industry (e.g. case studies), according to our research question we used the inclusion and exclusion criteria in order to selectconsider only the publications which report case studies. Therefore, the "quality threshold for selection of studies" is defined as the studies which present a "evidence obtained from case series".

New paragraph in Pg. 23. Line 45.

Since search strings are not intended to search empirical studies in the software industry, we defined the inclusion and exclusion criteria in order to include publications that report case studies.

(4)

comments

4. It is not clear to me how the manuscript defines "small"

(or "medium" for that matter). Page 2 refers to Fayad et al.'s 'definition' of small being companies with fewer than 50 employees, however Figure 4 reports on companies with up to 249 employees. What's the definition of small and medium?

Presumably a definition of "small" and "medium" should form part of the inclusion or exclusion criteria? For example, of the publications that have been selected through searching,

*include* those publications that refer to companies who have less than 50 employees?

Answer In order to clarify how the manuscript defines “small” and “medium” companies, definitions of small and medium-size companies has been included.

Changes in

the paper Include paragraph in Pg. 4 Line 35.

In the context of the present work and according to (EU, 2005) small enterprises are companies with fewer than 50 employees and medium enterprises are companies which have between 50 and 249 employees.

In References page

EU, 2005. The new SME definition. User guide and model declaration. Available on:

http://ec.europa.eu/enterprise/enterprise_policy/sme_definition/sme_user_guide.pdf.

Accessed: April, 2007.

comments 5. I am not sure that the research question (as stated in section 3.1 and Appendix A) clearly relates to the results reported in section 4. Section 4 reports on: publication trends (figure 2), countries from which SMSEs have

undertaken SPI ( figure 3), number of employees (figure 4), standards (Table 4) and various models (Figure 5 and 6), improvement proposals (Table 5), etc. . The research

question is: what approaches concerning SPI have focused on SMSEs and at the same time present a real study case? (taken from section 3.1). For this question, Figure 4, Table 4 and Figures 5 - 6 seem relevant.

Answer According to this comment we have eliminated the figure 3 whose results are less related to the research question. The figure 5 has been also deleted as their results can be extracted from table 4.

However, we think that the information provided by tables 4, 5 and figure 4, 6 (standards used by each study, assessment and reference models, improvement proposals and size of companies) is related with significant issues to discuss to inside this knowledge area in order to providing an up-to-date framework in which to position new research activities.

From the perspective of found primary studies we also consider that the trend of

publications shown in Figure 2 provides useful information to justify the growing interest of the research community on this area.

In order to clarify this aspect the paragraph has been re-written.

Changes in

the paper Pg. 9. Lines 29-39

With the information extracted from the studies, a statistical analysis was conducted whose purpose was to show the relevant findings of the systematic review that had been carried out. These results (presented in this section) give an answer to the original research question: What approaches concerning SPI have focused on and been applied to SMSEs and at the same time include a study case? Based on aspects such as publication trends, companies involved, models used, process measurement, improvement proposals, improved processes and success factors.

(5)

In this systematic review a research question, which is of interest in the industry applied SPI research area, has been formulated. The aim is to identify new research activities to do in the field of SPI through the current state of art. The overview of the state of art presented in this paper comes from the information extracted of the 45 primary studies found as a result of the systematic review. With the information extracted from the studies, a statistical analysis was conducted whose purpose was to show the relevant findings of the systematic review that had been carried out. These relevant findings is based on the discussion of significant aspects related to this knowledge areaaspects such as publication trends, size of companies involved, models used, process measurement, improvement proposals, improved processes and success factors. The results, presented in this section, try to give an answer to the original research question: What approaches concerning SPI have focused on and been applied to SMSEs and at the same time include a study case?

comments

6. Related to point 5, the research question is phrased differently in different parts of the manuscript. For

example, section 4 (p. 9) refers to the "original question":

"What approaches concerning SPI have focused on and been applied to SMSEs in the aspects of publication trends, companies..." which is differently phrased to the question in section 3.1 and to the question in Appendix A. In some respects this is a minor point, but it does increase confusion over what exactly is being investigated in the systematic review.

Answer The research question statement is now homogeneous in the whole article.

Changes in

the paper To see the following changes in the paper: Research question clarified in Pg. 4, Lines 19-21; Pg. 22 Line 29-32 and Pg. 9, Lines 33-39 from R1.

comments

7. Section 3.4 refers to "Figure A": should this be Table 3?

Answer Fixed.

Changes in

the paper Pg 8. Line 7.

Figure A by Table 3 ID_revision R10

Reviewer comments

8. I am not sure that presenting the figures (which essentially plot only two dimensions of data) in three dimensions adds anything.

Answer The figures has been changed to two dimension versions.

Changes in the paper

comments

9. I am not sure what the benefit is in reporting each company in Table 4. Perhaps it would be just as effective to just report the total number of companies for each Standard.

I also note that the label in Table 4 refes to "Standars"

rather than "Standards"?

Answer The table 4 has been changed to address this comment.

Changes in

the paper Pg. 12. Lines 46-56 ID_revision R12

Reviewer

comments 10. Table 5 is some what confusing. What does the percentage column refer to? Percentage of publications? Percentage of companies? How have you extracted the improvement proposals

(6)

from the publications? How did you define the 6 different categories of improvement proposals? On this item, Table 3 simply refers to "Use an improvement framework in order to guide the programme SPI."

Answer The percentage columns refers to percentage of publications. To clarify this point the label % has been added. On the other hand, a new and short paragraph has been included to explain how the categorieds presented in table 4 were defined.

Changes in

the paper The label % of studies in the first column of Table 4 has been included. Pg. 15 Lines 13-14.

New paragraph in Pg. 15. Line 33.

The improvement proposals categories (from IP1 to IP6) shown in table 4 were defined through a deep analysis and later synthesis of the SPI proposals extracted from every primary study. First of all, every publication was completely analysed and then a summary of the main ideas and contributions was extracted. The next step was to group the SPI contributions which were related to a more general concept in different

categories, resulting eight categories in total. For example, the improvement proposals “to use an improvement framework in order to guide the programme SPI” (see table 3) and

“to follow the concept of action packages in order to carry out the implementation of the SPI in SMEs” (Calvo-Manzano et al., 2002) can be grouped under the category IP4 “to guide improvement efforts”.

comments

a) review the conduct of the systematic review, taking account of Kitchenham's guidelines and advice (e.g. "lessons learned") from other published systematic reviews;

Answer The Kitchenham’s guidelines and other the works followed to carry out the current systematic review have been explicitly stated in this new version. Additionally, the context under which the review was conducted is explained

Se aclara el contexto de la revisión sistemática y los trabajos seguidos.

Changes in

the paper Pg. 4 Lines 7-9.

To carry out the systematic review of SPI in SMSEs, the protocol template presented by Biolchini in (Biolchini et al., 2005), was followed.

The systematic review which follows has been carried out in the context of a PhD Thesis whose aim is to provide small companies in Latin America with a process improvement model tailored to their special characteristics and which is part of the project

COMPETISOFT (Oktaba et al., 2007). For a successful application of the method the guidelines presented in (Kitchenham, 2004), the protocol template presented in (Biolchini et al., 2005) and (Biolchini et al., 2007) were followed and other similar systematic reviews were considered (Jørgensen et al., 2007; Hannay et al., 2007)

Paragraph in Pg.22 Line 8.

To carry out the systematic review of SPI in SMSEs the protocol template presented by Biolchini was followed.

To carry out the systematic review of SPI in SMSEs the protocol template presented in (Biolchini et al., 2005), the Kitchenham's guidelines for systematic reviews presented in (Kitchenham, 2004) and some advices from other published systematic reviews were followed.

In References page

Kitchenham, B., 2004. Procedures for performing systematic reviews, Keele University and ICT Australia Ltd. Pp: 1-28.

Biolchini, J., P. Gomes, A. Cruz, T. Uchôa and G. Travassos, 2007. Scientific research ontology to support systematic review in software engineering. Advanced Engineering Informatics Vol. 21(2) April pp. 133-151.

Hannay, J. E., D. I. K. Sjøberg and T. Dybå, 2007. A Systematic Review of Theory Use in Software Engineering Experiments. IEEE Transaction on Software Engineering Vol.

33(2) February pp. 87-107.

Jørgensen, M. and M. J. Shepperd, 2007. A Systematic Review of Software Development Cost Estimation Studies. IEEE Transaction on Software Engineering Vol. 33(1) January

(7)

pp. 33-53.

REVISOR # 2.

ID_revision S1 Reviewer

comments - page 5, line 7 talks about the "planned list", but no rationale for selection is given.

Answer - The rationale for selection of sources list is given.

Changes in

the paper Include paragraph in Pg. 5. Line 21.

The sources have been identified according to expert opinion in the work area which is the scope of the systematic review. Inside these sources there are very important journals in which the research area is treated, as such as: Information and Software Technology, Software: Practice and Experience, Software Process: Improvement and Practice, IEEE Software, Software Technology and Engineering Practice, Computer, among others.

comments

- page 19, line 36 talks about the identification of new sources being identified

Answer The paragraph has been clarified.

Changes in

the paper Pg. 19. Lines 35-40.

Using as a starting point the results of this systematic review, new information sources will be reviewed in new iterations following the procedure for execution of the systematic review suggested.

Using as a starting point the protocol of this systematic review, new studies sources (e. g.

Springer or conferences related to SPI) will be reviewed in new iterations following the procedure for execution of the systematic review suggested.

comments

- page 22, line 58 states that the source selection criteria is "electronic journals which can be access via the Internet from the library digital service of the University of

Castilla-La Mancha

Answer The paragraph was deleted. A new paragraph has been written.

Pg. 22. Lines 58-62.

Electronic journals which can be accessed via Internet from the Library digital service of the University of Castilla-La Mancha (http://www.biblioteca.uclm.es).

The sources have been identified according to expert opinion in the work area which is the scope of the systematic review.

Pg. 23. Lines 58-62. The paragraph was deleted.

The sources have been identified according to expert opinion in the work area which is the scope of the systematic review.

comments

There is no real discussion of why the specific keywords were used, nor the specific logical combinations used in the search strings. At the very least, some sort of pre-scan to determine the utility and effectiveness of both parts of this could have been considered and described. The specific focus on SPI and no use of common synonyms could easily have prejudiced the quality and utility of the work.

Answer A new paragraph has been included to explain how the evaluation of the protocol was performed. Se ha incluido un párrafo explicando la forma como se realizó la evaluación del protocolo planeado.

Changes in

the paper Include this paragraph in Pg. 24. Line 24.

The protocol evaluation of this systematic review was performed through: La evaluación del protocolo de ésta revisión sistemática se realizo a través de:

− To test the pProtocol excecution testing oon one source in the first iteration of the

(8)

review. During this iteration the initially planned protocol was evaluated and improved for the achievement of the review objectives, giving as a result the protocol presented in this work.

− El resto no me convence demasiado. Lo hablamos:

− en la primera iteración de la revisión. Durante esta iteración se evalúo, modificó y refinó el protocolo planeado inicialmente para que cumpla con los objetivos

concebidos en la revisión. Esto permitió la evolución del protocolo hasta convertirse en el que se muestra en éste paper. Los elementos más dinámicos en un principio del protocolo son las palabras clave y las cadenas de búsqueda. Por ejemplo, con respecto a las palabras claves se tuvo en cuenta inicialmente la palabra “medium”, así pues al hacer la búsqueda con las palabras claves small y médium ésta generaba menos resultados los cuales estaban contenidos en la búsqueda realizada con la palabra small. También se realizó una prueba considerando reemplazar la cadena de búsqueda “software process improvement” by “software process assessment”, se pudo observar que con la nueva cadena de búsqueda se obtenían los mismos estudios primarios que con la inicial, sin reportar ningún nuevo estudio primario.

− A través de reuniones periódicas con dos expertos en área investigada para determinar si el protocolo y la ejecución de la revisión se estaba llevando a cabo adecuadamente.

comments - the references in the manuscript to government support for SPI in SMEs: the authors ascribe cause and effect, whereas it is not clear to me whether this is just self-fulfilling;

Answer These referentes have been eliminated in the paper, es they do not provide an added value to the main contribution of the paper. Se elimina los párrafos relacionados con el government support.

Changes in

the paper Pg. 10 Lines 27-62

The main software improvement initiatives in these countries have been promoted by means of the real commitment of governmental programmes, as for instance:

− Software Engineering Australia – SEA – is an industry-led body with the charter of improving the quality and reliability of software in Australia. SEA is supported by the Commonwealth through the Department of Communications, Information Technology, and the Arts. SEA provided funding for the Software Quality Institute – SQI– to deliver a process improvement programme to 22 small software development firms. SQI has recognized the need for approaches that can generate incremental improvements to demonstrate business benefits, particularly for smaller developers.

− Countries as Ireland, Finland and Sweden have participated in the European Software and System Initiative (ESSI) of the European Union the which have promoted different projects for strengthening SPI in SMEs, such as SPIRE (Software Process Improvement in Regions of Europe) (SPIRE, 1993) and TOPS (Toward Organized Software Processes in SMEs) (Esprit_Project, 1999), among others.

− In Brazil, the government has subsidized the implementation of the PBQP–Software (Productivity and Quality Software Programme) (Bedini et al., 2005) and the

“MPS.BR” project has been developed (melhoria do processo de software brasileiro) (Weber et al., 2005), both of which strive towards the same objective.

Pg. 11. Lines 7-13. The word “Too, ” has been deleted.

comments

- the assumption by the authors that the literature is specifically representative of real life, in respect of the proportionality of different kinds of SPI approach - I strongly doubt that this is the case, and in any case it is surely impossible to prove or establish without a huge amount of additional work.

Answer We comepletely agree that such statement is strong and huge quantity of additional work would be necessary to assure it. So, a new paragraph has been included to explain that the presented results actually provide an overview of the field based on selected primary studies.The whole article has been reviewed to clarify this

(9)

assumption. Se ha incluido un párrafo aclarativo que los resultados presentados no representan todo el mundo real, sino solo muestran un panorama obtenido desde los estudios primarios encontrados. Además se elimina todo el apartado 2, en el cual el primer párrafo se hacia esta suposición.

En todo el artículo se ha corregido la redacción para evitar que se dé esta interpretación.

See changes in paper from R7.

The section 2 in Pg. 2 Lines 16-60 has been deleted.

comments

- much of page 3 seems to be low-value-add about the methodology, and could perhaps be assigned to an appendix;

Answer Se ha eliminado todo el apartado 2 Changes in

the paper The section 2 in Pg. 3 Lines 16-60 has been deleted.

comments

lines 45 & 46 of page 9 notes that there is "an increasing interest on ... the whole area of SPI and SMSEs", whereas we are dealing with very small amounts of data (or evidence from the study). This looks like an overstatement;

Answer En todo el artículo se ha corregido la redacción para evitar generalizar.

Pg 9. Line 46. The word whole has been deleted.

comments - line 24 of page 10 notes the six countries which report most improvement efforts in SMSEs, and then goes on to note the specific support from governments in those same

countries. Cynically, I wonder if what is really being observed here is a requirement to publish results being a standard requirement of such funding;

Answer Se elimina los párrafos relacionados con el government support.

Changes in

the paper See changes in the paper from S5.

comments

- line 9 of page 11 refers to government "organisms", which I think should be "organisations".

Answer Fixed, Thanks.

Changes in

the paper Pg. 11. Line 9, “organisms” by “organizations” has been changed.

comments - the first paragraph of page 11 has some interesting

assertions such as "company commitment is vital" without any citations or references. These look inappropriate without such references (whilst they may well be correct, they just look like assertions of the authors, and thus out of place);

Answer Thanks. TheA suitable reference has been included.

Changes in

the paper In references page.

Dyba, T., 2005. An Empirical Investigation of the Key Factors for Success in Software Process Improvement. IEEE Transactions on Software Engineering Vol. 31(5) May pp.

410-424.

comments Figure 3 on page 11 doesn't seem to add much value; if I read it correctly then 2% simply means one reference found?

(10)

Answer Yes., it does The Figure 3 has been deleted.

Changes in

As can be seen in Figure 3, the six countries which report most improvement efforts in SMSEs are, in descending order, Australia, Ireland, Finland, Denmark, Brazil and USA.

The countries which report most improvement efforts in SMSEs are, in descending order, Australia, Ireland, Finland, Denmark, Brazil, USA, Sweden and Mexico.

comments

- Section 4.3 starting on page 12 and going through to page 14 just made me think: "so what?". Not enough effort has been put into drawing useful conclusions - so perhaps there aren't any?

Answer Changes in the paper

comments

- line 28 of page 14 asserts "The motivation for this type of companies should therefore be to improve their processes, in order to make them effective and efficient units, rather than to gain the certification itself". Generally, I find that the authors do understand the differences between certification and non-certification use of standards (a common failing in articles submitted to SQJ), but I can't see where the authors conclude "therefore" in the quoted statement. Is this editorialising, or pulled from a specific report that they studied in more detail?

Answer We thank for the suggestion. The paragraph has been clarified.

Changes in

the paper Pg. 14. Line 28.

The motivation for this type of companies should therefore be to improve their processes, in order to make them more effective and efficient units, rather than to gain the

certification itself.

Together with the above considerations, we consider that the motivation for this type of companies should therefore be to improve their processes, in order to make them more effective and efficient units, rather than to gain the certification itself (Paulk, 1998).

In references page

Paulk, M. C., 1998. Using the Software CMM in Small Organizations. Proc. Joint 16th Pacific Northwest Software Quality Conf. and 8th Int’l Conf. Software Quality, 350-360.

ID_revision S15 = nS14????

Reviewer

comments - lines 34 & 35 of page 14 state that "ISO 9001 helps to provide better support to the CMM initiative and to improve areas of the organisation that are not directly related to the development of the software". The authors clearly mean that compliance with ISO 9001 should have a broadly

beneficial effect on organisations beyond the scope of coverage of the CMM (so the shorthand they have used is poor), but such an assertion is beyond the scope of the study/article submitted for publication, and looks like editorialising. It could be argued that, without specific guidance applicable to software development, ISO 9001 on it's own has little relevance to software development

organisations, but arguing that is outside the scope of the article and this review;- line 28 of page 14 asserts "The motivation for this type of companies should therefore be to improve their processes, in order to make them effective and efficient units, rather than to gain the certification itself". Generally, I find that the authors do understand the differences between certification and non-certification

(11)

use of standards (a common failing in articles submitted to SQJ), but I can't see where the authors conclude "therefore"

in the quoted statement. Is this editorialising, or pulled from a specific report that they studied in more detail?

comments

- lines 37 & 38 point out that ISO 9001 also exists as harmonised national standards, but so what? By pointing this out, the authors further call into question the scope of their keyword search!

Answer Thanks. The sentence has been deleted. Porque la sección solo quería hacer una pequeña introducción a lo que es una revisión sistemática pero no aportaba nada nuevo.

Changes in

the paper Pg. 14. Lines 37-38. The sentence has been delete.

It should be remembered that in many countries the ISO 9001 standard is a national standard.

comments

- line 48 of page 14 uses the word "ratifies". This is surely far too strong, and should at best be "tends to support";

Answer Thanks. Fixed.

This ratifies the assumption that the SMSEs’ efforts are still insufficient to establish formal software measurement processes.

This tends to support the assumption that the SMSEs’ efforts are still insufficient to establish formal software measurement processes.

comments

- the final assertions made at the bottom of page 14 look far to strong & sweeping to come out of a sample size of 2!

Answer These strong assertions Some sentences from the paragraph has been deleted . Changes in

the paper , constructed from measurement frameworks such as Goal Question Metric (GQM) or Practical Software Measurement (PSM). The definition of these measurement models would help to make an evaluation more objective.

comments

- the final paragraph of page 15 looks like editorialising - there are no references or citations to back it up;

Answer Es una opinión nuestra a partir del estudio realizado. This paragrpah includes our a personal opinión of the authors from the obtained Se hresults, We have clarified it.

Changes in

the paper Pg. 15. Line 49.

From the analysis of the extracted primary studies, we A partir del análisis de las

propuestas de mejora extraídas desde los estudios primarios, nosotros consideramos qu eit can be concluded that the improvement approaches can be generated at an organizational or a technical level. At an organizational level, the improvement approaches are cheaper, but the improvements cannot be seen in the short term. At a technical level, the

improvement approaches are more expensive but the improvements are seen in the short term.

comments - lines 40 to 43 of page 18 talk about the "current state of affairs" as if this is a universal statement, whereas it actually only applies to the literature that has been

studied (see earlier remarks about the representativeness of the literature searched);

Answer The paragrpah writing has been refined in order to avoid strong

(12)

generalizationThanks. La redacción del párrafo ha sido mejorada para evitar generalizar.

Changes in

In this article a systematic review of the literature on SPI efforts carried out in SMSEs has been presented, which has allowed us to obtain a comprehensive and thorough view of the current state of affairs in this area.

In this article a systematic review of the literature on SPI efforts carried out in SMSEs has been presented, which has allowed us to obtain a view about SPI in SMSEs through a representative set of related works obtained as a result of the systematic reviewcreada desde la perspectiva de la información extraída de los estudios primarios encontrados por el protocolo de la revisión sistemática.

comments

- lines 47 to 48 of page 18 state an assertion that there is an "ever-increasing interest in tackling SPI in SMSEs" - forgetting to say that it is the literature search that indicates this (see earlier remarks about the

representativeness of the literature searched);

Answer A new statement at the beginning of the paragraph has been included to tackle this aspect.

Thanks. Se ha insertado una oración al principio del párrafo para aclarar que los resultados son a partir de los 45 estudios encontrados.

Pg. 18. Lines 47-48

From the obtained results in the current review, De acuerdo a los resultados obtenidos a partir del análisis de los 45 estudios primarios hallados con el protocolo de la revision sistemática, the first observation from the study carried out is that, in the last 5 years, the Software Engineering community has shown an ever-increasing interest in tackling SPI in SMSEs.

comments - lines 50 & 51 of page 18 talk about the "real commitment of governmental programmes" - this looks like

editorialising;

Answer Thanks. The sentence has been deleted.

comments

- line 59 of page 18 uses the word "then", but the

associated argument is not presented - decisions about this are surely the remit of SMSEs and the commercial and other pressures they face concerning certification, not for the authors to editorialise about!

Answer This paragraph has been deleted Changes in

the paper Paragraph deleted Pg. 18. Lines 59-61.

Following these models, should then be a means towards, and not an end of, improvement.

comments

- line 9 of page 19 uses the words "where the use of the three models" whereas I think it should read: "where the use of all three models";

Answer Thanks. Fixed.

Pg. 19. Line 9.

REVISOR #3.

(13)

ID_revision T1 Reviewer

comments

The authors emphasise the importance small and medium sized software enterprises; claiming that these companies are the mainstay of the software industry. However, this claim is not substantiated.

Answer The A reference has been included to support such assertion.

In References page.

Fayad, M. E., M. Laitinen and R. P. Ward, 2000. Software Engineering in the Small.

Communications of the ACM Vol. 43(3) March pp. 115-118.

comments

A weakness of this paper is the assumption that the results of a Systematic Literature Review are generalisable (i.e.

apply to the whole population). It might help the reader if we are given a general idea as to how many SMSEs there are (roughly), so that the 45 selected 'different' studies are placed in context with the population. The conclusions discuss finding in terms of what is happening with SMSE's and SPI - whereas the results are in fact attempting to summarise the literature on SMSE's and SPI.

Answer This aspect has been deeply tackled in the new version of the paper. Ok, se ha dejado claro que los resultados del artículo se obtienen de solo 45 estudios primarios. Se ha mejorado la redacción del articulo para evitar generalizar.It has been improved the writing of the article in order to avoid strong generalization according to the scope of the review.

To see Review S6, S8, S20, S21.

comments

A further concern is the lack of attention to the quality of the publications included in the systematic review. The authors quote Kitchenham's SLR guidelines (2004), but have not followed her recommendation that all papers are

subjected to a quality assessment. Although the authors state that all papers have already been peer reviewed, it is likely that they are of variable quality. For example are they all based on randomised trials? Are the sample sizes all the same (we know they are not from the analysis)? How did the authors explain their data collection/analysis? I would imagine that the SQJ would find this of particular importance.

Answer A paragraph about the quality of the publications has been included.

Changes in

the paper To see Review R5. New paragraph in Pg. 5. Line 4.

New paragraph Pg. 9. Line 24.

Con respecto a la calidad de las publicaciones incluidas en la revisión sistemática se puede decir que los estudios primarios son observational studies with design type case series, mediante el cual las SMEs fueron intervenidas con la propuesta de mejora formulada. De estos design type case series 82% son post-test and 18 % son pre-test post- test. Para presentar los resultados se utiliza el análisis cualitativo, sin embargo el 27% de los estudios presentan análisis cuantitativo de algún resultado. Además, el 42% de los estudios reportan evidencia obtenida desde 2 ó más SMEs.

comments

Would like a definition of SME earlier in the paper (what size is a 'small' company; what size is a medium company, what is the size of a 'micro' company?) - sizes given on page 11, section 4.2.

(14)

Answer Fixed.

Changes in

the paper To see Review R6.

comments

Some grammar/definitions need checking, e.g. the word 'this' is often applied to a plural. Also, does the acronym SMSE really stand for Small and medium software 'organisations' [as given in the abstract], and not 'enterprises'?)

Answer Fixed.

comments The abstract might be misleading when it states that the systematic literature review is a real study case, (which suggests primary research); the systematic literature review aggregates findings of several studies to plot the

landscape/give an overview and patterns of

behaviour/practice as reflected in the published work in the area of interest.

Changes in

the paper To see change in the paper Pg 1. Lines 52-56, Pg 3. Lines 7-9 from revision R1.

comments

Section 3: 3.1 why isn't ''medium' one of the key words? - Also the research question might be slightly less cumbersome if you use the word 'empirical' rather than real study case?

- If empirical is what is meant by 'study case' of course!

Changes in

the paper To see revision R3 and S4.

To see change in the paper Pg 1. Lines 52-56, Pg 3. Lines 7-9 from revision R1.

comments

3.2 Workshop selection is unclear in final paragraph.

Changes in

the paper Pg. 5 Lines 21-25.

we chose this workshop because an overview of research from the world-wide

community which addresses the unique issues of process improvement in small settings (including small teams, small projects, small organizations, and small businesses) is presented.

we chose this workshop because es organizado por el SEI y su area de interés es presentar an overview of research from the world-wide community relacionado con process improvement in small settings (including small teams, small projects, small organizations, and small businesses) is presented.

comments 3.3 What about secondary searches, did authors find any papers referenced in primary studies that were worth

following up? - It looks as if this is covered in the flow chart under "more discovered studies" perhaps? Later, at end of page 7, authors do state they do not do a follow up search. Perhaps explain why.

Answer No se realizó secondary searches, sin embargo la tendremos en cuenta como trabajo furturo.

Changes in

the paper New paragraph Pg. 19. Line36.

Using as a starting point the results of this systematic review, new

(15)

information sources will be reviewed in new iterations following the procedure for execution of the systematic review suggested.

Using as a starting point the protocol of this systematic review, secondary searches from primary studies and new studies sources (e. g. Springer or conferences related to SPI) will be reviewed in new iterations following the procedure for execution of the systematic review suggested.

Pg. 7. Line 54

In this systematic review potential references inside primary studies were not searched, la intención es administrar mejor las búsquedas evacuando primero las fuentes definidas y en una iteración futura realizar búsquedas secundarias a partir de los estudios primarios identificados.

comments

To give more confidence in the paper selection, it would help if the authors confirmed whether their protocol was validated, and if so how was it validated and by whom? What were the plans for disagreements between researchers? How many researchers took part in the selection? Did they perform any inter-rater reliability tests to validate inclusion/exclusion decisions?

Answer El protocolo fue validado, un párrafo explicativo se ha incluido.

Additionally, the context under which the review was conducted is explained Changes in

the paper To see Revision S4 and Revision R13.

comments

Figure 1: This 'flow chart' works well as a means of

representing the complex, iterative review selection process and I found this a main contribution of the paper. (Although perhaps using black as the background doesn't help the presentation). Other researchers, unfamiliar with conducting SLRs could follow this flow chart that on preliminary

inspection looks accurate (I haven't tested it out fully).

It may not be complete however, as it is lacking a 'quality' check. (The assumption that the work has been peer-reviewed and therefore doesn't need quality assessment is weak, especially as these authors have included the grey literature in their searches). The flow chart is also lacking a validation or inter-rater reliability process.

comments

Another strength of this paper is the concise reporting of the extraction process as shown in Table 2. This approach helps other researchers/practitioners replicate the study.

However, the category 'proceedings' is fairly vague.

Answer Se ha aclarado en la tabla que el proceeding es del SEI.

Changes in

the paper Pg. 7 Line 48.

comments

Section 3.3: page 7: second bullet point: The second level studies: do the authors mean a theoretical study, not based on direct/empirical observation?

Answer Fixed.

Changes in

the paper Pg. 7 Line 48.

A second level, where the study deals with SPI initiatives for SMSEs which nevertheless

(16)

have never been applied in such companies (is a theoretical study not based on direct/empirical observation, there is no study case applied to SMSEs, for instance (Brodman et al., 1994)).

comments

Section 4: Interpretation of results: The authors need take care in how they present their results: For example, in Figure 2, they say there is a growing interest in SPI as shown in the increase in publications - however, the increase in publications in this area might be due to the fact that more papers are being published in many areas of software engineering?

Answer Se ha reforzado la razón describiendo que también han aparecido diferentes estándares, proyectos y propuestas relacionadas con este tema.

Changes in

the paper Pg. 9. Line

This is indicated by the increasing number of studies that deal with the subject, according to the analysis of the trend in publications that was carried out (see Figure 2).

This is indicated by the appearance of a great number of standards and proposals related to SPI for SMEs, and also the increasing number of studies that deal with the subject, according to the analysis of the trend in publications that was carried out (see Figure 2).

comments

4.2 Companies involved. The searches picked up that micro- companies are also applying SPI. However, as the term micro wasn't in the original search, it is possible that not all micro companies that have published case studies are

represented here. This is not noted.

Answer Se ha definido solo dejar los términos small and maedium companies para mayor claridad del articulo.

To see Revision R3.

comments

The conclusion: If the results are based on following the review protocol, we need to know that the protocol has been validated in a 'systematic' way.

Answer El protocolo fue validado de manera sistemática.

Changes in

the paper To see Revision S4 and Revision R13.

comments The conclusion: If the results are based on following the review protocol, we need to know that the protocol has been validated in a 'systematic' way.

Answer Se ha redactado de nuevo eliminado la palabra comprehensive Changes in

In this article a systematic review of the literature on SPI efforts carried out in SMSEs has been presented, which has allowed us to obtain a view about SPI in SMSEs through a representative set of related works obtained as a result of the systematic review.

comments All results should be in terms of what the literature is saying - that may or may not apply to the whole

population. There is no note of publication 'bias' - as companies who 'fail' at SPI are less likely to publish their work?

Answer The scope has been clarified and the limitations related to “publications bias” of the systematic review have been included in the new section.

(17)

comments

Page 18: "Following these models, should then be a means towards, and not an end of, improvement". The fact that companies receive certification (or not) does not suggest an 'end' to improvement. It is a continuous process.

Answer This paragraph has been deleted Changes in

the paper

Paragraph deleted Pg. 18. Lines 59-61.

comments

Page 19: Not sure of these categorisations (top of page), e.g. CMM process model, ISO.. assessment method; IDEAL model guides improvement (CMM also guides improvement doesn't it?)

Answer This paragraph has been better explained.

Changes in

the paper Paragraph deleted Pg. 19. Lines 5-9.

Another relevant fact ascertained is that the improvement models used most in the SMSEs are CMM as a process model, ISO 15504 as an evaluation method and IDEAL as a model for guiding improvement.

Another relevant fact ascertained is that the improvement models used most in the SMSEs are CMM as a process model (el cual presenta buenas practicas de ingenieria del software) , ISO 15504 as an evaluation method and IDEAL as a model for guiding improvement (a través de la definición de un ciclo de vida para la mejora de procesos).