Característica 33: organización, administración y gestión

The evaluation employed a balance of qualitative and quantitative lines of evidence, enabling the triangulation of research findings. Mitigation strategies were used to address the limitations, and along with the triangulation of multiple lines of evidence, were considered sufficient to ensure evaluation findings can be used with confidence. Yet, the following limitations and mitigating strategies of the evaluation methodologies should be considered:

 Gaps in literature and document review: Although CIC has a good collection of documentation on the BIP, there is a lack of academic literature focused on the economic activities of business immigrants. Rather, the available literature examines more generally the self-employment activities of immigrants, with no distinction made about the class under which they were admitted to Canada.

Mitigation strategy: This deficit was addressed by gathering direct information on the economic outcomes of BIs through the telephone survey and administrative data analysis of incomes they report to the Canadian Revenue Agency on their income tax report. However,

information obtained though these other lines of evidence only apply to the specific cohorts studied, and cannot be generalized to all cohorts admitted since the program was introduced.

As such, mitigation strategies could not completely counter the lack of relevant academic literature on economic activities of BIs.

 Representativeness of survey respondents: The survey sample obtained through the informed consent process differed from the total BI population admitted between 2007 and 2011, with ENs being over-represented and INs being under-represented. Additionally, there was an under-representation in the survey sample of respondents from Asia.

It is also possible that the results of telephone survey of BIs could have been positively skewed by those respondents more compliant with the terms and conditions of the EN program. This self-selection bias is however the same that would be expected in any survey undertaking.

Mitigation strategy: The survey results were weighted to address these imbalances identified in order to ensure that they were representative of the total BIP population (for more

information on weighting, see Appendix D). The sample size in the BIP survey was sufficient to report averages for BIP and its three classes for all indicators.

 Gaps in administrative data: Gaps in several administrative data sources were noted. First, because of inconsistencies in selection criteria point information in CAIPS and GCMS, the evaluation was unable to use this information to get a description of the points received by BIs on the selection grid and to explore which selection criteria would explain better economic performances.

Mitigation strategy: The application and admission records capture similar types of information (i.e., information related to the socio-demographic characteristics of individuals) although in some instances to a lesser level of detail, which was at least sufficient to support socio-demographic analysis of the BIP cohort. In addition, the evaluation gathered qualitative data on this topic.¹²

Second, the evaluation had planned to use information from the Entrepreneurship

Monitoring Information System (EMIS) and the Investment Monitoring Information System (IMIS) to assess the economic activities of ENs and INs, as well as the degree of monitoring of and compliance with EN terms and conditions. Unfortunately, these CIC databases had a high rate of missing values, especially for the years under study. As a result, the evaluation was unable to draw upon these sources of information.

Mitigation strategy: This limitation was anticipated when planning the evaluation. As such, the evaluation was designed in a way that allowed to gather the information needed through other lines of evidence, namely from the KIIs with CIC representatives and the telephone survey with BIs.

Third, although administrative data on application, admission and tax records became

available to cover additional years after data collection for the evaluation had been completed, analysis focused on the original scope period for the evaluation (2007 to 2011). This ensured alignment with the time frame covered by other methodologies (such as key informant interviews, case studies and the survey), so that the respondents across lines of evidence were reflecting on the same period of the BIP.

 Definition and interpretation on economic performance: Economic establishment and performance are concepts difficult to define. Once defined their operationalization and interpretation also poses challenges. Given the nature of the business activities that BIs are likely to undertake in Canada and of the Canadian tax system (such as deductions that can be obtained), the extent of BIs economic performance is difficult to fully assess.

Mitigation strategy: The evaluation adopted a conservative approach in defining and measuring the economic performance of BIs. As such, results provide a minimum estimate of their economic performance in Canada. For more details on the assessment of the economic performance, refer to section 5.1.

12 Key informant interviews and interviews conducted as part of the case studies asked whether the selection criteria in place allowed the program to meet its policy objectives and if any alternative selection criteria should be

considered.

 Gaps in financial data: In CIC’s Cost Management Model (CMM), Federal and Quebec business immigrant financial data are captured in the aggregate form. This limited the Evaluation’s ability to focus only on costs incurred by the federal part of the program. As such the examination of costs presented in this report includes both Quebec and Federal costs associated with the BIP. Additionally, the scope of the financial information available was limited due to the absence of CMM data for the 2008/09 fiscal year.

Despite all the limitations mentioned above, the design of the evaluation and data collection approaches were able to mitigate the impacts. Strategies were taken to ensure the evaluation presented valuable, reliable information to support strong findings. Where possible, multiple lines of evidence were used to inform the evaluation questions. Despite the different methodological approaches, the results from the different lines of evidence generally converge towards common and integrated findings. The use of different lines of evidence was also designed in a way that they would complement each other in order to fill information gaps.

The evaluation methodologies and limitations are discussed in greater detail in the Technical Appendices.