Impact evaluation through counterfactual analysis

European standards and pioneering practice in Croatia

An overview of European practices and standards for evaluation of various interventions and funds in the last several years indicates a rising focus and popularity of impact evaluation, as well as its attributed method of counterfactual analysis. In this light, European Commission established the Center for Research on Impact Evaluation at the Joint Research Center whose primary goal was to provide support in application of counterfactual analysis to interventions’ impact assessments, notably the European Social Fund. The Commission presented its dedication to making this assessment method a part of policy implementation process. An example of such dedication can be seen in the evaluation of cohesion policies in which the Commission published a practical guidance document complemented with an annex exclusively directed towards design and usage of counterfactual analysis, since it represents newer and complex method.

In accordance to the EU standards, the promotion of the counterfactual analysis could also be seen at the Croatian institutional level within social development interventions. For instance, the Evaluation Strategy for European Structural Instruments in 2012 explicitly mentioned counterfactual analysis as one of the two fundamental types of impact evaluation, while the Assessment Strategy for the Implementation of European Structural and Investment Funds in Financial Perspectives 2014 – 2020 mentioned counterfactual analysis as one of desired evaluation methods. However, this method is only starting to be more significantly used in practice in Croatia. One of the examples is certainly the official Evaluation Plan of the Operational Programme Efficient Human Resources 2014-2020 which prescribes the use of counterfactual analysis in 7 out of 14 planned evaluations; whose majority is currently a part of contracting process through the public procurement procedure Realization of evaluations of Operational Programme Efficient Human Resources 2014 – 2020.

 

What is a counterfactual method?

The counterfactual analysis primarily measures the intervention’s effect and answers the question of how much change did the intervention cause. If a certain group of individuals goes through an intervention, this analysis investigates characteristics of that group after the intervention and estimates what would those characteristics be if there was no intervention at all. Therefore, counterfactual analysis implies a selection of a treatment group i.e. a group that undertook the intervention and a selection of a control group composed of individuals that did not undertake intervention, but are resemble the first group in characteristics that could influence the intervention impact or participation in it. This is usually followed by a method of pairing the units of the analysis from the treatment group with units form the control group according to the resemblance in selected characteristics. The selection of these two groups is usually followed by quantitative analysis and their statistical comparison.

When observing its application, the need for a database providing data for a larger number of individuals (or other units of analysis) necessary to measure or ‘control’ is probably the strongest limit for the wider use of the counterfactual analysis, meaning unavailability or incompleteness of data hugely influences the applicability and practicality of this method.

 

Current Croatian examples of counterfactual analysis

Though counterfactual analysis in evaluation in Croatia is in its pioneering phase, the evaluators’ community can expect an increase in a demand for its use. So far, we have noted three significant evaluations that used this method, including the following:

1.    Ipsos. (2016) External Evaluation of the Measures of Active Labour Market Policies 2010 – 2013 (Vanjska evaluacija mjera aktivne politike tržišta rada 2010. - 2013.).

Evaluation of labour market policies measures, implemented by the Croatian Employment Service for the period 2010 and 2013, used the counterfactual analysis to measure the impact of active employment measures, including acquisition of professional experience in the workplace – occupational training without commencing employment, subsidies for (self)employment, education of unemployed and public works. For the purpose of the counterfactual analysis, two groups of the labour market participants were selected for each of the analysed measure. The treatment group gathered participants that participated in the intervention, and the control group those who were, based on a number of key characteristics, paired with the participants of the treatment group, but did not participate in the intervention themselves.

 

2.    Ecorys. (2019) Impact Evaluation of the Priority Axis 3 Business Competitiveness of the Operational Program ‘Competitiveness and Cohesion’ 2020: Final report (Vrednovanje učinka Prioritetne osi 3 Poslovna konkurentnost Operativnog programa „Konkurentnost i kohezija“ 2020.: Završno izvješće).

Upon the assessment of the priority axis 3 of the Operational Program “Competitiveness and cohesion”, the evaluation used counterfactual analysis in order to measure the effect of the grants to small and medium businesses. The analysis implied a comparison of a treatment group of entrepreneurs who received the grant with those who did not but are similar to the individuals from the treatment group according to county, number of employees and sector. After questioning the participants, the data was complemented by the database from the Digital Chamber of the Croatian Chamber of Economy.

 

3.    Ipsos. (2019) Evaluation of the Initiative for Youth Employment as part of the Operational Programme Efficient Human Resources 2014-2020 for 2018: Final report (Evaluacija Inicijative za zapošljavanje mladih u sklopu Operativnog programa Učinkoviti ljudski potencijali 2014. – 2020. za 2018. godinu: Završno izvješće).

Another example of counterfactual analysis, in which MAP’s experts also took part, refer to the Evaluation of the Initiative for Youth Employment, i.e. an evaluation of specific measures of active employment policies directed towards integration of youth in the labour market. In order to measure the impact of these interventions, the group of young participants that took part in the measures were paired with individuals who did not partake in the interventions, but are similar according to their sex, age, education, county, date of entering the intervention/job market, handicap, homelessness, social disadvantage, national minorities and foreign descent. Primary data source was the internal user’s database and database of the Croatian Employment Service.

Finally, regardless of the limited use of counterfactual analysis in evaluations in Croatia, there are signs of rising demand for this method. Former pioneering evaluations which relied on this analysis, as well as every future attempt, inevitably build upon knowledge base and available designs for this method and will gradually ease its future use in Croatia. However, its outspread and popularity will heavily depend on the availability and quality of data in specific national databases.