Three papers on evaluations : the "what if" in the evaluation of public programs

Abstract

SUMMARY:

The “What if” in the evaluation of public programs: An appraisal of methodologies and practices

Three papers on related topics concerning methodologies and practices of the evaluations of public programs constitute the dissertation. The first paper demonstrates the applicability of observational methods for the assessment of the program level contributions of two Financial Schemes and two Governmental intervention programs from the prede- cessors of Innovation Norway in the early nineties: The Regional Venture Capital Loans program aims at compensating presumed regional funding disadvantages, the Investment Grants at stimulating physical investments in buildings, machinery, and equipment. The FRAM program aims at enhancing leadership skills while the Network program tries to encourage cooperation between companies. The raison d’être for all four initiatives is a presumed market failure and the public benefits from compensating an assumed funding gap when positive externalities are expected. The paper is based upon the view that matching routines are nonparametric pre- processing methods that facilitate further analysis. A combination of covariate matching and difference-in-differences analysis is employed for finding the best possible estimates for the effects of the programs under scrutiny. The analyses suggest that at least two of the four initiatives produce positive contribu- tions and produce lasting impacts that are observable for a considerable period following par- ticipation.

The second paper concerns a leadership-training program for SME’s. Over a period of eight to fourteen years after key managers completed the program we have traced the effects of the program on the performance of participating firms. Transition rate methods are applied in order to assess the survival of participating firms. The key findings are that the impact of the training program with respect to survival is negligible and that this overall effect can be decomposed into a positive impact upon the mature participating firms and a catastrophic negative effect on the newcomers; the newly established firms that participated. The third paper discusses the methodological cleavages within the evaluation commu- nity. The history of evaluation has produced a number of sub-fields based on different parent disciplines. Strong paradigmatic commitment to own sub-discipline and corresponding rejec- tion of competing world-views may lower the overall trust in evaluations. Since New Public Management predominantly is based on a popular notion of economics, this is the story of how pluralism meets monism. The conclusion is that the methodological heterogeneity of evaluation research may be deteriorating with respect to public trust in social science based evaluations.