Evaluation of the Results and Anticipated Socio-Economic Impact of Completed Projects of the ...

205 Pages·2009·1.43 MB·English

EVIMP -2

Evaluation of the results and
anticipated socio-economic impact of
completed projects of the Growth Programme

Lot 7 report
December 2006
Contents Page

1. Context 6
2. Evaluation questions and method 10
2.1. Key evaluation questions 10
2.2. Evaluation methodology 10
3. Logic model 18
3.1. Explanation of the model 18
3.2. Linking the model to the structure of the report 20
4. Profile assessment 25
4.1. Introduction 25
4.2. Technical characteristics 27
4.2.1. Managerial challenge 27
4.2.2. Motivational challenge 30
4.2.3. Technical challenge 33
4.3. Exploitation characteristics 37
4.3.1. Leadership 37
4.3.2. Structure 39
4.3.3. Orientation 41
4.4. Comparison of the funding instruments 41
5. Evaluation findings 47
5.1. Introduction 47
5.2. Management performance 47
5.3. Effectiveness by instrument, action and type of project 49
5.4. Success classification by instrument, action and type of project 54
5.5. Relation between effectiveness and success classification 58
5.6. Success and failure factors 59
6. Impact assessment 63
6.1. Introduction 63
6.2. Impact on partners 63
6.3. Impact on industry 66
6.4. Impact on environment and society 69
7. Analysis of hypotheses 72
7.1. Does effectiveness increase the probability of impact? 72
7.2. What are the determinants of effectiveness? 72
Final report EVIMP-2 Lot 7 Page 2 of 203
7.2.1. Preliminary experiments 72
7.2.2. What are the determinants of technical effectiveness? 73
7.2.3. What are the determinants of exploitation effectiveness? 73
7.2.4. What are the determinants of success/failure? 74
7.3. Conclusions 75
8. SMEs and NAS 79
8.1. SME participation 79
8.2. NAS countries 82
9. Conclusions 85
9.1. Relevance of the results of the evaluated projects to the objectives of the
KA/GA, as stated in the work programme 85
9.2. Usefulness of the evaluated projects compared with the funding involved 89
9.3. Issues of synergy among evaluated projects 89
9.4. Lessons learnt 90
10. Appendices 93
Statistical annex 93
Short project descriptions 99
List of reclassified projects in KA 3 and KA 4 and dedicated calls in GA2 196
List of projects that do not correspond to logic model conditions 199
Glossary 202

Final report EVIMP-2 Lot 7 Page 3 of 203
List of figures

Figure 1: evaluation method.....................................................................................................................................11
Figure 2: logic model...............................................................................................................................................24
Figure 3: instruments by action................................................................................................................................25
Figure 4: what if the EC had not funded..................................................................................................................30
Figure 5: type of players in consortium by instrument.............................................................................................34
Figure 6: type of player involved in consortium by action.......................................................................................34
Figure 7: size class of main industrial partners who are also coordinator by instrument.........................................35
Figure 8: size class of main industrial partners who are also coordinator by action................................................36
Figure 9: size class of main exploitation partners who are coordinator by instrument............................................38
Figure 10: size class of main exploitation partners who are coordinator by action..................................................39
Figure 11: size classes of industrial partners by instrument.....................................................................................40
Figure 12: size classes of industrial partners by action............................................................................................40
Figure 13: type of research by instrument................................................................................................................41
Figure 14: technical risk...........................................................................................................................................43
Figure 15: exploitation capability.............................................................................................................................44
Figure 16: exploitation risk......................................................................................................................................45
Figure 17: management performance.......................................................................................................................48
Figure 18: effectiveness and project phases.............................................................................................................51
Figure 19: effectiveness by instrument....................................................................................................................52
Figure 20: effectiveness by action............................................................................................................................53
Figure 21: effectiveness by type of research............................................................................................................53
Figure 22: effectiveness of the programme..............................................................................................................54
Figure 23: success and failure by instrument...........................................................................................................55
Figure 24: success and failure by action..................................................................................................................56
Figure 25: success and failure by type of research...................................................................................................57
Figure 26: success rate of the programme................................................................................................................57
Figure 27: relation of success with effectiveness.....................................................................................................58
Figure 28: impact on partners...................................................................................................................................63
Figure 29: impact on partners by action...................................................................................................................64
Figure 30: estimated monetary value.......................................................................................................................65
Figure 31: impact on industry..................................................................................................................................67
Figure 32: impact on industry by action...................................................................................................................67
Figure 33: impact on job creation............................................................................................................................68
Figure 34: impact on safeguarded jobs.....................................................................................................................69
Figure 35: size classes of SMEs...............................................................................................................................81
Figure 36: NAS partner activity by instrument........................................................................................................83
Figure 37: NAS countries per instrument.................................................................................................................83
Figure 38: relevance of project s in KA1.................................................................................................................85
Figure 39: relevance of projects in KA3..................................................................................................................86
Figure 40: relevance of projects in KA4..................................................................................................................87
Figure 41: success and failure by budget size..........................................................................................................89

List of tables

Table 1: possible project objectives.........................................................................................................................18
Table 2: possible impacts.........................................................................................................................................19
Table 3: rationale for including variables in the logic model...................................................................................22
Table 4: overview of projects by action and type of research..................................................................................25
Table 5: sectors in RTD where research was carried out.........................................................................................26
Table 6: sectors in CR where research was carried out............................................................................................26
Table 7: sectors in TN where research was carried out............................................................................................27
Table 8: average budget size per project in EUR.....................................................................................................27
Table 9: minimum budget size per project in EUR..................................................................................................28
Table 10: maximum budget size per project in EUR...............................................................................................28
Table 11: % of projects with(out) differences between requested and granted funding...........................................28
Final report EVIMP-2 Lot 7 Page 4 of 203
Table 12: number of partners per project.................................................................................................................29
Table 13: number of countries per project...............................................................................................................29
Table 14: budget concentration by Herfindahl index...............................................................................................29
Table 15: project management capability.................................................................................................................29
Table 16: extensions in months................................................................................................................................30
Table 17: number of projects coherent with the core business in % of total...........................................................31
Table 18: motivation factors RTD............................................................................................................................31
Table 19: motivation factors CR..............................................................................................................................32
Table 20: motivation factors TN..............................................................................................................................32
Table 21: research capacity......................................................................................................................................33
Table 22: value chain integration.............................................................................................................................33
Table 23: coordinator is the main industrial partner.................................................................................................35
Table 24: concentration of research partners per project........................................................................................37
Table 25: coordinator is main exploitation partner..................................................................................................37
Table 26: number of SMEs coordinating as exploitation partner in RTD................................................................38
Table 27: final user inclusion...................................................................................................................................40
Table 28: comparison between instruments concerning managerial challenge........................................................42
Table 29: comparison between instruments concerning motivational challenge.....................................................42
Table 30: comparison between instruments concerning technical challenge...........................................................42
Table 31: comparison between instruments concerning leadership.........................................................................43
Table 32: comparison between instruments concerning structure............................................................................44
Table 33: comparison concerning orientation..........................................................................................................44
Table 34: summary table on profile characteristics..................................................................................................45
Table 35: distribution of projects used for evaluation findings................................................................................47
Table 36: projects with TIP seminar........................................................................................................................48
Table 37: success and failure definitions.................................................................................................................54
Table 38: most important success factors.................................................................................................................59
Table 39: most important failure factors..................................................................................................................60
Table 40: success factors for RTD...........................................................................................................................60
Table 41: success factors for CR..............................................................................................................................60
Table 42: failure factors for RTD.............................................................................................................................61
Table 43: failure factors for CR...............................................................................................................................61
Table 44: key impact figures for partners.................................................................................................................65
Table 45: key impact figures on industry.................................................................................................................69
Table 46: environmental and societal impact by action...........................................................................................70
Table 47: environmental and societal impact by instrument....................................................................................70
Table 48: key impact figures for environment.........................................................................................................71
Table 49: key impact figures for society..................................................................................................................71
Table 50: significant technical effectiveness...........................................................................................................75
Table 51: exploitation effectiveness.........................................................................................................................75
Table 52: effectiveness of projects by instrument....................................................................................................75
Table 53: additional success factors.........................................................................................................................76
Table 54: number of SMEs per project...................................................................................................................80
Table 55: roles for SMEs as % of total of those roles..............................................................................................80
Table 56: number of NAS partners per project........................................................................................................82
Table 57: roles of NAS partners in numbers............................................................................................................82
Table 58: Relevance GA1 RTD objectives..............................................................................................................87
Table 59: relevance GA1 socio-economic objectives..............................................................................................88
Table 60: relevance GA 2 and INFRA RTD objectives...........................................................................................88
Table 61: relevance GA 2 and INFRA socio-economic objectives..........................................................................88
Final report EVIMP-2 Lot 7 Page 5 of 203
1. CONTEXT
The “Competitive and Sustainable Growth Programme” was initiated by the European Commission in
1997-1998. It was an important component within the Fifth Framework Programme as the budget for
the Growth projects and activities amounted to 2 705 million Euro for the period from 1998 to 2002. It
brings together work under three previous programmes: BRITE-EURAM, SMT (Standards,
Measurement & Testing) and Transport.

Within the objectives of the Lisbon strategy, the targets of the Competitive and Sustainable Growth
Programme are clear: to increase economic growth and to create new jobs in Europe.

The need and motivation for deploying the Growth programme was the understanding that the future
economic power of the European Union will be based on a range of goods and services, both
competitive and environmentally friendly, in order to help improve the quality of life and beyond. The
ambition was, therefore, both to support European industry to become more competitive and to achieve
sustainable development.

The main goals of the Competitive and Sustainable Growth Programme were:
• to produce, disseminate and use the knowledge and technologies needed to design and develop
processes and produce high quality, environment- and consumer-friendly products which will
be competitive on tomorrow's market;
• to help increase economic growth, maintain and/or create new jobs in Europe;
• to sustain the continuing innovation and modernization efforts of manufacturing, processing
and services enterprises (including SMEs) so as to improve their competitiveness;
• to support the development and implementation of Community policies that enable competitive
and sustainable development.

These goals went hand in hand with the development of related services, including transport, which is
economic, safe and protective of the environment and quality of life as well as with the development of
quality materials, reliable measurement and testing methods and the optimal use of specific research
infrastructures.

In order to operationalise the aims described above, several different types of instruments
(project types) were applied, including R&D projects, demonstration projects, thematic networks,
concerted actions, etc.

R&D, demonstration and combined projects (RTD) were carried out by industrial or service
organizations, universities and research centres covering research targeted on strategic objectives with
significant potential for socio-economic and industrial impact. They demonstrated the participation of
economic entities, whenever appropriate, as well as potential for new policy options and Community
policy. SME participation in RTD actions was to be encouraged and special attention was given to the
utilisation and diffusion of technologies developed under RTD projects.

Thematic Networks (TN) were designed to facilitate networking of organisations, co-ordination of
activities and exchange and dissemination of knowledge with the intention to optimise research efforts,
reach critical mass, and enhance impact at European level. They were meant to bring together industry,
universities, research centres, users, research infrastructures, and other relevant stakeholders around a
common S&T objective related to the priorities of the programme.

Co-operative Research proposals (CR) were designed to enable at least three mutually independent
SMEs from at least two different Member States, or one Member State and an associated state, to
jointly seek the resolution of their common technological problems by entrusting it to third legal entities
(the "RTD performers"), including industrial entities, with appropriate research or technological
validation capacities.
Final report EVIMP-2 Lot 7 Page 6 of 203
Concerted Actions were used for the co-ordination of research activities already funded within
individual Member States, in order to channel efforts efficiently, to exchange and complement
experience, or to disseminate results.

The aim of Virtual Institutes was to improve the interaction between industry and research and to
facilitate the transfer of technology and research results, in particular to SMEs, for quick exploitation.
Using the latest information and communication technologies to ensure effective transnational
networking, they were meant to become self-funding after the period of EU support. Virtual Institutes
have been described as a pilot instrument and the Union has been gathering valuable experience in their
creation.

Accompanying measures were aimed at contributing to the implementation of the programme, to the
up-dating of the Work programme, the preparation of future activities and the dissemination of results.

The Programme was structured in three main interconnected elements:

1. A set of four key actions, helping to develop critical technologies, concepts and policies to
solve clearly identified problems:

Key Action 1 (KA1): Innovative products, processes and Organisation

As part of the GROWTH Programme, Key Action 1 (KA 1) focuses on „innovative products, processes
and organisation“. The special emphasis of KA 1 was to support the production of industrial goods and
services contributing towards solving problems with strategic implications of society, the economy and
Europe. The aims of KA 1 can be clustered into the four categories:
• “high quality and internationally competitive“;
• “clean“;
• “safe”;
• “user-friendly“.

Within KA 1, concentration on a few most relevant priorities was achieved through calls for project
proposals targeted on clearly identified objectives, the “Targeted Research Actions” (TRAs). These
TRAs were modified during the course of the GROWTH programme:

TRAs according to the original work programme (Call 1, March 1999):
1.1 Customer oriented and high-tech production;
1.2 Towards new and miniaturised products and processes;
1.3 Machinery, production equipment and systems for manufacturing;
1.4 Towards zero-waste in manufacturing and processing promoting eco-efficient industries.

TRAs according to the work programme, version Dec 1999 and later (all Calls after Call 1):
1.5 Products Services;
1.6 Machines;
1.7 Extended Enterprise;
1.8 Modern factory;
1.9 Infrastructure.

Within each TRA, R&D activities and clusters of projects are covering and integrating certain key R&D
areas, each with further subdirectories as well:
• Efficient production including design, manufacturing, and control;
• Intelligent production;
• Eco-efficient processes and design;
• Organisation of production and work.

Final report EVIMP-2 Lot 7 Page 7 of 203
Key Action 2 (KA2): Sustainable Mobility and Intermodality

Key Action 2 on Sustainable Mobility and Intermodality, comprising
• a regulatory and accountable framework reflecting socio-economic objectives;
• an interoperable infrastructure which allows the operation of attractive, environmentally
friendly and efficient transport means;
• modal and intermodal systems for managing operations and providing services.
KA 2 is not part of this study.

Key Action 3 (KA3): Land Transport and Marine Technologies

As part of the Growth Programme, Key Action 3 focuses on “land transport and marine technologies”.
The main emphasis of KA3 is to support research contributing to solve the challenges of society, the
economy and enhance the competitiveness of Europe in the specific technical areas.

Within KA3, the priorities were clearly identified as “the targeted research areas (TRA)”. Proposals
requesting EU support had to refer to one of these objectives:
• Improved fuel efficiency and reduced emissions - cutting CO2 emissions and developing and
validating zero-emission vehicles;
• Improved performance - increasing safety, reliability, maintainability, availability, operability,
energy efficiency and adaptability;
• Improved system competitiveness - reducing both time to market and development costs.

Key Action 4 (KA4): New perspectives for Aeronautics

As part of the Growth Programme, Key Action 4 focuses on “New perspectives in aeronautics”. The
aim is to strengthen the competitiveness of the European aeronautical industry, while ensuring
sustainable growth of air transportation with regard to environmental and safety issues.

Within KA4, the priorities were clearly identified as “the targeted research area’s (TRA)”. Proposals
had to refer to one of these objectives:
• Cut procurement costs;
• Improve efficiency and performance, with reduction in fuel consumption by 20% and general
improvement in reliability and direct operating costs;
• Reduce noise and climate impacts as well as improve passenger environment.
• Improve operational capability and safety.

2. Research and technological development activities of a generic nature, with potential
multi-sectoral applications, helping to develop the scientific and technological base in a
limited number of critical areas:

These encompass:

Generic Activity 1 (GA1) new materials and their production and transformation (including
steel): the main specific objectives are to:
• support advanced materials applications needed for improved quality of life;
• develop sustainable materials production and transformation technologies;
• improve safety and reliability;
• promote the efficient use and reuse of materials.

Generic Activity 2 (GA2) measurement and testing: the three socio-economic objectives are:
• technical support to standardisation and Community policies;
Final report EVIMP-2 Lot 7 Page 8 of 203
• the fight against fraud;
• improvement of quality.

Within this activity, Dedicated Calls were initiated by the European Commission on specific objective
related to the priorities of the programme like technical standardisation, EU legislation, testing
(searching for best available technologies), public awareness raising or innovation practices, support in
antifraud activities etc. in order to facilitate establishing of multinational working organisations,
coordination of activities and exchange and dissemination of knowledge with the intention to optimise
research efforts, reach critical mass, and enhance impact at European level. Sometimes the calls resulted
from meetings of EU policy-makers with research community and industry senior management tackling
high impact issues.

3. Support for research infrastructures (INFRA).

This action stimulates infrastructure operators to co-operate and pool resources among themselves, as
well as with users of the infrastructure. In addition it will help to provide a relevant networked
environment in the fields covered by the Programme.

Following types of projects were supported:
• Support activities to medium and large-scale facilities;
• Setting up of virtual institutes;
• Reference databases;
• Measurement and quality management infrastructure

Due to the limited budget for this objective (compared to the other main Programmes), the support was
limited to a number of specific projects encouraging “virtual institutes”.
Final report EVIMP-2 Lot 7 Page 9 of 203
2. EVALUATION QUESTIONS AND METHOD
2.1. Key evaluation questions
The evaluation of the different projects financed by the “Competitive and Sustainable Growth
Programme” (the GROWTH Programme) must contribute to the ex post evaluation of the results and
the anticipated socio-economic impact of the programmes as a whole.
This evaluation report will consequently focus on:
• the actual achievements of the different projects;
• the actual results and impacts of the projects;
• factors that have influenced the success or failure of the projects.
Being a judgement, the evaluation will deliver an expert opinion on:
• the relevance of the projects’ output for the objectives of the GROWTH programme and of each of
the Key-Actions or Generic Activities within it;
• their effectiveness in terms of achieving the declared objectives;
• their usefulness in terms of their achievements relative to the amount of funding received;
• their value for money;
• the impact which the project results are likely to achieve.

In addition, the evaluation is expected:
• to identify any issues to be retained as "lessons learnt";
• to identify success stories and illustrative failures to be delivered in a separate report.

2.2. Evaluation methodology
The European Commission has split up the evaluation of the programme in different lots:
• Lot 1: Key action 1;
• (Lot 2: Key action 2 was not part of the evaluation);
• Lot 3 – Key action 3;
• Lot 4 – Key Action 4;
• Lot 5 - Research and technological development activities of a generic nature - new materials;
• Lot 6 - Research and technological development activities of a generic nature - measurement and
testing and Support for Research infrastructures;
• Lot 7 - Support services, exploitation of the resulting database (including statistical analysis) ,
contribution to the evaluation of the programme and its key actions and generic activities, design of
a possible follow-up assessment, reports and presentations.
These lots have been awarded to 3 contractors:
• Lot 1 – VDI/VDE;
• Lot 3, 4 and 7 – Deloitte;
• Lot 5 and 6 – Technomar.
Final report EVIMP-2 Lot 7 Page 10 of 203