Evaluation of Innovation Programming

September 2017

View the print-friendly version: PDF

Table of Contents

• Section 1: Evaluation Recommendations
• Section 2: Introduction
• Section 3: Relevance
• Section 4: Performance: Achievement of Expected Outcomes
• Section 5: Performance: Design and Delivery

Western Economic Diversification Canada (WD) would like to thank all of the key informants and survey participants who generously gave of their time and knowledge to take part in the Evaluation of Innovation. Without their participation and their insights, this report would not have been possible. The evaluators acknowledge the work done by Goss Gilroy Inc. (the consultants) in collecting key informant, focus group and survey data.

Section 1: Evaluation Recommendations

WD's Innovation programming makes strategic investments with western Canadian businesses, industry organizations and post-secondary institutions to enhance business innovation through research and development, and technology commercialization. The evaluation concludes that WD's innovation programming is relevant and achieving results. The evaluation identified possible improvements to enhance the design and delivery of the programming and to strengthen achievement of longer-term outcomes.

Recommendation 1: The department should develop a system to track longer-term outcomes on projects after project completion.

Recommendation 2: The department should review its project intake process to improve upon its frequency and predictability, and its interaction with applicants during the project assessment stage.

Recommendation 3: The department should carefully assess how it will provide meaningful feedback to unsuccessful applicants on their applications

top of page

Section 2: Introduction

2.2 Background

The Department of Western Economic Diversification Canada (WD) was established in 1987 to lessen western Canada's (British Columbia, Alberta, Saskatchewan, and Manitoba) strong economic dependence on its natural resources. Under the Western Economic Diversification Act, 1988, the department is mandated to "promote the development and diversification of the economy of Western Canada and to advance the interests of Western Canada in national economic policy, program and project development and implementation."

Innovation programming within the department contribute to WD's (the department's) strategic outcome of growing and diversifying the western Canadian economy. The department uses two grant and contribution (G&C) programs to provide funding for projects under its innovation programming activities. The initiatives are:

• Western Innovation (WINN) Initiative – A $100 million five-year federal Initiative that offers repayable contributions for businesses with operations in Western Canada to move their new and innovative technologies from the later stages of research and development to the marketplace.
   
• Western Diversification Program (WDP) – WDP funding is provided to not-for-profit organizations (such as industry associations, incubators, accelerators, post-secondary institutions, and government) for innovation projects that either bridge the gap between western Canada's science and technology (S&T) strengths and industrial research and development (R&D) activities and/or build capacity in support of commercialization of knowledge-based products, processes and services.

2.3 Evaluation Context and Scope

This evaluation study was included in the department's evaluation plan (2015–20). This evaluation addresses the core issues of relevance and performance identified in the Treasury Board Policy on Results. The scope of the evaluation included all WD innovation projects (WDP and WINN) from 2010–11 to 2014–15. The scope was expanded to include completed projects implemented prior to 2010–11 specifically to gather data on longer-term results.

Characteristics of the Projects Selected for the Evaluation

• The evaluation focused on 311 WDP and WINN projects within the evaluation period, which involved $585 million in funding from WD.
   
• The overall project costs totalled $1.79 billion, which means that $2.07 in funding was contributed from other sources for every dollar approved by WD.
   
• The funding was allocated across the provinces with 33% of the funding recipients based in British Columbia, 30% in Alberta, 20% in Saskatchewan, and 16% in Manitoba.
   
• The 311 projects involved 179 funding recipient organizations. About 134 funding recipient organizations received funding for one project. The rest received funding for 2 or more projects within the evaluation period.

2.4 Evaluation Approach, Design and Methodology

The evaluation issues addressed in this study related to relevance and performance. The methodology for the evaluation include:

• Review of relevant documentation and data – Documents reviewed include background on WD innovation programming. Data reviewed include financial and performance information.
   
• Key informants interviews – In total, 53 interviews were completed with a broad cross section of stakeholder groups in order to contribute a line of evidence that will address many of the evaluation questions and indicators related to relevance, results and operating efficiency.
   
• Surveys – The survey was conducted with 179 funding recipient organizations with a response rate of 65%. At a confidence level of 95%, the sample of organizations that completed the survey achieves a margin of error of about ±6.5%. In addition, a sample of 41 stakeholders or businesses that derived direct benefits from funded projects and 42 applicants who were not approved for funding from WD were also surveyed.
   
• Case Studies – Twelve case studies (8 for WDP and 4 for WINN) were completed for the evaluation. Results from 8 case studies completed for the 2012 Evaluation of Innovation were updated for this evaluation.
   
• Focus Groups - Two focus groups (one focused on the WDP and one focused on WINN) were completed for the evaluation. The focus groups included a mixture of representatives from funded projects and representatives from federal and provincial government departments. In total, 23 representatives participated in the focus groups.

2.5 Limitations

• Potential participant bias as much of the primary data collection is with those with a vested interest in the program – Mitigation strategies for this included ensuring that at least 40% of key informants were comprised of representatives that are not associated with any funded project proponent but are knowledgeable of WD.
   
• Potential for non-response errors and respondent bias – Mitigation strategies included closely monitoring participation levels to ensure that the profile of responding organizations was consistent with that of the overall target population within the cluster. Where necessary efforts were undertaken to reach under-represented groups.
   
• Difficulties in identifying and aggregating impacts – Mitigation strategies included using multiple lines of evidence such as surveys and WD database to report on results, and including older WDP projects to assess longer-term results.

top of page

Section 3: Relevance

3.1 Continued Need for Innovation Programming

The report, Innovation Canada: A Call to Action[1] (commonly known as the Jenkins Report) concluded that productivity growth drives increases in average per capita incomes and business competitiveness, and that productivity growth is primarily the result of innovation. The report also concluded that innovation gains require an environment or ecosystem conducive to innovative activity, which is supported by both the public and private sectors (e.g., innovation requires sufficient investment in R&D, the presence of high quality scientific research institutions, extensive collaboration between universities and industry, the protection of intellectual property, high levels of competition, and access to various forms of capital and financing).

Overall, Canada tends to lag other OECD nations in terms of innovation performance. For example, according to the most recent international comparison conducted by the Conference Board of Canada in 2015[2], Canada ranked 9th among the 16 Organizations for Economic Co-operation and Development (OECD) countries in terms of innovation. The report noted that performance on a few innovation indicators has improved, but there are signs of emerging and persistent weaknesses on a number of others. If the four western provinces were included within 16 OECD countries, British Columbia would rank 8th amongst the 16 countries (after Sweden, Denmark, Finland, the US, Switzerland, the Netherlands, and Austria). Alberta would rank 11th, and Saskatchewan and Manitoba would each rank 15th.

Findings from research are consistent with findings from key informant interviews and focus groups conducted for the evaluation. When asked what they see as the primary needs and issues for innovation programs in Western Canada, key informants and focus group representatives highlighted the following:

• A lack of investment in, and limited access to capital for completing the final stages of technology development and moving products to market.
   
• The need to increase access to managerial skills related to innovation.
   
• A strong need for significant long-term investment in clusters that will drive future development of the economy.
   
• The need to further strengthen the innovation ecosystem across a range of economic drivers such as increasing access to specialized highly qualified personnel and specialized equipment and resources, and providing funding and support to assist the commercialization of products.
   
• Investments to maintain and expand the existing innovation ecosystem system as well as close key gaps ranging from earlier stage investments to providing investment that will help early-stage companies with initial products on the market to scale up operations.

3.1.1 Alignment of WD Innovation Programming to Western Canada Innovation Needs

Expected results for WD delivery of innovation programming include advanced technology development proven through successful deployment in an operational setting, enhanced financial capacity of businesses to pursue technology development and commercialization, and sales of knowledge-based products/processes/services by businesses. Other immediate outcomes are applied R&D technology development involving validation in a simulated environment, technology commercialization capacity by ensuring prototypes are ready for demonstration in an operational setting, and highly qualified personnel to support commercialization of knowledge-based products, processes, and services. These results aligned with innovation needs and support WD's mandate to grow and diversify the western Canadian economy.

The key informants noted that most of WD's investments in technology commercialization are well-aligned with the needs of businesses given that the investments are targeted directly or indirectly (i.e. with the support delivered through non-profit organizations) at businesses.

Applicants of WDP and WINN funded projects were asked to rate the extent to which there continues to be a need for WD to provide funding for projects that focus on innovation on a scale of 1 to 5 where 1 is not at all and 5 is to a great extent. The perceived need for innovation programming was high and funding recipients gave an average rating of 4.8. Applicants who did not receive funding for their projects provided a rating of 4.6 on the need for WD funding. Some of the reasons given for the need were shortage of capital, the need to diversify the western Canada economy and create strategic long term benefits, build on business clusters and fill gaps in innovation infrastructure.

3.1.2 Gaps in Innovation Programming

The gap most commonly mentioned by key-informants was the limited availability of funding relative to the high demand. As a result, many good projects are rejected. Some WD respondents noted that it was difficult for them to comment on whether the programs are aligned with the needs of businesses, given that staff have little access to first-hand information about the specific needs or major issues faced by businesses.

Other gaps noted by key informants include the need to:

• increase support for later stage commercialization activities;
   
• increase access to relevant business support services for businesses, and increase access to talent with specialized skills (e.g. management positions, and more incentives for experts and managerial staff to work); and
   
• making funding available for the improvement of technologies and not just new technology.
   

3.1.3 Other Innovation Programs

Several federal, provincial and industry programs provide funding to innovation initiatives in Western Canada. A few of the programs identified through document reviews and key-informant interviews are highlighted below:

1. The Industrial Research Assistance Program (IRAP) – A National Research Council of Canada program that provides financial support to small and medium-sized enterprises in Canada to help them undertake technology innovation
    
2. Centres of Excellence for Commercialization and Research Program – A federally funded program that bridges the gap between innovation and commercialization by matching clusters of research expertise with the business community.
    
3. Natural Sciences and Engineering Research Council of Canada (NSERC) – A federal government agency that provides grants to postsecondary students and postdoctoral fellows to research in the natural sciences and in engineering.
    
4. Canadian Foundation for Innovation (CFI) – Created by the Government of Canada, the CFI operates as an independent, non-governmental body and funds innovation research by universities, colleges, research hospitals and non-profit research organizations.
    
5. Sustainable Development Technology Canada (STDC) – A Government of Canada funded initiative to support the development and demonstration of innovative clean technology projects.

Mechanisms have been implemented by WD to ensure complementarity and coordination of funding for projects to the extent possible. For example, a Memorandum of Understanding was signed between WD and STDC to coordinate funding efforts. IRAP is a traditional grant program focused on early stages of commercialization and is perceived to be complementary to WD. IRAP officers are also engaged in assessment of the technical merit of the WINN applications.

Key informants perceived WD's innovation programming to be more flexible than most other identified programs and noted that WINN is the only government program that is providing repayable contributions specifically focused on commercialization of technology. It was noted that the WINN program is structured in a way to contribute to a variety of costs (capital, salaries, materials and equipment) while other funding programs may be more restricted to specific types of eligible costs.

Funding partners of projects viewed WD programs to be complementary and help augment other programs. Focus group participants indicated that WD funding played a critical role in leveraging funds from other partners. About half of federal and provincial government representatives noted that the presence of WD funding for a specific project may influence their decision to participate in projects by providing funding or other assistance to organizations. Most key informants noted that other similar funding programs overlap with WD to the extent that they are intentionally designed to be stacking programs and facilitate some risk sharing. Equipment grants for research institutions were identified by key informants as one area of potential overlap between WD innovation programming with other funders such as NSERC and CFI.

3.2 Alignment with Government Priorities

The federal government budget 2017 introduced an innovation and skills plan to build Canada as a world-leading innovation economy to create jobs and grow the middle class. The budget supports innovation in key growth industries such as clean technology, digital and agri-food. New measures in the budget are designed to improve access to financing, encourage investment, support the demonstration of technologies and create jobs. This aligns with key expected results identified for WD's innovation programming. Expected results include clean technology, access to financing for businesses, technology development and demonstrations, and the creation of highly qualified personnel to support technology commercialization.

Nearly all key informants perceive WD activities to be aligned with emerging government priorities and expectations. Currently much of program funding for WINN is focused on clean technology which is a major priority of the government.

3.3 Alignment with Federal Roles and Responsibilities

The roles and responsibility of the federal government (and WD) in innovation programming is set out in the federal government's innovation and skills plan. The plan states that to help Canada realize its potential as a global leader in innovation, the Government must ensure that its services best meet the needs of Canada's innovators and job creators. Stakeholders interviewed for the evaluation agreed that WD's innovation programming is aligned with federal roles and responsibilities.

The government (and WD) role in fostering an innovation ecosystem is supported by the OECD innovation strategy presented in 2015[3] at a meeting of the OECD Council in Paris in June 2015. The strategy was well received at the meeting and affected policy developments in many countries. The strategy concludes that governments play a key role in fostering a sound environment for innovation, investing in the foundations for innovation, helping overcome barriers to innovation, and ensuring that innovation contributes to key goals of public policy.

__________

top of page

Section 4: Performance: Achievement of Expected Outcomes

4.1 Performance Measurement Data

The performance measurement system used by WD requires funding recipients to collect and report data on a number of agreed upon performance indicators (usually up to a maximum of five indicators per project). When the projects are approved, targets are established for each indicator. Recipients provide periodic updates to WD on performance indicators and targets during the project and at its conclusion.

• Results for WDP projects in terms of dollar value, include:
   
  • $61.6 million of incremental private sector investment attracted;
     
  • $15.8 million off applied R&D related to initial technology development;
     
  • $36 million of sales tied to the knowledge-based product, process, service, or technology commercialized;
     
  • $5 million of applied R&D undertaken related to intermediate technology development;
     
  • $58.4 million of R&D undertaken in the new facility or using new equipment supported by WD funding;
     
  • $9.2 million of international business leads generated through missions, events, and marketing initiatives;
     
  • $8.4 million of advanced technology development undertaken; and
     
  • $19.2 million of R&D undertaken for a 3-year period following new facility completion or equipment set-up.

• Most WINN projects were on-going at the time of the evaluation, however progress is being made towards achievement of results. Results in terms of dollar value include:
   
  • $5 million of sales tied to the knowledge-based product, process, service, or technology commercialized;
     
  • $5 million value of incremental private sector investment attracted; and
     
  • $2.3 million of advanced technology development undertaken.

The purpose of most of WD innovation projects is to facilitate the development and/or commercialization of new technology. If successful, over the longer-term, most of these projects will contribute to the business revenues, attract new investment, and the create employment. The exercise of analyzing the survey data with data from WD database identified a variety of limitations with respect to reporting on results. These include:

• Significant impacts commonly occur beyond the reporting period and therefore are not captured.
   
• The diversity of projects supported by WD contributed to the use of a wide range of unique indicators for assessing project success. As a result it was difficult to roll up results in a meaningful way.
   
• Key indicators for reporting on results could be applicable to a wider range of projects than were reported against, resulting in under-reporting of results in some instances.
   
• Certain types of non-monetary indicators involve aggregating unequal outcomes rendering the data meaningless. About 5% to 10% of the projects with a given indicator (e.g. number of prototypes ready for demonstration in an operational environment) commonly account for approximately 60% to 80% of aggregated targets set, because of the nature of the projects. As such, achieving set targets often reflects the performance of one or two projects.
   
• Concerns about reliability (most of the data is self-reported by the funding recipients and not validated) and data availability (particularly with respect to projects where reporting is dependent on non-profit organizations being able to collect data from participating companies).

4.2 Innovation Projects Impact

In the survey, funded recipients were asked to rate the extent (on a scale of 1 to 5, where 1 is not at all, 3 is to some extent, and 5 is to a great extent) to which the project resulted in a variety of different types of impacts. The highest rated impact was the development of networks, partnerships, and alliances (4.4). Other highly rated impacts for recipients funded included on-going investment in R&D (4.0), development of new/improved technologies (4.0), adoption of new technologies (4.0), attraction/retention of HQP (3.9), development of technical or business skills (3.9), as well as development of prototypes or new technology demonstrations (3.9).

Where recipients reported that their project generated specific types of impacts, they were asked to quantify the impact (e.g. in terms of the number of new partnerships and alliances developed). Recipients expressed that impacts:

• led to the development of over 1,000 partnerships and alliances;
   
• led to the development of 744 technologies and over 8,000 prototypes;
   
• supported the adoption of 645 technologies and the commercialization of over 1,500 products and technologies;
   
• attracted and retained nearly 3,000 HQP;
   
• assisted nearly 8,500 people further develop their business and technical skills;
   
• contributed to signing more than 3,000 licensing arrangements or transfer agreements;
   
• generated 156 new businesses including spinoff companies; and
   
• led to the filing of more than 1,800 technology disclosures and 629 patent applications.

In terms of financial impacts, recipients reported that their projects contributed (or are expected to contribute in the short-term) to further investment of over $590 million, increased investment in R&D totaling $311 million, increased revenues of $177 million by new companies created, and costs savings of $58 million.

4.3  Longer-Term Results

Most significant results from projects are achieved after the end of WD funding. The 12 case studies conducted for the evaluation illustrated significant long term results after project completion. Most targets established for projects examined for the case studies were achieved or exceeded. In most cases, the success of the projects led to additional projects between WD and the organizations. Examples from the case studies include:

• Genome Prairie, which continues to be a success and has supported over $260 million of research activity with funding from federal, provincial, industry and international sources. WD provided $1.2 million for two projects with the organization to support the establishment of four pan-western networks, and bridge the pre-commercialization gap. To date, Genome Prairie estimates having created 1,900 jobs supporting over $260 million of research activity.
   
• WD completed 2 projects with the Saskatchewan Food Industry Development Centre in 2010 and 2012 to support equipment purchase and establishment of an innovation facility. An additional project for the establishment of a new agri-food innovation facility is on-going and expected to be completed by 2018. A review of the current annual report of the centre provides examples of the centre successfully working food processors to add value to their products for domestic and/or international markets. Since inception, the Food Centre has assisted over 260 clients in developing and processing over 700 products where 50% have been commercialised.

The on-going commercial success of projects such as the NeuroArm with the University of Calgary is one example of longer-term outcomes of funded projects. A review of the website for the project shows that the NeuroArm continues to be profiled internationally through conferences, workshops and publications. The NeuroArm have been successfully commercialised and is currently used in neurosurgical procedures in hospitals.

4.4 Incrementality

WD funding accounted for 32% of the total funding for WDP projects and 44% of the total funding for WINN projects included in the evaluation. Funded recipients indicated that most of their projects were dependent upon the availability of funding from WD. Projects were likely to have been delayed, cancelled or reduced in scope in the absence of WD funding. On average, the recipients estimated that there was a 40% likelihood that their project would have proceeded in some form (e.g. reduced or delayed) even in the absence of support from WD.

Over half of WINN applicants who did not receive funding from WD (55%) reported that their projects were implemented but with a reduced scope, a delayed start, or a longer implementation period. Some of the applicants were able to access funding from other sources including IRAP, Growing Forward (Invest Agriculture BC), private capital or private equity, foreign partners and NSERC. At the time of the survey, 7% of projects had been completed, with 52% of projects ongoing. Seventeen percent of respondents said their projects were suspended while another 7% reported that their project had not yet started.

4.5 Client Satisfaction

About 90% of funded recipients expressed satisfaction in their dealings with WD staff. They indicated that WD provided clear direction to them regarding what was expected in a proposal. Most unfunded applicants tended to be less satisfied with the application requirements and criteria as well as in their dealings with WD staff. Reasons for dissatisfaction include:

• receiving no feedback on the reasons for applications being rejected;
   
• the length of the application and approval cycle; and
   
• the time and expense to prepare an application.

When asked to rate the extent to which the assistance provided by WD met the needs of their organization on a scale of 1 to 5 where 1 is not at all and 5 is to a great extent, WDP recipients provided an average of 4.9 while WINN recipients provided an average rating of 4.6. In addition to the funding received from WD, the recipients attributed the success of their projects to a strong team, collaborations with partners external to the project, a strong vision and leadership, and organizational support. Some of the factors that were identified as constraining the success of projects included delays in project implementation, changes to the operating environment in which the project was designed (e.g., severe declines in oil prices and other commodity prices), and technical issues.

top of page

Section 5: Performance: Design and Delivery

In May 2014, WD moved to a call for proposals (CFP) process where applicants submitted project proposals during set time periods based on WD's Strategic Priorities. The CFP system is used for the majority of WD programming areas not just for innovation projects.

The first CFP for WDP closed in May 2014 while the second closed in February 2015. Under the two intakes, WD received 665 WDP applications and approved 111 projects seeking $127 million in total WD funding. Three CFPs have been held to date for WINN: closing in the fall of 2013, the fall of 2014 and spring 2016. Under the 3 intakes, WD received 797 applications and approved 85 projects seeking $96.9 million in total WD funding.

Stakeholders commented that the CFP process is open, transparent and fair process that allows for comparative analysis of projects and better alignment with Government of Canada plans and priorities. After the 2nd intake, improvement were made in areas such as the applications review and approval processes, scoring matrices, and communications

The majority of stakeholders across multiple lines of evidence for the evaluation (key informant interviews, focus groups, surveys and case studies) expressed concern in the following areas of the CFP process:

• Timeliness of the intake process both in terms of frequency and predictability was an issue. This means that projects from proponents may need to sit on the shelf for an extended period of time waiting for WD funding to be available.
   
• There was a lack of feedback on the project review process. Officers are not able to engage with project proponents or provide feedback to improve the projects.
   
• Short timelines (proposal time and time for project implementation) was a concern. The short timeline between the call and the closing meant that some of the proposals were very rushed which impacted on the quality. The short timelines also added to the difficulty for project proponents to leverage funding in support of their applications.
   
• Inadequate feedback to unsuccessful applicant discourages future applications. This view was expressed strongly by the majority of unsuccessful applicants and also mentioned in the focus groups.

Other concerns raised on the CFP process by some stakeholders across multiple lines of evidence include:

• CFP tends to be reactive rather than proactive, and less proponent-driven which can result in lower quality proposals.
   
• There were difficulties for applicants in meeting selection criteria in areas such as funding requirements, and timelines to commercialization.
   
• There were differences in stacking limits. The guideline regarding allowable stacking of funding tends to be lower under WINN than for other programs (e.g. 50% vs 75%).
   
• Coordination of WINN with the Scientific Research and Experimental Development (SR&ED) tax credits was a concern. The SR&ED Program is a federal tax incentive program designed to encourage Canadian businesses of all sizes and in all sectors to conduct research and development in Canada. Access to WINN funding impacted on SR&ED tax credits for companies.
   
• There was limited awareness of WD Innovation programs amongst potential applicants and intermediaries

Suggestions for Improvements to Design & Delivery

Suggestions for Improvements offered by stakeholders include:

• Staging more frequent intakes using a fixed annual schedule. Stakeholders suggested using a hybrid approach to program delivery (e.g. one or two annual intakes and continual intake or more frequent intakes for WDP applicants). This would increase the predictability of the CFP process to applicants.
   
• Improving engagement with project applicants during the application stage. Examples of such engagements may include presentations or an interview process during the assessment criteria stage, and working with applicants to further develop their proposals.
   
• Increasing engagement with applicants and stakeholders, and community level outreach, to better understanding needs, build coalition, and communicate government priorities and requirements.
   
• Introducing some type of tiered application system, where the concept is first reviewed before the proponent is invited to apply to minimize the amount of time spent on unsuccessful applications.
   
• Providing clear, detailed feedback to unsuccessful applicants (e.g., reporting on the results using a scoring grid), and providing reference to other funding agencies. This would help unsuccessful applicants to refine their proposals.
   
• Addressing stacking issue and better coordination of WINN funding with SR&ED tax credits.
   
• Reducing the time that elapses from the closing date to the first payment. A suggested option to mitigate the impact was to allow reach backs (i.e. allow companies to claim eligible expenses that were incurred after the application closing date but prior to the contract signing.
   
• Being more targeted in the CFP and seek to optimize the resources by setting parameters around the established priorities. An example provided is to make pre-screening processes more structured and focused so that applicants are not wasting their time writing full proposals and being rejected because the funds are limited.

Leverage Rate

WD approved funding for the 311 innovation projects totaling nearly $585 million. The WD funding was leveraged with significant funding from other sources including industry, non-profit organizations, and other government departments. The overall project costs totalled $1.79 billion, which means that $2.07 in funding was contributed from other sources for every dollar approved by WD. This compares well with findings from the 2012 Evaluation of Innovation[4] (each department dollar was matched by $2.50 from other contributors in 2012).

top of page