Methodology

Scope of Work

This summative evaluation of the WDP included a wide range of data collection methods, each of which are further detailed below. Please note that any category represented in the tables throughout the document with a percent less than three has been combined into an “other response less than 3%” category (with some noted exceptions).

The objectives of the evaluation and the core evaluation issues are presented in Table 2-1. The evaluation framework is presented in Appendix B.

Table 2-1 Core Evaluation Issues

Objectives	Evaluation Questions
Relevance	Do Western Canada’s current economic development and diversification needs align with those defined in the WDP 2003 Terms and Conditions? Was the WDP complementary, or did it overlap or duplicate other economic development programs? Do WDP sub-programs respond appropriately to the needs identified in the sub-program’s terms of reference, funding agreement, and/or approval authority? Do project assessments demonstrate consideration of appropriate levels of project funding in compliance with the Treasury Board Policy on Transfer Payments? Is there a legitimate and necessary role for government in this program area? Does the WDP model reflect federal and provincial priorities related to western economic diversification and development? Does the WDP reflect the needs of the western economic community?
Success	Has WDP contributed to improved coordination of federal economic activities and programs in the west? Has WDP improved understanding and awareness of western issues, including increased access to and participation in federal programs? Has WDP improved dialogue around and understanding of Western Canadian issues, challenges, opportunities and priorities? Have WDP funded programs/activities contributed to enhanced community planning, increased viability and diversification of local economies, and increased levels of community adjustment to mitigate economic crisis? Have WDP funded programs increased participation in international markets, successful partnerships and strategic alliances, and/or foreign investment in Western Canada? Have WDP funded projects increased technology adoption, commercialization and linkages, research and development, community innovation, knowledge infrastructure, or technology skill development? Were there unintended positive or negative impacts from the program?
Cost-Effectiveness	Were funds leveraged from other government sources and/or the private sector? Did cost-shared agreements allow WD and its funding partners to carry out activities that would not have been completed otherwise? Are Canadians getting value for tax dollars from the WDP? Is the package of sub-programs resulting from the WDP affordable and do they provide value for tax dollars from the WDP?
Design and Delivery	Does the program design and delivery remain appropriate? Are program recipients satisfied with the approval, monitoring, and payment processes? Has WD created an effective management structure for the various sub-programs and sub-agreements that are supported by the WDP? Have projects been adequately monitored, with project reports completed properly and in a timely fashion? Are project reports providing WDP with useful information for project management, evaluation, and monitoring? Does the current WDP reporting system demonstrate outcomes?

Literature and Document Review

A literature and document review was undertaken to better understand the activities and goals of each WDP sub-component analyzed in this evaluation. To better assess the relevance of the WDP, documents pertaining to economic needs and trends in Western Canada were examined. This allowed for an analysis of whether or not WDP activities and goals align with Western economic needs, and further elucidated whether or not there is a legitimate role for the WDP in the area of western economic development and diversification. The document and literature reviewed included, but was not limited to, the following:

• Information on sub-components;
• WDP Terms and Conditions;
• Articles pertaining to economic development programs;
• Information about the Blue Ribbon Panel (see Appendix E for more details on the Panel);
• Other economic development/diversification programming information; and
• Treasury Board policies and directives.

Administrative Data Review

The objectives of the WDP and the department’s strategic outcomes remained the same throughout the entire evaluation period. However, how the department collected administrative data changed in 2005. As such, the administrative data review had to take this into account.

Prior to 2005, the department had a logic model with indicators that linked to the department’s strategic outcomes. Departmental officers used these indicators during the project assessment and due diligence phase to ensure that projects would contribute to the achievement of the long-term outcomes. Departmental officers recorded these indicators manually in the project files. These indicators were not used as performance measurement indicators except on an ad-hoc fashion for departmental performance reports.

In 2005, the Treasury Board required departments to develop a Program Activity Architecture (PAA). The PAA was developed as a performance plan to reflect the activity, sub-activity, and project outcomes, as well as indicators and sources where performance information can be obtained. The PAA was designed to guide the measurement of success of the WDP down to the project level and based on the strategic outcomes of the department: policy, advocacy, and coordination; community economic development; entrepreneurship and innovation. Project-level indicators are also intended to reflect or link to PAA indicators. Projects do continue to have unique indicators and outcomes as well. See Appendix C, which outlines the alignment between WDP’s Terms and Conditions to WD’s PAA.

In 2005, the department developed an electronic database to record and track project assessment and performance indicators, all linked to the new PAA. This database, known as Project Gateway, consolidated and standardized both project assessment indicators and performance measurement indicators into one electronic source.

Projects analyzed for this evaluation were at varying stages of completion. When entered into Project Gateway, each project is assigned a code reflecting the status of the project. As the project moves forward, these status codes are updated. Within the time frame of this evaluation, the majority of projects were completed (74.7%).

Table 2-2 illustrates the strategic outcomes for each project. The majority of projects undertaken were in community economic planning, development, and adjustment related activities (43.9%). Other activities included innovation (30%), and business and entrepreneur-related activities (26%).

Table 2-2 Project Activity under the PAA

Region	Percentage of Projects	Expenditures
Policy, Advocacy and Coordination	0.7%	$1,339,436
Community Economic Planning, Development, and Adjustment	43.2%	$150,049,931
Entrepreneurship	26.1%	$95,787,384
Innovation	30.0%	$133,079,729
Total	100%	$390,256,480

n=3,484. Reference: Administrative Data

Activities in the PAA may be further broken down into sub-activity levels. Outlined in Table 2-3 are the sub-activities of the projects examined in the evaluation.

Table 2-3 Project Sub-Activity and Expenditures

Activity	Expenditures	Number of Projects
Partnership and Coordination	$177,725	8
Collaboration and Coordination	$81,999	4
Research and Analysis	$898,021	10
Economic Research and Analysis	$181,691	4
Community Planning	$8,823,666	171
Community Development	$88,182,436	954
Community Economic Adjustment	$984,620	16
Community Economic Development	$62,059,209	376
Business Development and Entrepreneurship	$1,588,098	117
Entrepreneurship	$13,924,037	373
Improve Business Productivity	$34,596,792	164
Market/Trade Development	$39,088,902	152
Industry Collaboration	$2,489,612	74
Foreign Direct Investment	$506,139	9
Access to Capital	$3,593,804	14
Technology Adoption and Commercialization	$29,005,999	193
Technology Linkages	$9,734,674	69
Technology Research and Development	$46,220,193	157
Community Innovation	$1,717,969	24
Technology Skills Development	$2,588,389	18
Knowledge Infrastructure	$9,697,784	25
Innovation	$34,114,721	552
Total for All Activities	$390,256,480	3,484

Reference: Administrative Data

File Review

To determine if WDP funded projects have been successful in obtaining the intended project-specific outcomes, a review was conducted of a sample of 110 files. Of the 110 files reviewed, 86 projects had been completed and 6 projects had been discontinued or cancelled. The rest of the projects were in-progress. The projects selected were mainly from the core Western Diversification Program and some sub-components (Western Economic Partnership Agreements and Urban Development Agreements). Table 2-4 outlines the number of file reviews completed in each region.

Table 2-4 File Reviews Undertaken

Sub-Program	Number of File Reviews Completed
Alberta	26
Saskatchewan	21
British Columbia	47
Manitoba	16
Total	110

n = 110. Reference: File review

Researchers obtained pertinent information in each region. In order to analyze the information collected, a database was developed and used to enter the information. In the course of conducting the file review, consultations were also conducted with program officers and Internet searches done to track long-term project outcomes.

Key Informant Interviews

The evaluators developed a key informant database, which included names and contact information for potential respondents. The research team completed semi-structured interviews with key-informants from the department (management and staff), community leaders, representatives from economic development organizations or programs similar to the WDP, experts in the field of western economic development and diversification, and individuals from organizations besides the department that partnered with the recipient.

Community leaders included individuals such as mayors, individuals from regional chambers of commerce, a regional health authority. Representatives from economic development organizations were from Community Futures Development Corporations (CFDCs), regional economic development authorities, economic development ministries, FedNor, and ACOA. Experts included individuals from various universities in the West as well as economists. Other partners interviewed were from provincial governments, municipalities, universities, corporations/businesses, and various councils.

Outlined in Table 2-5 is the number of interviews completed, by stakeholder groups.

Table 2-5 Completed Key Informant Interviews

Stakeholder Group	Completed Interviews
WD Management	11
WD Staff	11
Community Leaders	7
Economic Development Program Representatives	11
Experts	8
British Columbia Partners	12
Alberta Partners	8
Saskatchewan Partners	5
Manitoba Partners	9
Total	82

Funding Recipient Survey

A survey of WDP funding recipients was undertaken to explore critical evaluation questions. For the recipient survey, the evaluators sampled projects based on program expenditure and program funding areas. Thus, projects from programs receiving greater funding had a high probability of being sampled, as did projects funded under the following sub-components, or delivery tools: Western Diversification Program, Western Economic Partnership Agreements, Urban Development Agreements, Saskatchewan Northern Development Agreement, and the Women’s Enterprise Initiative.

However, because the WDP database did not contain recipient contact information for every project, the number of possible respondents was reduced to 2,148. The first stage of surveying was based on quotas set for the aforementioned sub-programs; however, in order to maximize responses, these quotas were removed after two weeks of surveying, and all projects were then made available for surveying.

The recipient survey was conducted using a “mixed-mode” methodology (on-line surveying with telephone follow-up). The evaluators sent recipients an email inviting them to participate in the survey. Recipients with available email addresses were sent a personalized URL link in the body of the email invitation, which enabled them to access their personalized surveys by clicking on the link.

Following the initial invitation from the evaluators, recipients were contacted by telephone to invite them to participate in the survey. Several email reminders were also sent to respondents. The mixed-mode survey methodology helped overcome the challenges associated with the contact information.

The recipient survey field test was conducted from December 17, 2007 to December 27, 2007. Full surveying began January 23, 2008 and ended February 20, 2008. In total, 1,042 surveys were completed, with a valid sample of 1,783, and a valid response rate of 58%. The high number of completions and high valid response rate suggest that the data from the recipient survey is a very good representation of recipient views and/or outcomes.

Among the survey respondents, 44.8% had been involved in the project as the project leader/director, 45.3% had been involved as funding applicants and the remaining 9.9% had been involved in project delivery.

Case Studies

Case studies were completed with both WDP funding recipients and with individuals from projects that did not receive funding. The evaluators developed a database of potential case study sites. The database included both non-recipient and recipient sites. A total of nine case studies were completed with funded projects, with at least two visits conducted in each province (Table 2-6). A total of four case studies (one per region) were conducted with non-recipients, or projects that were not funded by the WDP. The following table outlines the projects included in the case studies:

Table 2-6 Projects Selected for Case Studies

Region	Recipient Projects	Non-Recipient Projects
British Columbia	University of British Columbia Centre for Drug Research and Development	Haida Gwaii Community Futures Development Corporation (is funded through the Community Futures Development Corporation)
Alberta	Inunctus	Calgary Zoo (due to policy change, was not eligible for funding)
	TR Labs/NEWT
	TEC Edmonton
Saskatchewan	Canadian Light Source, Inc.	Saskatoon Zoo Foundation
	Aboriginal Human Resource Council
	SpringBoard West Innovations, Inc.
Manitoba	SmartPark at University of Manitoba	United Way of Winnipeg
	MB Audio Recording Industry Association/ MB Motion Picture Industry Association

At each site visit, interviews were conducted with key program personnel, administrative data pertaining to project outcomes was collected, and, if relevant, a tour of the site was completed. Findings in the case studies were further augmented with consultation done with program officers. Case study reports with thorough information regarding the projects and the activities undertaken during the site visits are included in Appendix D.

Focus Groups

During the survey, respondents were asked if they were willing to participate in a follow-up focus group or if the project in which they had been involved had an available client database to support a project-specific focus group. Based on these responses, participants’ lists were developed to support three recipient focus groups and two project specific client (end-user) focus groups. Participation in the focus groups was very low as shown in Table 2-7.

Table 2-7 Focus Groups

Focus Group	Location	Scheduled Date	Number of Participants Agreeing	Number of Confirmed Participants	Number of Actual Participants
Client Focus Group	Winnipeg	February 21, 2008	8	5	3
Client Focus Group	Vancouver	February 21, 2008	8	2	1
Recipient Focus Group	Vancouver	February 25, 2008	13	9	2
Recipient Focus Group	Saskatoon	February 26, 2008	9	7	1
Recipient Focus Group	Calgary	February 27, 2008	6	6	1
Total			44	29	8

Because of the extremely low turnout, the focus groups findings were limited in the evaluation to being merely a complementary line of evidence.

Methodological Limitations

The evaluation had a number of methodological limitations. These limitations impacted on the ability of the report to adequately address the issue of cost-effectiveness and to some extent the longer-term success of the program.

Some difficulty was encountered in scheduling some of the key informant interviews. This issue was addressed by supplementing the stakeholder database with replacements if individuals were unable to participate and by extending the time frame for interview completion in order to capture as many completed interviews as possible.

Due to the large size of the WDP and all its sub-components, the methodology for the evaluation did not include a detailed evaluation of each WDP sub-component. Where appropriate, results from sub-components in the evaluation and from other research studies were analysed and included in the report.

Another limitation was incomplete or incorrect contact information available for project funding recipients who were selected to complete the Recipient Survey. In order to mitigate this issue, the evaluators worked with potential respondents to secure correct contact information, to access forwarding information, or to locate replacement respondent information (in the case of incorrect or out-dated information). Individuals were also contacted, either via telephone or email, in order to ensure that as many potential respondents as possible were given the opportunity to complete the survey.

Finally, estimating the net impacts of a program ideally requires that cross-sectional comparisons be made between samples of recipients and non-recipients from similar populations to ensure that impacts can be attributed to the intervention (WDP funding) and not to differences between the groups. The WDP database did not include contact information for non-funded applicants for the purposes of a survey. This is because the department does not go out and publicly solicit proposals similar to what is done with other funding programs. Sometimes, organizations that approach the department for funding may be deemed ineligible. In those cases, departmental project officers will often redirect those organizations to more appropriate funding organizations. The department generally does not keep a written record of those organizations, as doing so is not seen to be cost-effective or relevant to the overall management of the WDP and achievement of its objectives.

However, because it was critical to the evaluation to examine non-funded projects, a small database of non-recipients was developed. Based on this list, case studies of these projects were undertaken. The non-recipient case studies allowed for some comparison and understanding of the impacts, or lack of impacts, for those projects that did not receive WDP funding.