Evaluation Methodology

Several research tasks have been completed to collect data providing multiple lines of evidence to address each of the evaluation issues and questions outlined in the Evaluation Framework (Appendix 1).  Table 3.1 provides a summary of the methodology used.

Table 3.1  NDA Evaluation Methodology

Document Review File Review Focus Group with NDBC (December 1, 2006) Key Informant Interviews (18) Successful Project Applicant Interviews (20) Unsuccessful Project Applicant Interviews (8) Training Program Participant Interviews (19) Delivery of Preliminary Findings/Feedback (NDBC, Feb, 2007, TMC, March, 2007) Preparation of Evaluation Report

3.1 Document Review

A number of relevant documents were reviewed in order to gain a full and complete understanding of NDA goals, objectives, and expected results, and to collect information relevant to the evaluation framework questions.

A sample of documents reviewed include:

• The Canada-Saskatchewan Northern Development Agreement;
• NDA Results-based Management and Accountability Framework;
• The Northern Development Accord;
• Northern Development Board Corp, 5-Year Strategic Plan;
• The Western Diversification Program Terms and Conditions;
• The Northern Saskatchewan Economic Development Act;
• NDA Communication plans; and
• Meeting minutes from the TMC, NDBC, and TMC Sub-committee.

3.2 File Review

Project files were reviewed in order to gather objective and quantitative data related to both approved and denied projects.  An initial review of twenty-five (25) project files was conducted at the Saskatchewan Northern Affairs office in La Ronge.  A second review of an additional fifteen (15) files was conducted at the Western Economic Diversification (WD) office in Saskatoon.  A project file review checklist was used during the review.

3.3 Focus Group with NDBC

An in-person focus group was conducted with members of the NDBC.  The focus group was convened on December 1st, 2006, in Prince Albert, Saskatchewan.  The focus group provided an opportunity to seek input from the NDBC and to explore programming and administrative issues in more detail.  (The results of the focus group discussion are attached as Appendix 2.)

3.4 Key Informant Interviews

Key Informant interviews were conducted with federal and provincial government officials and with community leaders representing northern Saskatchewan.  A total of eighteen (18) Key Informants were interviewed during the course of the evaluation.  (Appendix 3 provides a list of Key Informants, Appendix 4 includes a copy of the Key Informant interview guide.

3.5 Successful Project Applicant Interviews

Individuals who received project funding were interviewed to determine the success of their projects and the impact that these projects had on northern communities.  Interviews also focused on issues relevant to client satisfaction, the application process and program administration. Twenty-five (25) organizations composed the sample, with some interviewees representing organizations that had received funding for multiple projects (i.e. Northlands College).  A total of twenty (20) successful applicant interviews were completed.  (Appendix 3 -  successful project proponents interviewed, Appendix 4 - interview guide, Appendix 6 -summary of interview results.)

3.6 Unsuccessful Project Applicant Interviews

Interviews were also conducted with organizations whose projects were not funded by the NDA.  These interviews are important to the evaluation as they help to determine whether projects were able to move forward without NDA funding, whether funding was secured from other sources, or whether unfunded projects were abandoned.  A total of eight (8) interviews were conducted with unsuccessful project proponents.  (Appendix 3 - list of unsuccessful project proponents interviewed, Appendix 4 - interview guide, Appendix 7 - summary of interview results.)

3.7 Training Program Participant Interviews

Several NDA funded projects were designed to provide training and related employment services to Northern residents. While conducting interviews with successful project proponents, a list of fifty-four (54) training program participants was collected.  Of that group, twenty-six (26) had either moved or contact information was no longer available.  Nineteen (19) were interviewed to determine the value of the training, employment status, satisfaction levels, etc.