Evaluation Issues and Related Findings

In 2004, a Results-Based Management and Accountability Framework was prepared for the Northern Development Agreement.^[1]  As part of the Framework, evaluation questions were posed within each of four key evaluation issues.  The following chart highlights the four main evaluation issues and the types of evaluation questions that are typically posed during a final program evaluation study.

The NDA Evaluation Framework outlines a set of evaluation questions that have been organized under each of the four key evaluation issues; Relevance, Success/Impact; Cost Effectiveness; and Program Design.  The evaluation questions are posed in order to explore a range of program design, implementation and operational factors.  Data collected during key informant interviews, applicant surveys, file reviews and document/literature review have been analyzed to prepare a response for each of the evaluation questions found in the framework.  The questions and responses are summarized in Section 5.0 of this report.

5.1 Program Relevance

Is there a documented and identified need for the NDA in northern Saskatchewan?

The NDA was initiated in response to the vision, principles and goals of the Saskatchewan Northern Development Accord which was signed in 2002.  The Accord documented the federal and provincial governments" support for economic development as a tool to advance northern communities.

 

As was the case in 2002, the economy of the North continues to depend on economic development as a tool to grow and advance northern communities.  Current analysis of population trends and employment statistics indicate that there will be an ongoing and continued need for economic development programs in northern Saskatchewan.  The population is growing (5.8% growth between 1998 and 2005) with nearly half of the northern population under the age of 20, and unemployment at approximately 25%.   These demographics alone highlight the need for economic development programming to support current and future education, employment and community development needs.

NDA evaluation results support the rationale for future economic development programming.  Interviews conducted with representatives of northern communities indicate that there is an on-going desire to generate economic growth, community development and employment opportunities through targeted business development and key sector growth.  Both federal and provincial governments support the hypothesis that future prosperity in northern communities will be generated through business and sector specific development.  All key informant interview respondents identified the need for future programming and identified that future economic development potential exists in mining, oil and gas, forestry, tourism and in service-related business development.  To support future growth in the region, there will be an on-going demand for human resource capacity building, leadership development, skills training and entrepreneurship training.  For those with the skills and capacity to generate new business development, there will be a need to support business planning, market assessment, feasibility analysis and capital investment.

When asked whether there is a need for an NDA in the north, the majority of project applicants (88%) reported that the NDA has provided an important tool for northern economic development.  Without NDA funding, 68% of the projects would not have gone forward.  The majority of the applicants could not access funding through other sources to achieve their goals and objectives.

Have consultations been held between Federal, Provincial and Northern Development Board members, resulting in recommendations for programming and project selection? 

The NDA implemented a governance strategy designed to ensure representation from federal and provincial governments as well as from northern communities.  The governance model used to manage the NDA is unique to the program and represents a significant development in the involvement of Northerners in government funding and economic development programs.

At the onset of the NDA, the Northern Development Board Corporation (NDBC) was identified as an appropriate group to represent Northerners.  The NDBC is a twelve (12) member Board with Tribal Council, Métis Nation and northern association representation. As members of the NDA management team, the NDBC was identified as the first point of contact for applicants and provided communication and initial project screening services.

The role of the NDBC, specific to the NDA, has been “to provide input in selecting projects undertaken under the Agreement”. The NDBC participates on a Tripartite Management Committee (TMC) consisting of two federal, two provincial and two NDBC members.  The TMC provides final review and approval of all projects.

The responsibility to consult with northern communities, in order to ensure equitable representation and to bring forward northern concerns and development issues is retained by the NDBC.  To support the NDBC"s efforts to consult with their stakeholders, both federal and provincial governments have provided funding for Strategic Planning and Consultation.  In August of 2003, the NDBC prepared a five-year Strategic Plan to assist in providing direction for the NDA.  The 2003 plan outlines five broad strategic priority areas and identifies initiatives and timelines to address each:.

1. Develop the capacity of Northerners to be self-sufficient.
2. Establish and strengthen northern socio-economic infrastructure.
3. Promote northern business ventures and opportunities.
4. Participate in natural resource development.
5. Implement the Northern Development Accord.

The strategic priorities identified in the NDBC Plan are broad in scope, providing general guidance for project selection.  However, these priority statements do not focus on specific, targeted sector, community or economic development.  There is no evidence that the TMC used the Strategic Plan to inform their NDA decision-making and there is no evidence that the NDBC used the Strategic Plan to proactively interact with communities to encourage targeted proposal preparation which would respond to their strategy.  Although a consultation process was put into place, the value of the consultation was limited.

Throughout the implementation of the NDA, the NDBC has contributed as an advisory and administrative body.  The Board meets regularly to review proposals and to provide project selection recommendations.

The NDBC reports some frustration with their Advisory Board role.  Although they provide input and make recommendations, they feel that the final decision for project selection rests primarily with the federal and provincial governments, who have the financial capacity to support the projects.  Many NDBC Board members describe their role as a “rubber-stamp”.  Although the NDBC has influence over project selection by saying “no” to projects that they do not want to support, they do not feel that they have the capacity to say ”yes” to projects that they want.   The NDBC reports that governments have a greater degree of influence because they have the financial resources to say “yes”.

Are clearly defined Goals and Objectives developed? 

The purpose, goals and objectives for the NDA are documented in the Northern Development Agreement.^[3]   A review of the Agreement suggests that the designers of the NDA implemented a broad definition of economic development, so as to include a wide array of activities categorized within training, capacity building, employment generation, infrastructure development, and investment.  The following five funding priority areas are identified in the Agreement.

• Economic Infrastructure;
• Innovation
• Realizing Employment Opportunity
• Capacity Building; and
• Investment Attraction

The priorities are broad in scope, allowing for a wide range of projects and project applicants to participate in the program.  In reality, the scope and parameters of the NDA were significantly narrowed when governments applied their own criterion to project selection.  For example; W.D. used the Western Diversification Program (WDP) as their source for project funding.  The WDP is restricted to funding non-profit organizations.^[4]  The provincial government sourced their $10 million share of the NDA from seven different provincial departments. Each of these departments had their own parameters and funding criteria.  Upon reviewing the NDA Purpose and Objectives, it is clear that the original intent of the Agreement was to encourage small business development, private-sector growth and new ventures.  These objectives could not be effectively met when the funding constraints of government departments were applied to the Agreement.

 

Are NDA activities and funded projects consistent with the Strategic Priorities?

A comprehensive file review confirms that each project funded within NDA fits within one or more of the Strategic Priority Areas.  Table 5.1 provides a break down of the number of projects and the total funding allocated to each strategic priority as of December 8, 2006.  (The categorization of the projects was done by the TMC Sub-Committee.).

Strategic areas are broad in scope.  This allows for a wide range of projects to be included in the NDA and makes it possible to utilize a comprehensive definition of economic development which includes training and employment as well as small business development and investment.

A significant number of NDA projects (32/42, 76%) focused on capacity building, training and employment.  This represents 63% of the total funds allocated to date.   Although the projects fit within the Strategic Priorities of the NDA, it could be argued that the areas of Capacity Building and Employment Opportunity were over-weighted.

The NDA was designed to encourage small and medium-sized business development, to promote value-adding, research and development and marketing options, and to provide increased opportunities for the private sector.  The priority areas of Innovation and Investment Attraction have utilized less than 1% of the overall NDA fund. Key informants and program administrators report that it was difficult to attract projects in the Innovation and Investment Attraction Priority Areas when for-profit groups and private-sector applicants were not able to apply.

Are the Strategic Priority Areas Appropriate?

Overall, evaluation findings suggest that the priority areas of the NDA are appropriate.  When interviewees were asked if the priority areas should be changed and/or expanded, over 90% indicated that the current priority areas are appropriate.  However, many key informant interviewees identified that the broad scope of the Priority Areas has not resulted in targeted, "strategic" investment.  Rather, it has provided the NDA with an opportunity to fund a wide variety of projects.

Key informants recognize that the investment in capacity, training and employment is important to northern communities.  However, given the benefit of hindsight, and the realities of the projects proposals that have been funded by the NDA in the past, there is support for a more proactive, targeted approach to project funding in future.  In order to ensure a significant impact, within a particular sector of the northern economy, it may be appropriate to consider larger investments in mining, forestry, tourism and small business attraction.

Is there sufficient demand for the program?

The NDA is a $20 million fund, distributed to eligible applicants over five years.  Approximately $700,000 of the fund has been allocated to the NDBC to cover administrative costs.  To date, project commitments total $13,592,115 over forty-two (42) projects.  Just over $6 million remains to be allocated.  Project applications were greatest in the first two years of the program with declining numbers of applications ever year thereafter.

The fact that the financial resources have not been fully utilized over the life of the NDA could be seen as an indicator that there is not sufficient demand for the program.  However, key informants report that it could also be argued that demand for the program was muted due to the funding criteria applied to project selection and that funding requests declined because applicants became disillusioned with the amount of time that it took to get funding decisions.  Administrators also report that over the life of the NDA, applicants became more familiar with the types of projects that fit within the mandate of the NDA.  As a result, there were fewer applications submitted in later years, and there were fewer that did not meet the requirements and criteria for funding. Broader funding criteria may have resulted in a larger number of applications.

There is little evidence to indicate that NDBC members actively sought out project proposals to address their own Strategic Plan, or that addressed the funding criteria of the NDA.  A more proactive approach to solicitation of project proposals may be warranted.

Do NDA investments fit into federal, provincial and NDBC plans for northern Saskatchewan?

The signing of the Northern Development Accord, and in turn the design and implementation of the NDA, demonstrate that federal and provincial governments and Northerners support economic development as an important tool for the future of northern Saskatchewan.  The investments made through the NDA contribute to the economic growth of the north.

---

^[1] Results-Based Management and Accountability Framework,  Northern Development Agreement – Saskatchewan, Garven & Associates, September, 30, 2004  (Evaluation Matrix included as Appendix 1 to this document.

^[2] Saskatchewan Northern Development Accord between The Minister of Public Works and Government Services and Federal Interlocutor for Métis and Non-Status Indians (R. Goodale) and The Minister of Northern Affairs (B. Belanger), 2002

^[3] Canada-Saskatchewan Northern Development Agreement, October 17, 2002, between the Government of Canada and the Government of the Province of Saskatchewan.

^[4] Western Diversification Program, Terms and Conditions, Annex "A"