Executive Summary

This report presents the findings of an evaluation of the Community Futures (CF) program in Western Canada. An evaluation of the CF Program is required as part of the Department's responsibilities under the Treasury Board of Canada Evaluation Policy and the Federal Accountability Act.

The evaluation was conducted between November 2007 and August 2008. Findings from this evaluation will be combined with those from CF evaluations concurrently conducted in the other Regional Development Agencies (RDAs) in Atlantic Provinces; Quebec; and Ontario to produce a roll up national report on the evaluation of the national CF program.

The aim of Western Economic Diversification Canada's (WD) CF program is to:

• support local rural communities and small and medium-sized enterprises (SMEs) in meeting their economic needs;
• help rural communities to develop and implement long-term community strategic plans leading to the sustainable development of their local economies; and
• provide resources to local CF organizations to build community capacity to adapt to and manage change.

In Western Canada, the CF program is delivered through a network of 90 non-profit CF organizations (34 in British Columbia, 27 in Alberta, 13 in Saskatchewan and 16 in Manitoba) that are supported by four CF associations (one per province) and a Pan-West CF Network.

The CF program has received a total of $158M in operating costs over the last 6 years to support CF operations, the CF Associations and WD administration of the program.

Operating Costs of the CF Program (000s)

Fiscal Year	CFs	Associations	WD Admin	Total
2002-2003	$21,303	$840	$1,827	$23,970
2003-2004	$20,435	$840	$1,827	$23,102
2004-2005	$23,390	$840	$1,827	$26,057
2005-2006	$24,711	$850	$1,840	$27,401
2006-2007	$25,202	$1,227	$1,840	$28,269
2007-2008	$26,313	$1,244	$1,840	$29,397

The evaluation methodology integrates the use of multiple lines of evidence and complementary quantitative and qualitative research methods as a means to ensure the reliability of results being reported, and the validity of information and data collected. The research methods included: 1) document and literature review; 2) review and analysis of WD's administrative data; 3) interviews with WD senior managers, program managers, and a sample of managers and board members in the CFs, provincial association representatives and external stakeholders; 4) survey of representatives from all CFs; 5) survey of CF clients, including those who applied for and did not receive loans; 6) case studies; and 7) review and analysis of Statistics Canada labour market data for rural communities in the West.

The evaluation focused on four main areas: program relevance, design and delivery, program impact, and cost-effectiveness.

Program Relevance

The overall conclusions for the Relevance section are that:

• there is a continued need for the CF Program;
• there is a high degree of complementarity between the CF program and other programs; and
• the objectives and mandate of the Program remain relevant to Government responsibilities and priorities.

It was found that no organizations fully duplicate or overlap the CF Program.  However, numerous complementary services are provided by various organizations.  CF Organizations (CFs) are more aware of these organizations and the programs and services they provide than CF's clients are. The level and type of collaboration with these organizations varies from one CF to the other. All three core services of the CF Program services are needed under both strong and weak economic conditions.

The local CFs' objectives and activities were found to be consistent with the National CF Program.  However, the emphasis placed on each of the core service areas differ from one CF to another. The local CF objectives and activities are generally aligned with WD's objectives.  However, CF program managers will need to be mindful of a recent shift in departmental priorities and ensure that, going forward, the CF Program remains in alignment with these priorities.

There were two main recommendations related to the relevance section:

• WD should introduce one or two specific indicators to capture CFs' role in pursuing collaborative relationships with other organizations (i.e. informal arrangements, memorandums of understanding, etc.) and acting as a gateway to services for their clients (e.g. number of referrals).
• WD Program Managers need to review the recent shift in WD priorities to determine how the CF Program should respond in the future.

Program Design and Delivery

The overall conclusions for the Design and Delivery section are:

• the CF networks are working effectively primarily at the provincial and sub-provincial levels;
• overall, the investment fund is well managed but clarity around acceptable loan loss rates and cash on hand is needed;
• performance data is being collected and used but there is some room for improvement; and
• there is no consistent plan in place to increase CF awareness but CFs seem to be engaged in a variety of awareness building activities.

The value of investment fund has increased at a yearly average of 1.83%. The overall increase of cash on hand was strongly and significantly associated with the strong economic conditions in the western provinces in the last six years (2002/03 to 2007/08).  Seventy percent of the observed decline in the number of loans issued was explained by variations in credit market conditions (i.e., easing of business loan conditions by traditional commercial lenders). Interviewees demonstrated inconsistencies in their interpretation of “appropriate loss rate”, with some interviewees stating that a good rate is between 10-15%, recognizing that the CFs are high-risk lenders, while others state that the loss rate on loans should not surpass 4%.

The client survey, some case study participants and some interviewees expressed a need for a higher loan limit in certain instances. The higher loan limits would be consistent with WD's focus on supporting larger projects. CF terms and conditions allow for some flexibility around the $150K limit but it is not evident under which instances the limit can be surpassed.

The CFs are gathering performance data for evaluation and measurement purposes.  Performance data is taken into consideration by the CFs and by WD during the decision-making process.  However, the accuracy and integrity of data currently available in the administrative databases is questionable. A training program is currently underway that should resolve some of the issues, however further work will be required.

There were 2 main recommendations related to the design and delivery section:

• WD should take the necessary steps to further enhance the management of investment fund.
  • WD should determines acceptable performance measures such as loss rates, number of loans, and levels of cash on hand, taking into account local, regional, and provincial economic conditions. This decision should be clearly and consistently communicated to all CFs.
  • WD should clarify the CF terms and conditions surrounding when the $150K loan limit can be surpassed. The decision should be clearly communicated to all CFs.
• WD should take the necessary steps to enhance the reliability and use of performance data.
• • WD should devise a strategy to resolve performance-reporting issues related to system inconsistencies and data definitions. 
  • WD should review the continued relevance of elements currently collected through the TEA system.
  • In order to ensure the integrity of data, WD should implement and lead quality assurance practices (i.e. sample audits, etc.).

Program Impact

The overall conclusions of the program impact section are:

• the CFs are providing much needed information, referral and counselling to clients;
• clients are highly satisfied with the service and are applying what they learn from the CFs to their businesses;
• the CFs continue to provide highly needed loans to support both new and existing businesses;
• the CF program is serving clients in their language of choice;
• there is strong expressed need for community strategic planning and CED related activities;
• the CFs' level of involvement in CED varies from one CF to the next, highlighting one of the strengths of the program in terms of flexibility and response to community needs;
• there is evidence that CF activities support economic growth and diversification of communities and some evidence that CFs are strengthening community capacity; and
• the CFs' activities have moderately decreased in recent years due in a significant part to the economic conditions in Western Canada, on one hand; and to a variety of other factors inherent to the program on the other hand.

Clients are very satisfied with the services that they receive from CFs and they express a high level of need for all services. The CF program has issued over 8,500 loans from 2001/02 to 2007/08 worth more than $328M, and which have resulted in the creation of nearly 28,000 jobs. Business services have led to the creation of a further 27,000 jobs over this same time period. There is evidence that clients are applying what they learn from the CFs to their businesses. CF-supported businesses are active in a wide variety of sectors suggesting that diversification is occurring. Diversification is further assisted by the 1-5 CED projects that CFs are supporting and by CF involvement in the strategic planning decisions occurring in communities. Over the period 1999/00 to 2005/06, CFs have led or coordinated 4,902 long-term and 5,803 short-term community economic development projects.

Long-term census data specific to each of the 90 CFs suggest that economic diversification has occurred, and CFs have contributed to reducing unemployment rate in the communities. It is reasonable to believe that the loans, advice and support provided to new and existing businesses is increasing their survival rate; however, the administrative data is lacking to corroborate this expectation.

There were two recommendations related to the impact section:

• WD needs to further define the indicators associated with the long-term outcomes of the program and ensure that data is being tracked against the established indicators.
• Subject to WD preparing a plan to implement the other recommendations in this report, WD should continue its support and long term commitment to the three services provided by the CF program.

Cost-Effectiveness

The overall conclusion is that the CF Program is still cost-effective when the level of activity and return on investment (ROI) are compared to costs.

The extensive use of volunteers contributes to the overall program cost effectiveness. Over the period 1999/00 to 2007/08, each CF was able to generate an annual average of 268,983 volunteer hours.

We found that conservative estimates, have CF loan clients generating anywhere from $6 to $11 of gross revenue for every CF loan dollar issued.  Loan clients are leveraging an additional 40% to 60% of additional funding. When cost and level of activity are observed over time, it is apparent that operating costs have been on the rise over the last few years and the level of activity has been declining. The decline has been gradual (e.g. average number of loans per CF has been decreasing by less than one (0.74%) per year over the last 6 years). Minimum performance measures have been put in place and this combined with a conversation on acceptable levels of risk should help to reverse the trends.

There was one recommendation related to the cost-effectiveness section:

• WD should continue to monitor the trends in level of activity relative to cost to ensure that the measures put in place are having the desired affect and reversing these trends.

Independent observers continue to strongly support the delivery model, calling it an outstanding/unequivocal success, innovative, and one of the few signs of the Federal Government in rural Canada. There is no major appetite to change the program, rather comments relate more to revitalizing the CFs and the program. Those involved with the Program offered suggestions on ways to further improve it. Some suggested improvements and lessons learned include

Improvements, as suggested by a variety of clients and stakeholders include:

• Review the number of CFs and/or the geographical divisions of the program
• Potential for the creation of satellite offices that would be supported by a central office that houses experts
• Ensure that there is more consistency in the way services are delivered by the various CFs

Lessons Learned, as indicated by CF and WD staff include:

• The CFs should be aware of their resources limitations, not over-commit, and be able to say “no” when necessary.
• CFs should document and clarify the roles and responsibilities before engaging in any partnership.
• WD and CF should keep communication channels open and spend time to build long-lasting partnership based on trust and clarity of roles and responsibilities.