Western Economic Diversification Canada
Symbol of the Government of Canada

Conclusions and Recommendations Regarding Data Collection and Performance Reports

Our mandate required us to conduct an analysis of the data collection methods and performance reports submitted by the FEDOs to WD. WD then uses this information to compile its reports to Canadians.

In this section, we present the conclusions and recommendations regarding data collection and performance reports for the Community Planning, Community Development and Improvement of Business Productivity programs.

Conclusions and recommendations regarding data collection and reporting: Community Planning and Community Development

Conclusions

All the FEDOs produce detailed documentation on their work in these two programs. The documentation takes the form of applications for funding addressed to WD or other funders, correspondence and memorandums of understanding with partners who file their own funding applications, reports submitted to the FEDO boards of directors and public reports.

The quantitative dimension is relatively easy to capture. The FEDOs document the names of partners, number of activities implemented, number of projects successfully completed or under way, and so forth.

In terms of objective evidence, the only criticism heard at certain focus groups was from a few people who said that the FEDOs sometimes "take the credit" for projects that are not theirs. The discussions with these people allowed us to understand their points of view. They claim that the FEDOs should report to WD only the projects they initiated and managed, not all the projects in which they participate.

The indicator definitions reflect an important reality: the FEDOs rarely initiate projects by themselves. The majority of the projects mentioned by the focus groups and examined in the documentation review were initiated by local individuals or groups. Once communication is received from the local community, a FEDO officer makes an on-site visit and the process begins.

This is why the indicator definitions read: number of projects or activities or service improvements in which the FEDO participated or which it facilitated or organized. In that sense, the indicators measure the sphere of influence of the FEDOs. This is confirmed by the comments made by certain focus groups to the effect that the FEDO's presence in their file adds to their initiative's credibility, allowing them greater access to expertise and resources. For these people, the FEDO's participation in their project is important. Therefore, the indicator is valid and the FEDOs are reporting the data correctly.

The quality and richness of the documentation related to community planning activities vary from one FEDO to another. In our view, this documentation is produced in Alberta and Manitoba from a mainly commercial perspective, presenting business analyses. In Saskatchewan and British Columbia, the descriptions are prepared mainly from a community perspective, and so are more detailed and elaborate. In all cases, the information is complete and allows one to easily determine the work done by the FEDOs in terms of community development.

Lastly, the conclusions that we are obliged to draw under our mandate may include a comparison of impacts among the four regions. We have noted that the four FEDOs utilize identical community planning approaches. The realities in the field are such that the outcomes achieved will vary from province to province, even if each of the FEDOs has the same base budget.

The study cannot compare community-planning impacts among the four regions for the following reasons:

The majority of the Francophone population in Manitoba is concentrated within a radius of about two hours by road from Winnipeg. As a result, it is easy to reach all the communities, which are concentrated in the bilingual municipalities, with a team located in Winnipeg.

The Francophone population in Saskatchewan is the smallest of the four Western provinces, the most rural and the most dispersed. When the focus groups were held for three days in Saskatchewan in May 2008, we travelled 1,300 kilometres in a rental car to contact 26 persons in three focus groups. In addition, the people in Willow Bunch and Ponteix agreed to travel an hour by road so that the focus group could be held in a central location, Limerick: this avoided the need for us to travel a few hundred additional kilometres. (The focus group in Regina was held in June.) Under these circumstances, it must be understood that an organization with a fixed amount of financial resources will obtain relatively fewer results, since travel and office rental expenses in the regions will demand more resources than in Manitoba, and the population is not as large.

The Francophone population in Alberta is dispersed all over the province. For example, it takes five hours to travel from Edmonton to Falher, and three hours to travel to Bonnyville. As in Saskatchewan, a significant share of the fixed financial resources is used for the travel and relocation of the work teams.

The situation is similar in British Columbia. Comox is four hours and Nelson seven hours by car from Vancouver. For all practical purposes, one has to travel by air, something that substantially increases expenses. The people in these two regions were critical of the fact that the FEDO has no local personnel.

Given these circumstances, the study cannot determine whether a given FEDO is posting better results than another, as the elements of comparison are so different.

Recommendation

The results obtained through the implementation of these two programs are necessarily qualitative in nature, since they deal with community planning and community development.

The indicators used since 2006 give a good profile of the FEDOs' outputs—number of partnerships established, increased capacity of community organizations, etc.

While useful, these indicators do not give an overview of the quantitative results as noted by the focus groups themselves. The most important of these is the value of capital investments generated in projects that have received major FEDO support and the jobs created as a result.

As examples, we identified three community projects during the focus groups that, on their own, generated total investment of over $10 million in capital property. In these three cases, the project promoters said that the FEDO's support was the key determinant of success.

Each of these three centres is generating new permanent jobs. The impact of eight or ten permanent jobs created in a small village is very significant. And these projects generate temporary jobs as well. For example, construction of a $2-million centre can generate 30 jobs over a three-month period.

These are significant quantitative impacts that do not appear in the present reports.

Recommendation 1: that the FEDOs and WD develop instruments to capture the value of incremental investments, temporary jobs and permanent jobs generated by community capital projects that have received substantial FEDO support.

To avoid duplication, it would be necessary to clearly differentiate projects set up under these two programs from projects entrepreneurs implement under the Improvement of Business Productivity program.

Return to the top of this pagetop of page

Conclusions and recommendations regarding data collection and reporting: Improvement of Business Productivity and Entrepreneurship program

The production of this part of the study posed a number of challenges in terms of collection and validation of the data provided to WD by the FEDOs.

We conducted the entrepreneur survey and analyzed the documentation provided by the FEDOs in May and June 2008, as planned.

Next we analyzed the data provided by the FEDOs to WD. We very soon noted some major information discrepancies in certain line items between our observations in the field during the focus groups, the survey results, the FEDO documentation and the results reported to WD.

With the support of WD officers and FEDO senior managers, we worked diligently through July, August, September and October to correct certain data that was clearly wrong, and to better understand the reason for the discrepancies.

Certain data seemed disproportionate to the reality in the field for the years 2003 to 2006. This is not obvious in analyzing the data of each FEDO individually, or in analyzing only the cumulative data for the four FEDOs.

It becomes obvious only after analyzing the data of each FEDO and the cumulative total for each line item.

Here are four examples.

The FEDOs provided figures for the line item Number of businesses receiving assistance with planning, cash flow, market information, etc. between 2003 and 2006. The cumulative totals reported are 880, 958 and 1,047 for these three years. In principle, these totals do not pose a problem.

It is only after analyzing the data from each FEDO that it becomes clear that these numbers are meaningless.

In 2004-2005, Saskatchewan reported the number 257 under this line item. The same year, Manitoba reported 139.

Anyone with any knowledge of the realities in the field could legitimately question the accuracy of this data, since the Francophone population in Saskatchewan is far smaller than the Francophone population in Manitoba. As a result, this sort of number can affect the credibility of the FEDO in its community.

The fact is, Saskatchewan reported its data properly in conformity with the definition, which is the number of times a service was provided. The same business could be counted 20 times, for example, if 20 interactions were needed to complete a planning process. Manitoba provided only the number of businesses served, not the number of interactions with each business. In the case of Saskatchewan, subsequent discussions with FEDO employees revealed that, on average, they had 10 interactions with each business. Therefore, they would have worked with around 25 businesses during the year. If the figure in this line item reflected the number of businesses, i.e. 25 in Saskatchewan and 139 in Manitoba, it would be correct. If the figure reflected the number of interactions in both cases, i.e. 257 in Saskatchewan and 1,390 in Manitoba (assuming 10 interactions per business), it would be correct.

When the two are combined, the figure becomes meaningless: it does not reflect the impact of the FEDOs and it cannot be used.

Here is the second example: The FEDOs and WD became aware of these problems and changed the system in 2006, creating the line item Number of clients served.

The cumulative total reported by the four FEDOs in 2007-2008 is 2,593, broken down as follows among the four provinces:

  • Manitoba: 172
  • Saskatchewan: 375
  • Alberta: 899
  • British Columbia: 1,147

To repeat the definition of this line item: Total number of identifiable clients served during the period covered by the report. Includes people who received a loan, assistance to the self-employed, consulting services and requests for information. Each client is counted only once, regardless of the number of services the client received during the period in question. Anonymous clients are not included, to avoid double counting.

Once again, we know that Manitoba works with over 172 entrepreneur-clients in one year. Here it is reporting only new clients served. As required under the definition, the other provinces are reporting all the clients who walk through their doors and request some service beyond trivial one-time information, including former clients who are returning and seeking service.

Since the FEDOs do not all use the same methodology, the cumulative total in this line item is also meaningless.

Here is the third example under the item Requests for information. We repeat here the definition: May include requests for "general information" and requests for "core information":

  1. General information: number of general information services (on the economy, business community, etc.) provided to clients. Includes individuals and businesses, as well as inquiries by telephone, in person, by email or at group sessions (multiply by number of participants). Count only services affecting the FEDO's main activities (e.g. do not include questions such as "where is town hall," etc.).
  2. Core information: number of consulting services linked to specific skills that are provided to an entrepreneur or a business on an individual basis. Count each interaction with a client as a single service, regardless of the length of the interaction or number of different individual services provided during the interaction.

The total reported by the four FEDOs under this line item for 2007-2008 was 15,039 information requests. The breakdown by province is as follows:

  • Manitoba: 676
  • Saskatchewan: 517
  • Alberta: 5,038
  • British Columbia: 8,808

Once again, we see that the huge disparity in the results reported indicates a different understanding of the definition of this line item and the cumulative total reported for the four FEDOs. Assuming 220 working days per year, Manitoba would issue 3.07 items of information per day and British Columbia 40.04. Since the FEDOs do not all have the same understanding of this line item, the total cannot be used to demonstrate the impact of the four FEDOs. Incidentally, British Columbia reported the number 0 under the same line item for 2006-2007. Clearly there are major divergences in understanding here.

Here is the fourth example. This is for line item Investments generated through financial leverage thanks to the business assistance ($). The definition is: Investments generated through the financial leverage of the FEDO's clients, resulting in loans granted by another financial institution or by the entrepreneur, thanks to the business assistance. Sources may include financial institutions, other governments, other sources (e.g. venture capital), and equity.

The CDEM in Manitoba and the CDEA in Alberta report these data, but use two different ways of calculating them.

Return to the top of this pagetop of page

The CDEM reports only incremental investments following consulting services offered to entrepreneurs; therefore, these are loans or grants obtained by entrepreneurs, as well as investments made by the entrepreneurs themselves. This is what is required according to the definition. The total reported is $36,265,559. Our detailed analyses for this study identified the sum of $38,939,515. We were unable to determine the cause of this discrepancy. The survey confirms the CDEM's impact in this regard. A total of 51 of the 58 respondents confirmed that they had obtained public or private financing thanks to the CDEM's assistance.

The CDEA reports all financial leverage from all sources, including fixed assets of community projects obtained as a result of its intervention. For example, it reports the amount of $8,867,913 for 2006-2007. This includes the share of capital grants of a community centre obtained as a result of the organization's intervention, revenues from trade fairs, and other leverage.

We also noted that two FEDOs, the CCS in Saskatchewan and the SDECB in British Columbia, do not report incremental investments resulting from their work. They do not ask this question of the entrepreneurs receiving their services. We confirmed that one officer in Saskatchewan generated leverage of $12,254,529, as defined under this line item, over the five years examined. Eleven survey respondents in Saskatchewan, representing 44% of the total number of survey respondents in the province, provided the financial leverage generated by the CCS. In more than one case, the amount was in excess of $100,000. These are major impacts that are not captured by the organization and are therefore under-reported.

Once again, adding the data provided under this line item gives a cumulative figure that is meaningless and does not demonstrate the real impact of the Western FEDOs.

Notwithstanding the problems with these line items, the impact study concludes that the data reported on the number of businesses created or improved are valid. The total reported by the four FEDOs for the five years is 716. We saw the names of 649 businesses that received services over the last five years. These names did not constitute the entire FEDO data bank, but they made up the core of the data bank for carrying out the survey. All the respondents completing the survey confirmed that they had, in fact, received a service from the FEDO.

The impact study also concludes that the data reported concerning the number of jobs created or improved are valid as well. The total reported for the period of 2003 to 2008 is 1,610, broken down as follows among the provinces:

  • Manitoba: 940
  • Saskatchewan: 127
  • Alberta: 140
  • British Columbia: 403

The result for Manitoba, which created a simple template that allows for the collection of reliable data, makes up 58% of this total. We saw the names of the businesses that created or maintained these jobs. The survey reached exactly 20% of all businesses that received assistance; these entrepreneurs report an increase of 203 jobs. Assuming that the survey results can be extrapolated to all the businesses receiving assistance, the total number of jobs created would be 1,015 (203 x 5). Manitoba reports 940, which is a close approximation. The analysis of the CDEM data bank confirmed the creation or improvement of 1,028 jobs.

Saskatchewan has not yet developed a complete data collection template that is used by all officers in the same way. The CCS reports 127 jobs created or improved over the five years. In Saskatchewan, the entrepreneurs who answered the survey indicated that the number of employees changed from 22 prior to applying for FEDO assistance to 103 afterward, an increase of 81 jobs. In total, 40% of all the businesses that received CCS assistance were contacted during the survey. If the number reported can be extrapolated to all people receiving service, the total would be 203. The officer who keeps this data for his clients provided us with a database presenting the number of jobs created or maintained as a result of the work done. We saw 114 full-time jobs and 61 part-time jobs. We were able to validate some of these data during the survey with those entrepreneurs who agreed to take part in the interview. The study concludes that the total reported by Saskatchewan for the five years is valid. Since the CCS has not developed a rigorous methodology for this and the other officers did not report any numbers, it is reasonable to think that this total is probably under-reported.

Alberta reports 140 jobs created, maintained or improved over the five years. Sixty-six percent (66%) of this total (93 jobs) is reported for the years 2006 to 2008 only, after the CDEA introduced a new and more accurate method of data collection. For the previous years, the figures reported were 15, 25 and 7. The survey indicates that the number of employees in Alberta rose from 90 before applying to the CDEA for service to 114 after applying for service. That represents an increase of 14 jobs. In all, 40% of all the businesses served by the CDEA participated in the survey. If we extrapolate this figure to the total, the result would be 35 jobs—less than half the total reported to WD. On the other hand, Alberta was the only province in which entrepreneurs indicated that 100% of the increase in jobs was due to the assistance received from the FEDO. Also, 90% of the entrepreneurs who answered the question said that the CDEA helped to increase their sales. This is the highest rate of the four provinces. Half of these respondents said that the increase was 11% or more. Fifty-seven percent (57%) of all respondents said that the CDEA helped them increase their business' payroll. In recent years, Alberta has experienced technical difficulties and staff turnover. The entrepreneur lists had to be recompiled for the purposes of this survey. For the five years concerned, we received 89 names from Alberta for the survey, only 15 of them from the south of the province. Neither the documentation provided nor the survey results allow us to confirm the total reported by Alberta. We can confirm about 40 of the total jobs reported.

British Columbia reports 403 jobs created, maintained or improved during the five years. The SDECB created a network of Francophone female entrepreneurs in 2003. The documentation concerning this network's activities is complete and specific. We have reviewed the activities carried out with this network. In all, 116 Francophone women entrepreneurs created, maintained or improved 226 jobs between 2003 and 2008. Female entrepreneurs made up 44% of the total entrepreneurs we called upon for the purposes of the survey. These figures in themselves indicate that the total jobs for the entire SDECB could be around 500. The total reported by British Columbia seems reasonable to us, even though the documentation is less specific for the categories of entrepreneurs other than women.

The objective evidence represented by the survey and analysis of the documentation provided serves to conclude that the totals for number of jobs created, maintained or improved are correctly reported in Manitoba, probably under-reported in Saskatchewan and probably correctly reported in British Columbia. The study can make no judgment on Alberta in this regard.

To summarize, the study can confirm 93% of the reported total jobs maintained, created or improved, i.e. about 1,500 jobs out of the total of 1,610 reported.

The impact study concludes that the totals reported by the Department demonstrating the cumulative effects of the four FEDOs in the Improvement of Productivity program are not sufficiently reliable. The totals for certain line items are meaningless because the FEDOs report the data differently. The totals for value of incremental investments are under-reported because two FEDOs do not capture the information and one FEDO captures it incorrectly. Only the totals for number of businesses and number of jobs created, maintained or improved are properly reported.

Certainly these are the two most important items of line data. As a whole, the data collection method used by the FEDOs for business productivity affects the credibility of the results reported by each FEDO and the total results of the four FEDOs reported by WD to Canadians.

This is even more serious because the four FEDOs are obtaining good results, as confirmed by the objective evidence.

This is not a matter of human error, but of gaps in the system. For the entirety of the present template, a cross-analysis must be done of the content of some 950 different totals for a period of five years. For the purposes of this impact study, we analyzed 125 of these. It is important to note that the discrepancies in the reports are noticeable only if a person analyzes the each province's data and the cumulative total for each of the line items.

However, no one in the system conducts this type of cross-analysis.

The FEDOs' senior managers analyze their own totals and the cumulative total, but do not analyze the data of their counterparts in any detail. As for WD, each officer analyzes the data from the FEDO of his/her province, and the employees at the central office add the totals per province to obtain the cumulative total. The provincial officers do not have the tools to analyze data from the FEDOs in the other provinces, and the central office does not have the tools to break down each provincial total in each line item and check that each FEDO is following the common definitions.

Return to the top of this pagetop of page

Three difficulties were noted with respect to the FEDOs:

  • The data collection tools used by the four FEDOs are not the same; sometimes the tools used by officers at a single FEDO are not the same. It stands to reason that the data reported will not be exact if each FEDO uses a different tool than its counterparts.
  • There is no common understanding of definitions among the four FEDOs.
  • The methodology used to monitor entrepreneurs to determine the value of incremental investments and number of jobs created, maintained or improved is not the same among the four FEDOs.

The following difficulties were noted with respect to WD:

  • The WD officer responsible for a FEDO reviews the data submitted on a quarterly basis and approves it; then that officer's manager approves it and it is entered into WD's system. The WD officer does not have the means to cross-check the data submitted by the FEDO; the officer's approval of the data, and the approval of the officer's manager, are an expression of confidence in the figure submitted by the FEDO and not an authentication of that figure.
  • The computer system used by WD is subject to electronic glitches. For example, the Manitoba data for 2006-2007 indicated 0 jobs created thanks to assistance to businesses. The CDEM reported a total of 225 jobs and the WD officer entered that information correctly in the system. All the same, the cumulative table to this day still indicates 0. (We recorded the correct figure for the purposes of this report.)

We conclude that it is not the FEDOs' performance that should be the subject of a recommendation further to this impact study, but rather the methodology for collecting data from the FEDOs, WD's methodology for authenticating data and the WD computer system used to compile FEDO performance reports.

Further to these analyses, we propose the following recommendation:

Recommendation 2: that WD and the FEDOs develop and implement a completely new approach for assembling relevant data and reporting outputs and outcomes achieved in the Improvement of Business Productivity and Entrepreneurship program.

This recommendation has five major effects:

  1. We closely examined the outputs and outcomes of the four FEDOs. We find that even though the situations in each Francophone community vary from one province to another, the FEDOs' work methods and approaches are very similar. This similarity has made it possible to implement collective projects such as the Western Tourism Corridor and the young entrepreneurs' camp in Saskatchewan.

    The impact study concludes that only one data collection instrument should be used by the four FEDOs. This instrument would provide all FEDO employees with the same tool to report to their line managers and senior management. This tool must make it possible to determine, among other things, the number of jobs created, maintained or improved and the values of incremental investments generated following the assistance received from a FEDO. Senior management would use this data in its subsequent reports to its boards of directors and communities.

    The four FEDOs should use a common format, all the components of which are understood in the same way, to prepare their performance reports for WD. For a certain period of time, the senior managers of each FEDO should review the quantitative reports submitted by their counterparts so as to be able to identify and correct disparities caused by differing interpretation of line items.

  2. The FEDOs should ask each entrepreneur-client they serve to sign a form that authorizes a third party to subsequently communicate with that entrepreneur for evaluation purposes.

  3. WD must identify methods of cross-checking the data submitted by FEDOs, while respecting the anonymity of entrepreneurs. This task could be performed by a third party, for example, or WD might order an impact study every three years. At a minimum, WD officers should verify the data collection instruments used by the FEDOs and the data collection processes to ensure that all the parties are using the same definitions under the same line items and that the data collected is reliable. This assumes that all WD officers have the same understanding of the definitions for each line item.

  4. The impact study conducted a detailed analysis of 18 line items in the WD databank. The data bank, upgraded in 2006, contains a total of 36 line items. The question that arises is this: what value is added by collecting information for that many items? The analyses that will be conducted as follow-up to this impact study will be able to identify the line items that are essential for measuring progress in connection with the program indicators.

  5. The September 2004 evaluation made the following observation with regard to the performance reports: "The accuracy of the quarterly reports is questionable, as results appear inconsistent from one FEDO to another. FEDOs feel that the reports only partly reflect their success. There is a desire to work with WD to improve the reporting format."

    The impact study finds that progress has been made in this regard, particularly with the clarification given to the definitions in 2006. Those efforts notwithstanding, however, it is clear that the FEDOs and WD will have to invest the necessary funds to ensure that WD and FEDO staff are trained in using the data collection tool and in preparing performance reports.