Archived information

This content is archived because Status of Women Canada no longer exists. Please visit the Women and Gender Equality Canada.

Archived information is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

Planning and Reporting

Evaluation of the Women's Program

Volume I - Final Report

Table of Contents

5.0 Conclusions and recommendations

This final section of the report summarizes the evaluation findings, draws conclusions, and provides recommendations.

Design and delivery

Overall, stakeholders perceive the design and delivery of the WP to have several important strengths. These include the Program's flexibility, which enables it to accommodate a diverse range of groups, strategies and issues; its social development approach, which is designed to support long-term social change by supporting the capacity of women's organizations to participate more fully in Canadian society; its decentralized structure and presence in communities; and its unique focus on women's equality.

The structure of the WP, consisting of technical assistance and financial assistance components, is seen as basically sound. Program staff and managers and applicants likewise observed that the proposal development process can be demanding, and more so since the introduction of the outcome-based approach.

Successful implementation of the outcome-based approach is currently hampered by a number of significant obstacles. Perhaps most importantly, neither applicants nor all program staff have fully grasped and embraced the outcome-based approach. There is also continuing uncertainty about the appropriateness of the outcome-based approach to social development work among both applicants and program staff. Secondly, although the WP collects a great deal of outcome information via the standardized forms it introduced in 2003-2004, it lacks any strategy for analysing and reporting on these data, and moreover lacks a program database, accessible to regional staff. Some program staff and key informants cautioned against over-reliance on a reporting solution based on the standard forms and a program database. These stakeholders recommended that the WP implement a diversified performance measurement strategy consisting of both the planned "quantitative" methods and objective qualitative methods. A reporting strategy is currently under development, but progress toward a program database has reportedly stalled due to a lack of financial resources, technical capacity, and management commitment.

Recommendation 1. Mechanisms for greater program management accountability relative to the implementation of the outcome-based approach are necessary.

Management's contribution to the successful implementation of the outcome-based approach could be measured in terms of the availability, quality, and timeliness of appropriate documentation (hard copy and/or web-based) and training activities on this approach for program staff, applicants and potential applicants across Canada.
Management should be accountable for clarity and consistency in funding recommendations and reporting. In particular, Program Managers should ensure that proposals are not recommended until they clearly articulate an initiative's activities, objectives and expected outcomes. A standard protocol should to be established for changing activities, objectives and expected outcomes during the duration of an initiative. They should also ensure that an initiative's final report is not accepted until it includes the required information concerning outcomes achieved.
Management should also be accountable for sharing information regarding best practices in the implementation of the outcome-based approach across the regional and local offices of the WP.

Recommendation 2. The WP should provide training in the outcome-based approach and efficient proposal development to funding applicants and program staff and managers.

Training should focus on ensuring an adequate understanding of the outcome-based approach, including its key concepts, definitions, and requirements, in order to realize potential efficiencies in the proposal development process.
It should also specifically address the purpose and proper use of the standard forms. It should emphasize the importance of consistency in reporting.

The file review component of this evaluation was limited in the extent to which it could meet its second objective, that is, to examine the extent to which the outcome-based approach to reporting was being implemented and to identify any emerging issues that might benefit from further investigation or ongoing monitoring. This was due to the small number of "closed files" available for review from the 2003-2004 fiscal year, and was a consequence of the short period between the implementation of the outcome-based approach and this evaluation.

Recommendation 3. The WP should repeat such a file review as part of a regular evaluation cycle, as per Treasury Board guidelines, because it would undoubtedly prove much more informative in terms of the implementation successes and challenges related to the outcome-based approach and the extent to which it is understood by applicants and program staff.

Recommendation 4. The WP should develop a strategy for reporting on program impact. Consideration should be given to a diversified performance measurement strategy that focuses not only on summarizing effectively the information collected on the standard forms, but also additional objective qualitative methods to illuminate outcomes achieved.

Some stakeholders indicated that they required assistance to understand the information required on the various forms, and the evaluator did not find the forms useful because of inconsistencies in interpretation of the various requirements, be it by applicants, funded initiatives, or program staff. Stakeholders have also expressed a need for simplifying and making the application and reporting process more flexible.

Recommendation 5. Although they were introduced relatively recently, the WP could improve upon the standard forms at this stage, as well as address the forms specifically within the recommended training and in the documentation on the outcome-based approach. In order to enhance their value for performance measurement purposes, improvements to the standard forms should be made in consultation with evaluation experts and a representative sample of potential applicants.

Given the inherent challenges of measuring social change and the need for the WP to collect more useful and relevant outcome information, the Program should focus on more objective qualitative data collection methods.

Recommendation 6. In order to complement self-reporting by the funded initiatives, the WP should invest in a process of external review of funded initiatives whereby a sample of initiatives would be selected each fiscal year for a review by external evaluators. Given the diversity of initiatives, high-level indicators of each initiative's impact could be applied.

Recommendation 7. The WP should develop and implement a program database, accessible to staff and managers in all regions, to manage basic aspects of their work and to permit data collection, analysis, and reporting on outcomes.

The evaluation took place in the context of the ongoing investigations of the new Standing Committee on the Status of Women, which in May 2005, recommended that the WP introduce a mix of program funding and project funding. The Committee's hearings confirmed stakeholder views expressed during this evaluation in that the termination of program funding and its corollary, the introduction of initiative funding, remains the most controversial of the 1998 changes to the WP.

The evaluation found that the termination of program funding continues to be perceived as detrimental by some organizations. On the other hand, initiative funding is believed to have increased equity of access to WP funding for organizations that were previously unable to access funding from the Program, and increased the visibility and capacity of these organizations. Survey results confirm that funding applicants are divided in their opinion of the impact of the change.

Neither 18-month nor multi-year funding, both of which were introduced in 1998, are nearly as controversial. Both are seen as positive developments. To a certain extent, stakeholders see 18-month and multi-year funding as compensating for the termination of program funding. However, some program staff and managers noted that multi-year funding is limited by its eligibility criteria. Suggestions for improvement included making multi-year funding available to less experienced organizations and ensuring its availability across the country.

While there is clearly strong support within an element of the WP's constituency for restoring program funding, the larger constituency seems largely content with the existing funding mechanisms. Among program staff, there was little support for restoring program funding, although many acknowledged a need for longer-term funding.

Recommendation 8. The WP should consider extending multi-year funding beyond three years, consistently across Canada, to support initiatives with longer planning and implementation cycles. However, annual renewal of funding should be contingent on organizations fulfilling performance monitoring and reporting requirements.

Impact

Although this evaluation was intended to be a summative evaluation focusing on outcomes achieved by the WP, it was limited in its ability to discern the overall impact of the Program due to the limitations of the available administrative data and the relative infancy of the outcome-based approach. As a result, evaluation findings pertaining to program impact are based primarily on anecdotal information.

There is widespread, though not unanimous, agreement among stakeholders that the WP has helped to advance women's equality in Canada. However, it is considerably more difficult to pinpoint the extent to which the WP has achieved each of its four objectives - promoting institutional change, influencing public policy, increasing public understanding of action on women's equality issues, and building capacity among women's organizations. While key informants were able to give specific examples of changes related to each objective, they also pointed out that until implementation of the outcome-based approach in 2003-2004, the Program made no systematic attempt to determine its overall impact. Many also observed that it is virtually impossible to isolate the net contribution of the WP due to the equally significant role of external factors, primarily the political climate, in influencing social change.

The evaluation also sought to determine whether the WP has achieved its objectives under the federal government's Agenda for Gender Equality (AGE) initiative. Administrative data show that AGE funds have enabled the WP to realize some success by a total of 122 new groups and 123 files involving new and emerging issues. However, in interviews, many program staff and managers noted that the purpose of AGE was not explicitly identified when the initiative was introduced and that, therefore, any successes were more by accident than by design. The additional funds simply made it possible to invest more money overall.

The evaluation found some ambivalence among program staff and managers regarding the WP's contribution toward achieving SWC's strategic objectives. While most believe that the Program has contributed to a more equitable public policy and a broader range of informed stakeholders, they were divided on the matter of its contribution to departmental effectiveness. These key informants reported that the WP and the rest of SWC work in isolation from one another and do not communicate or collaborate effectively. In short, the WP is not well integrated into the agency, communications with other directorates within the agency are poor, and the WP lacks influence on departmental decision-making. There is a widespread perception that the poor relationship stems from a fundamental philosophical difference regarding the best way to achieve women's equality. The WP uses a social development approach, but this is not, nor reportedly has ever been, the approach taken by SWC. Key informants emphasized the need for improved communications and collaboration among all of the directorates within SWC.

Recommendation 9. SWC and the WP should work toward greater integration of the WP and developing more collaborative relationships between the WP and other SWC directorates to ensure that each is benefiting from the work and expertise of the others. For example, considering the wealth of information it collects and frequent contacts with women's groups and other equality-seeking organizations, the WP should be directly involved in policy development and implementation. Collaborative relationships between SWC directorates, including the WP, should be based on a continuous exchange of information.

Cost-effectiveness/alternatives

Key informants were unanimous that the WP is giving Canadians value for their tax dollar, observing that the WP encourages community-based organizations to work on issues that are important to Canadian society "on a shoestring," while relying heavily on volunteer contributions of time and effort. Available financial information presents a picture whereby a significant portion of the WP's limited budget goes toward administrative costs. This is likely due to the social development approach to the work of the WP. Thus, included in the administrative costs is the provision of technical assistance required for building capacity among women's groups and other equality-seeking organizations. The complexity of women's equality issues and the social change process, the fact that the WP deals with a number of marginalized organizations and/or new and emerging organizations, and the Program's decentralized delivery model can also contribute to increasing the costs associated with providing this form of assistance.

Recommendation 10. The WP should develop a procedure by which it would monitor the portion of administrative costs that are associated with the provision of technical assistance and other duties, which may not be directly related to proposal development, assessment or recommendation, or even related to the WP. Depending on time tracking systems or other administrative reporting requirements already in place at the WP, it could take the form of a fairly simple biannual estimate of the portion of the workload among program staff and managers associated with technical assistance, or at the other end of the spectrum, take the form of detailed daily tracking of time spent on technical assistance versus other tasks.

As well, efficiencies could potentially be gained by improvements to the standard forms, by simplifying and/or shortening the proposal preparation process, through training, by providing staff with better access to technologies, and greater dissemination of information to women's groups and other equality-seeking organizations as well as the general public via the web and other appropriate means.

The findings also suggest a need for greater transparency and increased communication with organizations during and after the assessment process, particularly when it comes to proposals that are declined. Stakeholders' suggestions for improving the proposal development and approval process included the provision of more staff assistance and clarifying the information available in program documentation and on the WP's web site.

Recommendation 11. In the case of proposals that are declined, the WP should provide clear feedback to all applicants on precisely how their proposal does not meet the eligibility criteria for funding. The WP should provide the results of the proposal assessment process to the declined applicant and always include information on how to appeal the decision.

Recommendation 12. Program staff and managers emphasized the need for access to appropriate and adequate technologies to improve the efficiency of program delivery. The WP should support increased efficiency and improved communications by enhancing, where necessary, the equipment, software and support available to program staff and managers, and providing additional access to technologies related to the demands of the program staff and managers' work (e.g., improved technical support at the regional level, cellular telephones and laptops for staff use while traveling, and redesigned electronic forms).

However, the most significant staff concerns had little to do with the efficiency of program delivery and more to do with its effectiveness. There is a general consensus among program staff and managers that effective delivery of the WP is hampered by problems related to program management. First, program staff and managers believe that the WP suffers from poor internal communications and information sharing among the regions and the national office. Secondly, there is a widespread perception that senior management within SWC does not truly understand, value and support the WP, and that as a result, the WP is not truly integrated into the agency.

Recommendation 13. The WP should take steps to improve internal communications by introducing mechanisms for timely communications among the regions and the national office.

The WP could make better use of its existing Intranet to improve the dissemination of information.
Meetings should be held as frequently as appropriate to disseminate information and engage program staff and managers on current issues faced by the Program as well as issues of a more administrative nature. The WP should use available technologies, such as telephone or video-conferencing, to substitute for or further supplement the more costly in-person meetings with program staff and managers across the national, regional, and local offices.

There is a general consensus among stakeholders that the WP does not duplicate any other programs to advance women's equality, primarily because very few other such programs exist. The WP is seen as unique in providing support for advocacy efforts, providing a high level of technical assistance, addressing a full range of equality issues, and focusing its mandate specifically on advancing women's equality.

Stakeholders had few suggestions for alternative delivery models. However, some suggested that the WP may wish to consider the use of contribution funding in certain circumstances, while others suggested community-based models. Of course, there was also considerable support for more sustained or longer-term funding.

The evaluation found little grounds for transferring the WP away from SWC. Stakeholders maintained that the Program should be retained within SWC since it is the only federal organization with a specific mandate to advance women's equality. However, stakeholders also emphasized the need for the WP and SWC to resolve their outstanding differences so that they may work effectively together. Many key informants pointed out that to be effective, the agency needs greater visibility and a stronger voice within the federal government.

Relevance

There is general agreement that the WP is still relevant to advancing women's equality in Canada, since, despite progress in some areas, this goal has not yet been achieved. However, some key informants also expressed concern that, for fear of controversy, the Program has become overly cautious in its funding decisions and therefore risks irrelevance. Fifty-eight percent of survey respondents believe that the Program responds well to new and emerging issues, while 51% agree that it responds well to new and emerging groups.

Recommendation 14. To be more transparent, the WP should disseminate reports on program impact to the WP's constituency and to the general public.

The evaluation found some disagreement among stakeholders regarding the WP's alignment with federal priorities and the strategic objectives of SWC. Some pointed out that the WP clearly aligns with federal priorities by addressing issues such as violence against Aboriginal women, childcare, trafficking in women, social capital, and democratic processes. However, others argued that the WP's mandate is to promote women's equality and that, quite often, this means supporting groups to challenge the status quo - that is, in terms of public policy and institutional decision-making. The crucial question, from their perspective, is the extent to which the federal government is aligned with the priorities of the WP and women's perspectives are integrated into federal policy - and not the reverse. Similarly, although some program staff and managers believe that the WP aligns well with SWC's strategic objectives, others said that the strategic objectives are vague and do not reflect the priorities of the grassroots women's movement.

Table of Contents