Common menu bar

AAND: Aboriginal Affairs and Northern Development
FaHCSIA: Department of Families, Housing, Community Services and Indigenous Affairs (Australia)
GBA: Gender-based Analysis
NCPC: National Crime Prevention Centre
NGO: Non-governmental organization
PAA: Program Activity Architecture
RBM: Results-based Management
SWC: Status of Women Canada
WCF: Women's Community Fund
WEF: World Economic Forum
WP: Women's Program
WPF: Women's Partnership Fund

List of Tables

Table 1.1: Number of Funded Projects by Year
Table 1.2: WP Total Annual Budget
Table 3.1: Total WP Applications and Projects Funded – 2004-2005 to 2009-2010
Table 3.2: Emerging Issues/Gaps: Program Applicants
Table 3.3: Nature of Barriers Addressed by Projects
Table 3.4: Best Practices: Funded Applicants
Table 3.5: Applicants' Experience Applying to the WP for Funding
Table 3.6: Administrative and Program Costs, Administrative Efficiency Ratio

Executive Summary

Overview

In support of the federal agenda for advancing the status of women, the Women's Program (WP) provides funding and professional assistance to organizations to carry out projects that advance equality for women across Canada through improvement of their economic and social conditions and their participation in democratic life. Status of Women Canada (SWC), the responsible agency for the program, has commissioned a summative evaluation of the WP.

Founded in 1973, the WP is a permanent program, with Terms and Conditions in effect for five years. The Terms and Conditions that are the focus of this evaluation came into effect in 2006-2007. The WP's current mandate is to advance equality for women across Canada through improvement of their economic and social conditions and their participation in democratic life. Its objective is to achieve the full participation of women in the economic, social and democratic life of Canada.

The WP provides both funding and technical assistance to funded organizations at the local, regional and national levels. The program supports a range of community-based projects that address women's economic security and prosperity, violence against women, and leadership and democratic participation. The program is delivered regionally through four offices, and an office that funds national projects. Program funding is $19 million annually. The program has two funding components:

The Women's Community Fund (WCF) accounts for the majority of spending, and seeks to address the economic and social situation of women and their participation in democratic life through projects that directly support women in their communities.
The Women's Partnership Fund (WPF) has the goal of building partnerships between SWC, eligible non-governmental organizations, public institutions and the private sector through collaborative projects that address the economic and social situation of women and their participation in democratic life.

Scope and Methodology

The evaluation covers the Terms and Conditions for the program in place between 2006-07 and 2010-11. Evaluation issues related to relevance and program performance were addressed, as well as design and delivery questions.

Multiple lines of evidence were used to increase the reliability and robustness of the evaluation of the WP. The evaluation included: a review of documentation, program data and literature; file review; key informant interviews; survey of program applicants; case studies; comparative program analysis; and cost-effectiveness analysis. Each of these methods is described below.

Document Review: Secondary sources were reviewed to develop a detailed profile of the WP and to contribute to addressing several of the evaluation questions including relevance, design and delivery, and program performance. A document review template was developed to summarize findings in the documents pertaining to each of the evaluation questions.

Program Data Review: SWC provided EKOS with a list of all projects approved for funding since September 2006 up to 2009-10 (n=398) for analysis. Coding was undertaken to transform some of the data for the purposes of analysis.

Literature Review: Recent literature was reviewed to contribute to the analysis of program relevance (need) and alternative delivery models. Sources included relevant literature and research on gender equality including research studies, position and policy papers, and domestic and international publications. Documentation and sources provided by SWC were complemented by an Internet search of academic, NGO research/ policy/advocacy organizations and government documents.

Review of Project Files: A stratified random sample of project files from SWC headquarters and regional offices was reviewed. In total, 40 files were reviewed, roughly 10 per cent of the projects funded during the period under study. The selection of files focused on closed projects and further considered the funding component, region and year of the agreement. An electronic data input form was used to capture file information in a consistent manner.

Key Informant Interviews: In total, 16 key informant interviews were completed with program managers and staff, partners, and external experts.

Online Survey of Program Applicants: The evaluation included an online survey of organizations that applied to the WP for funding during the period under study. The survey included a census of organizations that were funded by the WP between 2006-07 and 2009-10. For non-funded applicants, a census of eligible organizations from the third and fourth funding calls was surveyed.

Case Studies: Seven case studies of funded projects were conducted to illustrate impacts and sustainability of different representative approaches within the WP's funding activity. Each of the case studies was based on a review of the project file, a review of other project or organizational documentation; and key informant interviews with project proponents, partners and key stakeholders (the precise number of key informants was determined on a case-by-case basis).

Comparative Program Analysis: A review of program materials from federal grants and contribution programs with similar characteristics to the WP was conducted. Six programs were identified. For each of these programs, publicly available information was reviewed (e.g., program web-site, departmental planning and performance documents, program evaluation/audits) to compare programming aspects such as proposal solicitation and eligibility criteria, and to identify areas where the WP complements and/or overlaps with other federal programs. Interviews were also conducted with program staff for the comparative programs.

Cost-effectiveness Analysis: An analysis of the efficiency of the program was undertaken, including an examination of utilization of resources (including leveraged resources) and the administrative to program dollar ratio of WP (annually). This analysis was supplemented by qualitative feedback obtained in the key informant interviews about efficiency and economy (and where there are opportunities for efficiency gains).

Findings

Relevance

The evaluation evidence indicates that there is an ongoing need to address gender equality. While there have been significant strides toward gender equality in Canada, including improvements on a number of indicators in the last five years, inequalities remain in a number of areas. The program's priority areas – ending violence against women and girls, improving women's economic security and prosperity, and encouraging women and girls in leadership and decision making roles – are well-aligned with national data that show, for example, persistent issues of spousal and sexual violence against women, a gap in wage parity on the basis of gender, and an underrepresentation of women in senior political and business positions. These priorities are also consistent with those of other jurisdictions and governments internationally.
Over the program cycle, demand for the program has been high – owing, in part, to the expanded eligibility criteria under the Terms and Conditions of the program and a broader solicitation of applications through a call for proposals process. Also driving demand is a lack of funding programs targeted to women or with a gender equality mandate. The importance of WP as a source of funding is supported in the reports of program applicants – the majority of unfunded applicants indicated that their project did not proceed as proposed without WP funding and most funded applicants did not perceive there to be ready access to funding alternatives if WP funding not been available to them.
The program is consistent with SWC strategic priorities (the program and the Agency share the same strategic/ultimate outcome of equality for women). The WP aligns with federal priorities, particularly those related to economic prosperity, and the program also supports complementary federal strategies (e.g., the Family Violence Initiative) and international commitments related to gender equality.
The federal level makes a relevant and important contribution to gender equality, though the federal government, SWC and the WP does not bear this responsibility solely. Canada's involvement in advancing gender equality is consistent with practices internationally. A caution is that program data indicate that the program funds many service delivery projects, an area that is traditionally the role of province. While projects access complementary support from other funding sources (e.g., provinces, donations/fundraising), there were few concerns related to overlap or duplication.
Overlap and duplication was not identified as a significant concern (in large part owing to perceived limited efforts in the area overall). However, opportunities for enhanced collaboration with provincial colleagues and across the federal family were noted.
In addition to addressing persistent areas of gender inequality such as violence and wage parity, other emerging needs were identified in the evaluation. Examples include trafficking and sexual exploitation/hypersexualization of women and girls. Immigrant and Aboriginal women were also identified as continuing priorities. Given the breadth of the program's objectives and finite funds, a more definitive niche for the program was perceived to be important to drive more focused funding efforts for greater impact. This program niche would leverage federal strength and fund in areas where levers in other federal departments are absent.

Performance

Effectiveness

The program's immediate outcome of increased awareness among women to identify and/or remove barriers to their participation in their communities is measurable only by proxy through participation of women in the program's funded projects. Almost 400 projects were funded in the first four years of the program to address barriers related to leadership, economic security, violence, community awareness and access to services among others. The evaluation data indicated a wide distribution in the number of women impacted at the project level – from projects serving a handful of women to those impacting many thousands (with a median number of 125). Greater reach is evident among projects that are led by women's groups and that foster change in policy or institution- or sector-wide practices or services. Of note is that more often than not the number of women directly affected was greater than originally anticipated by the project.
There is good evidence that the program has been successful in stimulating partnerships with NGOs and other levels of government, while less so with other federal departments and the private sector. While an intended outcome for the WPF funding stream only, funded applicants from both the WCF and WPF streams report undertaking beneficial partnerships with these various sectors.
Projects also leverage funding from partners – WP funds, on average about 60 per cent of project costs, with proponent organization and their partners funding the balance (including financial and in-kind contributions).
With respect to the program's intermediate program outcome – participation of women in their communities – there is some evidence of this impact, though it is difficult to "count" or provide a roll up of these impacts across the great variety of projects that are funded. The evaluation noted many examples of impacts across a spectrum of indicators including impacts on the individual women who participated, as well as projects that involved more far-reaching impacts on policy, institutional practices and services.
The program's ultimate outcome – equality for women and their full participation in the economic, social and democratic life of Canadian society – is broadly stated, ambitious and difficult to attribute WP's contribution to its achievement. Key informants and project proponents were both cautious about the program's achievements in this area. Evaluation evidence to measure the program's performance on this measure was limited. There are also limitations on the program's potential impact given the finite amount of project-based funding available and the program's focus on a wide range of projects that are supported at the individual level.
Still, there is evidence that many projects funded by the WP (six in ten) are at least moderately sustainable. This longer-term impact occurs through ongoing availability of resources or tools, or continued delivery of aspects of the project by the proponent organization or their partners. Sustainability was reported somewhat more often by projects that were led by women's groups and those whose projects resulted in change at the institutional level.
There were a number of observations from the evaluation (e.g., from the literature, views of key informants and funded applicants) on promising practices. Examples such as partnerships, involvement of participants in delivery, culturally relevant programming, mentoring, and wraparound supports are being used by many projects already. However, the knowledge-building and knowledge sharing capacity within the program, including the identification and sharing of promising practices/proven resources and tools, is limited and identified as a gap by the program and stakeholders.
Unintended outcomes of the program were generally few, but largely positive. The experience at the project level has shown that many projects have garnered greater than anticipated participation and interest in their initiative by women, partners and the community.

Design and Delivery

The design of the WP changed significantly in 2006-07 and 2007-08, and program managers continue to make adaptive changes to improve delivery. Since 2010, the poorly distinguished WCF/WPF funding components are now obsolete. The program has also moved away from the general call for proposals process towards a dual approach of continuous intake and targeted call for proposals mechanisms to address difficulties with the management of periodic calls. Indeed, satisfaction with the application process for the general calls was only moderate among funded applicants who identified the potential to streamline, simplify and clarify the application process and improve timeliness. While a majority of funded applicants did receive assistance from WP staff during the application process, many suggested additional support or feedback as a potential design improvement. Most unfunded applicants did not feel adequately informed as to why their application was unsuccessful.
The changes in the program's Terms and Conditions, as well as operational funding reductions have had both positive and negative impacts. During the period under study, projects incorporated a more concrete focus on the tangible impacts of their funded activities on women and introduced many mainstream organizations to gender-based programming (improvements valued by both the program and funded applicants). However, operational funding reductions have created capacity constraints and placed increasing and changing demands on staff who have limited access to support. The capacity issues have had negative repercussions in terms of responsiveness to funded applicants and engagement of partners and stakeholders (applicants in the West were particularly affected).
The key themes that emerged in terms of design improvements had to do with: improving the knowledge building/knowledge transfer capacity of the program; extending/reinforcing collaborations with partners and stakeholders; streamlining applications processes; and support to staff.
Some of the program's funding eligibility criteria for organizations and for eligible activities were seen to be overly restrictive and limiting access to program funding to achieve objectives. Indeed, the criteria were found to be quite variable across a number of federal grants and contributions programs that were examined. Incorporating an emphasis on sustainability of projects was raised as a way to direct funds for maximum impact.
Performance measurement has improved over the current program cycle, with new tools and processes introduced to better capture results. Like other programs with broad and longer-term social objectives, the program faces challenges in measuring and attributing impact. Funded applicants are generally satisfied with the expectations and their requirements for monitoring and reporting (their level of effort, performance measures, accessibility of staff). While there are undoubtedly improvements that could be made to the tools (e.g., greater focus on outcomes as opposed to outputs, linkages across the program's intended outcomes statements), performance measurement is more seriously hampered by human resource constraints to undertake the time consuming analysis and synthesis of information to inform program decision-making and strategic planning.

Efficiency and Economy

The program's administrative efficiency ratio is reasonable – for every dollar of funding that is granted or awarded by the program, $0.13 is spent on program administration. A number of factors support the program's efficiency and economy: the program's lean operations and leveraging of resources (including expertise, networks, volunteers) from the community-based sector. The capacity issues of the program were perceived to be a drag on program efficiency – that is, the limited staff complement and supporting automated systems undermine the program's efficiency and economy.
Few alternative models or agencies are available to deliver on the WP's mandate. The WP is distinguished as a national program that addresses equality through engagement of multiple stakeholders in multi-jurisdictional contexts. Potential alternative program vehicles such as provincial or third party delivery were generally seen to have limited capacity or authority to deliver the program, and carry some additional risks and potential disadvantages (e.g., related to accountability and federal visibility).

Recommendations

Recommendations to improve the Women's Program are based on the findings and conclusions of the evaluation. The following recommendations focus on program improvements in program performance and program design and delivery.

Program Performance

1) In order to improve the overall effectiveness of the program in achieving its ultimate intended outcome of equality for women, the program should identify any necessary program design changes to fund activities that the evaluation found to have greater potential reach and sustainable impact. These evaluation results and the program's own internal analyses indicate that funded projects that include components that foster change at the level of policy or institutional practices have greater reach and sustainability. Funded projects that create resources or tools, or leverage proponents' existing regional or national networks/membership also have advantages in terms of reach and sustainability. While projects led by women's groups were also associated with greater reach and sustainability, any program design changes to the eligibility of organizations must be balanced against the benefits of introducing gender equality programming into mainstream organizations.

Design and Delivery

2) The program should clarify its theory of change from activities and outputs through to immediate, intermediate and ultimate outcomes. The program's current logic model does not adequately capture all activities carried out by the program, and the results chain does not reflect the program's evolving understanding of social change – i.e., based on a multi-dimensional approach that supports social change at multiple levels, including the participation of individual women, capacity and connections among organizations or networks of organizations, and change within communities and at the level of policy.

3) The WP should continue to initiate and strengthen relationship-building with provinces and other federal departments and agencies to seek input on program funding priorities. WP project activity in areas such as immigration and skills development to remove barriers to women's participation is diverting finite program funding to areas where other federal or provincial programs are in place and could address the participation of women. While other government departments are aware of the WP and have a favourable impression of the program, efforts moving forward would benefit from consultations with federal partners to ensure that WP's limited funding is deployed in areas where there is an absence of other levers to address barriers to women's participation.

4) The program should review deployment of its internal organizational resources to address significant capacity issues within the program, particularly a thinly-spread regional delivery structure. The program should continue its exploration of alternative delivery models or innovations with a view to addressing particular constraints in underserved regions (the West where applicant satisfaction levels tend to be lower and the North where there are few projects funded) (e.g., leveraging of provincial capacities, utilization of service delivery nodes in other federal departments, use of third party delivery for selected program focus areas).

5) The program should further leverage the project-level investments that it makes with enhanced mechanisms for knowledge building/sharing with and among stakeholders. Encouraging projects to share their results and embedding opportunities for dissemination and exchange at the program level would enhance the value of products developed by funded projects and keep the program funding focused on innovation. To support this:

Consideration should be given to adjusting the current funding envelope (e.g., O&M dollars made available) to permit activities to support dissemination/exchange activities, including via web-site postings, workshops/meetings or other mechanisms.
Review the program's Terms and Conditions to remove any restrictive eligibility criteria related to organization eligibility (e.g., academic organizations, cooperatives) and activity eligibility that inhibit undertaking knowledge building/sharing activities.

1. Introduction and Description of the Women's Program

The report is organized into the following sections: Chapter One provides a description of the WP; Chapter Two discusses the objectives of the evaluation and the methodological approach; Chapter Three presents the evaluation's findings related to relevance, design and delivery, and performance (effectiveness, efficiency and economy); conclusions are included in Chapter Four; and recommendations are presented in Chapter Five.

1.1 Program Description

a) Context

Since it was established in 1971, the Office of the Coordinator, Status of Women, (to be referred to as the "Agency") has played an important role in advancing equality for women in Canada. The Agency's strategic outcome is "Equality for women and their full participation in the economic, social and democratic life of Canada." SWC focuses its efforts on three key priority areas: ending violence against women and girls, improving women's economic security and prosperity, and encouraging women and girls in leadership and decision-making roles.

The Agency pursues its intended strategic outcome by way of two key program activities. First, SWC fosters inclusive and effective policies that advance gender equality and address those issues that affect women by enhancing the capacity of federal departments/agencies to integrate gender-based analysis (GBA) into all aspects of their work. Second, SWC seeks to ensure that women are equipped with the information, knowledge, skills and tools to be able to identify and address barriers that impede their full participation in society. The WP falls within the second program activity.

b) Overview of the Women's Program

Founded in 1973, the WP is a grants and contributions program that was created in response to a recommendation of the 1970 Royal Commission on the Status of Women.¹ Originally administered by the former Department of the Secretary of State, the WP has been housed within SWC since 1995. The WP is a permanent program, with renewal of its Terms and Conditions occurring every five years. The Terms and Conditions that are the focus of this evaluation came into effect in 2006-07. The WP's current mandate is to advance equality for women across Canada through improvement of their economic and social conditions and their participation in democratic life. Its objective is to achieve the full participation of women in the economic, social and democratic life of Canada.

The WP provides both funding and technical assistance to funded organizations at the local, regional and national levels. The program has two funding components:

The Women's Community Fund (WCF) accounts for the majority of spending, and seeks to address the economic and social situation of women and their participation in democratic life through projects that directly support women in their communities.
The Women's Partnership Fund (WPF) has the objective of building partnerships between SWC, eligible non-governmental organizations, public institutions and the private sector through collaborative projects that address the economic and social situation of women and their participation in democratic life.

Funded projects vary considerably in nature and scope and are undertaken by a variety of different proponents, address a range of issues and apply a variety of strategies to achieve their objectives. As noted above, the projects are designed to support women in their efforts to increase their awareness, knowledge and skills to identify and address barriers to their participation in society. The program does not provide core funding to organizations, nor does it cover ongoing administration costs. The maximum financial support that can be provided to an eligible recipient is $500,000 per fiscal year. The majority of the projects are funded through grants. Project duration can be up to a maximum of 36 months.

Those eligible for funding from the WP are incorporated not-for-profit and for-profit Canadian organizations whose mandates are consistent with the objective of the WP. The WP does not provide funding to individuals, unions, governments or government agencies, educational institutions, cooperatives or organizations mandated by governments (e.g., school boards, hospitals).

In 2006-07, applications for funding to WP were received through a process of continuous intake of proposals. This shifted to periodic general calls for proposals for the WCF in 2007-2008 (there were two calls during this year and annual calls in subsequent years). The WPF continued to be delivered through the continuous intake model. Note that in 2010-2011, the program revived the continuous intake model for the WCF.

The WP sets priorities for funding annually.² For some calls for proposals, the Program has also identified specific target populations in addition to funding priorities. In 2009-2010 and 2010-11, the Program's priority issues included: 1) ending violence against women and girls; 2) improving women's and girls' economic security and prosperity; and 3) encouraging women and girls in leadership and decision-making roles.

In 2007-08, the target populations were: Aboriginal women; immigrant women; visible minority women; and senior women. From 2008-2009 to 2009-2010, the target population was all women and girls. In 2010-11, for projects addressing the issue of violence against women and girls, the target populations were: 1) women and girls living in remote areas; 2) Aboriginal women and girls; and 3) immigrant women and girls.

c) Program Activity

During the first four years of the current program cycle, 398 projects were funded. Projects funded in 2006-2007 were granted under the Women's Program (WP) – prior to the creation of the WCF and WPF. Projects between 2007-2008 and 2009-2010 were funded primarily by the WCF (335 projects). There were 31 projects funded under the WPF. The average agreement value during this time was approximately $180,000.

**Table 1.1: Number of Funded Projects by Year**
	2006-2007	2007-2008	2008-2009	2009-2010	Total
Total	32	180	103	83	398
WP	32	-	-	-	32
WCF	-	166	91	78	335
WPF	-	14	12	5	31

With respect to the program's key priority issues, 45 per cent of projects addressed violence against women and girls, 40 per cent of projects were funded under the economic security priority and leadership was addressed by 10 per cent of projects (the remaining 5 per cent of projects addressed the WP priority area of health (2006-2007) that was named as a priority area for one of the Calls. Nineteen per cent of WCF funded projects were targeted to immigrant and refugee women, while thirteen per cent of projects were targeted to Aboriginal women. One in ten funded projects each targeted women who have experienced abuse, marginalized women or young women/girls.

d) Structure and Governance

Overall accountability for Program implementation resides with the SWC Coordinator. Reporting to the SWC Coordinator, the Director General of the WP is responsible for the delivery, performance and accountability of the Program.

Delivery of the WP is decentralized. Since 2007, the program has been delivered via a national office and four regional offices: Atlantic; Quebec/Nunavut; Ontario; and West/NWT/Yukon. Proposals for projects that are local, regional or provincial/territorial in nature are assessed for eligibility by the regional offices, while proposals for projects that are national in scope are assessed by the national office. All eligible project proposals are then considered against a comprehensive list of factors included in the project assessment guidelines. Final approval of applications rests with the Minister of Status of Women Canada. Of the 398 projects funded between 2006-2007 and 2009-2010, one-third were funded in the West/NWT/Yukon (though only a handful of these were led by organizations in the North). Approximately one-fifth were funded in Ontario and Quebec/Nunavut, respectively. Just over one-sixth were funded in the Atlantic region, and just under one in ten projects were national in scope.

e) Program Resources

Since 2008-09, the salary and operating costs of the program have remained stable at about $2.7 to $2.9 million per year. The WP annual budget is $18.95 million (grants and contributions), however the annual budget fluctuated during this period between approximately $17 and $24 million annually, due to various reprofiling exercises.

**Table 1.2: WP Total Annual Budget**
In $ millions	2007-2008	2008-2009	2009-2010	2010-2011	2011-2012
Total	19.15	27.05	22.85	22.75	21.65
Salary and operating costs	2.0	2.9	2.9	2.8	2.7
Grants	13.45	17.55	14.75	14.75	14.75
Contributions	3.7	6.6	5.2	5.2	4.2

Source: 2007-08 to 2010-11 Public Accounts and 2011-12 Main Estimates

f) Logic Model

A logic model is a visual representation of a program that identifies the linkages between a program's activities and the achievement of its outcomes. The Women's Program's logic model, presented in Figure 1, provides a graphical depiction of how the activities and outputs of the program relate to immediate, intermediate, and ultimate outcomes. Specifically, the intended immediate outcomes of the Program include:

Increased awareness among women in identifying and/or removing barriers to their participation in their communities;
[For the Women's Partnership Fund] Increased partnerships with other federal departments, levels of government, NGOs and the private sector.

The intermediate outcome for the Program is increased participation of women in their communities. The ultimate outcome of the program is equality for women and their full participation in the economic, social and democratic life of Canadian society.

Logic Model - Women's Program

Logic Model - Women's Program
Long description

2. Scope of the Evaluation and Methodological Approach

2.1 Evaluation Objectives and Issues

In the 2010–2011 Agency Report on Plans and Priorities, completing a summative evaluation of the Women's Program was identified as one of three management priorities for the SWC. The evaluation assesses the relevance of the program, program performance, as well as whether WP is delivered in an efficient and economical manner. Potential program improvements or alternative models that could enhance the achievement of outcomes are also examined. Furthermore, since the last evaluation of the program carried out in 2005, the WP has undergone significant changes and this evaluation will seek to assess whether the Program, in its current form, is meeting policy objectives. The evaluation covers the time period between 2006-07 and 2010-11. The Evaluation Framework is included in Appendix A.

2.2 Lines of Evidence

a) Document, Program Data and Literature Review

Document Review: Secondary sources were reviewed to develop a detailed profile of the WP and to contribute to addressing several of the evaluation questions including relevance, design and delivery, and program performance. SWC provided documents such as: program Terms and Conditions, program management documents (e.g., calls for proposals, application guides, and proposal assessment tools), internal analyses and performance reports and corporate/policy documents). A document review template was developed to summarize findings in the documents pertaining to each of the evaluation questions.

Program Data Review: SWC provided EKOS with a list of all projects approved for funding since September 2006 up to 2009-10. The list contained 398 funded projects (once amendments and supplements were removed). Coding was undertaken to transform some of the data for the purposes of analysis. Contents of the data file included: administrative information (e.g., call number, fiscal year, agreement number); proponent profile information (e.g., type of organization); financial data (e.g., funding component, amount of funding); and project information (e.g., priority area, project activities).

b) Review of Project Files

A stratified random sample of project files from SWC headquarters and regional offices was reviewed. In total, 40 files were reviewed, roughly 10 per cent of the projects funded during the period under study. Projects were sampled from the program's "master list" of funded projects, with a focus on completed projects (78 per cent) and active multi-year projects (22 per cent) in order to best document project outcomes. The selection of files further considered the funding component, region and year of the agreement. Selected files were profiled to ensure representativeness with the population (according to variables such as issue focus, dollar value of the agreement, primary target group). A profile of the characteristics of the files that were reviewed is included in Appendix C (under separate cover).

The project files were reviewed to obtain information on program design and delivery elements such as project profile data, project outputs and outcomes, leveraging of resources and sustainability. An electronic data input form, presented in Appendix D under separate cover, was used to capture file information in a consistent manner.

c) Key Informant Interviews

In total, 16 key informant interviews were completed with program managers and staff (6), partners (7) and external experts (3). The master key informant interview guide is included in Appendix E (under separate cover). This guide was further customized for each respondent group. All key informants were notified by SWC of the evaluation and invited to participate in the study. Key informants were then contacted by the consultant to schedule an interview and provided a copy of the interview guide prior to the interview. While most interviews were conducted by telephone, some interviews with respondents in the National Capital Region were conducted in-person; all interviews were conducted in the key informant's official language of choice.

d) Online Survey of Program Applicants

The evaluation included an online survey of organizations that applied to the WP for funding during the period under study. The survey included a census of organizations that were funded by the WP between 2006-07 and 2009-10 (n=335). For non-funded applicants, a census of eligible organizations that have never been funded, or not been funded by the WP since 2006, was surveyed (n=131), focusing on the third and fourth calls for proposals. Non-funded applicants that were deemed ineligible or who submitted proposals outside the funding parameters of the WP were screened out of the sample. Some organizations had submitted multiple applications for funding over several proposal calls. These duplicates were removed and the most recent application for funding retained. A portion of the sample – 11 per cent – had outdated or invalid contact information that could not be updated.

The survey instrument was designed for administration with both the funded and non-funded groups. Skip logic and branching routed respondents to the modules of the survey appropriate to their level of experience with the program. The survey obtained the perspective of applicants on evaluation issues related to relevance, design and project-level outcomes. The survey instrument is included in Appendix F under separate cover.

The survey duration was from June 6 to July 21, 2011. In total, 150 funded applicants and 51 unfunded applicants completed the survey. The response rate to the survey was 49 per cent. A comparison of the survey sample and population of funded applicants on key indicators (e.g., region, year of funding) indicated representativeness of the sample.

e) Case Studies

Seven case studies of projects funded by the WP were conducted to obtain detailed qualitative and quantitative data on the extent to which funded projects achieved their expected outcomes and contributed to the intended immediate and intermediate outcomes of the WP. The case studies bring together and analyze data collected at the project level.

Each of the case studies was based on a review of the project file, a review of other project or organizational documentation; and key informant interviews with project proponents, partners and key stakeholders (the precise number of key informants determined on a case-by-case basis). The case study interview guides are provided, under separate cover, in Appendix G.

WP management, in consultation with evaluation committee members and the consultant, selected projects to illustrate different approaches that are represented in WP's funding activity (as well as region and pillar of activity). Case study summaries are included in Appendix H under separate cover.

f) Comparative Program Analysis

In addition to Agency and program-related documents, the document review component also included a review of program materials from federal grants and contribution programs with similar characteristics to the WP. Six programs were identified.³ For each of these programs, publicly available information was reviewed (e.g., program web-site, departmental planning and performance documents, program evaluation/audits) to compare programming aspects such as proposal solicitation and eligibility criteria, and to identify areas where the WP complements and/or overlaps with other federal programs. Interviews were also conducted with program staff from comparative programs.

g) Cost-effectiveness Analysis

An analysis of the efficiency of the program was undertaken, including an examination of utilization of resources (including leveraged resources) and the administrative to program dollar ratio of WP (annually). This analysis was supplemented by qualitative feedback obtained in the key informant interviews about efficiency (and where there are opportunities for efficiency gains). The subjective assessments of key informants were also considered with respect to the issue of economy. Efficiency and economy were also examined at the project-level by SWC using a case study approach focused on a cluster of projects to address women in non-traditional occupations. The results of this internal exercise have been incorporated in this report.

2.3 Limitations

A number of methodological caveats should be noted for this evaluation. First, much of the information that was gathered for this review is drawn from internal sources – program documentation and administrative data, as well as interviews or surveys with program managers and project proponents who have a clear stake in the program. This potential bias was mitigated to some extent by including external experts as interview respondents and unfunded applicants in the online survey.

There was limited information available directly from the ultimate client group for the program – women themselves. The WP funds external organizations to undertake project-based work, and reporting on results for program participants is the responsibility of the funding recipient. Contact information for project participants is not supplied to the program. Thus, impacts for individual women are limited to those indicated by project proponents (in project reports or the survey).

Determining the performance of the program in achieving its intended intermediate outcomes – increased participation of women in their community – was difficult. The survey and case studies provide some evidence in this area; however, the vast number of highly disparate funded projects made it difficult to achieve a succinct roll-up to the program level.

Other limitations of the individual lines of evidence include:

Literature review: The literature review proved challenging in terms of scoping the boundaries of the review. Obtaining and reviewing literature on indicators of gender equality were straightforward and benefited from the release of Statistics Canada's Women in Canada series in early 2011. However, the literature review was also to contribute to the determination of successful WP project delivery models. This was more difficult as there are many different models used by WP projects and the literature was not readily available or sufficiently synthesized to determine best practice with the resources allocated to this component.
File review: In total, 40 WP project files were reviewed, or about 10 per cent of projects funded during the study period. The selection of files over-represented closed projects, but were chosen to broadly represent key criteria such as region, year of funding, funding stream and program pillar. Given the number of files that were reviewed, the sample may not be representative in terms of other unexamined project characteristics.
Key informant interviews: Although the key informants contacted for interviews represented a cross-section of internal and external stakeholders, the number of key informants who were contacted for interviews was somewhat low – 16 individuals.
Survey of applicants: The survey of applicants (funded and unfunded) used industry standard methods for survey administration, including email and telephone reminders and online and telephone assistance to support respondents. However, the initial sample contained a number of entries with inaccurate or out-dated email addresses or the primary project contact had left the organization. The response rate to the survey was 50 per cent, but given the finite initial pool of potential respondents, the final sample size for the survey was limited.
Case studies: The seven case studies were selected to represent the diversity of WP-funded projects across a number of criteria – region, size of grant, and delivery model – and drew on a review of project documentation and feedback from project proponents, partners and beneficiaries. Some case studies had less success in contacting partners and beneficiaries due to elapsed time since the end of the project or difficulty in identifying appropriate respondents. Given the diversity of project activity, case study findings may not be generalizable to the WP overall.

The limitations identified above are common ones for program evaluations of social programs. Like other studies, this evaluation uses multiple lines of evidence to address the weaknesses of individual methodologies. This process of triangulation strengthens the evidence to answer the evaluation questions. However, it should be noted that the determination of progress in achieving broad social objectives such as gender equality is difficult. This is due, in part, to the complex and long-term nature of the social change process, and the significant contribution required in other arenas (legislation, policy) and by other jurisdictions and sectors to achieve these objectives.

3. Findings

This chapter presents the findings of this evaluation of the WP, presented by evaluation issue (relevance and performance) and by the related evaluation questions.

3.1 Relevance

The evaluation examined the continued need for the WP, the alignment of the program with federal and departmental priorities, and the extent to which the WP is consistent with federal roles and responsibilities.

a) Ongoing need for the WP

Evaluation Question:

1. Is there an ongoing need for the WP to address the status and equality of women in Canada?

Overall findings:

The evaluation evidence indicates that there is an ongoing need to address gender equality. While there have been significant strides toward gender equality in Canada, inequalities remain in a number of areas. The program's priority areas – economic security and prosperity, violence and leadership – are supported by national data and are also consistent with those of other jurisdictions and governments internationally. Over the program cycle, demand for the program has been high, owing in part to a dearth of funding alternatives for programs targeted to women/gender equality.

The evaluation evidence supports the continued need to address the status and equality of women in Canada. National data and international measures indicate that while there has been progress on many indicators (e.g., women's participation in the workforce and post-secondary education), gender equality has not been achieved in Canada. The World Economic Forum (WEF), for example, tracks the gender gap in over a 100 countries using four key measures of gender parity: education, health, economics and politics. In its 2010 Global Gender Gap Report⁴, gender parity in Canada was found to exist in the areas of education and health. However, economics and politics represent two areas in which gaps in gender parity continue to exist in Canada. Based on these indicators, Canada ranked 20^th out of 134 countries - a marked improvement from the 2008 Report in which Canada was ranked 31^st out 130 countries, but a regression compared to Canada's 2006 ranking of 14^th out of 115 countries.

With respect to the WP's three pillar areas of economic security, violence and leadership, the literature indicates:

Economic security and prosperity. Employment and entrepreneurial activity among women has increased over the last five years. However, while average total income for Canadian women has increased at almost twice the pace as it did for men between 2000 and 2008, women continue to have lower absolute income levels.⁵ In 2008, women employed on a full-time, full-year basis earned about 71 cents for each dollar earned by their male counterparts, an earnings ratio that has remained relatively stable since 1999. Part of the difference in earnings for women and men is attributed to the number of hours worked⁶, as well as the gender-based distribution of occupations in which the majority of employed women continue to work in traditional and typically lower paying jobs and sectors. Women are also disproportionately living in poverty compared to the Canadian population as a whole. Poverty is more prevalent among female lone parent families, women with disabilities, racialized women and recent immigrants. Women living in poverty face intensified negative effects from both the ongoing economic restructuring and the recent downturn.⁷
Violence. According to the 2009 Statistics Canada General Social Survey, overall rates of self-reported spousal violence have decreased over the past decade. While women and men were equally likely to report some form of spousal violence (physical or sexual), spousal violence against women is more likely to be of a more serious nature. The proportion of Aboriginal women who reported spousal violence was double that of non-Aboriginal women, and Aboriginal women were also more likely to report more serious forms of spousal violence.⁸ A 2008 study on child and youth victims of violent crime found reported rates of violence were slightly higher for girls than boys across age categories (with the exception of adolescents 9 to 12 years of age), primarily due to their higher rates of sexual violence.⁹
Leadership. The World Economic Forum (WEF) measures the extent of female political empowerment in a given country using three indicators: women in parliament; women in ministerial positions; and years with a female head of state. In 2010¹⁰, Canada ranked 36 out of 134 countries in gender political empowerment. Similar research in Canada by Catalyst Inc. shows that after sharp gains in the last 30 years, women's progress on the electoral front has stalled at about 20 per cent of electoral seats, and women legislators tend to be underrepresented in the most senior positions. Similarly, although women account for almost half of the Canadian labour force and occupy over one-third of management occupations, they still experience a significant level of gender disparity in accessing senior management positions in FP500 companies. Women are also more vulnerable to job loss during company downsizing or closure than their male peers¹¹.

Often referencing these statistics, evaluation key informants across all respondent categories agreed that there is continued need for interventions to advance gender equality. This sentiment was echoed by surveyed program applicants – 96 per cent indicated that "there is an ongoing need for a federal initiative such as the Women's Program to fund projects to promote gender equality and the full participation of women in the economic, social, cultural and political life of the country" to a great extent (responded 5 on a 5-point scale). Like key informants, program applicants pointed to persistent barriers to equality (e.g., violence, poverty, underrepresentation in political office, business and the media) experienced by women and the need for a long-term commitment to solutions.

A second indicator of ongoing need is demand for and utilization of the program. Based on the number of applications for funding received and projects approved, demand for WP funding has been strong, and grants and contribution funding for the five-year program cycle has been fully committed.¹². Since 2007-2008 (and the implementation of the program's revised Terms and Conditions and call for proposals process¹³), the number of applications received annually increased dramatically, while the proportion of applications funded declined substantially. In between 2007-08 (the first full year of the program's new Terms and Conditions) and 2009-10, 26 per cent of applications received were approved. Despite the relatively low success rate, 80 per cent of surveyed applicants indicated that they plan to apply again for WP funding.

Table 3.1: Total WP Applications and Projects Funded — 2007-2008 to 2009-2010

Total WP Applications and Projects Funded - 2007-2008 to 2009-2010
Long Description

Program data. List of Applications Received and Projects Approved; compiled March 2010
EKOS Research Associates Inc.

b) Consistency with Agency Strategic Outcomes and Federal Priorities

Evaluation Questions:

2. Are the mandate, objectives, and programming elements of the WP consistent with SWC strategic outcomes and federal government priorities?

Overall findings:

The program is consistent with SWC strategic priorities (the program and the Agency share the same strategic/ultimate outcome of equality for women). The WP aligns with federal priorities, particularly those related to economic prosperity, and the program also supports complementary federal strategies (e.g., the Family Violence Initiative) and international commitments related to gender equality.

In addition to addressing persistent areas of gender inequality such as violence and economic security, other emerging needs were identified in the evaluation. Examples include: trafficking, sexual exploitation/hypersexualization of women and girls, immigrant and Aboriginal women. Given the breadth of the program's objectives and finite funds, a more definitive niche for the program was perceived to be important to drive more focused funding efforts for greater impact. This program niche would leverage federal strength and fund in areas where levers in other federal departments are absent.

Consistency with Agency Strategic Outcomes

The Agency's Program Activity Architecture (PAA) serves to provide an overview of how the programs and activities of SWC are linked and how their expected results are organized to contribute to achieving the outcomes and mandate of the SWC. SWC identifies two program activities in its PAA. The WP contributes to the second program activity which is women's participation in Canadian society and its expected result is "increased participation of women in their communities." In the immediate-term, the results of this program activity are measured by increased awareness of women to identify and remove barriers to their full participation in their communities.

The overall mandate and objectives of the WP are well-aligned with the SWC strategic outcome. The strategic outcome of SWC is "Equality for women and their full participation in the economic, social and democratic life of Canada", which is also the ultimate outcome of the WP.

Consistency with Federal Priorities

Program documents and key informant respondents indicate that the WP is aligned with federal government priorities in the following ways:

At the highest level the program is aligned with the Canadian Charter of Rights and Freedoms – Section 15 – Equality Rights;
The WP is aligned with the Government of Canada outcome: Economic Affairs (income security and employment for Canadians). This outcome area involves "Program activities (which) aim to improve the financial situation of Canadians and to provide basic income support to the most vulnerable of Canadian society". Budget 2007 announced increased funding to SWC of $10 million annually. This announcement was included among the government's Investing in Canadians priority. More recently, the problem of violence against women and girls was mentioned in the 2011 Speech from the Throne;
The program supports a number of federal strategies such as the Family Violence Initiative and the federal strategy to deal with missing and murdered Indigenous women and girls; and
Canada has commitments and obligations under international agreements that imply a domestic commitment to gender equality such as the Beijing Declaration and Platform for Action on Gender Equality (1995), and the Convention on the Elimination of all Forms of Discrimination against Women (1979).

Gaps/Niche of the WP

The priority areas that have been identified for the program (i.e., the three pillars – violence, economic security and prosperity, and leadership) are widely believed by key informants across all respondent groups to continue to be "logical", "coherent", "a way to frame the dialogue". The priorities are also consistent with the way other organizations and jurisdictions provincially and internationally organize their understanding of gender equality. However, WP managers and staff key informants also noted some challenges with the priorities –they do not convey the extent of interconnectedness or intersectionality of the issues (e.g., that issues of violence are inextricably linked to women's economic security), and tend to represent the symptoms rather than root causes of gender inequality. The current areas of focus are also seen by program staff as being extremely broad. In recent targeted calls for proposals (Blueprint projects) efforts were made to focus on areas where more work needs to be done.

At the programming level, emerging priorities where the WP could focus are identified by the program in two ways: from the experience of the program itself (i.e., "organically, from the ground up" based on proposals that are received, networking and partnerships with community-based organizations); and through exchanges with the policy area within SWC – a relationship that program key informants note as having steadily improved.

Trafficking, sexual exploitation/sexualization of women and hypersexualization of girls are examples of emerging areas that are currently receiving attention within the program. Other emerging or pressing needs that could be addressed by the program that were mentioned by key informants include: work-family balance, violence against Aboriginal women, stereotyping, women and the environment, and women in rural or remote locations. Program applicants were also asked in the survey to identify any emerging issues or gaps in activities that the WP is particularly well-positioned to address in the next five years. Applicants identified a wide variety of issues and target groups, many reflecting areas where the WP is currently funding - e.g., economic security, violence, initiatives for immigrant and Aboriginal women.

**Table 3.2: Emerging Issues/Gaps: Program Applicants**
Are there any emerging issues or gaps in activities that you feel the Women's Program is particularly well–positioned to address in the next five years?	Top 10 Responses Per cent of Applicants (n=201)
Women's economic autonomy	17
Violence against women	16
Issues of gender equality (general)	13
Women's leadership and empowerment	11
Immigrant women	11
Women's participation in politics and public policy	11
Aboriginal women	10
Women who need mental health or addictions support	8
Employment and training for women	8
Research and policy development on women's issues	8

Source: Survey of Program Applicants, 2011

Program managers and staff were of the opinion that there is work to be done in better defining the niche of the WP, given its current breadth and finite funds. The program's previous Terms and Conditions focused on work with equality seeking organizations at the systemic level. The emphasis in the new Terms and Conditions for projects to have a direct impact on women was viewed as broadening the scope of the program, both in terms of potential proponent organizations and types of projects. A potential niche for the program was commonly described by program managers and staff as areas of programming where there is a void or vacuum. For example, according to these key informants, the program should be funding in areas where the provinces do not have jurisdiction or authority, or where there is an absence of federal levers in other departments. Areas such as women's leadership, gender-based analysis, awareness-raising around gender equality issues (being "the voice of equality", "women's voices") are examples of areas mentioned by program key informants where the program could play a 'value added' role.

c) Consistency with Federal Role and Responsibilities

Evaluation Questions:

3. Is the WP consistent with federal role and responsibilities?

Overall findings:

The federal level makes a relevant and important contribution to gender equality, though the federal government and the WP do not bear this responsibility solely. Canada's involvement in advancing gender equality is consistent with practices internationally. A caution is that program data indicate that the program funds many service delivery projects, an area that is traditionally the role of provinces.

While projects access complementary support from other funding sources (e.g., provinces, donations/fundraising), there were few concerns related to overlap or duplication.

Federal role and responsibilities

Key informants were uniformly of the opinion that the federal government has a role to play in gender equality. Advancing gender equality and achieving the full participation of women in Canadian society is perceived to fall within federal roles and responsibilities, though it is not solely the responsibility of SWC or the federal government. Stakeholders such as federal departments/agencies, other levels of government, non-governmental organizations and the private sector are also seen as having a key role to play in addressing the economic, social and cultural situation of women.

Canada's federal level involvement in gender equality programming is consistent with practices internationally. A review of programs and policies in selected countries (e.g., US, Australia, Finland) consistently indicated federal commitment to gender equality expressed in a variety of ways (e.g., National action plans, funding to women's organizations, federal government coordinating councils).

A caution is that the program data indicate that many projects funded under the program's Terms and Conditions focus on direct interventions with women (for example, skills development, and employment supports). Though many of these projects also include other activities such as increasing awareness, access to or improvements to services, this funding emphasis has frequently led the program into supporting service delivery projects, an area that is more typically within provincial roles and responsibilities. Program key informants raised this issue and supported a refocusing of programming to better emphasize the strength and opportunities offered by federal level engagement in the gender equality issue (e.g., multi-jurisdictional initiatives, awareness-raising, and knowledge building/knowledge transfer).

Overlap/duplication

All provinces and territories in Canada have women's directorates or women's offices. However, attention to the issue of gender equality at the provincial level was widely thought to be uneven across the provinces and territories and waning by key informants who are familiar with activities in these jurisdictions. It was noted among key informants and program applicants that there is generally a dearth of available funding in Canada to support projects that address the participation of women and gender equality specifically. However, there are a variety of funding sources (that may or may not have gender equality as their raison d'être) that are complementary to the WP. These programs provide funding for a specific target group (e.g., Aboriginal people) or issue (e.g., criminal justice).

Funded applicants most commonly identified the provincial government (31 per cent), donations/ fundraising (30 per cent) and non-governmental/philanthropic foundations or organizations (e.g., the Canadian Women's Foundation, the United Way) as complementary sources of funding for their project.

While there are other federal departments offering programming that touches on WP priorities (such as Canadian Heritage, Citizenship and Immigration Canada, Human Resources and Skills Development Canada and Aboriginal Affairs and Northern Development (AAND)), few project proponents indicated that they had received other federal funding for their project (15 per cent).

There are limited formal mechanisms for coordination of funding across federal departments or jurisdictions. At the regional level, there is some informal coordination between regional WP staff and provincial officers. Some regions have a more developed culture of collaboration than others, however, according to program key informants (and verified in the comments of provincial respondents) this contact has diminished over time due to regional capacity constraints.

3.2 Program Performance: Achievement of Intended Outcomes

This section presents the evaluation findings related to the achievement of the WP's intended outcomes. The emergence of promising best practices, as well as unintended outcomes are also discussed. WP performance was explored in key informant interviews, survey of applicants, and case studies, as well as the review of program documents and file review.

a) Immediate Outcomes

Evaluation Questions:

4. To what extent has the WP achieved or made progress toward achieving:

its planned immediate outcomes (Increased awareness among women to identify and/or remove barriers to their participation in their communities; and increased partnerships with other federal departments, levels of government, NGOs and the private sector)
its planned intermediate outcomes (Increased participation of women in their communities.)

Overall findings:

The program's immediate outcome of increasing women's awareness and removal of barriers to participation is measurable only by proxy through participation of women in the program's funded projects. Almost 400 projects were funded in the first four years of the program. The median number of women directly benefiting from projects is 125. There is good evidence that the program has been successful in stimulating partnerships with NGOs and other levels of government, while less so with other federal departments and the private sector. Projects leverage funding from partners – WP funds, on average about 60 per cent of project costs, with the proponent organization, and their partners funding the balance (including financial and in-kind contributions).

With respect to the program's intermediate program outcome – participation of women in their communities – there is some evidence of this impact, though it is difficult to "count" or provide a roll up of these impacts across the great variety of projects that are funded. The evaluation noted many examples of impacts across a spectrum of indicators including impacts on the individual women who participated, as well as projects that involved more far-reaching impacts on policy, institutional practices and services.

Awareness of Barriers

Awareness among women to identify and remove barriers to participation is expected to flow from the participation of women in WP funded projects. However, "awareness" is not generally measured at the project level and, therefore participation (i.e., number of women directly impacted by the project) is used as a proxy measure. As reported previously, 398 projects were funded under the WCF and WPF funding streams during the first four years of the program. According to funded applicants, nine in ten (91 per cent) projects included an awareness and outreach component (such as engagement of women in issues and identification of barriers), and the same proportion (91 per cent) included an intervention component directly with participants (involving, for example, education and training, workshops, community-based activities, or development of relationships and networks).

The nature of the barriers to women's participation that projects were designed to address was varied, and projects often addressed multiple barriers (i.e., intersectional in nature). According to funded applicants, projects most commonly addressed women's leadership or other skills, financial security/autonomy and lack of awareness/understanding of problems within the community (e.g., violence) (each barrier mentioned by one in four applicants).

**Table 3.3: Nature of Barriers Addressed by Projects**
What was the nature of the barriers to women's participation in their community that your project was designed to address?	Per cent of Funded Applicants (n=150) (multiple mentions possible)
Leadership/other skills among women to bring about change	25
Financial security/autonomy	23
Lack of awareness regarding problems within the community	23
Addressing family/multigenerational violence and ensuring proper supports for victims	19
Barriers to basic needs (food, housing, transportation, child care, health care, etc.)	14
Access to services, capacity, and availability	13
Discrimination/marginalisation/oppression	13
Lack of access to training, education, work experience, etc.	13
Cultural barriers	12
Community isolation	11
Other	25

Based on the survey data, the median number of women that projects' directly affected was 125.¹⁴ Across projects, there is a great deal of variation in reach, with some projects impacting small numbers of women (e.g., more intensive intervention projects with multi-barriered women) to other projects benefitting 3,000 women or more (e.g., projects that improve practices, service, or policy). This number was slightly higher among WPF projects, projects led by women's organizations or with prior program funding, projects with institutional level impacts, and projects that included development of a resource or tool. In the case studies, there were several examples of projects that obtained significant reach in terms of their impact. This occurred when projects were led by national or regional umbrella organizations or coalitions that were able to influence the practices or services of their member organizations and other networks for greater impact. Similarly, a monitoring exercise undertaken by the program in 2010 analyzed the outcomes of 13 WP projects and found that projects that impact individual women by addressing multiple facets of the barriers they are facing over the long term (addressing individual and systemic barriers) and projects that achieve structural improvements which, in turn, directly affect individual women have the most far-reaching impact (in terms of number of women impacted)¹⁵.

About half of funded applicants (49 per cent) reported that the number of women directly affected by their project was higher than expected at the time of their funding application. One in ten (10 per cent) project proponents indicated that the number of women directly affected by their project was lower than expected.

Funded applicants were asked to rate their project's contribution to WP immediate outcomes. Most (89 per cent) felt their project contributed to increased awareness among women in identifying and/or removing barriers to their participation in their communities to a great extent. This was confirmed in a review of a sub-set of WP project files where nine out of ten projects were judged to have increased awareness among women in identifying and/or removing barriers to their participation in their communities.¹⁶

Increased Partnerships

The development of partnerships was identified as an immediate intended outcome of the WPF only. However, because partnerships represent a key activity of the WCF also, and the WCF represents the majority of activity under the program, this immediate outcome is discussed for both funding components. Nine out of ten funded applicants (and all WPF project proponents) reported forming partnerships as part of their WP funded project. Of those projects that formed partnerships, nine in ten project proponents reported that all (15 per cent) or some (75 per cent) of the partnerships established for their project were with new partners. The vast majority of funded applicants felt that partnerships with other organizations have been important to the implementation of their project and achievement of project objectives (to a "great extent" by 82 per cent of WCF projects and 100 per cent of WPF projects).

Project partners were drawn from the community-based sector (88 per cent) more than from any other sector. Partnerships with other levels of government, particularly the provincial government, were common (42 per cent), while partnerships with other federal departments and the private sector were less frequent (15 and 11 per cent of funded applicants respectively). WPF projects were more apt to have partners from all these sectors (with the exception of the private sector) compared to WCF projects. Examples of other partners included Aboriginal organizations, educational institutions and health institutions.

The most common type of contribution from project partners, as reported by project proponents, was promoting awareness of the project (including recruitment of participants or providing participant referrals) (85 per cent). Projects also formed partnerships to share resources and materials, and to seek advice and expertise.

With respect to leveraging of resources, 91 per cent of funded applicants received other financial or in-kind assistance for their WP funded project. Of these, eight in ten project proponents (79 per cent) said that their own organization had contributed funding to the project. Six in ten surveyed funded applicants indicated that they had received funding from partners (financial or in-kind). Just over half of projects (54 per cent) leveraged support from volunteers and one-third from donations or fundraising (30 per cent).

The review of a sub-set of project files indicated that, on average, WP funding represented 61 per cent of the total cost of the project. While the number of WPF projects available for this file-based analysis was too small (n=5) to conduct a separate analysis, internal analyses conducted by the program indicates that, among WPF projects closing in 2008-09 and 2009-10 (n=11), financial and in-kind contributions were leveraged from partners in a better than 1:1 ratio with WP funds.¹⁷

Funded applicants were asked to rate their project's contribution to WP's immediate outcome of increased partnerships. Two-thirds of funded applicants (66 per cent) indicated that their project increased partnerships with other federal departments, levels of government, NGOs or the private sector to "a great extent". Again, the review of project files was generally congruent – based on projects' final reports, six in ten projects were judged to have met objectives in terms of increased partnerships.

Almost all funded applicants reported that they will continue to work with project partners on other initiatives or build on results achieved in their WP project. In fact, six in ten (61 per cent) indicated that they are already working with partners on opportunities beyond the funded project. Another almost two in five (37 per cent) anticipated that there will be opportunities to work with the newly established partners in the future.

b) Intermediate Outcomes

The intended intermediate outcome of the WP is the increased participation of women in their communities. "Participation in communities" implies a broad spectrum of measures given the diversity of the projects that are funded by the program. Final reports that were examined as part of the file review provide a variety of indicators of community participation. Examples include:

Economic security/independence: training leading to employment opportunities; a better understanding of job markets, job prospects in different sectors, work culture; women placed in jobs or going to school; support for women pursuing non-traditional occupations; support for women entrepreneurs;
Knowledge and awareness/skill development: information helpful for newcomer women to increase their awareness of Canadian laws, their rights, existing support services and programs; women equipped with reintegration skills;
Addressing violence: increased knowledge and skills to respond to sexual violence; women taking steps to increase their safety; improving access to resources/tools that are culturally relevant; increasing awareness of violence at the community level;
Leadership: development and support in the areas of corporate governance; Aboriginal cultural leadership; supporting women's participation on boards;
Empowerment/confidence: Increased confidence and pride among participants; acquired skills in the area of leadership and self-expression; and
Linkage with other services: increased women's knowledge about key issues that affect their involvement with the community; improved health and confidence in managing daily life; women encouraged to access and become connected to their community and existing resources and support structures.

Key informants also commented on the variety of forms of participation resulting from funded projects, noting that many projects resulted in very profound impacts on the women participating (e.g., leaving a violent relationship, obtaining positions of leadership in their community, securing additional education or long-term employment). Other projects were described as having far-reaching impacts for many women (e.g., implementation of pay equity provisions on a province-wide basis, providing access to culturally-relevant resources and tools to communities of women that had limited access to these resources, improving skills/practice guidelines among service delivery staff in shelters).

Funded applicants were asked to rate their project's contribution to WP intermediate outcomes in the survey. Most (82 per cent) felt their project contributed to increased participation of women in their communities to "a great extent" (responded 4 or 5 on a 5-point scale), again, consistent with the evidence in the review of project files.

c) Ultimate Outcome

Evaluation Questions:

4. To what extent has the WP achieved or made progress toward achieving:

c. its planned strategic outcome¹⁸ (Equality for women and their full participation in the economic, social and democratic life of Canadian society

Overall findings:

Key informants and project proponents were both cautious about the program's achievements in this area. Evaluation evidence to measure the program's performance on this measure was limited. There are also limitations on the program's potential impact given the finite amount of project-based funding available and the program's focus on a wide range of projects that are supported at the individual level. Still, there is evidence that many projects funded by the WP (six in ten) are at least moderately sustainable.

Progress toward outcomes

The program's planned strategic outcome is 'Equality for women and their full participation in the economic, social and democratic life of Canadian society'. By definition, the program's ultimate outcome is a longer-term endeavour, and one that is not the sole responsibility of WP. As noted earlier (Section 3.1(a)), though there has been progress on some indicators, gender equality in Canada has not been achieved. On many indicators (such as wage parity and representation in business and leadership positions) there has been a plateau.¹⁹ In addition, new issues emerge over time (e.g., difficulties resulting from the economic downturn, honour-based violence).

In general, both funded applicants and key informants felt less confident in speaking to the impact of the program on gender equality. Funded applicants were cautious when asked in the survey to rate their project's contribution to the WP strategic or ultimate outcome. Six in ten (58 per cent) felt their project has contributed to the WP planned strategic outcome of equality for women and their full participation in the economic, social, and democratic life of Canada. Similarly, according to key informants, while the program can show concrete results in terms of participation of women in their communities, progress on gender equality was described as "slow" and in "bite sizes". Projects that are sustainable and/or lead to changes within communities, organizations/institutions or policies were perceived to have greater reach and importance in contributing to gender equality.

According to program key informants, the challenge in achieving the program's ultimate outcome is also symptomatic of a disconnect in the program's causal chain which does not move seamlessly from the awareness and participation of individual women to the achievement of the broader social objective of gender equality. Program managers and staff were of the opinion that while increasing the participation of women is one way of affecting social change, the program could be more effective in achieving its ultimate intended outcome by focusing on more sustainable projects with greater reach.

Sustainability of Effort

Two-thirds of funded applicants (65 per cent) reported that their project was completed at the time of the survey. For those projects no longer receiving WP funding, more than half of projects have been sustainable in some manner – that is, the project continued after WP funding ended (nine per cent) or the project is continuing on a limited scale (48 per cent). Projects that included impacts at the level of institutional practice were more likely to report sustainability. This effect was illustrated in several of the case studies of projects that created tools or resources to influence the way organizations deliver services. Gender-based analysis was a common technique to encourage organizations to review and change their practices or delivery, to better meet the needs of women.

Funded applicants with projects that continued in some form after WP funding ended were asked to identify the factors that support their project's sustainability. Just over half of these respondents indicated the project efforts will be sustained because their project activities have been incorporated or influenced their organization's operations (55 per cent) (e.g., project activities or aspects of them integrated into other programming, visibility of the organization/partnerships leading to other opportunities, proponent making investments in other program areas as a result of lessons learned from WP project). A similar proportion indicated that individuals trained by the project will continue to conduct activities with clients and communities (53 per cent) or sustainability will be supported through the continued dissemination of the resources and tools that were developed (52 per cent). Four in ten said their project activities have affected their partner organization's operations (38 per cent), additional funding has been received (38 per cent), or resources/tools are being used in other settings (36 per cent). These findings are very similar to those that were reported in the final reports – sustainability of the project most often occurring through integrating the project into the operations of the proponent organization or initiating knowledge-sharing activities to inform others about lessons learned or practices developed from the project.

Projects that have limited sustainability often are more resource-intensive projects that deliver a service to women. In the case studies that illustrated this approach, sustainability of these projects was a challenge. The project proponents in these instances were relatively small organizations, dependent on project funding with limited ability to continue the intervention (e.g., workshops, discussion sessions) in the absence of ongoing funding (a type of funding WP does not have the authority to provide).

Impact of External factors

Interview respondents in all categories were invited to identify external factors that may have a positive or negative influence on the effectiveness of the WP. Positive factors tend to be attributed to unpredictable opportunities or serendipity that will open a window of opportunity for collaboration or a change in the broader environment that creates demand for solution (e.g., labour shortage in Alberta providing opportunities for women to enter non-traditional occupations). Factors that were identified as negatively affecting WP's efforts include, for example, some feeling in the broader community that gender equality has already been achieved (the "job is already done"), thus undermining the perceived legitimacy of program investments. Several key informants noted that the economic climate (downturn) can negatively influence the amount of public funding that is available to address social issues, or the effectiveness of some projects, particularly in the skills development/employment area (i.e., limited employment opportunities available to project graduates).

At the project level, the survey of program applicants and the review of project files indicate a number of other, more practical challenges in project delivery. These challenges were often related to the recruitment of the target group. For example, one in five funded applicants identified difficulties with reaching their target groups (19 per cent) as a challenge for their project.

One in five funded applicants also identified financial constraints (19 per cent) as a challenge (e.g., budgetary shortfall, for example, due to higher than expected demand or unanticipated costs, need for ongoing funding) and one in ten experienced difficulties in finding partners or establishing networks (13 per cent). Also commonly cited were logistical challenges, such as securing childcare or transportation for participants.

d) Delivery Models/Best Practices to Achieve WP Intended Outcomes

Evaluation Questions:

5. What models have emerged as best practices? How are they contributing to the achievement of WP outcomes? How can these models be used to inform future program design and delivery?

Overall findings:

There were a number of observations from the evaluation (e.g., from the literature, views of key informants and funded applicants) on promising practices. Examples such as partnerships, involvement of participants in delivery, culturally relevant programming, mentoring, and wraparound supports are being used by many projects already. However, the knowledge-building and knowledge sharing capacity within the program, including the identification and sharing of promising practices/proven resources and tools, is limited and identified as a gap by the program and stakeholders.

The WP funds a wide array of projects utilizing various common activities or approaches. Examples of these that have been used by projects include: culturally adapted workshops; peer mentoring. In addition, several models have been used: asset-based model (Sustainable Livelihoods); co-operative model; 'stages of change' model; 'theraplay' model and a 'grassroots models of service delivery'. The evaluation evidence could not definitively determine among the various approaches and models which would be considered best or promising practices to inform future program/project design and delivery.

Program managers and staff were uniformly of the opinion that identifying 'best' or 'promising' practices among the interventions that it funds and disseminating these broadly is a time-consuming endeavour and is currently limited due to capacity issues. The recent Blueprints call for proposals (February 2011) is an example of the program's efforts to build this knowledge base and knowledge transfer capacity. Blueprints is funding projects to develop and pilot tools or approaches to address various priority areas identified by the program (e.g., women in non-traditional occupations). The projects include an external evaluation of the effectiveness of the model used and sharing of results.

However, based on the experience of funded applicants (survey and case study results), the program's own internal analyses and the literature, some overarching programmatic best practices could be identified.

Partners/networking: The importance of partnered approaches was highlighted by many funded applicants – one in five funded applicants noted the importance of partner, network or champion support to extend the reach of the project (19 per cent) and another 13 per cent mentioned beneficial collaborations with partners to increase effectiveness. As noted elsewhere, these partnerships take many forms at the project level and with a variety of sectors (e.g., community agencies and governments). The project file review and case studies similarly highlighted the value of partnerships with other organizations and consultative approaches as a means to develop resources and carry out activities effectively. Partners could be a means of accessing participants, sharing the delivery of project activities, and ensuring knowledge transfer.
Involvement of participants in an active way: When asked to identify the best practices or lessons learned to improve projects to address barriers to women's participation in their communities, one-quarter (24 per cent) of funded applicants reinforced the program requirement to involve women themselves in the project at all stages (e.g., through peer support or peer advisory councils). This observation was echoed in the project files that were reviewed where projects often featured enhanced flexibility in delivery to adjust the approach/curriculum based on participant feedback. 'Hands on' types of activities or exercises with relevance to the lives of women were also noted as particularly successful.
Responsiveness to community needs/cultural sensitivity. One in five funded applicants noted the importance of the projects' responsiveness to community needs (19 per cent). There are many examples of projects where responsiveness has taken the form of culturally-sensitive or culturally-relevant interventions. Providing programming that is respectful of the diverse backgrounds of participants (sensitive to linguistic needs or incorporating cultural content, for example) can be successful both for individual level interventions and promoting community-level dialogue and community knowledge.
Removing or managing logistical barriers to women's participation. Both funded applicants in the survey and in the review of project files identified the importance of attending to practical considerations in fostering women's participation. This includes planning and resources to provide childcare, transportation, meals, and so on, as a means to fully reach an intended audience.
Mentoring approaches. The concept of women helping women, mentorship and the importance of community and mutual support is another common recurring theme that was identified in WP's internal analyses and is supported in the literature, as well. The literature review included several successful mentorship approaches that were used to attract young women or new entrants into traditionally male-dominated occupations or as a means of reducing or even overcoming the gender-based challenges faced by women in business.
Screening and assessment. In the case studies, there were several examples of projects that highlighted the importance of early screening of participants, particularly for projects offering more intensive interventions to multi-barriered women. These projects indicated the importance of the recruitment and selection process to ensure high levels of completion and success (e.g., committee based process, validation of employer support/other supports in place to ensure retention in the program).

Table 3.4: Best Practices: Funded Applicants
What best practices or lessons learned can be identified based on your organization's experience to improve the design or delivery of projects to address barriers to women's participation in their communities?	Per cent of Funded Applicants (n=150)
Ownership/involvement of participants in the project	24
Partners, networks, champions are helpful to expand the project's reach across the community	19
Programs need to respond to community needs/community-driven	19
Ensuring adequate supports are in place for participation (e.g., transportation, child care)	19
Collaboration with partners and clients can strengthen the project's impact	13
Long-term, sustainable programs are needed to address some issues	9
Other	23

e) Unintended Outcomes

Evaluation Questions:

6. What unintended outcomes has the WP had? What are the lessons learned?

Overall findings:

Unintended outcomes of the program were generally few, but largely positive. The experience at the project level has shown that many projects have garnered greater than anticipated participation and interest in their initiative by women, partners and the community.

Four in ten funded applicants did not mention any unintended outcomes of their project, while the remainder reported largely positive unintended outcomes of their project. One-quarter (25 per cent) reported greater than expected interest in the project, while one in five (20 per cent) noted new or improved partnerships. One in seven identified organizational improvements (15 per cent) or improved capacity and additional funding initiatives (14 per cent) as an unintended (or unexpected) outcome of their program.

Of the minority of projects that mentioned negative unintended or unexpected outcomes, the risks or difficulties of participating in interventions to the women themselves was raised in some project contexts (e.g., that family or community members may feel threatened by the project, or women's participation may raise troubling issues for them – reliving trauma associated with abuse, sexual exploitation).

3.3 Design and Delivery

Evaluation Questions:

7. Are Program design, delivery mechanisms and structures now in place appropriate and effective? What has been the overall impact of the ongoing transitional changes? What opportunities exist to improve the program design?

8. Is the level of access to program funds by potential recipients sufficient to achieve program outcomes?

9. Does the performance measurement strategy/framework allow for the capture of adequate and useful information for assessing Program impact? How could performance reporting be improved?

Overall findings:

Design and delivery mechanisms are generally appropriate, and the program has made adaptive changes to improve delivery. Satisfaction with the application process was only moderate among funded applicants who identified the potential to streamline, simplify and clarify the application process and improve timeliness. Most unfunded applicants did not feel adequately informed as to why their application was unsuccessful.

The changes in the program's Terms and Conditions, as well as operational funding reductions have had both positive and negative impacts. The current Terms and Conditions have served to ensure a more concrete focus on the tangible impacts of programming on women, and introduced many mainstream organizations to gender-based programming. However, operational funding reductions have created capacity constraints and placed increasing and changing demands on staff, who have limited access to support.

The key themes that emerged in terms of design improvements had to do with: improving the knowledge building/knowledge transfer capacity of the program; extending/reinforcing collaborations with partners and stakeholders; streamlining applications processes; and support to staff.

Some of the program's funding eligibility criteria for organizations and for eligible activities were seen to be overly restrictive and limiting access to program funding to achieve objectives. Indeed, the criteria were found to be quite variable across a number of federal grants and contributions programs that were examined. Incorporating an emphasis on sustainability of projects was raised as a way to direct funds for maximum impact.

Performance measurement has improved over the current program cycle, with new tools and processes introduced to better capture results. Like other programs with broad and longer-term social objectives, the program faces challenges in measuring and attributing impact. Funded applicants are generally satisfied with the expectations and their requirements for monitoring and reporting (their level of effort, performance measures, accessibility of staff). While there are undoubtedly improvements that could be made to the tools (e.g., better reporting on outcomes as opposed to outputs), performance measurement is more seriously hampered by human resource constraints to undertake the time consuming analysis and synthesis of information to inform program decision-making and strategic planning.

a) Appropriateness/Effectiveness of Program Design

Program Design

Key informants were asked to comment on a number of aspects of the program's design, notably: the structure of the program's funding streams; proposal solicitation and assessment; roles and responsibilities; and program capacity. With respect to the WCF/WPF funding streams, interviewed program key informants uniformly indicated that the two programming streams did not unfold the way they were intended. According to these respondents:

The WCF/WPF components were not sufficiently differentiated in terms of their programming target and intended outcome;
The design of the WPF was overly optimistic in its underlying assumption that funding partnerships could be forged among government funders in various jurisdictions (i.e., that the timing and priorities of funders/funding opportunities could be synchronized, which proved to be very difficult); and
The WPF presented a challenge for the skill set and capacity of staff. With the operational reductions to the program in 2006-07, remaining staff had limited time available to develop relationships/networks and the required social development skills to proactively engage stakeholders around the more complex WPF projects.

With respect to processes for promotion, proposal solicitation and assessment, program key informants noted that the program has limited capacity to conduct formal promotion of the WP to potential grantees. Surveyed program applicants were most often aware of project funding opportunities available through the WP because they had previously received funding (38 per cent). Other common sources of awareness included word of mouth, the WP website or through contact with the WP.

Satisfaction with the program application process among applicants was moderate: half of the applicants surveyed were satisfied with the overall process of applying for project funding. Considering various aspects of the application process, satisfaction with the guidance or support received from WP staff during the proposal process (among those who received it) and with the amount of funding awarded in relation to the amount requested (for most projects, amount received was similar to amount requested) was comparatively high (over seven in ten expressed satisfaction on these aspects). Weaker elements of the application process were level of effort required to prepare an application and the timeliness of learning the result of the funding decision which had lower satisfaction ratings.

**Table 3.5: Applicants' Experience Applying to the WP for Funding**
How satisfied were you with...?	Per cent of Applicants* (n=201)
The guidance or support received from WP staff during the proposal process (n=140)**	77
The amount of funding awarded in relation to amount requested	73
The clarity of the requirements to complete application	58
The eligibility criteria for funding	58
The ease of learning about funding opportunities under the WP	54
The level of effort involved to prepare application	42
The way the program defines and assesses risk as a criterion	38***
The timeliness of learning the result of the funding decision	38
The overall process of applying for project funding under the WP	51

* Represents per cent satisfied (rated 4 or 5 on a 5 point scale)
** Those who received guidance or support
*** 17 per cent of applicants responded "don't know" to this question

There is a gap in program communication with respect to unfunded applicants – most unfunded applicants surveyed felt that they were not adequately informed as to why their application for funding was not successful (only one in four felt adequately informed).

A key issue for program managers and staff with respect to proposal solicitation and application was the shift in the program's approach to proposal solicitation for WCF which began as a continuous intake process, shifted to call for proposals and now more recently has included a revival of the continuous intake model. In a Comparative Review of Funding Delivery Models conducted internally, a reported strength of the call for proposals model was the transparency of the approach and the consistency in the review and approval process. However, the lack of mechanism to pre-screen potential proposals or applicants resulted in a large numbers of applications and a lengthy review process (e.g., 4-6 month process to review and approve projects). The call for proposals model was also believed to represent a higher level of risk exposure given the volume of applicants and limited intelligence gathering/interaction between program staff and applicant organizations during the proposal process. Conversely, the strength of the continuous intake model was the short 4-6 week review and approval process. As well, workload – receipt of proposals, approvals, review of reports – was spread more evenly throughout the year.

Findings from the comparative program analysis conducted as part of this evaluation indicates that slightly more of the federal grants and contribution programs reviewed use a call for proposals or request for letters of interest approach than a continuous intake model (although one program reviewed uses both). Having experience with both types of processes, the current consensus view of program managers and staff interviewed is to favour a dual model – continuous intake of proposals to enable more proactive work with proponent organizations, together with strategic periodic calls for proposals.

With respect to roles and responsibilities and program capacity, program key informants raised a number of challenges, stemming largely from the changes in the program's Terms and Conditions and operational funding. These are described in more detail in the following section.

Program Transitions and their Impact

The WP Terms and Conditions and program administration changed significantly with program renewal in 2006-2007.²⁰ These changes were informed by a number of processes (the 2005-2006 Summative Evaluation of the WP, consultations with stakeholders and the recommendations of the Standing Committee on the Status of Women) and were influenced by reductions in SWC's operating budget in 2006-2007.

According to program managers and staff, the impact of changes to the program's Terms and Conditions has been mixed. On the positive side, the increased focus of programming to directly impact women has encouraged project proponents to think more concretely and to more effectively link their project activities to tangible results for women. Also positive, the extension of the eligibility criteria (beyond equality-seeking organizations) has broadened the pool of organizations with which the WP has contact to include mainstream organizations. Though these organizations may not have a formal mandate with respect to gender equality, their participation was perceived by the majority to extend the awareness of and attention to gender equality beyond women's groups (although the evaluation data indicate generally less reach and fewer sustainable impacts for projects led by mainstream groups).

Negative impacts of the program transitions identified by staff and management included:

The shift towards projects having a direct impact on women in their communities was widely seen to have been interpreted very narrowly during the early years of the program cycle and heavily curtailed the selection of projects for funding to individual level, service delivery types of interventions.
The extension of the eligibility criteria to include a broader pool of eligible organizations, coupled with the new call for proposals solicitation process created an enormous workload for WP staff. The number of applications for funding increased dramatically, but included a significant proportion of poor quality proposals and proposals falling outside the mandate of the program.
The reduction in program operational funding and the number of regional points of service created capacity challenges for the program:
- Regional delivery resources are now thinly spread. According to key informants, regional staff lack the time and resources to realize the benefits of a regional delivery structure (e.g., working closely with regional partners/ and potential applicants during the call for proposals process, managing project risk through relationship-building and monitoring, intelligence gathering/identifying emerging issues) or to serve remote areas such as the North.
- With operational resources focused on delivery, there is limited time for the program to "step back", analyze the type of projects funded and their value, with a view to informing decision-making and the effectiveness of the program. One of the tools that the program is lacking is an automated system to assist with program management (an improvement that was also recommended in the 2005 Summative Evaluation of the WP).
- The changes in Terms and Conditions and the switch to calls for proposal from continuous intake (and back again) have demanded new and changing skills from staff. Several program managers and staff noted that these capacity issues are further exacerbated by SWC's small agency status, making it difficult to access training or IT support.

Surveyed applicants were divided in their assessment of changes to the WP since 2006. The most positive change was seen to be the shift in the focus of the program from funding projects that foster institutional change to ones that have a direct impact on women; which was viewed as positive by about half of the applicants. Only one-quarter favoured replacing the continuous application process with an annual call for proposals (four in ten rated this change as negative). Similarly, the change in funding eligibility criteria was viewed positively by only one in five applicants (and viewed as negative by a similar number). Those who had previously received funding from the WP were more likely to see the change in eligibility criteria as negative.

The program's transitional changes appear to have had a moderate, though largely negative, impact on the level of service provided by WP staff. Among program applicants who had also applied to the program before 2006, two in five applicants noted a difference in the information, guidance or support they received from WP staff. Of those who noted a difference (n=80), the nature of the change was largely negative - nearly half noticed a decrease in ability to contact or receive support from staff (particularly among proponents located in the West), one in six noticed changes in funding and the application process (e.g. less funding available, more stringent eligibility criteria, tighter competition for funds), and one in ten commented on a deficit in the quality of informed or well-connected staff.

Design Improvements

A number of themes emerged with respect to suggested design changes to improve the effectiveness of the Program:

Enhance knowledge building/knowledge sharing capacity. A concern for many key informants is the absence of capacity within the program for conducting knowledge building/knowledge sharing activities to enhance the value of the piloted and tested tools, resources and models being developed by funded projects. This role is currently limited within the program due to human resource constraints and lack of program dollars or O&M support for this kind of activity. Key informants recommended design improvements (such as O&M resources) to support the program in its role as a knowledge broker. Mechanisms such as a 'clearinghouse' or virtual library are currently under development. Engagement of stakeholders and participation in conferences/forums/ networking was noted as another way to enhance knowledge sharing.
Extend collaboration. Program key informants saw an important role for the WP as a federal lead to engage and coordinate stakeholders in multi-jurisdictional, multi-stakeholder contexts. Provincial stakeholders interviewed concurred, recommending that the program work more closely and collaboratively with them to improve communications so provinces are "plugged in" to WP activities; opportunities to pool resources are identified; and WP is able to leverage the provincial office's knowledge of local issues and organization networks. Provincial non-governmental representatives also highlighted a gap in the area of research/policy analysis that the WP should address.
Streamlined application process. A number of funding applicants argued for the need for a simpler and clearer application process and clearer guidelines and criteria. This was echoed by several program staff as well. Funded applicants surveyed provided some additional suggestions for improvements to the application process, including additional support from SWC staff when completing their application (more common in the West) and additional feedback or follow-up from SWC after the application has been reviewed.
Support for staff. In addition to capacity issues within the program, there were a number of suggestions from program key informants around aligning staff skills to the evolution of the program. For instance, the continuous intake proposal process demands additional skills related to outreach to organizations, collaboration and project development. To be successful, these key informants suggested that changing expectations must be accompanied by guidance and support for staff to adjust and be successful in new roles.

b) Level of Access to Program Funds

As reported previously, the WP received 1400 applications between 2007-08 and 2009-10, and funding was approved for 26 per cent. A detailed application form must be completed by all applicants. The application form collects information on the organization, the project, budget, and anticipated impact and reach. The application form must be accompanied by a project proposal (five pages or less) and a project work plan, results framework and performance measurement plan. The proposal must also provide information on the project context, a justification for the project; detail the population to be reached and anticipated results of the project. All proposals (to both the WCF and WPF) are assessed following a multi-stage process that considers general eligibility requirements of the WP; the quality of the proposal; and level of risk.

Projects are not selected for funding for a variety of reasons: they fall below the program's rating threshold; the proposed project is judged to be outside the scope or mandate of WP, the organization or proposed activities are ineligible for funding by the WP; or deficiencies are evident in the proposal (e.g., duplicates other work, unable to secure other partners, unable to demonstrate need or justify funding request).

For projects that are approved, incrementality of WP funding appears strong. Very few funded applicants surveyed believe that their project would have proceeded in the absence of WP funding. More than three-quarters of surveyed funded applicants believe that their project would not have proceeded; while one in five believe it would have proceeded on a reduced scale. Consistent with this finding, close to half of the unfunded applicants surveyed indicated that their project did not proceed. Only one in six unfunded applicants surveyed indicated that their project proceeded although on a reduced scale (e.g., shorter duration, fewer participants or fewer activities), while only one in ten proceeded as planned.

Program managers and staff had two concerns about proponents' access to funds. First, program managers and staff felt that certain restrictions with respect to organization eligibility criteria (e.g., exclusion of academic/educational organizations and cooperatives) and activity eligibility criteria (e.g., awareness campaigns, research) were limiting the potential effectiveness of the WP, particularly in the area of knowledge sharing. The comparative analysis of other federal programs indicates that federal grants and contributions programs are not consistent in defining these eligibility criteria (i.e., other programs allow applications from academic institutions, cooperatives and other government agencies, and/or include broader categories of activities).

Second, some program key informants favoured the notion of sustainability of projects as a criterion for access to project funds; that is, moving away from projects that are 'one-off' or have very small numbers of participants, to focus more strategically on projects that have greater reach and the potential for ongoing or longer-term impact. As noted previously, projects that report impacts both at the level of individual women, as well as at the level of communities and institutional practice/policy, are more likely to be sustainable.

c) Performance Measurement

Following the 2005 Summative Evaluation, the WP logic model was revised and the program strengthened its commitment to results-based management (RBM). At the project level, for contribution funding, payments are normally based on achievement of performance objectives and/or reimbursement of costs incurred or expenditures made by the recipient. Grant funding typically entails fewer reporting requirements, however, WCF projects are also required to submit interim and final reports which demonstrate progress in project implementation. The recent Blueprints call for proposals (February 2011) – particularly because it involves testing models – incorporates a strong evaluation component (there is dedicated funding to external evaluation and sharing of results).

Performance measurement templates and tools available to the program include:

Results Framework and Performance Measurement Plan completed by project proponents as part of their application for funding.²¹
Monitoring and reporting templates for projects' Interim Report and Final Reports.²²
An Interim Report Review Form and Closeout Assessment Form completed by the WP officer.²³

Based on these project-level data sources, the program has produced some higher level analyses based on the closeout assessment forms. For example, the 2009-2010 Departmental Performance Report examined short-term outcomes for 22 projects organized according to the Program's three priority areas.

Most funded applicants provided positive ratings of the WP's performance measurement requirements and tools. Eight in 10 agreed that the level of project reporting required is reasonable, the required reporting format adequately reflects project activities, that WP staff were available to answer questions and provide expertise in addressing performance reporting questions, and that reporting requirements allow for the opportunity to demonstrate project impact. Of those who did not agree with the statements (n=12), the most common complaints were that reports were too complicated or required too much detail, or the reports took too much time or effort to prepare.

The consensus view of program managers and staff is that while monitoring and performance reporting have improved over time, there continue to be challenges in assessing the impacts of the program:

The program's logic model does not currently represent all program activities and outputs, and some internal key informants felt the statements related to intended program outcomes could be strengthened. An example is the immediate outcome "increase awareness among women in identifying/removing barriers to participating in their community". Projects do not typically measure 'awareness' of barriers to participation, though participation numbers are typically used as a proxy. However, "number of women directly impacted" is an output-based measure that does not adequately convey a project's success or link well to the program ultimate outcome of gender equality.
Evidence is often preliminary and anecdotal, or does not adequately convey the impacts of projects. The conduct of the case studies confirmed that tracing outcomes that may materialize in the longer-term (e.g., for individual women, take up and utilization of resources, changes in policy or practice) is a challenge for projects. Funding agreements do not typically allow for the time or expense to document these outcomes.
The program's ultimate outcome – gender equality for women – is very difficult to measure and to attribute changes to the efforts of the program (there are many other factors at play).
While the program has some tools and is obtaining a great deal of information from projects (e.g., closeout assessments, templates for final reports), the time available to extract and synthesize and understand the information is limited.

Funded applicants offered few suggestions to improve performance measurement and reporting. A minority of applicants – about one in ten or less – suggested more flexibility in reporting, or allowing for more customized reports, more contact with staff to help clarify requirements, or a more simplified and less repetitive reporting format. Key informants made suggestions for improvement to performance measurement that included: improve project-level reporting to obtain better information on how funded projects lead to longer-term or organizational change (i.e., emphasize reporting on outcomes as opposed to outputs); and analysis leading to more information on best practices and model approaches (e.g., who WP funds, for what models, and to what effect).

3.4 Efficiency and Economy

a) Program Costs

Evaluation Questions:

9. How does the cost of the WP delivery compare to other federal grants and contributions programs?

Overall findings:

The program's administrative efficiency ratio is reasonable – for every dollar of funding that is granted or awarded by the program, $0.13 is spent on program administration. Program comparators could not be identified.

A common indicator of program efficiency is administrative efficiency (the ratio of operational costs to program dollars), which can be computed for the WP grants and contribution program.²⁴ Note that program leveraging is often identified as an indicator of efficiency. This analysis may be found in Section 3.2(a).

The average annual administrative cost for the WP (administrative costs as a proportion of total program budget) has varied between 10.4 per cent and 12.7 per cent between 2007-2008 and 2011-2012 (average of 11.7 per cent).

**Table 3.6: Administrative and Program Costs, Administrative Efficiency Ratio**
In $ millions	2007-2008	2008-2009	2009-2010	2010-2011	2011-2012	Total
Salary and operating costs	2.0	2.9	2.9	2.8	2.7	13.3
Grants	13.45	17.55	14.75	14.75	14.75	75.3
Contributions	3.7	6.6	5.2	5.2	4.2	24.9
Total	19.15	27.05	22.85	22.75	21.65	113.45
Administrative costs as a proportion of total program costs (salary and operating costs/total program costs)	10.4%	10.7%	12.7%	12.3%	12.5%	11.7%
Administrative efficiency ratio (salary and operating costs as a proportion of grants and contribution dollars awarded)	$0.12	$0.12	$0.15	$0.14	$0.14	$0.13

Source: 2007-08 to 2010-11 Public Accounts and 2011-12 Main Estimates

With respect to an administrative efficiency ratio, on average, the administrative cost for program delivery is $0.13 for every program dollar allocated. The administrative efficiency ratio for the WP compared to other grants and contribution programs of a similar nature in government is difficult to ascertain. For example, of those programs selected as "comparator programs" for the comparison review component of the evaluation, none of these programs publish administrative efficiency ratios, nor were they presented in evaluations of these programs (where the reports were provided or could be located).

b) Program Efficiency and Economy

Evaluation Questions:

10. To what extent has economy and efficiency been demonstrated by the WP?

Overall findings:

A number of factors support the program's efficiency and economy: the program's lean operations and leveraging of resources (including expertise, networks, volunteers) from the community-based sector. The capacity issues of the program were perceived to be a drag on program efficiency – that is, the limited staff complement and supporting automated systems undermine the program's efficiency and economy.

Few alternative models or agencies are available to deliver on the WP's mandate. The WP is distinguished as a national program that addresses equality through engagement of multiple stakeholders in multi-jurisdictional contexts. Potential alternative program vehicles such as provincial or third party delivery were generally seen to have limited capacity or authority to deliver the program, and carry some additional risks and potential disadvantages (e.g., related to accountability and federal visibility).

Factors that promote/hinder efficiency and economy

Program managers and staff key informants were asked to indicate the extent to which WP program activities have been delivered in an efficient and cost-effective manner. The views of key informants on the efficiency of delivery at the program-level were generally positive. Among the key factors that were seen to be contributing to the program's efficient delivery were the program's lean operations. As well, key informants noted the leveraged investment that occurs at the community level through project-based funding. Community-based proponent organizations typically have limited administrative costs and can leverage resources from local partners, as well as contribute their own expertise, networks and volunteers.

At the project level, projects that were included in the case studies also indicated efficient delivery. One of the projects that was examined noted their successful leveraging of significant resources in addition to WP funding from sources such as the province, and foundation, industry association and employer partners. Other case studies projects pointed to their ability to obtain other (non-monetary) contributions such as the expertise of their organization and networks at a low cost in the community sector. Also, utilizing proponents' existing dissemination channels and reach was noted as a cost-effective way to raise awareness and broaden project results.

It should also be noted that some clusters of projects funded by the WP potentially derive a social return on investment. Internal analyses conducted by SWC using a cluster of projects that support women's participation in non-traditional occupations found a substantial return on project investments for this cluster. This analysis examined the program's investment in women's employment in terms of social returns derived from increased wages for women and returns to employers, which yielded estimates of returns in the order of 1:1.5 to 1:2.²⁵

The lean operation of the program, while providing low program costs, was noted by most program key informants as also a hindrance to the effective delivery of the program. Limited human resources and technological support are persistent challenges to deliver the program in a way that is strategic and derives full value from the experience, resources and tools developed by projects through knowledge sharing activities.

Alternative potential delivery agents/alignment with other program delivery models or best practices

The international literature indicates a wide variety of approaches in other jurisdictions to gender equality. In the US, for example, federal funding of programming in support of gender equality tends to be dispersed among government agencies and departments. However, since March 2009, these efforts are coordinated by the White House Council on Women and Girls. Australia, like Canada, has a Women's Office, housed within the federal Department of Families, Housing, Community Services and Indigenous Affairs (FaHCSIA) that is mandated to advance the status of women through programs, services, grants and funding. Like SWC, the Women's Office is the principal agency responsible for delivering women's programs and for consultations between the Australian government and women's organizations. These programs tend to deal with specific areas such as representation and policy-making, leadership and development, sport and violence against women. Australia (and other countries such as Finland) have funded national groups to take the lead in some areas. In both countries, strategic direction is provided by national action plans (e.g., Time for Action: The National Council's Plan for Australia to Reduce Violence against Women and their Children, 2009-2022, Australia). There is little evidence internationally, however, on the relative merits of these various approaches or their transferability to the Canadian context.

Alternative potential delivery agents or alternative models to the WP that were suggested by key informants included program delivery by provinces or by a third party organization. However, program alternatives were generally not seen to be equivalent in terms of infrastructure or programming breadth to the WP. As mentioned, provincial women's directorates/offices were not seen by key informants, including the provincial representatives themselves, as being in a position to undertake national initiatives, given the limited capacity of these offices and the absence of levers outside their own province. Delivery of the program by a third party organization (such as a community-based organization) was identified as a second potential alternative, though again, limited capacity and infrastructure of voluntary organizations represents a disadvantage of this approach. The program has also explored delivery through regional service delivery locations of other federal departments. SWC control and visibility, as well as accountability would be more limited for these alternatives (in comparison to an internally-delivered program).

Another possibility, suggested by some program managers and staff, would be to provide funding to a third party organization(s) with specific expertise to deliver a portion of WP grants in their area of expertise (and within a set of broader priorities and guidelines outlined by the program). This approach could ease internal capacity issues within the program and potentially contribute to program efficiency and effectiveness.

4. Conclusions

SWC's Women's Program was renewed in 2006-07 with revised program Terms and Conditions and a mandate to advance the equality of women and their participation in social, economic and democratic life. The program's funding cycle was completed in September, 2011 and, therefore, a summative evaluation of the program was identified as a priority in the SWC's 2010–2011 Agency Report on Plans and Priorities. The findings of the evaluation lead to the following broad conclusions about the relevance, design and delivery, and performance (effectiveness, efficiency and economy) of the Women's Program.

4.1 Relevance

The evaluation evidence indicates that there is an ongoing need to address gender equality. While there have been significant strides toward gender equality in Canada, including improvements on a number of indicators in the last five years, inequalities remain in a number of areas. The program's priority areas – – ending violence against women and girls, improving women's economic security and prosperity, and encouraging women and girls in leadership and decision making roles – are well-aligned with national data that show, for example, persistent issues of spousal and sexual violence against women, a gap in wage parity on the basis of gender, and an underrepresentation of women in senior political and business positions. These priorities are also consistent with those of other jurisdictions and governments internationally.
Over the program cycle, demand for the program has been high – owing, in part, to the expanded eligibility criteria under the Terms and Conditions of the program and a broader solicitation of applications through a call for proposals process. Also driving demand is a lack of funding programs targeted to women or with a gender equality mandate. The importance of the WP as a source of funding is supported in the reports of program applicants – the majority of unfunded applicants indicated that their project did not proceed as proposed without WP funding and most funded applicants did not perceive there to be ready access to funding alternatives if WP funding not been available to them.
The program is consistent with SWC strategic priorities (the program and the Agency share the same strategic/ultimate outcome of equality for women). The WP aligns with federal priorities, particularly those related to economic prosperity, and the program also supports complementary federal strategies (e.g., the Family Violence Initiative) and international commitments related to gender equality.
The federal level makes a relevant and important contribution to gender equality, though the federal government, SWC, and the WP does not bear this responsibility solely. Canada's involvement in advancing gender equality is consistent with practices internationally. A caution is that program data indicate that the program funds many service delivery projects, an area that is traditionally the role of provinces. While projects access complementary support from other funding sources (e.g., provinces, donations/fundraising), there were few concerns related to overlap or duplication.
In addition to addressing persistent areas of gender inequality such as violence and wage parity, other emerging needs were identified in the evaluation. Examples include trafficking and sexual exploitation/hypersexualization of women and girls. Immigrant and Aboriginal women were also identified as continuing priorities. Given the breadth of the program's objectives and finite funds, a more definitive niche for the program was perceived to be important to drive more focused funding efforts for greater impact. This program niche would leverage federal strength and fund in areas where levers in other federal departments are absent.

4.2 Performance

Effectiveness

The program's immediate outcome of increased awareness among women to identify and/or remove barriers to participation in their communities is measurable only by proxy through participation of women in the program's funded projects. Almost 400 projects were funded in the first four years of the program to address barriers related to leadership, economic security, violence, community awareness and access to services among others. The evaluation data indicated a wide distribution in the number of women impacted at the project level – from projects serving a handful of women to those impacting many thousands (with a median number of 125). Greater reach is evident among projects that are led by women's groups and that foster change in policy or institution- or sector-wide practices or services. Of note is that more often than not the number of women directly affected was greater than originally anticipated by the project.
There is good evidence that the program has been successful in stimulating partnerships with NGOs and other levels of government, while less so with other federal departments and the private sector. While an intended outcome for the WPF funding stream only, funded applicants from both the WCF and WPF streams report undertaking beneficial partnerships with these various sectors.
Projects also leverage funding from partners – the WP funds, on average, about 60 per cent of project costs, with proponent organization and their partners funding the balance (including financial and in-kind contributions).

With respect to the program's intermediate program outcome – participation of women in their communities – there is some evidence of this impact, though it is difficult to "count" or provide a roll up of these impacts across the great variety of projects that are funded. The evaluation noted many examples of impacts across a spectrum of indicators including impacts on the individual women who participated, as well as projects that involved more far-reaching impacts on policy, institutional practices and services.
The program's ultimate outcome – equality for women and their full participation in the economic, social and democratic life of Canadian society – is broadly stated, ambitious and difficult to attribute WP's contribution to its achievement. Key informants and project proponents were both cautious about the program's achievements in this area. Evaluation evidence to measure the program's performance on this measure was limited. There are also limitations on the program's potential impact given the finite amount of project-based funding available and the program's focus on a wide range of projects that are supported at the individual level.
Still, there is evidence that many projects funded by the WP (six in ten) are at least moderately sustainable. This longer-term impact occurs through ongoing availability of resources or tools, or continued delivery of aspects of the project by the proponent organization or their partners. Sustainability was reported somewhat more often by projects that resulted in change at the institutional level, were led by women's groups or leveraged existing relationships or networks.
There were a number of observations from the evaluation (e.g., from the literature, views of key informants and funded applicants) on promising practices. Examples such as partnerships, involvement of participants in delivery, culturally relevant programming, mentoring, and wraparound supports are being used by many projects already. However, the knowledge-building and knowledge sharing capacity within the program, including the identification and sharing of promising practices/proven resources and tools, is limited and identified as a gap by the program and stakeholders.
Unintended outcomes of the program were generally few, but largely positive. The experience at the project level has shown that many projects have garnered greater than anticipated participation and interest in their initiative by women, partners and the community.

Design and Delivery

The design of the WP changed significantly in 2006-07 and 2007-08, and program managers continue to make adaptive changes to improve delivery. The poorly distinguished WCF/WPF funding components are now obsolete. The program has also moved away from the general call for proposals process towards a dual approach of a continuous intake and targeted call for proposals mechanism to address difficulties with the management of periodic calls. Indeed, satisfaction with the application process was only moderate among funded applicants who identified the potential to streamline, simplify and clarify the application process and improve timeliness. While a majority of funded applicants did receive assistance from WP staff during the application process, many suggested additional support or feedback as a potential design improvement. Most unfunded applicants did not feel adequately informed as to why their application was unsuccessful.
The changes in the program's Terms and Conditions, as well as operational funding reductions have had both positive and negative impacts. During the period under study, projects incorporated a more concrete focus on the tangible impacts of their funded activities on women and introduced many mainstream organizations to gender-based programming (improvements valued by both the program and funded applicants). However, operational funding reductions have created capacity constraints and placed increasing and changing demands on staff who have limited access to support. The capacity issues have had negative repercussions in terms of responsiveness to funded applicants and engagement of partners and stakeholders (applicants in the West were particularly).
The key themes that emerged in terms of design improvements had to do with: improving the knowledge building/knowledge transfer capacity of the program; extending/reinforcing collaborations with partners and stakeholders; streamlining applications processes; and support to staff.
Some of the program's funding eligibility criteria for organizations and for eligible activities were seen to be overly restrictive and limiting access to program funding to achieve objectives. Indeed, the criteria were found to be quite variable across a number of federal grants and contributions programs that were examined. Incorporating an emphasis on sustainability of projects was raised as a way to direct funds for maximum impact.
Performance measurement has improved over the current program cycle, with new tools and processes introduced to better capture results. Like other programs with broad and longer-term social objectives, the program faces challenges in measuring and attributing impact. Funded applicants are generally satisfied with the expectations and their requirements for monitoring and reporting (their level of effort, performance measures, accessibility of staff). While there are undoubtedly improvements that could be made to the tools (e.g., greater focus on outcomes as opposed to outputs, linkages across the program's intended outcome statements), performance measurement is more seriously hampered by human resource constraints to undertake the time consuming analysis and synthesis of information to inform program decision-making and strategic planning.

Efficiency and Economy

The program's administrative efficiency ratio is reasonable – for every dollar of funding that is granted or awarded by the program, $0.13 is spent on program administration. A number of factors support the program's efficiency and economy: the program's lean operations and leveraging of resources (including expertise, networks, volunteers) from the community-based sector. The capacity issues of the program were perceived to be a drag on program efficiency – that is, the limited staff complement and supporting automated systems undermine the program's efficiency and economy.
Few alternative models or agencies are available to deliver on the WP's mandate. The WP is distinguished as a national program that addresses equality through engagement of multiple stakeholders in multi-jurisdictional contexts. Potential alternative program vehicles such as provincial or third party delivery were generally seen to have limited capacity or authority to deliver the program, and carry some additional risks and potential disadvantages (e.g., related to accountability and federal visibility).

5. Recommendations

Program Performance

1) In order to improve the overall effectiveness of the program in achieving its ultimate intended outcome of equality for women, the program should identify any necessary program design changes to fund activities that the evaluation found to have greater potential reach and sustainable impact. These evaluation results and the program's own internal analyses indicate that funded projects that include components that foster change at the level of policy or institutional practices have greater reach and sustainability. Funded projects that create resources or tools, or leverage proponents' existing regional or national networks/membership also have advantages in terms of reach and sustainability. While projects led by women's groups were also associated with greater reach and sustainability, any program design changes to the eligibility of organizations must be balanced against the benefits of introducing gender equality programming into mainstream organizations.

Design and Delivery

2) The program should clarify its theory of change from activities and outputs through to immediate, intermediate and ultimate outcomes. The program's current logic model does not adequately capture all activities carried out by the program, and the results chain does not reflect the program's evolving understanding of social change – i.e., based on a multi-dimensional approach that supports social change at multiple levels, including the participation of individual women, capacity and connections among organizations or networks of organizations, and change within communities and at the level of policy.
3) The WP should continue to initiate and strengthen relationship-building with provinces and other federal departments and agencies to seek input on program funding priorities. WP project activity in areas such as immigration and skills development to remove barriers to women's participation is diverting finite program funding to areas where other federal or provincial programs are in place and could address the participation of women. While other government departments are aware of the WP and have a favourable impression of the program, efforts moving forward would benefit from consultations with federal partners to ensure that WP's limited funding is deployed in areas where there is an absence of other levers to address barriers to women's participation.
4) The program should review deployment of its internal organizational resources to address significant capacity issues within the program, particularly a thinly-spread regional delivery structure. The program should continue its exploration of alternative delivery models or innovations with a view to addressing particular constraints in underserved regions (the West where applicant satisfaction levels tend to be lower and the North where there are few projects funded) (e.g., leveraging of provincial capacities, utilization of service delivery nodes in other federal departments, use of third party delivery for selected program focus areas).
5) The program should further leverage the project-level investments that it makes with enhanced mechanisms for knowledge building/sharing with and among stakeholders. Encouraging projects to share their results and embedding opportunities for dissemination and exchange at the program level would enhance the value of products developed by funded projects and keep the program funding focused on innovation. To support this:
1. Consideration should be given to adjusting the current funding envelope (e.g., O&M dollars made available) to permit activities to support dissemination/exchange activities, including via web-site postings, workshops/meetings or other mechanisms.
2. Review the program's Terms and Conditions to remove any restrictive eligibility criteria related to organization eligibility (e.g., academic organizations, cooperatives) and activity eligibility that inhibit undertaking knowledge building/sharing activities.

Appendix A Evaluation Framework

**Summative Evaluation of the Women's Program 2010 - Evaluation Framework**
Evaluation Issues and Questions	Indicators	Litera- ture Review	Docu- ment Review	File/ Data Review	KI Inter- views	Tar- geted Survey	Case study
Relevance
1. Is there an ongoing need for the WP to address the status and equality of women in Canada?	Changes or trends over 2006/07 to 2010/11 in indicators of gender equality Support for addressing equality issues through an instrument such as WP Evidence of program demand (e.g., applications, program funding spent)	X	X	X	X
2. Are the mandate, objectives, and programming elements of the WP consistent with SWC strategic outcomes and federal government priorities?	Degree of correspondence between mandate, objectives, and programming elements of the WP with SWC strategic outcomes Assessment of alignment between mandate, objectives, and programming elements of the WP with federal government priorities. Identification and analysis of gaps that could be addressed by the WP	X	X		X
3. Is the WP consistent with federal role and responsibilities?	Degree of correspondence between program mandate and federal jurisdiction, roles and responsibilities Degree or nature of overlap of federal role and responsibilities with roles/responsibilities of provincial orders of government. Analysis of other potential delivery agents/ alternative model, their advantages and disadvantages identified	X	X		X
Performance
4. To what extent has the WP achieved or made progress toward achieving: a. its planned immediate outcomes²⁶ (Increased awareness among women to identify and/or remove barriers to their participation in their communities. AND increased partnerships with other federal departments, levels of government, NGOs and the private sector)	Number and nature (including activities) of funded projects Reach of funded projects (# and profile of women participating) Types of barriers identified by projects Types of resources (outputs) produced that identify and address barriers to participation; perceived quality and usefulness Number and nature of funded projects that demonstrate increased awareness and knowledge of barriers to participation Assessment of leveraging of WP funds Number and nature of partnerships created: through Partnership Fund at the project level through the Community Fund		X	X	X	X	X
b. its planned intermediate outcomes²⁷ (Increased participation of women in their communities.)	Extent to which results achieved at project level with respect to women's participation have been achieved Changes in capacity to identify and address barriers Number and nature of partnerships created: through Partnership Fund at the project level through the Community Fund		X	X	X	X	X
c. its planned strategic outcome²⁸ (Equality for women and their full participation in the economic, social and democratic life of Canadian society.	Extent of contribution of Program to equality/ participation Assessment of sustainability of effort Extent of perceived impact of external factors (mitigating/enhancing)		X	X	X	X	X
5. What models have emerged as best practices? How are they contributing to the achievement of WP outcomes? How can these models be used to inform future program design and delivery?	Identification of best practices supported by evidence from peer- based and relevant government literature Analysis of factors of success (nature and reach) Cost benefit analysis of selected model	X	X	X	X	X	X
6. What unintended outcomes has the WP had? What are the lessons learned?	Identification and analysis of unintended outcomes and effects Analysis of impact on the WP and/or how dealt with		X	X	X	X	X
Design and Delivery
7. Are Program design, delivery mechanisms and structures now in place appropriate and effective? What has been the overall impact of the ongoing transitional changes? What opportunities exist to improve the program design?	Extent of change in impact / effectiveness of Program design and delivery on achievement of results Analysis of perceived impact of various factors related program design and delivery (all actors) Analysis of processes for promotion, proposal solicitation and assessment		X	X	X	X
8. Is the level of access to program funds by potential recipients sufficient to achieve program outcomes?	Analysis of project funding decisions (what is and what is not) Extent to which program design supports project/activity sustainability Degree to which eligibility criteria are appropriate measured against applicant Extent to which program delivery approach contributes to project/activity sustainability Degree to which internal selection criteria are appropriate measured against applicant Changes in recipient perspectives on project / program information, processes and mechanisms (e.g., quality, usefulness of program material; project performance measures, selection criteria, selection of funding instrument, project performance reporting requirements, program response times, etc.) Analysis of processes for promotion, proposal solicitation and assessment Assessment of ability of project applicants to secure other funding		X	X	X	X
9. Does the performance measurement strategy/ framework allow for the capture of adequate and useful information for assessing Program impact? How could performance reporting be improved?	Analysis of project recipients' capacity to respond to performance reporting requirements (performance measures, templates, interim and final activity reports and evaluations) Assessment of quality of program administration, information and reporting tools and processes Extent to which current performance mechanism supports program analysis, policy and management (decision making)		X	X	X	X
Efficiency and Economy
10. How does the cost of the WP delivery compare to other federal grants and contributions programs?	Analysis of WP administrative costs using those of similar federal programs including an assessment of mitigating factors (e.g., size, delivery of program, orientation)	X	X	X	X
11. To what extent has economy and efficiency been demonstrated by the WP?	Assessment of alignment with other program delivery models or best practices identified from the literature Analysis of perceptions of factors that promote/ hinder efficiency and economy	X	X	X	X

Appendix B Bibliography

Aruna Papp, "Culturally Driven Violence Against Women in Canada's Immigrant Communities", Frontier Centre for Public Policy Series No. 92, July 2010.

Bramham, Daphne. 2010. "Someday There Will Be No More "Firsts." Ottawa Citizen, Sept. 9. http://www.ottawacitizen.com/life/Someday+there+will+more+firsts/3786640/story.html

Burghardt, J., & Gordon, A. (1990). More jobs and higher pay: How an integrated program compares with traditional programs. New York: Rockefeller Foundation.

Canadian Centre for Justice Statistics. "Victimization and offending among the Aboriginal population in Canada", Jodi-Anne Brzozowski, Andrea Taylor-Butts and Sara Johnson. Juristat. Vol. 26, no. 3, 2006.

Canadian Centre for Policy Alternatives. "Women's Poverty and the Recession." Monica Townson. Sept. 2009
http://www.policyalternatives.ca/sites/default/files/uploads/publications/National_Office_Pubs
/2009/Womens_Poverty_in_the_Recession.pdf

Carter, Ph.D., Nancy M., and Christine Silva. Pipeline's Broken Promise. The Promise of Future Leadership: Highly Talented Employees in the Pipeline, Catalyst Inc.
http://www.catalyst.org/file/340/pipeline%27s_broken_promise_final_021710.pdf

------. 2009. Opportunity or Setback. High Potential Women and Men During Economic Crisis. Catalyst Inc.
http://www.catalyst.org/file/305/opportunity_or_setback_final_081209.pdf

Castaño, C. et. al. Policy towards Gender Equity in Science and Research, Seventh Framework Programme, October 2010.

Catalyst Research and Communications, Monitoring Mission. Final Report. March 31, 2010

Centre for Policy Alternatives. "Women's Poverty and the Recession", Townson, Monica. Ottawa: 2009.

CIBC Small Business. "Women Entrepreneurs: Leading the Charge, 2005" http://www.cibc.com/ca/pdf/women-entrepreneurs-en.pdf

Clark, M., & Huston, T. (1993). Assisting the smallest business: Assisting microenterprise development as a strategy for boosting poor communities. An interim report. Washington, DC: Self-Employment Learning Project, Aspen Institute for Humanistic Studies.

Cooper, M. et. al. Review Of Best Practices In Family And Sexual Violence Programming, Family and Sexual Violence Review Advisory Committee, City of Calgary, 2004.

de la Rey, C. C., Jankelowitz, G. G., & Suffla, S. S. (2003). Women's leadership programs in South Africa: a strategy for community intervention. Journal of Prevention & Intervention in the Community, 25(1), 49-64.

European Social Fund, Equality and Diversity Good Practice Guide: Gender, 2007

Ibarra, Herminia, Nancy M. Carter and Christine Silva. 2010. "Why Men Still Get More promotions than Women, Your high-potential females need more than just well-meaning mentors." Boston, Mass: Harvard Business School Publishing Corp. http://hbr.org/products/R1009F/R1009Fp4.pdf

Indian and Northern Affairs Canada. "Aboriginal Women: A Demographic, Social and Economic Profile". Ottawa: Summer 1996.

Industry Canada. "Key Small Business Statistics," July 2010
http://www.ic.gc.ca/eic/site/sbrp-rppe.nsf/vwapj/KSBS-PSRPE_July-Juillet2010_eng.pdf/$FILE/KSBS-PSRPE_July-Juillet2010_eng.pdf

Justice Canada. Research and Statistics Division. "Examination of Declining Intimate Partner Homicide Rates: A Literature Review." Dawson, M. 2001.
http://www.crvawc.ca/documents/Myrna%20Dawson%20- %20An%20Examination%20
of%20Declines%20in%20Intimate%20Partner%20Homicide%20Rates.pdf

Livingstone, Anne-Marie, Lucie Chagnon et al. From Poverty to Empowerment: A Research Report on Women and Community Development (CED) in Canada.
http://www.cdnwomen.org/PDFs/EN/CWF-frompoverty-report.pdf

Lugo, Nancy Rudner. Empowerment education: A case study of the Resource Sisters/Compañeras Program. Health Education Quarterly. Vol 23(3), Aug 1996, pp. 281-289

McNish, Jacquie and Janet McFarland. 2010. "Why the executive suite is the final frontier for women." Globe and Mail, Friday, Oct. 08, 2:43PM EDT.

Morris, Marika, et al. 2007. "Integrating the voices of low-income women into policy discussions on the Canada Social Transfer (CST)." Canadian Research Institute for the Advancement of Women. Fact Sheet #9.

Murray, Janet, et al. Women in Transition Out of Poverty: A guide to effective practice in promoting sustainable livelihoods. Women and Economic Development Consortium, January 2, 2002.
http://www.cdnwomen.org/PDFs/EN/CWF-WIT-guide.pdf

National Council on Welfare, Presentation to the Standing Committee on the Status of Women, June, 2006.

National Council on Welfare, release of 2006-2007 report, December, 2008.

Native Women's Association of Canada. 2009. Voices of Our Sisters In Spirit: A Report to Families and Communities. http://ywcacanada.ca/data/research_docs/00000022.pdf

Okagaki, A. (1989). Women and self-sufficiency: Programs that work, policy that might. Washington, DC: Corporation for Enterprise Development.

Raheim, S. S., & Bolden, J. J. (1995). Economic empowerment of low-income women through self-employment programs. AFFILIA: Journal of Women and Social Work, 10(2), 138-154.

Sanders, C. K., Weaver, T. L., & Schnabel, M. M. (2007). Economic education for battered women. AFFILIA: Journal of Women and Social Work, 22(3), 240-254.

Smith, E., Nowhere to Turn? Responding to Partner Violence Against Immigrant and Visible Minority Women, Canadian Council on Social Development , 2004.

Statistics Canada. "Women in Canada, A Gender-based Statistical Report." http://www.statcan.gc.ca/pub/89-503-x/89-503-x2010001-eng.htm

Statistics Canada, "Family Violence in Canada: A Statistical Profile, 2009".
http://www.statcan.gc.ca/pub/85-224-x/85-224-x2007000-eng.pdf

Statistics Canada. "National Trends in Intimate Partner Homicides, 1974-2000", Pottie-Bunge, V. 2002. Juristat. Vol. 22, no. 5. Catalogue no. 85-002-XPE. Ottawa.

Statistics Canada. Canadian Centre for Justice Statistics. "Family Violence in Canada:

A Statistical Report, 2007". http://www.phac-aspc.gc.ca/ncfv-cnivf/pdfs/fv-85-224-XIE2007001.pdf

Statistics Canada. 2003. "Canadians and their non-voting political activity." The Daily. June 19.
http://www.statcan.gc.ca/daily-quotidien/070619/tdq070619-eng.htm

Status of Women Canada. 2010. "2010-2011 Report on Plans and Priorities", http://www.tbs-sct.gc.ca/rpp/2010-2011/inst/csw/csw-eng.pdf

Status of Women Canada. The Honourable Rona Ambrose, Minister of Public Works and Government Services and Minister for Status of Women. "2010, November 25". Government of Canada calls for an end to violence against women [Press release].
http://www.swc-cfc.gc.ca/med/news-nouvelles/2010/1125-eng.html

Townson, Monica, "Women's Poverty and the Recession." Canadian Centre for Policy Alternatives, Sept. 2009, isbn 978-1-897569-72-6, p. 15

Trimble, Linda, and Jane Arscott. Still Counting: Women in Politics Across Canada, 2003. Alberta, Canada: Athabasca University. http://stillcounting.athabascau.ca/index.php

Tutty, L. et al., An Environmental Scan of Programs that Address Domestic and Sexual Violence: What's Working in Alberta?, 2005 Online: http://www.child.gov.ab.ca/home/documents/familyviolence/rpt_opfvb_DVScan_2005.pdf

Weiss, C. (1990). The role of intermediaries in strengthening women's self employment activities. In Local initiative for job creation: Enterprising women (pp. 59-74). Paris: Organisation for Economic Co-operation and Development.

World Economic Forum, The Global Gender Gap Report 2010, Switzerland, 2010.

YWCA Canada, Turning Point For Women Effective Practices In Sheltering Women Leaving Violence In Intimate Relationships, Phase II Report, 2006

1 The Commission called for the creation of a federal mechanism of support for the direct participation of Canadian women in efforts to advance women's equality.

2 Prior to 2007-08, funding priorities were referred to as 'areas of focus'

3 The comparative program analysis included: Public Safety Canada's National Crime Prevention Centre (NCPC), Community Development Program at Agriculture and Agri-Food Canada, Inter-Action (former Multiculturalism Program) at Citizenship and Immigration Canada, Aboriginal People's Program at Canadian Heritage, the Family Violence Initiative under the Justice Partnership and Innovation Fund at Department of Justice, and Social Development Partnerships Program – Disability at Human Resources and Skills Development Canada.

4 World Economic Forum, The Global Gender Gap Report 2010, Switzerland, 2010.

5 Statistics Canada, Women in Canada: A Gender-Based Statistical Report, Paid Labour chapter, 2011.

6 Even among full-time workers, women work fewer hours than their male counterparts attributed, in part, to childcare and/or other family responsibilities.

7 Townson, M., Women's Poverty and the Recession, Canadian Centre for Policy Alternatives, Sept. 2009.

8 Statistics Canada, Women in Canada: A Gender-Based Statistical Report, Women and the Criminal Justice System chapter, 2011.

9 Statistics Canada, Child and Youth Victims of Police-reported Violent Crime, 2008.

10 World Economic Forum, Global Gender Gap Report 2010, Switzerland, 2010.

11 Nancy M. Carter, Ph.D. and Christine Silva, High Potential Women and Men During Economic Crisis, from Catalyst Inc. The Promise of Future Leadership: A Research Program, August 2009 Talented Employees in the Pipeline, http://www.catalyst.org/file/305/opportunity_or_setback_final_081209.pdf

12 Note, however while all funds were committed, in some years the funding was not fully expended as the development of proposals took longer than expected which led to delays in submission and approval and several organizations were unable to spend their approved budgets.

13 Under continuous intake, interested applicants contacted Program staff as a first step, making a pre-proposal application. Staff would screen out ineligible organizations and projects, and records of rejected proposals were not maintained.

14 The median is the value that separates the responses in half (i.e., the middle value). Median was used, as opposed to mean due to the wide distribution in responses. The mean (350) was skewed upwards as a result of a small number of cases with very large values.

15 Catalyst Research and Communications, Monitoring Mission. Final Report. March 31, 2010

16 Note that the response categories between the funded applicant survey and review of project files are not strictly comparable (surveyed applicants rated their project on a five point scale, while the file review used a categorical yes/no/insufficient evidence categorization).

17 Total amount invested across the two years was $1.1 million, while total leveraged was $1.4 million.

18 This is SWC's strategic outcome

19 Women in Canada, A Gender-based Statistical Report, Statistics Canada, publication 89-503-X, Date Modified: 2010-12-16 (wage gap);and Nancy M. Carter, Ph.D. and Christine Silva, High Potential Women and Men During Economic Crisis, from Catalyst Inc. The Promise of Future Leadership: A Research Program, August 2009 Talented Employees in the Pipeline, http://www.catalyst.org/file/305/opportunity_or_setback_final_081209.pdf

20 The changes included: 1) the Program's resources and activities were re-focused towards three key areas – the economic, social and democratic participation of women; 2) the Program mandate and objectives were streamlined to link WP more closely with outcomes for women participating in funded projects; 3) advocacy and lobbying activities were no longer eligible for funding; 4) eligibility of program recipients (formerly women's organizations and other equality-seeking organizations in the voluntary sector) was expanded to include all not-for-profit, as well as for-profit organizations, whose mandates are consistent with the objectives of the WP; 5) the WP budget increased to $18.75 million from $10.8 million (though program administrative capacity was reduced from 55 positions to 22 positions, then increased again to 32 positions, and 16 regional points of service were consolidated into four regional offices); and 6) shift in Program application process from a continuous intake model to call for proposals, with identified funding priorities and target populations.

21 The Framework encourages proponents to systematically list their activities, outputs and planned results, as well as develop performance indicators and data collection strategies that would help to gauge progress towards planned results. Proponents must also identify risks and mitigation strategies to deal with risks.

22 The reports ask projects to: describe any changes to the project (as proposed and since the last reporting period), circumstances that helped or hindered the work, outcomes of the project (listed against each of the intended objectives from the funding application) and including unexpected outcomes; partnerships; implications for future work (plan to continue work); lessons learned/best practices; and financial summary.

23 In addition to administrative information for internal reporting purposes, the closeout form provides information on: the outcomes of the project in point form and narrative form; which short-term, intermediate or long-term WP intended outcomes were met by the project; description of other expected outcomes of the project in the longer-term; listing of publications/resource materials created by the project; lessons learned; follow-up; and leveraging.

24 Operational costs typically include administrative expenses associated with proposal solicitation and selection (e.g., coordination of assessment processes, risk and quality analyses), management of grants and contribution agreements, and broader program monitoring and reporting requirements.

25 Women's Program, Status of Women Canada, An Application of a Social Return on Investment (SROI) Methodology to Women in Non-Traditional Occupations Cluster (Draft), 2011.

26 Increased partnerships with other federal departments, levels of government, NGOs and the private sector and increased awareness among women in identifying and/or removing barriers to their participation in their communities.

27 Increased opportunities for women to participate fully in economic, social, cultural life of Canada.

Common menu bar

Breadcrumb

Summative Evaluation of the Women's Program Final Report - 2012

Table of Contents

List of Acronyms

List of Tables

Executive Summary

Overview

Scope and Methodology

Findings

Relevance

Performance

Effectiveness

Design and Delivery

Efficiency and Economy

Recommendations

Program Performance

Design and Delivery

1. Introduction and Description of the Women's Program

1.1 Program Description

a) Context

b) Overview of the Women's Program

c) Program Activity

d) Structure and Governance

e) Program Resources

f) Logic Model

Logic Model - Women's Program

2. Scope of the Evaluation and Methodological Approach

2.1 Evaluation Objectives and Issues

2.2 Lines of Evidence

a) Document, Program Data and Literature Review

b) Review of Project Files

c) Key Informant Interviews

d) Online Survey of Program Applicants

e) Case Studies

f) Comparative Program Analysis

g) Cost-effectiveness Analysis

2.3 Limitations

3. Findings

3.1 Relevance

a) Ongoing need for the WP

Table 3.1: Total WP Applications and Projects Funded — 2007-2008 to 2009-2010

b) Consistency with Agency Strategic Outcomes and Federal Priorities

Evaluation Questions:

Overall findings:

Consistency with Agency Strategic Outcomes

Consistency with Federal Priorities

Gaps/Niche of the WP

c) Consistency with Federal Role and Responsibilities

Evaluation Questions:

Overall findings:

Federal role and responsibilities

Overlap/duplication

3.2 Program Performance: Achievement of Intended Outcomes

a) Immediate Outcomes

Evaluation Questions:

Overall findings:

Awareness of Barriers

Increased Partnerships

b) Intermediate Outcomes

c) Ultimate Outcome

Evaluation Questions:

Overall findings:

Progress toward outcomes

Sustainability of Effort

Impact of External factors

d) Delivery Models/Best Practices to Achieve WP Intended Outcomes

Evaluation Questions:

Overall findings:

e) Unintended Outcomes

Evaluation Questions:

Overall findings:

3.3 Design and Delivery

Evaluation Questions:

Overall findings:

a) Appropriateness/Effectiveness of Program Design

Program Design

Program Transitions and their Impact

Design Improvements

b) Level of Access to Program Funds