Evaluation

Evaluate and Demonstrate Impact

After you have had a chance to establish and operate your programs, you will need to determine whether the programs are effectively meeting your outcome goals. An independent evaluation is an important way to find answers to questions about your impact. An evaluation can also help gain supporters and funders, as it can help demonstrate the impact of our program.

In 2017, NYC Service engaged Urban Institute, a leading research organization based in Washington, D.C., to conduct an evaluation of the first two years of the City Service Corps program, looking specifically at whether the agencies/organizations benefit from the members’ service, whether members gain skills, and whether they continue to be civically engaged after their service. In other words, Urban Institute evaluated the impact of the trifecta of benefits: improving the communities in which they work; developing a workforce of highly skilled people; and creating a new generation of leaders who prioritize civic engagement.

Overall, the evaluation found that, based on the first two years of the program, City Service Corps is making an impact on New York City and its members’ future.

Who are City Service Corps members?

According to the evaluation report (available at https://www.urban.org/research/publication/city-service-corps-americorps-program-outcome-evaluation), City Service Corps aligns with general AmeriCorps demographics in many ways; for instance, almost 2/3 of members are female. However, City Service Corps is made up of a more racially and ethnically diverse group than AmeriCorps in general, reflecting New York City’s population.

  City Service Corps AmeriCorps
Female 65% 65.3%
Male 35% 35.7%
Non-Hispanic Black or African American 33.3% 17.2%
Non-Hispanic White 21.9% 52.4%
Hispanic 15.8% 17.1%
Asian 7.9% 6.7%
Other, Multiple Races, or No Answer 22.8% 9.1%
Average Age 27 25
Years of Education 15 16

Source: Urban Institute (based on the Corporation for National and Community Service member exit survey)

Developing a skilled workforce

Prior to their service, half of City Service Corps members were in the workforce, about one-quarter were in college/ graduate school, and 7 percent were serving in another AmeriCorps program.

A year after they completed their service, most alumni were at nonprofits or in government/public sector, according to the evaluation. And 70 percent of alumni said that serving in City Service Corps had an impact on their career pathways.

Impacting the community

Most City Service Corps alumni – 86% – believed that their service made a positive difference in their communities. And their host site supervisors agreed that their work had a definite impact. 91% of site supervisors reported that City Service Corps members helped them work more efficiently. 80% of site supervisors reported that their members contributed in some way to helping the agency or organization expand into new program or service areas.

Creating a new generation of civic leaders

Many City Service Corps alumni are active members of their communities, according to the evaluation. Following their city government-focused term of service, 83% reported that they have a clear understanding of city government. 75% of City Service Corps alumni said they voted in local elections during the previous year. And at least 40% say they are more likely to volunteer, keep informed about public issues, and participate in community organizations than they were before they participated in City Service Corps.

Uncovering challenges

The evaluation also uncovered challenges with City Service Corps members. The evaluation found the challenges seemed to center on soft skills, including members’ capabilities, communication, and professionalism. Because of this information from the evaluation, the City Service Corps team has added additional trainings around these topics.

70%

City Service Corps affected the career pathway I chose

74 responses

85%

City Service Corps made me more interested in the types of jobs that I pursued

73 responses

83%

City Service Corps made my personal goals for occupational attainment increase

70 responses

Question Icon

Meaningful Questions for a Successful Evaluation

Selecting the right independent evaluator is a critical component in the effort to measure your program’s true impact on your town or city. The following questions will help you formulate a robust request for proposals from potential evaluators.

  • What kind of evaluation are you looking to conduct?

    For instance, NYC Service decided upon an outcome evaluation for City Service Corps.

  • What is the evaluation plan being proposed?

    Specifically, what combination of focus group/survey/ interview/data review does the evaluator plan to use? In City Service Corps’ case, evaluators used the following: performance measure data collected by City Service Corps; program analysis pre- and post- City Service Corps intervention; surveys of members and alumni; samples of client and host site satisfaction data; analysis of data collected by the program’s partner, City University of New York, about trainings.

  • What experience does the evaluator have with AmeriCorps programs?

    City Service Corps worked with Urban Institute, which has done extensive work with AmeriCorps programs throughout the country.

  • What is your timeline?

    City Service Corps planned for a 16-month turnaround from awarding the contract through completion.

  • How much are you budgeting for the evaluation?

    City Service Corps budgeted $125,000 for the evaluation, but costs may differ depending on the type of evaluation conducted.

  • What specific questions are you looking to have answered with the evaluation?

    NYC Service posed the following questions to guide the evaluators:

    • What outputs and outcomes might reasonably be attributed to the members’ service?
    • Do City Service Corps members add value/build capacity at their host sites in ways that could not happen without their presence?
    • How does the program affect post-career choices made by members?
    • What are the tangible skills that alumni gained from their experience serving with City Service Corps?
    • Do alumni continue to be engaged in civic life?

  • Where is the evaluator based, and would they be willing to come to your town or city to conduct focus groups?

    Urban Institute is based in Washington, D.C., but representatives did travel to New York City to conduct focus groups.

Practical Advice from Aaron Miner,
Managing Director, Service Year Programs

Begin with the what and why: what do you want to evaluate and why are you doing it at this time? We chose to look at the first two years of the program so we could test the theory of the case we had made for the program. We then went about writing detailed research questions so we would be prepared to work with whatever evaluator we selected. After putting out a Request for Proposal (RFP), we selected Urban Institute. One of the benefits was that the evaluator from Urban Institute had a history of evaluating AmeriCorps programs, which meant we did not have to cover the basics of AmeriCorps programs and how they work.

Evaluation

Create Sustainability

As you plan the future of your program, you will want to take a step back to think strategically about how to make the program last for the long term.

City Service Corps continues to review and make adjustments to make the program stronger. These adjustments include:

1. Alignment with city priorities.

City Service Corps is continuing to ensure that it aligns with the City’s priorities, including OneNYC, the citywide strategic vision and plan.

2. Program size.

City Service Corps is still working to find the right size for the program, but 80-120 members are close to the ideal size.

3. Professional development.

City Service Corps has overhauled its professional development program, which is now called the “Leadership & Development Series.” The new professional development program changes are based on staff feedback from host sites and members from the past few years. As part of the series, City Service Corps will be launching a skills assessment of members to equip members and host site supervisors with more information to set expectations about areas of growth and development.

4. Higher stipends.

City Service Corps is a continuous advocate of raising AmeriCorps member stipends so that they provide a more realistic cost of living for higher cost cities such as New York City. NYC Service has thus increased the stipend for our programs to better align with New York City’s cost of living, but this means that agencies may have a harder time finding the budget to provide the members with stipends. As such, staff are exploring dedicated funding within the City’s budget to offset some of the stipend costs undertaken by agency host sites.

5. Diverse funding streams.

While AmeriCorps does generally provide federal funding for its programs, it may not be enough to sustain full program operations. A municipality should think about how to build sustainable revenue streams to cover the cost of operating an AmeriCorps program. City Service Corps does not use any funding from AmeriCorps; it uses a mix of funds from private funders and City agencies.

Question Icon

6 Questions to ask yourself when thinking about your program’s future:

  1. What is the mission and vision of your program, and will they still be relevant to the program in three years?
  2. What impact has your evaluation uncovered? Are there any adjustments your program should make as it moves forward?
  3. How committed are your champions in your program? Can they help to expand the program’s reach?
  4. Is your current funding stream sustainable as is, or does it need to change as the program continues to operate?
  5. Does your agency need to make any changes (to staff, budgeting, etc.) to allow for an expansion or continuation of your AmeriCorps program?
  6. Does your agency need to make any changes (to staff, budgeting, etc.) to allow for an expansion or continuation of your AmeriCorps program?
Practical Advice from Aaron Miner,
Managing Director, Service Year Programs

We continue to work to align City Service Corps with City priorities, including the City’s strategic plan, OneNYC. As priorities of the city change, we hope that our colleagues see City Service Corps as a way to build capacity on high priority programs, supporting City agencies through challenging times, such as the COVID-19 crisis, when effective resident engagement is critical.

Conclusion

City Service Corps quickly evolved from an idea coming out of a service summit to a successful AmeriCorps program operating within the constructs of the city government, with public and private contributors funding program operations. From 2015 until today, NYC Service has built a robust administrative and leadership development program within City Service Corps.

While the funding and operational models were unique to City Service Corps, its continued success and impact on New York City illustrate that the program is successful, replicable, sustainable, and impactful.

Service is a powerful element in creating a stronger civic engagement fabric in a city. We hope that you find this guide useful as you make decisions on whether creating a similar program in your municipality is something to pursue and then use our learning and tools as resources.