Evaluation of Walnut Terrace After-School Program

FINAL REPORT
EVALUATION OF WALNUT TERRACE AFTER-SCHOOL PROGRAM

Submitted to:
Raleigh Parks and Recreation Department

by

Beth D. Kivel, Assistant Professor
Greg Todd, Graduate Research Assistant
Department of Recreation and Leisure Studies
University of North Carolina
Chapel Hill, North Carolina
October, 1999

Executive Summary

Between February and April, 1999, the Department of Recreation and Leisure Studies at UNC-Chapel Hill and the Raleigh Department of Parks and Recreation, evaluated the after-school programs at Walnut Terrace Community Center. Because there were no explicit measurable performance outcomes, it was difficult to determine the precise impact of the program on participants. Yet, based on interviews with and observations of participants and leaders, we did determine that participants enjoyed the program and they perceived that they learned the importance of doing their homework and respecting others. While the overall goals of the program were educational, behavioral and attitudinal, there were no explicit measurable performance outcomes for participants. While the program met its overall goals, it could easily be strengthened with more explicit measurable performance outcomes and no explicit focus on incorporating structured recreational activities into the program. Recommendations include trainings that would teach after-school leaders how to develop measurable performance outcomes and how to facilitate small and large group structured recreational activities for participants, and additional resources to update the on-site library and provide at least one operational computer on-site. There seems to be no question that this program is a good one. Given the commitment and enthusiasm of the current leaders and the willingness of the Raleigh Department of Parks and Recreation to provide its support, this program has the potential to be great.

Introduction

Profile of Raleigh and Wake County

Raleigh, known as the City of Oaks, is located in central North Carolina. The city offers a myriad of cultural activities with a variety of places to visit from historic plantations to state of the art sports arenas. Raleigh is part of Wake County and recent census data show that approximately 275,000 people live within the city limits and 305,000 live in surrounding communities. It is anticipated that by the year 2025, there will be more than 580,000 people living in and around Raleigh. While 67% Raleigh’s residents are of European American descent, Raleigh is a racially diverse city compared with the rest of North Carolina. Approximately 28% of its residents are African American; and 5% are classified as other, including 3% Hispanic. Approximately 300,000 people are employed in Wake County and unemployment rates are low, holding steady at approximately 2.9% over the past five years. The average household size has remained constant at 2.3 since the 1990 census.

In 1976, Wake county schools and Raleigh public schools merged. A nine-member board of education sets policy and determines educational philosophy for the county. There are 106 schools in the county: 69 elementary schools; 20 middle schools; 13 high schools and 4 special option schools. Wake County public school system is ranked in the top three school systems in the state. In 1994 and 1996, two new bond issues provided funds for 25 new schools and improvements to 59 others. In 1997, enrollment in the public school system reached 89,441 with a total budget of $62.4 million. The annual per pupil expenditure was $4,722. Four thousand (4,000) students graduated from high school that year and 84% of these students indicated an intention to pursue higher education. The racial composition of students attending these schools is: 69% white, 26% African-American and 5% other minority. The teacher to student ratio is between 1:22.5 and 1:25.5 depending on the grade.

Raleigh Parks and Recreation Department

Raleigh Parks and Recreation Department (RPARD) oversee 56 major city parks, 19 staffed community centers, eight neighborhood centers, two art centers, 104 tennis courts, 150 park sites, and approximately 36 miles of greenways. Since 1990, RPARD has sponsored educational programs for children who have been identified as being at-risk for not making a successful transition from childhood to adulthood. RPARD continues to provide programs for “high risk youth, latch key kids, children from single-parent homes, youth in need of further educational opportunities” and all youth in general. The department works in conjunction with other city funded agencies that share their purpose to offer special programming for youth, such as D.A.R.E drug education programs, fire prevention and other safety education programs, reading programs, mentoring, assistance with homework, and opportunities for structured and unstructured play and recreation.

During the 1998-1999 school year, RPARD sponsored 13 after-school programs in Raleigh and/or in Wake County. Approximately 260 children participated in these after-school programs. General goals of the program were to provide young people with:

1. To provide a holistic program that involves students, parents, teachers and the after-school staff;

2. To reinforce the importance of learning and of education;

3. To provide a support system for participants;

4. To increase peer cooperation and interaction;

5. To teach respect and responsibility outside of the classroom;

6. To encourage appropriate behavior and conduct by encouraging participants to follow rules and directions; and

7. To teach participants to understand boundaries and how to operate within a structured environment.

During the 1998-1999 school year, after-school activities were offered at the center four days a week (except Fridays). The after-school program had two primary activity components: education and recreation. The education component involved program leaders working with participants to complete their homework and encourage those who had completed their homework to read ahead for school or to read one of the books provided by the center. The children also participated in educational activities such as spelling tests and learning geography by making and coloring maps. In addition, the teachers and teacher assistants reviewed the students’ homework every day that they attended the program and Center director Ethel Evans also had parental permission to review the grades of participants. The second component involved structured and unstructured recreation activities. Approximately once a week an arts specialist would visit the program to show students how to make balloon animals or how to create a map of the United States or how to make collages.

Students usually arrived at the center between 3:00 and 3:30 p.m. and worked with staff to complete their homework between 3:30 and 4:30 p.m.. After they had finished their homework, they participated in recreation activities such as bingo and board games from 4:30 to 5:00 p.m.. The participants then left the facility between 5 and 5:30 p.m. The center occasionally provided snacks.

Participants in the program ranged in age from 7-17; most of them attended Washington Elementary School that is located approximately one block from the center where the after-school program is held. A few students from neighboring schools also attended the program. Once students have completed their homework, they could elect to go back to Washington Elementary School gym to participate in supervised, but unstructured activities such as pick-up basketball games. The RPARD does not provide transportation to enable participants to attend the gym program.

Program Staffing

The program is typically staffed with three to five individuals. At least three staff are paid by RPARD and two staff members work are paid through their universities (Shaw University and North Carolina State University; Federal work study, mentoring programs, etc.). Some university students also volunteer to complete service requirements for their degree programs.

Center Director Ethel Evans has worked with the community center for 10 years. She oversees the scheduling, programming and supervision of the after-school program. .

Evaluation Project

Purpose

The purpose of this evaluation project was to work with RPARD to evaluate an after-school program that targets at-risk youth. Beginning in the Spring of 1999, faculty from the Department of Recreation and Leisure Studies at UNC-Chapel Hill (UNC) met with staff from RPARD to discuss the possibility of a collaborative evaluation project. Faculty again met with RPARD staff in the fall, 1999. During that meeting an evaluation plan was developed. Between February and April, 1999, the project investigator (PI) and graduate research assistant (GRA) conducted a formative evaluation to assess what the participants were learning as a result of their participation in the program.

There were two phases for the evaluation. The first phase involved an evaluation of the after-school program; the second involved a follow-up to this evaluation to determine how UNC and the RPARD could work together collaboratively in the future.

The evaluation took place between January and June, 1999. One researcher and one research assistant from University of North Carolina at Chapel Hill conducted the evaluation at the Walnut Terrace Recreation facility. They conducted six observations on Monday and Thursday afternoons between February and April, 1999.

Sampling

Fourteen participants (ages 7 to 17 years) were interviewed; the majority of participants had been enrolled for at least one year. Five of 14 interviewees had been enrolled for at least four years. All but one of the participants attended the program four days a week. Consent forms were given to the program supervisor who distributed them to the parents of participants. Participants’ parents kept one copy and returned the other to the program supervisor. In addition to interviews, program staff were observed and interviewed. Due to time constraints, parents were not interviewed.

We used convenience and purposive sampling methods. Between February and April, 1999, we observed approximately 30 children and five staff. We conducted semi-structured interviews with 14 participants (seven girls and boys) and three leaders. All of the participants and staff we interviewed were of African American descent.

Methods The PI and GRA used two methods of evaluation for this project: observations and interviews.

Observations

The PI and the GRA conducted field observations and during two of his six observations, the GRA was a participant observer. Qualitative observations were conducted and the PI and GRA jointly observed on one occasion and the GRA conducted observations during five additional visits between February 25-March 29, 1999. The PI and GRA observed participants and leaders and took detailed notes that described what saw and heard during visits to the program. Field notes were later typed, coded and analyzed.

Interviews

The PI and GRA interviewed four children together and then the GRA interviewed ten children on his own. Two semi-structured interview guides were developed, one for the participants and one for program leaders (See Appendices A and B respectively).

The primary focus of the evaluation was to assess what participants were learning as a result of being in the after-school program. Participants were asked to give their impressions of the program, the program leaders, what they were learning from the program, the kinds of activities they were participating in while at the program and what they would be doing if they were not in the program. Verbatim notes were taken during the interviews. Interviews lasted between 20-30 minutes. Participants were interviewed alone or in pairs.

Data Analysis

We used the process of triangulation – field notes, interview data and multiple observers and interviewers – to collect data. The GRA took detailed field notes during observations and interviews and field notes were then typed at the end of each session. In addition, we analyzed interview and observation data. The observation and interview data were analyzed inductively, using constant comparative analysis (Henderson & Bialeschki, 1995). Using this method, we identified and developed common themes that emerged within the data.

Findings

The findings of the evaluation are consistent with the informal goals of the program. The program seemed to impact students’ attitudes and behaviors in two areas: education and “conduct”/”manners”. While our analysis of the observations did not clearly assist us with determining what the participants learned as a result of being in the program, subsequent interviews did reveal that participants and leaders felt the program was beneficial and that they internalized some of the program goals.

Findings Based on Observations

Our analysis of the observations revealed more about general behaviors and issues that emerged from the program than the impact of the program An analysis of the observations revealed several themes:

Participants

a. enjoyment

b. distraction

c. disruption

Leaders

a. caring/attentive

b. staggered leadership

c. absence of structured recreation

Space/resource issues

a. space constraints

b. adequate funding

These themes were developed to help us organize and make sense out of the data we gathered. Our analysis was based on limited observations and does not represent the full spectrum of activities that participants engaged in over the entire length of the program. The themes should not be used to make any conclusive judgments about the program, the participants or the leaders.

Participants

Enjoyment : Participants seemed to “enjoy” themselves and having a “good time.” Most of the participants interacted with one another in a friendly manner, they were respectful to the leaders and they were attentive and engaged when visitors, such as the arts specialist, would come to the center. We noted a great deal of laughter and good humor among participants.

Distraction : While participants seemed to understand the structure of the program (i.e., they were to come in, hang up their coats and begin doing their homework), we observed many instances in which they would begin to do what was expected and then become distracted. They seemed easily distracted by what their peers were doing and by anyone who came through either the front or side door. Also, if one person at a table finished their homework, that person would get up and begin walking around the room talking to and/or distracting others.

Disruption : It did not take long for distracting behavior to sometimes turn in to disrupting behavior. Disrupting behaviors included talking to peers who were still doing homework, playfighting in the room, and interrupting leaders several times as they attempted to work one on one with students who were completing their homework. Thus, leaders would be torn between the need to intervene and the need to work with students who wanted assistance with their homework. Boys were more likely to engage in these behaviors.

Leaders

Attentive/Caring : Our field notes consistently revealed that the leaders of the program were attentive to and they cared about participants. This was demonstrated in the time that an individual leader would spend helping a child with her/his homework. At times, a leader would spend up to 15 minutes working individually with one child. The leaders were patient and would explain how to solve a problem four or five times if they needed to help the student understand a concept. The leaders used the children’s first names and would often attach endearments to their names. In turn, the children addressed the leaders with respect, calling them Ms. or Mrs. ___________. Clearly, the leaders and participants had based on mutual respect.

Staggered leadership : We noted that while the Center Director was usually the first person to greet participants, there was often a time lapse between when the children arrived and when additional leaders would arrive. This did not seem to be a problem when only a few children were present, but when more than 10 children would be in the center at one time with one leader, chaos would sometimes ensue. At times there were up to 25 children in a very small space at one time and it took leaders many attempts to get participants to listen, quiet down and cooperate with requests to do activities.

Absence of Structured Recreation: While the opportunity to participate in structured and unstructured recreational activities was one of the goals of the program, we did not observe many instances in which the leaders led participants in these types of activities. Occasionally, participants would play musical chairs or the leaders would assist the arts specialists who visited the program on a weekly basis. Otherwise, we did not see leaders engage in additional structured recreation opportunities.

Adequate Resources

Space Constraints : The space seemed too small to adequately accommodate 25-30 children who completed their homework at different times and then were permitted to engage in recreational activities. With warm weather, participants could go outside to play, but in the fall, winter and early spring, the center was not conducive to a variety of simultaneous activities. The linoleum floor and the vast numbers of windows made the space acoustically-unfriendly and prone to a great deal of noise. The design of the center – front doors going in and several side doors leading out of the main room where participant gathered—sometimes made it difficult for center staff to know when participants had left the room.

Adequate Funding : The community center did not seem to have adequate resources to ensure smooth functioning of the site. Although the center had six computers, all were broken. There was one large bookshelf that was three-quarters full, but it was unclear the extent to which the books were current or that they accommodated children from a variety of age groups. The space itself was sterile, the atmosphere was not very warm and inviting, nor was it very child-friendly. In back of the center was a playground with a small amount of playground equipment.

Findings Based on Interviews :

We conducted 14 interviews with participants and three interviews with program leaders. After the interviews, we analyzed the transcripts and developed themes, most of which focus on the participants’ attitudes and behaviors. These themes include: homework first, play later; school matters; getting along with others; it’s fun; somebody’s got your back.

Homework First, Play Later

Almost all of the participants we interviewed said that they saw a connection between doing homework first and then being able to do what they wanted afterwards. This theme was illustrated by comments by participants when they were asked to describe what they liked about the program. Two 10-year old boys said what most of the participants echoed: “It was fun because we get to do homework and then go outside” and “We do our homework so when we get home we can play N64 Playstation.” It was unclear the extent to which participants may have made this connection prior to being in program. The three leaders articulated their hope that participants would learn the “importance” of homework. This theme was consistent with the goals of the program and what the leaders had hoped participants would learn.

School Matters

Related to issues of homework, participants implicitly said that they thought school mattered. We noted this theme based on two factors: first, what participants said they liked about the program and what that learned in the program to do, and receive help with, homework. Second, one participant explicitly said the program taught her that school matters. A 17-year old young woman said that as a result of coming to this program for seven years, she learned that “grades do count in life and attitude does make a difference. You have a chance, you just have to take it.” Several of the participants also identified new information they learned as a result of this program. Many of them spoke about a map-making project through which they learned about different countries and the states and state capitols in the United States; others spoke about learning to improve their spelling; and others simply said, they learned “new stuff.” To determine the long-term impact of this theme, it would be important to interview participants as they entered the program at 7 or 8 years of age and again as they left the program in high school. Such a focus, however, was beyond the scope of this evaluation.

Getting along with Others—Respect, Cooperation, Manners

After identifying the need to do homework, participants said they also learned a lot about getting along with others, and the importance of respect. This point is illustrated in this young person’s comment: “I learned how to respect people, raise my hand to get the leader’s attention.” Two additional comments by participants illustrated the extent to which they learned how to respect others. A 7-year old girl said, “I learned to respect people and helping other people.” And an 8-year old girl said, “I learned to listen to teachers and I learned how to be quiet.”

Two additional quotes illustrate some of the manners that participants said they learned. One 8-year old girl said, “I learned, ‘please can I use a pencil, not, can I get a pencil’” and one of the 8-year old boys in the program said, “They taught me to say please and not be rude.” A 10-year old boy said that he learned the importance of cooperation. “I learned how to be a good sport. In basketball one day I wasn’t passing the ball and the day I passed the ball [I learned that] it’s good sportsmanship, people appreciate it and appreciate working together.”

It is also important to note what participants learned to “not do.” A nine year old boy said he learned about not using physical force. He said, “I learned ‘don’t kill each other. . . no fighting and no hitting or punching.” Participants also learned the relationship between good behavior and the positive consequences that may follow. “One 10-year old boy said that he learned that “if our behavior is good (listening, doing our homework, not saying shut up) then we get games, trips and skating.”

During their interviews, the leaders spoke about their expectation of participants relative to getting along. One leader said, “I hope they learn respect for each other, how to follow directions and learn how to get along.” Another leader said, “I hope they learn that “Respect is #1.” And another leader echoed these comments by saying, “I hope they learned from me how to respect each other and the instructors . . . and how to get along with each other. I wanted the kids to basically work together. We have a lot of problem children in the area and they do not get along… so to basically have them get along.”

Based on this small sample and there were no follow-up interviews, it is difficult to assess the long-term impact of the program and the extent to which participants internalized some of these attitudes and behaviors. However, the fact they identified and articulated these issues illustrated that they are taking away positive attitudes that may influence subsequent behaviors. It’s Fun

Virtually all of the participants said the program was fun. They provided a variety of explanations as to “why” they thought it was fun. Generally speaking, they said they liked the leaders because they were “cool” and they had fun because they got to do “cool” stuff. One 10-year-old boy’s comments illustrated this point: “We do cool stuff, meet new people, we do activities like dance performances; we get to do homework, go outside and we also play with our friends.” Another child echoed his comments: “We go on field trips and do fun activities. We eat potato chips and they let us play games.”

“Somebody’s Got Your Back”

Participants felt that they were valued and that the leaders cared about them. Several of them said that what they liked about the program was that leaders helped them with their homework and it was “okay” to ask for help. One participant said, “I learned that people take time to help us and leaders take time from work to be with us.” This theme is also consistent with what the leaders hoped participants would take away from the program: “I hope they learn to trust the person sitting next to them. . . I hope they learn that someone has got their back.” Another leader said, “I hope they learn that someone cares about them.” This theme is also consistent with our observations that the leaders demonstrate care and concern for participants.

If Not this Program, Then What?

When participants were asked “what would you be doing if you were not in the program,” we received inconsistent answers. In general, participants said they would most likely be “do nothing,” be at home and/or would probably play with their friends or play video games. Discussion

It is difficult to assess the actual impact of the program on participants, but the findings can provide insight into what participants “did” in the program, how they behaved and how the leaders interacted with them.

The participants’ behaviors that we observed –enjoyment, distraction, disruption—are typical of what would be observed at many after-school programs. Participants seemed to be having a good time, especially when they were engaged in an arts and crafts activity or when they had concluded their homework and were then able to do what they wanted to do. Interviews with participants revealed that they thought the program was “fun” because they got to make balloon animals and maps and hang out with their friends.

The other behaviors that we observed, distraction and disruption, might be due less to any fundamental problems with the program and had more to do with the fact that it might be difficult to sit still after being in school all day. After sitting in a classroom for eight hours, it is not unusual for students to be restless after classes conclude. Thus, while interviews revealed that participants may have internalized the belief that they should do “homework first and play later,” such a message may be diminished by the reality that it is difficult to maintain concentration and one’s attention span after sitting all day in a classroom. Although the after-school program offers participants a structured environment to assist them with learning to do their homework and internalize more positive attitudes toward education, it seems as if participants might be somewhat resistant to a structure that is too similar to their classrooms.

In addition, some of the disruptive and distracting behaviors may be due, in part, to the fact that there may not be enough leaders working with participants at the same time. What seems most beneficial about this program is that the leaders do spend a great deal of time, one on one with participants to assist them with their homework. We observed, and participants spoke about, how much they liked the fact that the leaders would spend time with them. Yet, it is difficult to provide one-on-one time when there are 25-30 young people in a very small space at one time and with a small number of leaders who are scheduled to arrive at different times.

During interviews, participants said they learned about the importance of respecting other people and the need to get along with one another. While this may seem somewhat contradictory to our observations of disruptive and distracting behaviors, we also noted that participants were respectful to the leaders. For example, they responded to the leaders by saying “yes maam” and they would say “please” when making requests and “thank you” when given something by the leaders. Participants did respond to requests to be quiet and/or to be cooperative with one another, but such requests would sometimes require leaders to raise their voices. The noise level in the room was exacerbated by the fact that the room was made of brick, it had many windows, floor to ceiling glass panes at the front of the room to separate it from the hallway and linoleum on the surface of the floor. Since there was nothing to absorb sound, noises that might have gone unnoticed, reverberated throughout the room. The lack of something to absorb sound may have contributed to some of the distracting and disruptive behaviors.

We also noted that participants would sometimes pick up coats or pencils on the floor without being asked to do so and there were many instances of cooperative (sharing of pencils and crayons) and kind behavior. While some of the participants said that they learned how to get along during recreational activities in the program, we primarily observed participants engaging in school work and on very few occasions did we observe them engaging in structured or unstructured recreational activities. Given the fact that we visited the program during only one part of the year and since we did not observe participants outside of this context, we did, nevertheless, conclude that participants attributed their “good” conduct and respectful behaviors to what they had learned in the program. This finding was consistent with what the leaders had hoped participants would learn.

Findings about the importance of homework do seem to reflect the primary focus of the program. Participants did homework first and then were allowed to play. Based on our observations and interviews, it seems that participants understand that this is required of them and they do it without complaint or without resistance. It is difficult to assess the extent to which this behavior would continue if the young people did not participate in this program but it is clear that while in the program, they understood what was expected of them and they complied with the structure. Moreover, participants said that one of the main reasons they liked the program was because they had a chance to work on their homework and they learned that school matters—two of the central goals of the program.

Participants enjoyed the program – they had fun – and felt the leaders cared about them. This echoes what Heath & McLaughlin (1993) found in their studies of youth programs: the most successful programs are structured, have clear rules and expectations, and the leaders demonstrate commitment, are reliable, and show care and concern for participants. It was clear that this after-school program has these components. The leaders of the program are committed—most had worked with the program between three and five years—and they provided a supportive environment for participants. While the leaders of the program are interested in assisting participants with their homework and providing them with recreational opportunities, they realized that there were space and resource issues that limit the potential impact of the program.

Conclusions

The findings revealed that participants liked the program, perceived they have learned to do their homework, and they learned how to get along with and respect others. What participants did not talk about and what we did not observe was their participation in structured recreation activities. Despite the fact that participants “liked” the program, it is difficult to assess the actual impact of the program on participants. Although most of participants said if it were not for this program, they would be spending time alone or with friends, talking on the telephone or playing video games, it is, nevertheless, important to determine the actual value of the program. Such a determination can only be made by first identifying measurable performance outcomes that reflect behaviors, attitudes, knowledge, skill levels, etc.

Recommendations

While the program appears to be beneficial, it seems important to develop a process for determining the “actual” impact of the program on participants. Our recommendations are meant to provide suggestions for how to quantify and qualify the value of the program and build on the strengths of the program. The educational component of the program seems to function quite well, but it is less clear how the program incorporates structured recreational activities into it. Thus, we have also offered recommendations that address this missing component. The recommendations are divided into three topical areas: future evaluations, leadership, and resources and facilities.

One reason it is so difficult to measure the impact of the program on participants is because there are no measurable outcomes, the program, the facilities and/or the leaders. If RPARD staff want to measure actual program, the program planning process should include goals and measurable performance objectives (e.g., behaviors, attitudes, etc.). Two explicit recommendations are offered:

• Is there anything that Raleigh Parks and Recreation cans do to support you and this program?