The Challenges of Measuring the Impact of Recreation Programs on Youth Resiliency: Designing culturally sensitive and age-appropriate evaluation instruments

by Rene Fukuhara Dahl, Ph.D., and Jane Reed, M.S.

This article originally appeared in California Parks and Recreation magazine, Summer 1999. Documents may be downloaded or printed (single copy only). You are free to edit the documents you download and use them for your own projects, but you should show your appreciation by providing credit to the originator of the document. You must not sell the document or make a profit from reproducing it. You must not copy, extract, summarize or distribute downloaded documents outside of your own organization in a manner which competes with or substitutes for the distribution of the database by the California Park & Recreation Society. CPRS, 7971 Freeport Blvd, Sacramento, CA 95832-9701, magazine@cprs.org.

In this article we discuss some of the challenges we encountered when we conducted evaluation research with children. First, we provide an overview of the evaluation project in which we were involved and then address the challenges of using standardized instruments to collect data with children. In addition, we suggest strategies for administering evaluation instruments to young children.

Background

This evaluation project began almost three years ago as part of the National Recreation and Park Association (NRPA) At-Risk Youth Recreation Consortium. The consortium came into being with a three-year grant from NRPA, called "Project Outcomes: Making the Case for the Impact of Park and Recreation Services for At-Risk Youth." The grant’s goal was to gather, through controlled studies, "scientifically legitimate evidence" that could not be challenged. Witt states that park and recreation agencies have conducted relatively few controlled studies on at-risk youth. Initial documentation efforts were undertaken by researchers at Texas A&M, Arizona State University West, Clemson and Penn State, resulting in the following findings: a) increased perceptions of self-worth by participants in a benefits-based, summer rural recreation program; and b) increased commitment to educational achievement in a basketball/tutoring/community service program, plus an increased orientation to positive role models and authority figures. According to Witt, "a larger set of studies is needed to achieve the threshold volume which will make the cumulative evidence convincing."

San Francisco State University (SFSU) was one of eight participating universities in a consortium which also included the four universities above, University of Georgia, University of Illinois and University of North Carolina, Chapel Hill. Dr. Peter Witt, of Texas A&M, initiated the project and coordinated the consortium. A total of 14 park and recreation agencies partnered with the universities; SFSU’s partners included Concord Leisure Services Department, Modesto Community Services & Neighborhood Connections Department, San Jose Parks, Recreation and Neighborhood Services Department and CPRS. Called the Northern California Collaborative, its purpose was to "develop effective evaluation systems, methods and instruments by which to measure program and individual outcomes and to disseminate this knowledge to public parks and recreation agencies."

In cooperation with its three parks and recreation department partners, SFSU selected three projects to evaluate: 1) afterschool recreation programs at two elementary school sites, one of which was part of a Healthy Start Program; 2) an afterschool enrichment program for children; and 3) a teen center drop-in program. Only programs which served children and youth who were at risk of not achieving productive adulthood were eligible. An additional selection criteria for two of the agencies was the desire to evaluate their programs for the first time. The lessons shared in this article come from the integration of our experiences with the afterschool programs.

One of the goals of the evaluation project was to develop evaluation instruments that could easily be used by park and recreation staff to conduct their own program evaluation. In addition to these locally developed instruments, participating universities and agencies agreed to use common measures in order to compare results between programs. One such instrument, the "Protective Factors Scale," was designed to measure program outcomes in 10 areas related to developing resiliency in youth. Before continuing with the project description, we digress briefly to discuss risk factors that are considered to be threats to youth resiliency.

Risk factors and high-risk environments

Some of the behaviors that interfere with healthy development include substance use, adolescent pregnancy, juvenile delinquency and school failure and/or dropping out. Six factors associated with high rates of those risky behaviors are: 1) early age of initiation of the behavior; 2) poor achievement in school and low expectations for achievement; 3) acting out, truancy, antisocial behavior and conduct disorders; 4) low resistance to peer influence; 5) lack of parental support; and 6) living in an economically deprived neighborhood. We were interested not only in the extent to which children in the recreation programs exhibited or were exposed to these behaviors, but what factors in the recreation programs helped children’s resiliency in the midst of those risks.

A high-risk environment might be one in which children and youth are regularly exposed to gangs, drugs, alcohol and other threats to their safety. They may have witnessed violence or have a high probability of becoming victims of violence. The risk may come from an unstable home where physical care and emotional support are unpredictable, where children do not have positive adult role models, where children are left alone with little or no supervision and with nothing productive to do.

Designing standardized instruments

In examining two standardized research instruments, the "Protective Factors Scale" and the "Life Events Scale," three issues arose: 1) the instruments’ cultural/social sensitivity; 2) its age-appropriateness; and 3) the dilemmas of administering instruments to young children. In each of these sections we offer suggestions and solutions that we found to be useful.

Cultural & social sensitivity

One of the axioms of conducting research is that the researcher should be aware of his/her beliefs and assumptions that might create systematic bias in the research process. The researcher takes steps, via the scientific process, to minimize this potential. Our concern about bias centers on the assumptions we make about social institutions such as the family.

The family is a social construct laden with personal and societal assumptions. Normative values, i.e., our beliefs about how things should be, shape our image of what a "typical" family is like. For many years, it was a middle-class, nuclear family with a father and mother who were married and lived together in a single family dwelling with their biological children, all of whom shared the same family name. The father was the primary wage earner and the mother was the caretaker working in the home without economic remuneration. Major changes in family structure have occurred in the past two generations, resulting in a wider range of familial arrangements, such as single parent/adult families; adoptive and blended families; families with gay and lesbian couples as parents; dual career families; and families with multiple family names. Even with these changes, the normative view is deeply woven into society’s belief about what the "traditional" family is.

Normative assumptions about the family are given credence when they are included on an evaluation instrument. The researcher must be concerned about the biased image that is created and reinforced about children and families who do not match the ideal; they are considered to be at risk socially, culturally or economically, and must not be systematically stigmatized. Instead, we must try to capture the broader range of family types in our work.

The "Life Events Scale" (Table 1) provides examples of normative beliefs that are operationalized as measures on an evaluation instrument. In this instrument, elementary school children were asked to respond with a "Yes" or "No" to 26 statements to indicate which of those events occurred to them during the past year. These items dealt with changes in family, changes in friendship, health problems, problems in school, and experiences with violence. The instrument was designed to give researchers a sense of the risk factors and level of turbulence in a child’s life that could threaten resiliency. In Table 1, we list items which contain normative assumptions about the family, what those assumptions are, as well as the problems and/or potential impact of those items on the children who use the instrument.

After discussions with school site personnel, it became apparent that many of the children in the afterschool recreation programs came from families that did not resemble the idealized, middle-class, nuclear family, but lived in single parent/adult families, headed by a mother, grandmother, or other relative. In some instances, a father or mother had never lived with the family. Many came from high-risk environments, some with a panoply of risk factors, while others had only one or two of these features. Our concerns about the instrument, then, had to do with statements about the family. If a child read statement after statement on an evaluation instrument that did not reflect his or her life circumstances, what would be the impact on the child? Would this instrument reinforce any feelings of marginality that the child had from being on the economic and social fringe of society? Would the child receive a negative message about how he or she did not fit into the world?

To paraphrase the poet Adrienne Rich, when a person in authority describes the world and you are not in it, you experience psychic disequilibrium and feel invisible, as if you looked into a mirror and saw nothing. When assumptions are codified in the evaluation instrument, we unwittingly reinforce hegemony and create a place where children do not feel validated or safe from exclusion. Our concern centers on children for whom these social assumptions are not true; we question the impact of these normative questions on the children. Among the goals of parks and recreation is the desire to help children feel good about themselves; reinforcing such hegemonic notions is antithetical to that desire.

A second concern is the instrument’s effect on the child’s sense of safety. One of the purposes of the recreation programs at school is to create a sense of safety for the child. The school becomes a refuge from problems in the child’s home or immediate environment. Some of the items in the "Life Events Scale" dealt with troubling topics in an attempt to determine the risk factors and environmental dangers in the child’s life (Table 1). Examples of those items are "Someone threatened me (with a gun or wanted to fight);" "Someone I know was injured because of violence by other people;" and "One of my parents had a serious accident."

When we saw how seriously children took these written instruments, how well they wanted to "perform" and how much they wanted to please the data collectors, we understood the strong impact of these evaluation tools. Many children asked, "Is this a test?" "Is this the right answer?" "What if we don’t pass?" and "Do we get a grade?" These questions persisted even after the children were repeatedly reassured that the instruments were not tests and there were no wrong answers. Because the children also wanted to please the team of data collectors, the impact of these authority figures added legitimacy to the evaluation tools and the message contained therein.

Age-appropriate instruments

To measure the impact of the afterschool programs on young children’s resiliency, we used the "Protective Factors Scale." This instrument was previewed and deemed to be appropriate by the teachers at both afterschool sites. Numerous concerns emerged about the suitability of this instrument with young children, many of whom read below grade level and speak and read English as their second language. Those concerns were: a) the difficulty of terms and phrases; b) the large number of response choices; and c) the length of the instrument (47 items). We use the "Protective Factors Scale" as an example of how we redesigned an evaluation instrument to be more appropriate for its target audience. To date, we have not analyzed the validity and reliability of the instrument as a result of these changes.

During the first of two rounds of data collection, we found that concepts like "succeed," "teamwork" and "goals" were too complex for the children to understand. They asked for explanations which the data collectors provided. The data collectors knew they risked biasing the children’s responses by providing them with detailed explanations and examples, yet they deemed it more important that the students both understood the questions and experienced success in the process. The need to explain vocabulary and concepts lengthened the data collection process considerably so that by the end of the session, many of the children were restless and tired and began to circle the same number choice for the questions in order to finish more quickly. Because of these difficulties, most of the data collected in the first round was unusable.

Even though a Likert-type scale with seven to five response choices was used, the children still had difficulty understanding the scale, which ranged from 1=Strongly Agree to 5=Strongly Disagree. We found that expecting them to discern the difference between a "2" and "3," for example, was an unreasonable expectation on our part.

Before we administered the instrument again, we made additional changes to make it more age-appropriate. First, the number of items on the instrument was cut in half and the font size was increased so that the instrument looked less like a test than the previous one. Second, the vocabulary was simplified and the concepts were made more concrete and specific. Third, the Likert-type scale was reduced to three response choices of "Yes," "Sort of" and "No," with circles to color in, rather than having to choose a number. The last change we made was to have the instrument translated into Spanish, so that children who preferred to use that version of the instrument could do so.

In this second round of data collection, we also incorporated interviews with the children to gain understanding, in their own words, of the impact of the recreation program on their resiliency and sense of safety.

The data collection process

The first round of data collection took place in third-grade classrooms during the school day. Our goal was to collect data from two groups: the students who attended the afterschool recreation program and a comparison group of students who did not. We hoped to track students in both of these groups for two years. The data collection process proceeded as follows: a team of data collectors, one English and one Spanish speaker, arrived at the classrooms at prearranged times and explained the instruments to the children. Simple instructions about how to use the instruments were written on the chalkboard and verbally reviewed. The children expressed concern that they were being tested, as this was the setting in which they took tests during the school day. Only English versions of the instruments were available, and the children who preferred Spanish received a verbal translation of each item on the instrument.

To make the data collection process less tedious and more relevant for the children, we made the following changes for the second round of data collection. We took the process out of the classroom and collected data during the afterschool program, which reduced the children’s anxiety level. To encourage student interest and involvement in the data collection process, staff from SFSU’s Training Resources for Recreation in Urban Environments (TRUE) organized games and activities for the children. After total group warm-up games, some of the children continued to play games with the TRUE staff while other children completed the instruments or were interviewed in Spanish or English. With the children engaged in interesting and fun activities, managing the data collection process was much more manageable than it had been in the classroom. Moreover, the children experienced much less anxiety about the process.

There is little extant information about successful group data collection strategies to use with young children. The strategies we used to create a more child-friendly process were successful in putting the children at ease, making the process fun and treating the children equally.

Conclusion

Our experience has taught us the importance of being culturally and socially sensitive when designing evaluation instruments, particularly when the goal is to assess positive impacts on youth resiliency. To accomplish this task, we must frame questions so that no child feels invalidated, and we must look for hidden or subtle messages we send through the tools that we use. We are still developing ways to ask questions about sensitive and unsafe topics without threatening the child’s sense of safety.

We learned the importance of avoiding a test like setting and atmosphere which increases the children’s anxiety levels. In addition, we gained a new awareness of the importance of making data collection more child-friendly and fun. As we continue to incorporate these lessons from the field into our work, our goal is to extend the positive impacts of recreation programming to all children.

About the authors

Rene Fukuhara Dahl, Ph.D., is professor of Recreation and Leisure Studies at San Francisco State University (SFSU) and Director of the Northern California Collaborative of the NRPA evaluation project.

Jane Reed, M.S., is program director of TRUE (Training Resources for Recreation in Urban Environments), a leadership training program at SFSU and coordinated data collection for the NRPA evaluation project.

References

1. Witt P. "Project Outcomes: Making the case for the impact of park and recreation services for at-risk youth." Grant proposal to NRPA, August 1996.

2. Dryfoos, J. Adolescents at risk: Prevalence and prevention. New York: Oxford University Press, 1990.

3. Rosaldo, Renato. Culture and Truth. Beacon Press: Boston, Mass., 1993.

Acknowledgments

The authors wish to acknowledge the helpful contributions of the following people in this evaluation project: Dr. Peter Witt, Texas A&M, plus other consortium members; Mark Deven, Vicki Matheny, Mac Kaiser; Bruce Stanley; Kathy Sweeney, Lynda Hampton, Stephanie Barring-ton and the third-grade teachers and school children at the two elementary school sites.

Thank you’s also to Doug Gaynor, Wendy Byrd, John Cincinato, Mark Linder, Terry Eberhardt, Ron Soto, Gary Okazaki, Marcos Orros, Sharyl Geisert, Ellis Mitchell, Jane H. Adams and Anne Seeley. Thank you to Marcela Arregui Reyes for data collection assistance and data entry, and also to Cecilia Abasolo for data collection assistance.

TALE 1

Critique of "Life Events Scale"

Table 2

Sample Items and Scale from Protective Factors Scale