The Journey Continues: Ensuring Cross-Culturally Competent Evaluation - Case Study B: Evaluation of Grant Project to Increase Senior Volunteerism

The situation

In another hypothetical example, a senior volunteerism grant project, established by a local foundation, encouraged senior centers to develop programs to increase volunteerism among seniors. The foundation was motivated to create this grant project, following a national research study that demonstrated a strong correlation between volunteerism and improved health among seniors. The foundation distributed a Request for Proposals and selected 10 senior centers in urban neighborhoods as grantees; two of these centers were located in areas that were experiencing an influx of Korean immigrants. Most of the residents in the other neighborhoods are low- to middle- income, and primarily African-American and White.  To support the effort, the foundation engaged a management organization to provide ongoing oversight and another organization to evaluate the overall project.

The foundation’s evaluation questions were:
  • What programs were developed by the centers to increase volunteerism among seniors?
  • Did volunteerism increase among the seniors served by the centers?
  • What capacities did the centers require to implement and sustain the programs they developed?
  • What supports were required to assist the centers in developing these capacities?

The evaluation

The evaluation firm hired a 65-year-old, White female, to be the evaluation director. This person brought with her more than 30 years of experience delivering services to seniors, had overseen program design and implementation for this population, and had experience evaluating programs for seniors. The firm believed that this person’s experience made her the ideal evaluation director. The rest of the evaluation team was comprised of four evaluators who ranged in age from 30 to 48, two of whom were White, one African-American and one Korean (bilingual), three women and one man, all with doctoral degrees.

The evaluation involved: a survey of seniors at each center about their volunteer activities; interviews with all levels of staff (from executive director to front-line staff) at each senior center about their volunteer programs, accomplishments and lessons learned; focus groups with selected seniors at each center to discuss their perceptions of the center’s volunteer programs and why (or why not) the programs met their needs; review and analysis of technical assistance requests and the management organization’s responses; and a group interview with the management organization’s staff about their lessons learned regarding the technical assistance process.

About three months into the evaluation data collection process, the management organization received calls from grantees that the seniors were upset about the focus groups and interviews. They complained that the focus group facilitators (i.e., evaluation team members) were condescending and made the seniors uncomfortable. According to the seniors, whenever they discussed their volunteer activities, the facilitators responded with disbelief, almost as if they thought the seniors were incapable of the activities. Some of the seniors added that the facilitators did not seem to take them seriously when they discussed the legacy they wished to leave behind and, therefore, why certain volunteer activities (e.g., helping other older immigrants, mentoring the younger generation) were important to them. Consequently, some of the seniors became hesitant to share their opinions, because they felt that the facilitators did not understand them or value their input.

The management organization’s team director shared this information with the foundation’s senior program officer for the grant project, adding that the team was concerned about the accuracy of the data, if seniors were reluctant to share their views. These complaints, according to the team director, were particularly apparent in two senior centers that served primarily Asians.

At the same time, and without any knowledge about the complaints, the evaluation team director shared her concerns with the foundation’s evaluation manager about the low response rate for the survey and the last five focus groups.

Commentaries by the experts

The experts discussed the following issues as critical considerations in evaluations involving seniors.

Focus group facilitators’ and underlying assumptions about the seniors. The focus group facilitators’ assumptions (i.e., stereotypes) about the seniors and their capabilities may have contributed to the participating seniors’ complaints, according to the experts. It is possible that the facilitators assumed that people 65 years and older are typically not active, especially as volunteers. Therefore, the facilitators may have been both surprised and impressed to hear about some of the participating seniors’ activities. Their reactions were interpreted by the seniors as condescending.

As noted in The Importance of Culture in Evaluation, evaluators need to pay attention to their assumptions about the group of people involved in the evaluation; they could be caught off guard and respond inappropriately, causing situations similar to the one in this example.  

Cultural differences between the focus group facilitators and evaluation participants. While the focus group facilitators shared similar racial and ethnic characteristics as the participating seniors, they differed in age (except the evaluation team director) and education. Both the similarities and differences between evaluators and evaluation participants need to be equally considered. For instance, it shouldn’t be assumed that the Korean evaluation team member could easily establish rapport with the Korean seniors at the two centers. On the contrary, the differences in age, education and professional status could complicate the dynamics due to expectations about how younger Koreans should behave around their elders. It is possible that someone who deviates from the norms of their social identity group could be perceived more negatively than an outsider, who isn’t expected to fully understand the norms. It also is possible that the older Korean adults view the bilingual Korean evaluation team member as their voice and advocate, and consequently, have unreasonable expectations of this person. It may be helpful to pair up members of the evaluation team for the data collection to check each other’s assumptions and alternate their roles, if necessary, to minimize the potential for such incidents. Or, the team could hire an advocate for seniors who is not employed by any of the centers to accompany the data collector.

Don’t use off the shelf survey questionnaires. There are several possible reasons why the survey response rate was low; one of them could be the inappropriateness of the questionnaire items, which made the questionnaire invalid. It is not unusual for evaluators to use an existing survey questionnaire because of a preference for validated instruments or budget constraints that prevent the development of a whole new instrument. These decisions, however, could compromise the quality of the data collected. Terms like “volunteerism” and “community engagement” can mean different things to people from different cultural backgrounds. It would have been preferable for the evaluation team to seek initial input on how these terms may be interpreted by people from different backgrounds, and then apply that information to develop a more culturally-sensitive questionnaire and to pilot-test it with a small sample. The evaluation team could accomplish this by conducting a couple of focus groups prior to actual data collection, or by establishing an advisory committee comprised of seniors from diverse backgrounds for guidance.

Other critical cross-cultural competency issues. The experts identified additional cross-cultural considerations in evaluations that involve seniors. First, terms like “aging,” “older adults,” “seniors” or “the elderly,” have all been used in the literature and in practice. It is important to explore which terms are preferable, by engaging a group of seniors to advise the evaluation, or asking the people who work closely with the seniors. An advisory committee comprised of older adults may require more individual or small group telephone conversations, unlike the typical advisory committees, made up of people who can easily travel and regularly meet in person.  

Second, evaluators need to be cognizant of how their interactions with the older adults at the centers could be influenced by their personal experiences (e.g., with their grandparents or aging parents, their own sense of mortality) and how their responses could affect the data collection. For instance, it is not unusual for some older adults, who are accepting of death, to speak openly about the subject. In the case study presented here, the seniors may have touched on the issue of death in discussing the legacy they wish to leave behind. This could have made the facilitator uncomfortable and consequently avoid asking in-depth questions about the support these adults might require to become active volunteers.

The reverse is also true. The older adults’ interactions with the evaluation team members could trigger memories of their children or grandchildren and cause them to respond in a certain way. Or, they might be reminded of their active years and develop a sense of resentment toward the younger team members.

Back to top