The Journey Continues: Ensuring Cross-Culturally Competent Evaluation - Case Study D: Evaluation of an Initiative to Increase Health Services for Lesbian, Gay, Bisexual and Transgender (LGBT) Individuals

The situation

A hypothetical state-wide foundation funded a five-year grant project to increase health resources and services for LGBT individuals by:
  • Addressing the most urgent health challenges faced by this population
  • Establishing a system for monitoring future health trends in this population.

The evaluation

The foundation hired an evaluator who worked with the foundation’s staff to create a logic model for the project. During the logic modeling process, the two health issues in LGBT communities that emerged as critical for the project to address were: tobacco use and sexually transmitted diseases. Existing studies have shown a high prevalence of these conditions among LGBT adults in the state and nationally.

The foundation issued a Request for Proposals, and funded 10 organizations to implement health interventions that promote smoking cessation and safe sex practices. The 10 grantee organizations included community health clinics, public health departments, family resource centers and community-based organizations that provide some type of health services. Some of these organizations primarily served LBGT individuals, while others served broader populations.

The evaluator developed an evaluation design that included annual interviews with grantee organizations’ staff and focus groups with program participants. In addition, grantees were required to track and survey, on a quarterly basis, the number of LGBT individuals who accessed and used their services (e.g., health education workshops, smoking cessation programs, counseling sessions) and changed their health behaviors. The evaluator developed a standard tracking form and a survey questionnaire for the grantees to administer, in order to ensure that they all collected the same data. The survey questionnaire asked the grantees’ LGBT service recipients about their consumption of tobacco products, safe sex practices and other behaviors that indicated the extent to which they live a healthy lifestyle, such as routine health check-ups and screenings along with physical activity.

After the first year, the evaluator analyzed and compared the results from all 10 sites. These results formed the baseline for the project, particularly the number of people served by each grantee, their satisfaction with the services and their health behaviors. The results indicated that a high percentage of the LGBT service recipients smoked and engaged in unprotected sexual activity. At the same time, a relatively high percentage of them also engaged in regular physical activity and routine health check-ups and screenings.

The evaluator submitted an annual report to the foundation. The report included an executive summary. The foundation made the executive summary available to the public by announcing it in its newsletter and posting it on its website.

A few weeks after the evaluator submitted the annual report to the foundation, a reporter contacted the evaluator. The reporter had heard about the evaluation in an encounter in the community, read the executive summary on the foundation’s website and was curious to learn more. The evaluator elaborated on the findings for the reporter.

The next day, an article about the findings was published in a local community newspaper (one of the 10 grantees was located in this community) and within a few days, more exchanges about the grant and the evaluation report appeared in the newspaper. The articles and op-eds tended to be negative in nature, expressing anger about the foundation’s support of LGBT individuals and their “unhealthy” habits. The foundation also received numerous calls about its support of the project, which some of the callers perceived as condoning homosexuality, as well as calls from a couple of LBGT leaders who wanted to know more about the design of the evaluation. These LGBT leaders recognized that the study’s findings could be used to raise awareness about the ways in which health disparities affect LGBT communities and, therefore, were interested in learning more about the evaluation approach.

To further intensify the situation, there was a legal challenge to limit health care coverage for same-sex couples, and the findings from the evaluation were used – out of context – to support this legal challenge.

Commentaries by the experts

The experts shared the following ideas for how the situation could have been prevented.

Establishment of an advisory committee. It appeared that the project and evaluation were designed with minimal, if any, input from leaders in the LGBT community. Their input is essential for several reasons. First, they can help the foundation staff and evaluator understand the diversity within the community. They also can lend credibility to the effort, which in turn, can increase the participation of LGBT individuals in the project. Because such leaders tend to have more experience than outsiders about the various ways in which their community has been exploited by researchers and evaluators,7 they are also more likely to consider issues that need to be addressed in advance, to prevent harm. For instance, they may be aware of a potential legislative proposal that an “outsider” would not know about. This knowledge is critical for developing a dissemination strategy for the evaluation findings, as well as measures that the foundation and evaluator can take to prevent the findings from being used inappropriately to ostracize the LGBT community.

Finally, the advisors can help the foundation and evaluator develop a more comprehensive and nuanced understanding of the health issues experienced by LGBT individuals in the state. Studies have shown a higher prevalence of smoking among LGBT people; however, they frequently fail to explain the unique social or behavioral dynamics in LGBT communities that are linked to this widespread behavior. Evaluators might consider how LBGT communities have been stigmatized and discriminated against for their sexual identities and behaviors, and consequently, any other behaviors associated with their communities. Health issues facing the LGBT community should be carefully categorized into: 1) areas where LGBT people are at higher risk because of unique circumstances associated with their sexual orientation and identities (e.g., exposure to anti-gay violence), and 2) areas where they are at risk not because of their unique circumstances, but which nevertheless require culturally competent attention (e.g., smoking, obesity, alcohol use).8

Analysis of the political context. Like any group of people that has been marginalized and devalued in society, research about the behaviors of LGBT people and evaluation of programs that serve them have typically been conducted within a hostile environment (similar to the Native-American community described in the second case study).2  Findings have frequently been used to demonstrate social deviance and to stop funding for programs that serve marginalized groups.8 In this situation, the findings were taken out of context and used by anti-LGBT groups to justify their actions.

Funders and evaluators must do their homework and learn about any policy implications that could result from their work, and to strategize early on with the right parties to discuss this possibility and how to move forward. Such analysis should be required as part of the design for evaluations that involve any type of minority group that experiences discrimination. The American Evaluation Association’s fourth guiding principle for evaluators (Respect for People) states that, “where feasible, evaluators should attempt to foster social equity...and ensure that [participants] have full knowledge of and opportunity to obtain any benefits of the evaluation.” The fifth guiding principle (Responsibilities for General and Public Welfare) further states that evaluators “…usually have to go beyond analysis of particular stakeholder interests and consider the welfare of society as a whole.” In The Importance of Culture in Evaluation, it was suggested that evaluators consider questions like the following, because of the privilege and power they have as information holders:
  • How might the study’s findings be used by community members, politicians, policymakers, journalists and special interest groups?
  • Will the findings potentially place a stigma on a certain group, or give the group power to access resources and improve their situations?

Development of a thoughtful and deliberate dissemination strategy. It is never too early to plan for how to disseminate evaluation results, even if the findings were originally envisioned as only being used by the funder and grantees. A dissemination strategy has to be part of an evaluation that addresses sensitive issues and groups that have traditionally been harmed by research. It is the responsibility of everyone involved, particularly the funder and the evaluator to ensure that such a strategy is developed and implemented properly. An advisory group could certainly help with the creation of such a strategy. It would be helpful if all the major stakeholders, including the foundation staff, evaluator, advisory committee members and a designated spokesperson from each grantee organization, received training in how to communicate the project and evaluation findings to reporters and other interested parties. 

The experts who commented on this situation suggested that a press kit be developed and a press conference held with speakers who can address the findings and put the findings into context (i.e., circumstances in the LGBT community, systems that affect this community and how these factors contributed to the findings). They also suggested that the findings could be timed with the release of other national data related to LGBT communities, such as the annual reporting of hate crimes against LGBT people, in order to provide a more comprehensive analysis of barriers faced by LGBT people (e.g., lack of access to recreational facilities where they feel safe and to culturally competent health care providers).

Diversity within the LGBT community. A comparative analysis across the 10 grantees (as conducted by the evaluator) could be harmful if the analysis did not account for any other type of diversity within the LGBT community, such as race and ethnicity, income, generation and education level. Further, the term “LGBT” contains a broad diversity of difference across sexuality and gender identity and expression. Too often, the realities of transgendered, lesbians and gays, and bisexuals are not disentangled from the term and studied for their unique societal barriers. For instance, a Caucasian gay man has different experiences from an Asian-American lesbian who has different experiences from an African-American transgender. It is also possible that a LBGT community has more in common with a non-LBGT community of the same race and ethnicity, than with another LBGT community of a different race and socioeconomic status.8 As implied in The Importance of Culture in Evaluation, cross-culturally competent evaluators recognize the complexity of multiple social identities and how they affect data collection and analysis and their interactions with program participants. Evaluators have to be cautious about controlling for certain variables (e.g., race, education level, income, social support) to carefully determine which demographic characteristics and social conditions may have contributed to the outcomes. At the same time, they also have to be careful about disaggregating the data so much that they end up with small sample sizes, which can inadvertently make the participants’ identity known, placing them at risk for harm. As a potential solution, evaluators should take extra measures to protect the participants’ confidentiality (e.g., an on-line survey that ensures anonymity, by allowing participants to complete the questionnaire, at their convenience, using a temporary user name), understand the political context within which the evaluation is occurring (as mentioned above), and explicitly recognize the evaluation’s limitations (e.g., sampling bias).

Sampling and other biases. The experts cautioned about certain biases inherent in research and evaluations concerning LGBT communities. They note that the typical respondents are people who feel comfortable about self-identifying as a lesbian, gay, bisexual or transgender, especially if the respondents are recruited from places where they are known to gather. Therefore, individuals who are not comfortable about exposing their sexual and gender identities will not likely participate in the survey. Evaluators need to acknowledge this bias in their reports.

An article that summarized the ethical standards for research and evaluations involving LGBT populations also warned against the use of stigmatizing language that implies abnormality or social deviance among LGBT individuals.7 Evaluators also should avoid the assumption that people identify only as female and male, or married and single.9 One solution is to create an open-ended question that asks participants to self-identify their gender.

Back to top