This day-long meeting involved 24 participants with extensive experience and methodological and evaluation expertise in various types of health promotion interventions, especially those in worksite, school, healthcare setting and community-based settings.  The day focused on approaches to, and recommendations for, addressing a number of challenges in evaluating the external validity of interventions, and ways to facilitate translation of health promotion research into practice.

 Following a brief discussion of work being conducted as part of the RWJF grant awarded to the Representativeness and Translation work group of the BCC, the day focused on in-depth discussion of three issues, as summarize below. (See also agenda for meeting and handouts attached as part of this report).
   

 There was lively discussion, concerning the RE-AIM framework used by the investigators in the RWJF grant, as well as more general issues. RE-AIM specific terminology and methodological issues: 

• Should Effectiveness be defined at the site or setting level also, in addition to the individual level?

• Adoption, as typically used in the literature, refers to a final stage of organizations or settings choosing to begin a program that has previously been validated; in the RE-AIM reviews, adoption has been used to refer to the number and representativeness of settings that will take part in a research evaluation of a potentially effective intervention. Perhaps a different or qualified term should be used to characterize this type of adoption/organizational participation.

• Others were concerned ways of thinking about the context for reaching individuals not be restricted to physical settings; but evaluation models be broad enough to include mediated interventions that do not occur in a particular setting, and possibly a term like “channel” might be best used to characterize this dimension.

• The Implementation dimension of RE-AIM has been restricted to actions of the program and intervention agents; perhaps it should also include “receipt” of intervention and the extent to which participants “use” the intervention materials.

General Methodological Issues: 

• There was general consensus that census, epidemiological, marketing, and economic databases be available which can be used to compare the characteristics of one’s sample of individuals and settings/sites participating to a broader population (e.g., regional, state, etc). It is more difficult to assess representativeness for community-based research programs and to identify characteristics of certain high-risk groups. Several participants felt it is important to describe the incentives used to get participants (and sites) to take part in a study as this may have implications for translation.

• There was a difference of opinion as to whether there was one key RE-AIM dimension or component for evaluating public health impact, or if several different dimensions were equally important.  Some felt that the dimension of Reach, at the individual level, was the “bottom line” for public health impact and the other factors or evaluation components served primarily as ways to impact Reach. Others felt all the various external validity dimensions were equally important, and it would be a mistake to focus solely or primarily on Reach and to de-emphasize the setting level factors.

• In general, the group concurred that there is insufficient knowledge or data at present to define a metric to empirically describe the status of an intervention on most dimensions related to external validity; rather, it makes most sense to qualitatively characterize intervention status.  Similarly, there was no agreed upon weighting system or model for combining dimensions to assess the overall public health impact of an intervention.  Some felt a type of multiplicative model would be appropriate, but others felt the relative value or importance of different external validity criteria would differ across applications.

Issues Around Using Guidelines or Templates for Reporting on External Validity Issues
(see attached sample list of questions and “flow chart” templates, which were used to focus discussion):

•  It was felt that having a standard way of reporting results would be helpful to researchers, but this should be framed as a guide or help to focus attention, rather than as a restrictive requirement that might limit innovativeness.

• In general, participants liked the flow chart template approach to reporting, but were skeptical that journals would support this much additional information being reported. Instead, needed changes in the incentive structure for investigators and the peer review process for both grants and journal articles were discussed.

• There was quite a bit of interest in the suggestion of a website on which investigators could post the information in the flow chart.  A user-friendly website could facilitate several objectives including facilitation of reporting and common usage of terms and definitions, providing feedback on how an intervention compared to others, providing a resource for meta-analyses, and generating discussion and interaction around these issues.

• It was pointed out that the external validity template can also be useful in the early stages of planning a project or proposal to include attention to reach and translation issues.

Discussion of Workgroup Survey Activity and Directions for the Future: 

• Participants provided feedback on several issues related to the survey of decision makers in organizations in BCC communities.  In general, reaction was that a more open-ended and concrete approach might elicit more than the structured questions we are currently using.

• There was a lengthy discussion of the role of cost and economic issues in adoption of health promotion programs.  All agreed cost issues are important, but there was a diversity of opinion concerning the specific aspects most important (initial start-up costs vs. ongoing implementation costs).  It was suggested that a broader term, such as resources, should be used to cover the variety of issues involved.

• It was felt that proposed meetings with groups of potential decision makers to explore issues of adoption and maintenance might not be the best focus for our grant.  A variety of recent activities related to dissemination among decision makers were described; participants suggested that the unique strength and target audience for RE-AIM was researchers and reviewers. 

General Discussion and Future Directions:

•  There was discussion of the increased attention translation issues are receiving from funding agencies, and the timing might be right to bring these issues to researchers’ attention.

• There was lengthy discussion of the need to change both the infrastructure and the incentives that serve to continue the tendency for behavioral researchers to focus predominantly or exclusively on short-term and internal validity issues.

• In general, there was consensus that the activities of the BCC RE-AIM project were helpful in raising the awareness of translation issues among researchers.  Most felt that our grant activities might be most effective if they focused on providing models, data, and ways to facilitate the reporting of this information among researchers.