How do we Evaluate Programmes?

Types of Communication Research
Differences Between Qualitative and Quantitative Research Methods
Qualitative Research Methods
Quasi-Quantitative Research Methods: Pretesting Messages and Materials
Quantitative Research Methods
Additional Research Methods

Types of Communication Research

Research into intended audiences’ culture, lifestyle, behaviors and motivations, interests, and needs is a key component to a health communication program’s success. This section describes communication research methods commonly used throughout program planning. See the chart Types of Research and Evaluation for more detail about research conducted in each of the stages of health communication program planning.

Most programs use more than one research method. For example, conducting exploratory focus groups with an intended audience at the start of program planning can orient you to the types of approaches, messages, and channels that are most likely to be successful with a particular group. In some cases, focus groups might be augmented with in-depth interviews to learn more about intended audience members’ motivations. Later, messages and materials might be pretested approximate how an individual would encounter them in "real life." Theater-style testing also approximates reality, using a simulated television-viewing environment. Clearly, some methods are better suited to specific purposes than others. Using multiple methods can help ensure that you get an accurate picture of your intended audience members and their likely responses to your program.

Differences Between Qualitative and Quantitative Research Methods

There are two basic types of research you might conduct with intended audiences: qualitative and quantitative. You will use methods from one of these two types depending upon what you want to learn. See the sidebar below, Qualitative Versus Quantitative Methods, for common distinctions between qualitative and quantitative research.

In this section, you will learn when to use each type of research, how to conduct research with members of your intended audience, and how you can use the data you collect to inform your project. Qualitative, quasi-quantitative, and quantitative research methods are discussed separately.

Qualitative Research

Use qualitative research methods when:

• You are planning a communication program and developing materials for it
• When the goal of your research is to explore a topic or idea
• When the goal of your research is to gain insights into an intended audience’s lifestyle, culture, motivations, behaviors, and preferences

Conduct qualitative research by:

• Selecting a small group of people chosen for particular characteristics
• Convening a discussion (i.e., a focus group or in-depth interview) or observing individuals’ behaviors in schools, malls, supermarkets, etc.
• Keeping the discussion fairly unstructured, so that participants are free to make any response and are not required to choose from a list of possible responses
• Choosing which question to ask next based on your participants’ previous responses

Qualitative research results cannot be:

• Quantified or subjected to statistical analysis
• Projected to the population from which the respondents were drawn because participants are not selected randomly (to be representative of the population as a whole) and because not all participants are asked precisely the same questions

Quantitative Research

Use quantitative research methods when:

• You are planning a communication program (e.g., to measure the prevalence of a particular behavior) or assessing a program already in place
• The goal of your research is measurement of particular variables Conduct quantitative research by:
• Selecting a large group or groups of people
• Using a structured questionnaire containing predominantly forced-choice or closed-ended questions

Quantitative research results can be:

• Analyzed using statistical techniques
• Considered representative of the population from which the respondents were drawn if each person in the population had an equal chance of being included

Qualitative Research Methods

Use qualitative research methods during the following parts of your program:

• Stage 1—to find out more about your intended audiences and to learn what the priorities and approach should be for trying to influence their awareness, knowledge, attitudes, intentions, or behaviors
• Stage 2—to determine whether your materials communicate the intended messages effectively and persuasively
• Stage 3—to understand why the program is or isn’t working as expected
• Stage 4—to learn more about what worked and what didn’t, and why certain outcomes occurred

Focus groups and in-depth interviews are the most common methods used in qualitative communication research. However, there are many innovative methods, some described here, that can help you learn about an audience. Because the methodologies for each are very similar, they are discussed together here, using instructions for focus groups as a guide.

About Focus Groups

Working from a discussion guide, a skilled moderator facilitates a 1- to 2-hour discussion among 6 to 10 participants, which can be conducted either in person or by telephone (ideally in person). The moderator keeps the session on track while participants talk freely and spontaneously. As new topics related to the material emerge, the moderator asks additional questions to learn more.

Common Uses

• Developing a communication strategy:
  • Learning about feelings, motivators, and past experiences related to a health topic
  • Exploring the feasibility of various potential actions (from the intended audience’s viewpoint)
  • Identifying barriers to those actions
  • Exploring what benefits the intended-audience members find compelling and what results they expect from taking a particular action
  • Learning about the intended-audience’s use of settings, channels, and activities
  • Capturing the language used by the intended audience to discuss issues
• Exploring reactions to message concepts (concept testing):
  • Identifying concepts that do or do not resonate and understanding why
  • Triggering the creative thinking of communication professionals
  • Illustrating to others how the intended audience thinks and talks about a health issue
• Developing hypotheses (or broad issues) for quantitative studies and identifying the range of responses that should be included in closed-ended questionnaires
• Providing insights into the results of quantitative studies by obtaining in-depth information from individuals typical of the intended audience to help understand why individuals responded in certain ways

Pros

• Group interaction can help elicit in-depth thought and discussion.
• Group interaction can help with brainstorming because respondents can build off one another's ideas.
• Moderators have considerable opportunity to probe responses.
• Focus groups yield richer data than surveys about the complexities of an intended audience’s thinking and behavior.

Cons

• Findings are not generalizable to the population.
• Focus groups can be labor intensive and expensive, especially if sessions are conducted in multiple locations.
• Group responses do not necessarily reflect individuals’9 opinions because some individuals in the group may dominate the discussion or may influence others’ opinions.
• Each person is limited to about 10 minutes of talking.

About In-Depth Interviews

The process, benefits, and drawbacks of in-depth interviews are similar to those of focus groups, except that the interviewer speaks with one person at a time. In-depth interviews can take place at a central facility or at the participant’s home or place of project/programme purpose. As with focus groups, when individual interviews cannot be conducted in person, they can be conducted by phone or over the Internet. Although these interviews are more time intensive, one of their key benefits is that each respondent is isolated from other respondents and therefore not influenced by what others say.

How to Design and Conduct a Focus Group or In-Depth Interview Study

To design and conduct a qualitative research study, complete the following steps.

Plan the Study

Determine the following:

• What you want to learn. Determine the objectives of your study at the outset, and then check to make sure that the moderator’s/interviewer’s guide includes lines of questioning that will provide the answers.You may also use the objectives to help analyze the results of the discussions and to organize the focus group or in-depth interview report.
• When you need to have that information.
• How you will apply what you learn. It is important to decide how you will use your focus group or in-depth interview results before you conduct your study.
• Your budget.
• Your criteria for who should participate.
  Select people who are:
  • Typical of your intended audience (the same behavioral, demographic, and psychographic characteristics).You may want to conduct separate groups with "doers" (those who already engage in the desired behavior) and "nondoers" (those who do not) to help identify what actions the doers take, and why, so that those approaches can be explored with the nondoers.
  • Not experts. Exclude market researchers and advertising professionals (because of their familiarity with the methodology) and those who have, or might be perceived by other group members as having, expertise in the subject matter (e.g., exclude health professionals from focus groups when the topic is a health issue).
  • Relative newcomers to focus groups or interviews, so that their reactions will be spontaneous. This will help you avoid "professional" respondents (i.e., those who have participated in many previous focus groups or individual interviews before) who may lead or monopolize the discussion.
• The number of groups you will convene.
  • Divide participants into different focus groups based on their gender, race, age, level of formal education, or any other variable likely to hinder their freedom of expression (e.g., teenage girls will be more comfortable discussing sexual activity if teenage boys or college-age women are not in the group).
  • Conduct a minimum of two focus groups with each intended audience segment (e.g., if you are conducting separate groups with men and women, you will need at least four groups–two with men, two with women). If intended audience perceptions vary or the audience feedback is unclear, you may want to conduct additional groups with each segment, especially if you revise the discussion guide to more fully explore unresolved issues.
  • If you are using in-depth interviews, conduct approximately 10 interviews per intended audience segment. If common themes do not emerge or the interviews, especially if you revise the interview guide in between interviews.

Choose the Location

You can convene focus group discussions or in-depth interviews in a variety of ways:

• Commercial focus group facilities can recruit participants for you (for both focus groups and interviews) and offer audio and video recording equipment as well as observation rooms with one-way mirrors for viewing. However, these facilities are usually available only in larger metropolitan areas.
• Teleconference services can set up telephone focus groups for you. Most allow observers to listen without being heard, and some provide remote viewing programs to allow the moderator to see a list of participant names (with a symbol next to the one currently speaking) or notes sent in by a technician from observers listening to the call. Some teleconference services can recruit participants; with others, you will have to recruit participants or contract with a recruiter separately.
• You can also conduct focus groups or in-depth interviews in meeting rooms at churches, office buildings, or other locations. If an observation room with a one-way mirror is not available, staff may still listen in through speakers hooked up in a nearby room or by audiotaping or videotaping the session. In some cases, one or two quiet observers may be allowed in the room to take notes.

See the sidebar Pros and Cons of Different Formats below for the advantages and disadvantages of different formats for focus group and in-depth interview research.

Draft a Recruitment Screener

A recruitment screener is a short questionnaire that is administered to potential participants, typically by telephone, to ensure that they meet the criteria you developed during step 1. Ideally, the screener should help you exclude participants who are already familiar with the specific subject of the sessions, or who know each other. If participants know the subject in advance, they may formulate ideas or may study to become more knowledgeable about the subject than the typical intended audience member. If participants know each other, they may speak less freely. See Appendix A for a sample screener.

Recruit Participants

Recruit participants by telephone one to three weeks in advance of the sessions. You can identify potential participants in different ways depending upon the type of people you are seeking and the resources you have available. Identify members of the public through focus group facility databases, by running an ad in a local publication, by working with community organizations , or by using the phone book (although the latter will be extremely time consuming if you have stringent recruitment criteria). Identify professionals through a relevant association or mailing list service or through a focus group facility’s recruiting databases. Depending on your budget and internal resources, you may choose to recruit in one of the following ways:

• Use your own organization’s staff to recruit participants.
• Hire a focus group facility or independent recruiter (to identify such facilities, consult a directory such as the American Marketing Association’s Green Book or the Marketing Research Association’s Blue Book). If you have many facilities and recruiters to choose from, consider getting recommendations from local organizations or organizations that conduct qualitative research.
• Get help from a university marketing research or advertising class.
• Work through gatekeepers such as teachers (for students), health care providers (for patients, physicians, or nurses), religious institutions or community organizations (a small donation may encourage them to participate), and instructors of English as a second language.

Regardless of how the recruiting is done, ensure that the screener is followed carefully so that only individuals who qualify for participation will be included.

Getting People to Show Up

To ensure that enough people show up, offer an incentive (usually money) and recruit two or three more people than you actually need. If all show up, select those who best fit the screening criteria, thank the extra participants, give them the agreed-upon incentive, and dismiss them. Other ways to increase participation include:

• Scheduling sessions at times that are convenient for your potential participants (e.g., at lunch or after work)
• Choosing a safe and convenient site with easy parking
• Providing transportation (or reimbursement for agreed-upon transportation costs)
• Arranging for childcare, if necessary
• Letting them know you will be serving refreshments

Recruiting Patients and Their Families

Recruiting patients and their families requires special consideration. Contact clinics, hospitals, or local HMOs for help and make adequate plans to ensure that the participants and their family members are not inconvenienced or upset. Some organizations may require institutional review board (IRB) approval of your research. Gaining IRB approval is often a long process, so be sure you check with the organization early in the planning stage of your study to find out whether you will need IRB approval.

Recruiting for Telephone Interviews

If you are recruiting for in-depth interviews to be conducted on the telephone, create a spreadsheet that includes spaces for the following information about each potential participant:

• Time zone in which the person is located
• Date, time, and telephone number at which he or she should be called for the interview
• Disposition of each call (e.g., scheduled an interview, no answer, busy, refused)

Develop a Moderator’s Guide

The moderator’s guide tells the moderator what information you want from the groups and helps him or her keep the discussion on track and on time. It is only a guide, however. During the focus groups, experienced moderators flow with the conversation, asking questions in the prescribed language and sequence when possible but sometimes deviating from the guide to avoid awkward transitions or unnecessary back-and-forth between topics.

Before you draft the moderator's guide, answer the following questions:

• What do we want to learn from the focus group?
• How will we apply what we learn?
• What tools (e.g., descriptive information, message concepts, or other draft creative work) will we need to provide?

Then, write questions for the guide that relate to the purposes you have identified. Make most questions open-ended so that participants can provide more in-depth responses than just "yes" or "no," but make sure the questions are not leading. This will help you get answers that reflect participants’ true feelings and not what they think you would like to hear. The amount of time and depth of questions devoted to each issue should reflect the value of the issue to the research. See Appendix A for an example of a moderator’s guide.

Do not include questions for group discussion when you need individual responses. However, you can have the moderator give self-administered questionnaires to each participant to be completed prior to conducting a focus group, or participants can be asked to individually rank items on paper—such as potential actions, benefits, or message concepts—during a group to obtain both individual and group reactions.

Conduct the Focus Groups

Focus groups typically begin with the moderator welcoming participants and briefing them on the process (e.g., all opinions welcome—there are no right or wrong answers; the presence of audio- and videotaping and observers; the importance of speaking one at a time; confidentiality). Participants introduce themselves to the group by first name, usually including some information relevant to the topic of discussion (e.g., number of years with glaucoma, amount/type of insulin used each day). Next, the moderator asks a few simple "ice-breaker" questions to help participants get used to the group process and to reduce participant anxiety. This also helps the moderator develop rapport with the participants.

Continuing to follow the moderator’s guide, the moderator manages the group and ensures that all topics are covered without overtly directing the discussion. Participants are encouraged to express their views and even disagree with each other about the discussion topics. The moderator does not simply accept what participants say but probes to learn more about participants’ underlying thinking and attitudes. The moderator also seeks out opinions from all participants so that all are heard and a few do not dominate the discussion.

Near the end of the discussion, the moderator will often give participants an activity or simply excuse him- or herself from the room for a moment to check with the observers and obtain any additional questions. Alternatively or additionally, notes can be sent in to the moderator while the group is in process if the observers would like different questions asked or other changes made to the group.

One advantage of focus group methodology is that the moderator’s guide, and any materials presented, can be revised between groups if necessary.

Analyze Results

The easiest and most thorough way to analyze focus groups is by reviewing transcripts, although groups can also be analyzed (albeit less thoroughly) by reviewing notes taken during the discussion. In many analyses, the goal is to look for general trends and agreement on issues. At the same time, it is important to note divergent opinions. Don’t ignore individual comments that raise interesting ideas or concerns such as lack of cultural sensitivity or difficulty in comprehension. In some instances, the goal is to capture the range of opinions about an issue, rather than to look for evidence of agreement or consensus.

Avoid counting or quantifying types of responses (e.g., "75 percent of participants preferred concept A"). Attempting to quantify the results—or suggesting in other ways that they represent the opinions of the intended audience as a whole—is inappropriate for qualitative research.

Quasi-Quantitative Research Methods: Pretesting Messages and Materials

Some commonly used communication research methods, such as central-location intercept interviews and theater tests, are best termed quasi-quantitative. While these methods are used in situations in which the goal is measurement and typically involve a questionnaire with mostly forced-choice questions, the results cannot be projected to the population as a whole (as with true quantitative surveys) because of the way in which participants are selected. For centrallocation intercept interviews, the only people who have a chance to participate are those who go to the location where the interviews are being held and who go there during the times they are conducted; this is not a truly representative sample of the intended audience. For theater tests, the only people who have a chance to participate are those who are recruited for the test, and recruitment does not follow a truly representative sampling design.

Quasi-quantitative methods are most often used during Stage 2 to pretest messages and materials. If your intended audience is geographically dispersed or it is difficult for them to get to a central facility, you can use telephone interviews and send participants any materials in advance. This type of pretest typically resembles an in-depth interviewing project in price and number of interviews, although there may be more closed-ended questions and the question sequence may be adhered to more closely.

Central-Location Intercept Interviews

Central-location intercept interviews consist of stationing interviewers at a point frequented by individuals from your intended audience and asking the individuals to participate in a study. If they agree, they are asked specific screening questions to see whether they fit the study criteria. If so, the interviewer takes them to the interviewing station (a quiet spot at a shopping mall or other site), shows the pretest materials, and then administers the pretest questionnaire.

For intercept interviews to be effective, you must obtain results from a minimum of 60 to 100 respondents from each intended audience segment you want to test.

Pros

• You can connect with harder-to-reach respondents in locations convenient and comfortable for them.
• The interviews can be conducted quickly. (The interview should be no longer than 15 to 20 minutes.)
• The interviews are a cost-effective means of gathering data in a relatively short time.
• If you choose an appropriate location, you will increase your chance to interview respondents who are among your intended audience.

Cons

• You must train interviewers.
• Your results are not representative or generalizable.
• Intercept interviews are not appropriate for sensitive issues or potentially threatening questions.
• Intercept interviews do not allow you to probe easily for additional information.

Central-location intercept interviews should not be used if respondents must be interviewed in depth or on emotional or sensitive subjects. The intercept approach also may not be suitable if respondents are likely to be resistant to being interviewed on the spot. In cases in which central-location intercepts will not work well, schedule interviews with respondents instead.

 

Questionnaire Contents

Unlike focus groups or in-depth interviews, the questionnaire used in central-location intercept pretesting is highly structured and contains primarily multiple choice or closedended questions to permit quick response. Open–ended questions, which allow free–flowing answers, should be kept to a minimum because they take too much time for the respondent to answer and for the interviewer to record. Questions that assess the intended audience's comprehension and perceptions of the pretest materials form the core of the questionnaire. A few additional questions, tailored to the specific item or items being tested ("Do you prefer this picture—or this one?"), may also be included to meet your program planners' particular needs. The questionnaire should be pretested before it is used in the field. See Appendix A for a sample questionnaire.

Interview Setup

A number of market research organizations throughout the country conduct central–location intercept interviews in shopping malls.You can also conduct these interviews in clinic waiting rooms, religious institutions, Social Security offices, schools, work sites, train stations, and other locations frequented by members of your intended audience. Be sure to obtain permission well in advance of the time you want to set up interviewing stations in these locations.

If you are using a market research organization to conduct the interviews, you will need to provide screening criteria, test materials, and the questionnaire. In some cases, market research organizations have offices in shopping malls, and you can watch the testing through a one-way mirror.

Participant Recruitment

If you or someone in your organization is recruiting the participants, you will need to develop a script and provide training in approaching members of the intended audience. For example, if you are recruiting participants in a clinic waiting room, the interviewer should be familiar with the screening criteria (e.g., women under 60 years of age) and approach only those people who appear to fit the criteria. When, after screening, individuals do not qualify to participate, the interviewer should thank them for their time and indicate that this study is not the right fit for them but that their willingness to participate is appreciated. If they do qualify, the interviewer can bring them to a designated location (e.g., another room or corner of the waiting room) and proceed with the study.

University and college departments of marketing, communication, or health education may be able to provide interviewer training or trained student interviewers. Pretesting is an excellent real–world project for a faculty member to adopt as a class project or for a master's student to adopt as a thesis project. However, this approach may mean that it takes longer to accomplish the research, and you could compromise the quality of the results if the individuals are not experienced in this type of research.

Theater-Style Tests

Theater testing is often used in the commercial arena to test advertisements for products and services. Theater testing can also be used to test the effectiveness of PSAs. In this methodology, participants are invited to a central location to respond to a pilot for a new television show; in the midst of viewing the TV pilot, they are shown your PSA or advertisement along with other ads. Participants complete a questionnaire following the presentation, first answering questions about the show and then answering questions about how effectively your message was communicated to them and what their overall reactions were.

Common Uses

Theater–style tests are most commonly used to test TV advertisements and PSAs. For theater–style tests to be effective, you must obtain results from 50 to 100 respondents from each segment you want to test.

Pros

• You can obtain responses from a large number of respondents at the same time.
• Theater–style tests more closely replicate what goes on in someone's home when they are watching TV, so you can accurately judge people's reactions to your message.
• Theater–style tests can be cost–effective if you use donated facilities and equipment.

Cons

• It is expensive to rent a facility and equipment (if necessary).
• Your results are not representative of the general population.

General Format

Individuals typical of your intended audience are invited to a conveniently located meeting room. The room should be set up for screening a television program. Participants should not be told the real purpose of the session, only that their reactions to a television program are being sought.

At the session, participants watch a television program. The program can be any entertaining, nonhealth video approximately 15 to 30 minutes in length. The videotape is interrupted about halfway through by a sequence of four commercials. Your message should be inserted between the second and third commercials. See Appendix A for a description of how to create a roughcut video for theater-testing your message.

At the end of the program, participants receive a questionnaire and answer questions designed to gauge their reactions, first to the program and then to the advertisements. Finally, your ad is played again and participants complete several questions about your ad. The majority of these questions should be closed-ended to enable an easy and accurate summary of participant responses.

In more sophisticated theater testing, participants use automated intended audience response systems to answer questions. Participants are provided with a small device that has response keys. Once a question is asked, they push a key to respond and the data are automatically tabulated.You have instant access to the numbers using this system. In addition, questions can be instantly added or deleted from the questionnaire based on the previous responses. Using an automated system is much more costly than using a standard paper-and-pencil questionnaire.

Other Media You Can Test

This methodology can also be used to test videos by asking participants to view a series of videos in which yours has been included. Examples of videos that might be tested include a 15- to 30-minute breast cancer awareness video that will be played in a clinic or a "how-to" video on administering epinephrine. These testing sessions will, of course, last longer than those testing ads. Participants evaluate the videos as described above.

Print advertisements can also be tested using a variation of this methodology. Several ads, including yours, are inserted into a magazine. Participants are given an adequate amount of time to read through the article, which includes your ad and others. After reading the article, participants receive a questionnaire and answer questions designed to gauge their reactions, first to the article and then to the advertisements. Finally, your ad is displayed alone and participants complete several additional questions.

Designing and Conducting a Theater-Style Pretest

The process for conducting a theater–style test includes the following steps:

1. Planning the pretest
2. Developing the questionnaire
3. Recruiting respondents
4. Preparing for the pretest
5. Conducting the pretest
6. Analyzing the pretest

You may find step 2 also useful for central–location intercept interviews.

Plan the Pretest

Determine:

• The purpose of the study (e.g., what do you want to learn?)
• When you need to produce results
• What your budget is
• The type and number of people who should participate in the pretest
• The locations where the pretest will be conducted

To conduct theater testing, you must have a large enough space to accommodate all of your participants at the same time. You must also ensure that you have several video monitors so that all participants can adequately view the program. Space constraints may be overcome by seeking out low-cost facilities such as a school auditorium or church hall.You may be able to borrow the audiovisual equipment from these facilities as well. You can also rent space, such as a hotel ballroom, if you want to test a large number of people. Hotels often rent audiovisual equipment as well. Reserve facilities and equipment well in advance of your pretest.

Develop the Questionnaire

To gather useful information from the pretest, you must carefully construct the questionnaire. See the sidebar Components Used in Most Questionnaires on the next page for general guidelines. Once you have written your questionnaire, be sure to test and revise it before you use it with a large number of respondents.

Recruit Respondents

Participants may be recruited through a market research facility or through local community organizations . In either case, you will need to provide an incentive for participants. If using a market research facility, you will also incur recruiting expenses. If you are working with a community organization, you may choose to make a donation.

Prepare for the Pretest

• Is the pretesting videotape ready for use? Are the video and audio portions clear?
• Is the videotape equipment—recorder and television monitors—functioning properly?
• Is the facility set up? Is the room reserved? Are there enough chairs? Are extra chairs available in case more people show up than you expect? Do you need another monitor so that everyone will be able to see the program? Is the heat or air conditioning working properly? Do you know where the light switches are? If a microphone is needed, is it set up and functioning properly?
• Have you made all the necessary staffing arrangements? Are your assistants coming to the session? Do they have transportation and directions for getting there?
• Have you made enough copies of your pretest questionnaire (including some extras)? Is each questionnaire complete (with no pages missing)? Do you have pencils for participants? Will they need clipboards or pads?
• Has participant recruitment taken place as scheduled? Did you call and remind participants to attend? Do they have transportation and directions?
• Has the moderator rehearsed?

Conduct the Pretest

• Be friendly and courteous to participants from the moment they arrive until they leave (remember to say "thank you").
• Keep calm and cool-headed throughout the session.
• Anticipate problems in advance (conducting a rehearsal to make sure that both equipment and timing work is a good idea).

The test session should take no more than 1 hour and 15 minutes if you are organized and well prepared. Follow the steps below to conduct your test:

1. Encourage participants to take a seat as they arrive. Close the doors no later than 10 minutes after the
   scheduled starting time.
2. When everyone is seated, introduce yourself by your name only (assuming you are the host). Do not
   tell participants the name of your organization during the session because it might bias their responses to your test ad.
3. Thank participants for coming and assure them that the evening should be enjoyable and that they will have
   a chance to give their views to the producers of "new" television program material. Discourage participants from talking to one another during the session. Tell them you are interested in their own individual views and that there are no right or wrong answers. Also, encourage them to write their answers clearly in the space provided on the questionnaire.
4. After your introductory remarks, have your assistants hand out the questionnaires (see Appendix A for a sample), pencils, and clipboards (if needed). Instruct the participants not to open the questionnaire until you ask them to do so. Turn on the video recorder and monitor to begin the test session.
5. Be attentive and watch for any problems with the sound or picture on the monitor. Be sure that the equipment is functioning properly throughout the program.
6. Be prepared to stop the recorder when the television program has ended. Introduce the questions, and thank the participants for their help so far. Ask them to open their questionnaires and complete the questions on the first page.
7. When the participants have finished Part I of the questionnaire, tell them that you would like to gather their reactions to the messages/PSAs that were shown during the program. Have them turn to Part II and instruct them to fill out the questions about the messages.When they have completed these questions, tell them that you want to obtain their reactions to one particular message in the series of messages they viewed.
8. Start the video. (Note: To avoid an awkward pause in the session’s pace, be sure there is not too much lead tape before the message starts.) After your PSA/ad has been replayed, ask participants to turn to the next page of the questionnaire and complete the remaining questions. Encourage them to answer every question and to avoid giving more than one answer, except when this option is indicated on the questionnaire.
9. Circulate through the room to monitor progress and to be sure participants are not discussing their responses. Collect the questionnaires as participants finish.
10. Thank participants for their cooperation. If you have incentives or token gifts, distribute them to participants as they leave. If you have provided a donation to a group in lieu of payment to participants, mention that you hope the group will find the donation helpful.

Analyze the Pretest

At this point, look at the overall results:

• What did you learn from the pretest?
• Did your message receive a favorable and appropriate intended audience reaction?
• Did your message fulfill its communication objectives?
• What are your message's strengths? Weaknesses?
• Did answers to any particular question stand out?

Use your answers to these questions to decide whether your message is both effective and appropriate and whether you need to revise your message prior to program implementation.

Diaries and Activity Logs

Other tools you can use to evaluate your program are diaries and activity logs. If you plan to use these tools to gauge the quality of program planning or execution, be sure to start keeping the diaries and activity logs as soon as you begin program planning. For each activity, request information in a specific format from program managers or participants. This information may cover issues such as the quality of program components or track how your intended audience uses the components.

Common Uses

• Track program implementation
• Assess effectiveness of program implementation
• Monitor whether planned activities are being conducted on schedule and within budget
• Learn what questions program participants had
• Learn what technical assistance was needed by program staff
• Track intended audience exposure to program components

Pros

• Allow respondents flexibility in their responses
• Enable researchers to observe behavior over time, rather than only once

Cons

• Require considerable effort on respondents' parts (for this reason, consider offering incentives for completion of the diaries/logs)
• Require staff able to code voluminous and challenging incoming data
• Are not appropriate for respondents who have low literacy skills or who have poor writing skills or penmanship

Instituting Diary/Activity Log Use

Steps in instituting the keeping of diaries and activity logs are:

1. Planning the study
2. Identifying who will participate
3. Developing and pretesting the form you will use
4. Collecting data
5. Analyzing results

Follow the steps below to institute the keeping of diaries and activity logs.

Plan the Study

Determine the following:

• What you want to learn
• How much information you need to collect
• How you will apply what you learn
• When you need the information
• What your budget is
• Your criteria for who should participate

Identify Who Will Participate

The sample you select depends on the goals of your study. If you are focusing on program implementation, you will want the diaries/logs to be completed by program staff (e.g., nurses in a clinic). In this case, you may have some control over the quality of responses you receive.

When planning the study, you must obtain permission from a manager or supervisor on site for staff to complete the diaries/logs during the study.You should provide an estimate of the amount of time and effort participation will entail (e.g., 15 minutes per day, 1 hour per day). Share drafts of the diaries/logs and get input from the supervisor prior to the study. This will help to ensure cooperation during the study.

Before the study begins, you should train staff to complete the diaries/logs. Even if it seems obvious to you, it is essential that you explain exactly what you want recorded in the diary/log. (See the sample log in Appendix A.) In addition, you should provide detailed, written instructions for future reference. These instructions can be used in lieu of training if you cannot physically get to the study site.

If you are focusing on participant experience with a program, you will want the diaries/logs to be completed by people who were exposed to program components. In this case, you will have much less control over the quality and quantity of responses.

Obtaining cooperation from participants may also be more difficult in this situation. For example, people attending an educational program on nutrition might be recruited to complete a diary of what they eat for a week and send it back to the researchers.You will likely need to provide an incentive (e.g., a gift certificate upon receipt of the completed diary), and you may also need to remind participants to send back the diaries at the end of the study period.

Develop and Pretest the Form You Will Use

Create questions. Write questions that are specific to your study objectives.

Examples of the types of information you might collect include:

• For a toll-free hotline activity log:
  • Date?
  • What is the gender of the caller?
  • Where did the caller get the number?
  • Did the caller request any educational materials?
  • What questions did the caller ask?
• For a health education program diary:
  • Date?
  • Which module was used?
  • Which of the suggested activities were completed?
  • How long did it take to complete the suggested activities?
  • Would you use these activities again?
  • Why or why not?
• For a clinic observation activity log:
  • Date?
  • How many people passed by the display containing the new breast cancer information?
  • How many stopped to look at the information?
  • How many took a brochure?

Pretest the diary/log. Once you have created the draft diary/log, you must pretest it with individuals who represent your intended audience. Describe the scenario for them before the pretest. For example, in the case of a hotline, you might say, "You are an operator on a hotline. People will be calling in, and you will need to fill out this activity log as you complete each call." Sit together with them and ask them to read each question aloud and tell you what they think they are supposed to do. Do not correct them if they do not say what you intended. This probably means that your diary/log is unclear. Continue through the entire diary/log and then ask them if there was anything that they found confusing or unclear. Pretest the diary/log with everyone as planned before you make any changes.

Revise the diary/log. Revise questions that people found confusing during the pretest. If a question was confusing only to one person, use your judgment to decide whether to change the question. Ask yourself whether there is something you can easily fix that would have helped that one person understand the question (e.g., providing an example). If so, you may be able to make a simple change or addition to clarify the question. Also consider whether this respondent found many of the questions confusing while other respondents had no problem with them. If this is the case, you may not want to make changes.You will have to decide on a case-by-case basis. If you make substantial changes to the diary/log, you should conduct another pretest before finalizing the form.

Collect Data

Produce diaries/logs in sufficient quantities so that respondents have several extra forms in case they make errors or need more space. Deliver the diaries/logs to respondents, along with detailed written instructions, prior to training (if applicable) or at least 1 week before the study begins. If you are asking program participants rather than program staff to complete diaries/logs for you, you will have to distribute the materials on site. Give respondents a fixed time frame to complete the diaries/logs and provide them with a means (envelope/postage) to return the data to you. If your study is longer than a week or two, you may want to ask respondents to ship the first week of data to you so that you can review the logs for accuracy and completeness and even begin to tally some of the information.

Analyze Results

In the planning phase, you determined what you wanted to learn from the study. Now you will have the chance to look through the diaries/logs to answer these questions. Diaries generally contain qualitative information (e.g., how food choices were made that day, evaluation of programs completed). Activity logs may contain several types of information—quantitative information you can tabulate easily (e.g., how many people called a hotline each day, whether people picked up a brochure) as well as qualitative information (e.g., reasons that students liked or participated in an activity).

Analyzing qualitative responses. The best way to analyze qualitative information is to read through the information, searching for similarities and differences between diaries. You will need to consider all of the questions that you determined were important in the planning phase. Once you have reviewed several diaries, you should be able to pull out general themes or patterns from the information. The best way to analyze these themes is to develop categories for the responses. For example, if you want to know why teachers thought their students liked or disliked a certain educational module in your program, you might group responses into categories such as "challenging," "fun," "too much work," "boring." Continue reading through the remaining diaries and see how many responses fall into these categories. As you go along, you may come up with additional categories or decide to collapse several categories together.You can certainly make inferences (e.g., "Teachers liked the module becau...") about diary information, but resist the temptation to quantify this information.

Analyzing quantitative responses. The easiest way to analyze these types of responses is to create a coding sheet for each quantitative question. Use a separate sheet for each question, writing the question at the top and creating columns for each possible response. For example, for a question about how many people picked up particular brochures, you could create columns for the following categories: 0, 1–5, 6–10, 11–15, 16–20, >20.

Use the following procedure to record the responses:

1. Take the first activity log and record the response by making a check mark in the appropriate column.
2. Repeat this procedure for every questionnaire.
3. Tally the total number of check marks in each column and then calculate the percentage of participants who gave each type of response.

Quantitative Research Methods

Use quantitative research methods during the following parts of your program:

• Stage 1—to obtain information on prevalence of relevant knowledge, attitudes, behaviors, and behavioral intentions
• Stage 3—to monitor usage of materials and intended audience awareness of the communication program and its various tactics
• Stage 4—to measure progress toward objectives

Two different quantitative research methods, surveying and readability testing, can be used.

Surveys

Surveys are characterized by large numbers of respondents (100 or more) and questionnaires that contain predominantly forced-choice (closed-ended) questions.

Common Uses

Used in planning and assessment to obtain baseline and tracking information on knowledge, attitudes, behaviors, and behavioral intentions

Pros

• Provides generalizable results (to generalize to a broader population, you must have a statistically valid random sample)
• Can be anonymous (useful for sensitive topics)
• Can inorganizational visual material (e.g., can pretest prototype materials)

Cons

• Limited ability to probe answers
• Potential bias from possible respondent self-selection

Most surveys are custom studies that are designed to answer a specific set of research questions. Some surveys, however, are omnibus studies, in which you add questions about your topic to an already existing survey. A number of national and local public opinion polls offer this option.

Follow these steps to conduct a survey:

1. Plan the study
2. Determine how the sample will be obtained and contacted
3. Develop and pretest the questionnaire
4. Collect the data
5. Analyze results

Sampling size and composition, questionnaire design, and analysis of quantitative data are complex topics beyond the scope of this book. If you are planning a quantitative study, see the reference list at the end of this book for additional information.

Additional Research Methods

Gatekeeper Reviews

Public and patient education materials are often routed to their intended audiences through health professionals or other individuals or organizations that can communicate with these audiences for you. These intermediaries act as gatekeepers, controlling the distribution channels that reach your intended audiences. Their approval or disapproval of materials can be a critical factor in your program’s success. If they do not like a poster or a booklet or do not believe it to be credible or scientifically accurate, it may never reach your intended audience.

Common Uses

Gatekeeper review of rough materials is important and should be considered part of the pretesting process, although it is not a substitute for pretesting materials with intended audience members. Neither is it a substitute for obtaining clearances or expert review for technical accuracy; these should be completed before pretesting is undertaken. Sometimes, telling gatekeepers that technical experts have reviewed the material for accuracy will reassure them and may speed their approval of your material.

Methodology

The methodology you should use for gatekeeper review depends upon your available resources, time, and budget. Common methods include:

• Self-administered questionnaires—
  Participants are sent the materials and the questionnaire at the same time.
• Interviewer-administered questionnaires—
  Typically, an appointment for the interview is scheduled with the gatekeeper, and the materials are sent for review prior to the interview.

Develop questionnaires that ask about overall reactions to the materials and for assessment of the information’s appropriateness and usefulness.

In some cases, you might not use a formal questionnaire (especially if you don’t think the reviewer will take the time to fill it out) but will instead schedule a telephone conversation or a meeting about the materials. If you are not using a questionnaire, consider in advance what kind of questions you want to ask in the meeting or interview and determine whether you need formal approval of the materials. A discussion with gatekeepers (e.g., a television public service director, the executive director of a medical society) at this point can also be used to solicit their involvement in a variety of ways that extend beyond materials development.

Readability Testing*

Readability formulas often are used to assess the reading level of materials. Fry, Flesch, FOG, and SMOG are among the most commonly used. Applying these formulas is a simple process that can be done manually or by using a computer software program. Each method takes only a few minutes.

Typically, readability formulas measure the difficulty of the vocabulary used and the average sentence length. In addition, computer software programs analyze a document’s grammar, style, word usage, and punctuation, and assign a reading level. These formulas, however, do not measure the reader’s level of comprehension.

Readability software programs are available at computer stores. Some software programs, such as Microsoft Word, include a readability-testing function. (Note: Mention of software products does not constitute an endorsement by the National Cancer Institute.)

Researchers James Pichert and Peggy Elam suggest three principles for using readability formulas effectively:

1. Use readability formulas only in concert with other means of assessing the effectiveness of the material.
2. Use a formula only when the text’s intended readers are similar to those on whom the formula was validated.
3. Do not write a text with readability formulas in mind.

Before you choose a readability testing method, decide on an appropriate reading level for the materials you’ve written. Then use readability testing to determine whether your text corresponds to the reading level you want.

The term reading level refers to the number of years of education required for a reader to understand a written passage. Some experts suggest aiming for a level that is two to five grades lower than the highest average grade level of your intended audience to account for a probable decline in reading skills over time. Others note that a third- to fifth-grade level is frequently appropriate for low-literacy readers. Keep publications as simple as possible to increase reader comprehension of the material.

Readability Testing Methods

You can test readability easily using a formula such as Fry, Flesch, FOG, or SMOG. These tests can be done quickly to indicate any problems with the drafted text. They do not involve the intended audience.

SMOG

To calculate the SMOG reading grade level of a written sample, begin with the entire written work that is being assessed, and follow these four steps:

1. Count off 10 consecutive sentences each near the beginning, in the middle, and near the end of the text.
2. From this sample of 30 sentences, circle all of the words containing three or more syllables (polysyllabic), including repetitions of the same word, and total the number of words circled.
3. Estimate the square root of the total number of polysyllabic words counted. Do this by finding the nearest perfect square and taking its square root.
4. Finally, add a constant of three to the square root. The resulting number is the SMOG grade or the reading grade level that a person must have reached to fully understand the text being assessed.

A few additional guidelines will help to clarify these instructions:

• A sentence is defined as a string of words punctuated with a period, an exclamation point, or a question mark.
• Hyphenated words are considered one word.
• Numbers that are written out should also be considered, and if in numeric form in the text, they should be pronounced to determine whether they are polysyllabic.
• Proper nouns, if polysyllabic, should be counted, too.
• Abbreviations should be read as unabbreviated to determine whether they are polysyllabic.

Not all pamphlets, fact sheets, or other printed materials contain 30 sentences. To test a text that has fewer than 30 sentences:

1. Count all of the polysyllabic words in the text.
2. Count the number of sentences.
3. Find the average number of polysyllabic words per sentence as follows:
   Average = Total # of polysyllabic words
                          Total # of sentences
4. Subtract the total number of sentences from 30.
5. Multiply that number by the average.
6. Add that figure to the total number of polysyllabic words.
7. Find the square root and add a constant of 3.

Perhaps the quickest way to administer the SMOG test is by using the SMOG conversion table. Simply count the number of polysyllabic words in 30 sentences (chains of 10 each from the beginning, middle, and end of the text) and look up the approximate grade level on the chart. See the sidebar Example Using the SMOG Readability Formula on the next page for an example of how to use the SMOG Readability Formula and the SMOG conversion table. In the sidebar, each of the 3 sets of 10 sentences is marked with brackets.

Readability Testing With the Intended Audience*

Other methods of evaluating reading levels and comprehension include having your intended audience pretest your materials. The WRAT or the Cloze technique can be used to do this. These types of testing are useful when you suspect that the intended audience may encounter difficulties with the material. Including pretest participants who have the same characteristics as the lowliteracy intended audience you are trying to reach is critical to the validity of your pretest results. Recruiting participants through groups or settings that include people with limited literacy skills is a logical starting point. But the only way to be sure your pretest volunteers read at the same level as your intended audience is to test their reading skills.

The Wide Range Achievement Test (WRAT) is used to measure reading levels, and the Cloze technique is used to measure comprehension. To avoid offending or causing discomfort to those whose reading ability you are testing, you can integrate a WRAT or a Cloze test into the pretest interview. For example, in a recent pretest conducted by the National Cancer Institute, the interviewers introduced the WRAT test as the last part of the pretest. They stated, "Thank you for helping with the questions on the chemotherapy booklet.We need your help with one last part—a word list. This will take only a few minutes. The word list will help us understand how difficult the words are in the chemotherapy booklet." This integrated approach spared participants the pressure or potential embarrassment of "failing a reading test."

The WRAT is based on word recognition and does not measure comprehension or vocabulary. The WRAT is an efficient way to determine reading levels and takes only a short time to administer.

The WRAT involves listening to the participant read from a prepared list of words, arranged in increasing order of difficulty. Pronouncing the word correctly shows that the reader recognizes the word. The WRAT focuses on recognition because, at the most basic level, if a person does not recognize a word, comprehension is impossible.

The test is over after the reader mispronounces 10 words. The test administrator notes the level at which the last mispronunciation occurred. The "stop" level equates to a grade level of reading skills.You can compare this level with the reading level of your intended audience to see if your pretest readers are a representative match for that audience.

The Cloze technique measures the reader’s ability to comprehend a written passage. Because it requires readers to process information, it may take up to 30 minutes to administer.

In a Cloze test, text appears with every fifth word omitted. The reader tries to fill in the blanks. This task demonstrates how well he or she understands the text. The reader’s ability to supply the correct word also reflects his or her familiarity with sentence structure.

While packaged Cloze tests are available, Leonard and Cecilia Doak’s Teaching Patients with Low Literacy Skills explains how to make up and score a Cloze test yourself, based on the materials you are pretesting. The book also discusses use of the WRAT to assess reading levels.

* Adapted from Clear and Simple: Developing Effective Print Materials for Low-Literate Readers (NIH Publication No. 95-3594), by the National Cancer Institute, 1994. Bethesda, MD. In the public domain.