Ssmon.chb.kth.se

Chapter 7
Qualitative methods for
effectiveness evaluation:
When numbers are not enough
Introduction
Methods of collecting qualitative information
7.2.1 Interviews and focus groups
7.2.2 Questionnaires with open-ended questions
7.2.3 Observations
7.2.4 Document analysis
Using qualitative methods in evaluation
7.3.1 Identifying implementation and intermediate outcomes
7.3.2 Verifying and complementing quantitative outcome measures
7.3.3 Eliminating threats to internal validity
7.3.4 Identifying unintended outcomes
7.3.5 Developing quantitative measures
Selecting a sample for qualitative studies
Qualitative data management and analysis
Ensuring good quality data
Qualitative Methods
Chapter 7
7.1 Introduction
The types of questions used in interviews willdepend on the purpose of the data- gathering.
Qualitative methods play an important role in They could be about any of the following: safety intervention evaluation. Although in most knowledge (e.g., What did you learn about in situations, numbers are necessary to prove effectiveness, qualitative methods can yield experience (In what ways, if any, have things We first describe four methods of gathering practices (In what way, if any, has the training qualitative information: 1) interviews and focus program influenced your safety practices on groups; 2) questionnaires with open-ended questions; 3) observations; and 4) document analysis. We identify and illustrate several different ways in which these types of data can be beliefs (What do you think the company’s used in an effectiveness evaluation. We follow goals are in providing you this program?) with some details of how to select study subjects, feelings (How do you feel about participating analyze the collected data, and ensure good A good interviewer is sensitive to the mood andfeelings of the interviewee(s), listens well, and Methods of collecting qualitative data
encourages them to elaborate on the topicdiscussed. Better interviews will result from 2) Questionnaires with open-ended questions interviews and has practiced with the interview questions. There are a number of approaches for 7.2 Methods of collecting qualitative
Structured interviews
Structured interviews contain a standardizedmeans of getting information. The same set of 7.2.1 Interviews and focus groups
carefully worded and ordered set of questions are used with each respondent. This technique information is through in-depth interviewing.
reduces the level of skill required for the This involves open-ended questions, where interviewer to do a good job and curtails the interviewees can answer questions on their own influence of any particular interviewer on the terms and in as much detail as they like. This is results. Structured interviews are useful where in contrast to the typical questions found on several people are conducting the interviews or employee surveys, that prompt for yes/no, if the interviewers are inexperienced. On the multiple choice or very short answers. For other hand, there is less opportunity to learn example, a truly open-ended question asks “what do you think about the new safety program?”.
In contrast, only a limited range of answers is allowed if you ask, “how useful was the newsafety program?” or “was the new programuseful?” Chapter 7
Qualitative Methods
Semi-structured interview
information, since extreme or false views tend tobe challenged by others in the group. A skilled A semi-structured approach to interviewing facilitator can guide the group’s dynamics so that the participants stay on topic and people who are standardization and flexibility. Here, an either shy or have less popular opinions are interview guide is used, which is basically a checklist of the issues explored during theinterviews. There is no set order to the topics, Exert some caution in selecting individuals for a and specific questions are not necessarily worked focus group. First, this format is not advisable if out ahead of time. However, before ending the sensitive information of either a personal or interview, the interviewer makes sure all the organizational nature is sought. People might be items have been covered, through the natural reluctant to speak up and could be vulnerable to course of the conversation. Any topics not yet repercussions if they do. For similar reasons, and, covered can then be addressed. As with the depending on the subject of the interview, you structured interview, this method ensures that should probably group people together with the same type of interview data are gathered similar positions within the organizational hierarchy. In particular, consider separating laborand management; and supervisors and those Unstructured interview
they supervise. In some cases, you might want togroup men and women separately.
The unstructured interview is more like aninformal conversation, where questions are Guidelines for obtaining good interview
data
conversation. Although certain topics arecovered, there are no predetermined questions.
1. Let the subject(s) know at the outset how long The data varies with each person interviewed.
the interview will last, its purpose and general This makes the analysis more challenging. As outline. Explain how confidentiality will be well, more opportunity exists for an interviewer’s bias to influence the results. The strength of this 2. Obtain consent (preferably by signing a consent approach though is that the interviewer can tailor form) for participating before starting the the approach and line of questioning to each 3. Start off the interview with non-controversial questions that require minimal recall. More Focus group interview
sensitive topics, including questions onknowledge, should be covered once a rapport A focus group is like an interview with a small group of people rather than only one person. A 4. Create an atmosphere of having a conversation.
semi-structured approach is most useful. About You do not want people to feel as if they are six to ten people can be interviewed together and the interviews usually last from one-and-one-half 5. Ask clear, truly open-ended, questions.
to two hours. This allows time for participants to discuss about eight to ten questions.
7. Be attentive. Indicate interest through your The focus group technique is a highly efficient 8. Tape record the interview in order to have a way to collect data. You receive the opinions of detailed record for analysis. Record important several people at the same time. The social setting provides a measure of validation for the Qualitative Methods
Chapter 7
7.2.2 Questionnaires with open-ended
Observational data is especially helpful in questions
evaluating safety programs as an externalevaluator. An understanding of the physical and social environment will be increased. You will questionnaires - even with truly open-ended catch issues that might go unreported during the questions - to be the most effective way to gather interviews because the insiders are too close to qualitative information. It is assumed that many their situations. As well, people might not speak people do not want to take the time to write out freely during interviews in fear of reprisal from a response. As well, it cannot be sensitive to co-workers or management. Finally, an on-site interviewee differences, since everyone gets the same question. The depth of responses is limited intervention activities are occurring as described.
because there is no opportunity to follow up onan interviewee’s statement with other questions. If you are an internal evaluator planning to useobservations, be aware that one’s view of things is influenced by one’s background and position questionnaire to measure the quantitative within the organization. Thus, if observations objectives in the project, you can then quite are going to play a large role in an evaluation, economically expand the breadth of the results consider bringing in an external, more neutral by including a few open-ended questions. These observer. Similarly, you might have to choose can be useful for gauging participant reactions, between being an observer or a participant, or identifying program barriers, bringing out something in between. The more you participate, unintended consequences of the intervention, the more first-hand your knowledge will be. The disadvantage is that it becomes more difficult to quantitative measures. Furthermore, the results maintain “objectivity” and your presence could of this initial screen can help you decide on the nature and extent of any follow-up qualitativestudies.
Tailor the length and frequency of observations toyour requirements. This can range from a single 7.2.3 Observations
two-hour site visit to verify programimplementation to a full-time, year-long presence Another way of collecting qualitative data is to to fully understand, for example, a change in actually go on-site and observe what is going on.
safety climate. Field notes are the primary means Depending on your needs for the evaluation, of recording observational information. This can everything can be captured, including the be supplemented with photographs or videos, physical environment, social organization, although such methods are often obtrusive.
program activities, as well as behaviors and Good field notes require a selectivity that can interactions of people. Or you can take a more focus on the important details, yet not severely narrow focus. The type of observational data used in qualitative analyses can be different thanthat used in quantitative analyses. In the latter, 7.2.4 Document analysis
specific observations are always being sought:e.g., whether a particular procedure is being Documents of interest in workplace safety done correctly or if a particular work-site intervention evaluations can include material condition is observed. In contrast, for the containing policies or procedures related to the purpose of qualitative analysis, specific types of intervention, safety records, committee minutes, observations might not be defined beforehand.
correspondence, memoranda, or reports. Theycan suggest topics to include in interviews or Chapter 7
Qualitative Methods
questionnaires and offer evidence of intervention important addition to an evaluation, since it is implementation, barriers to implementation, or not usually possible to quantitatively measure other events in the workplace that could threaten every pertinent intermediate effect of the the evaluation’s internal validity.
intervention. It can be difficult to anticipate themall and measure them quantitatively. You Be aware that documents are never more than a especially want to find out the extent to which partial reflection of reality. Some are normative; the intervention was implemented as planned.
such as procedures documents. They tell what Document analysis, observations and interviews should be done, not whether it actually is done.
can be used to check on program activities.
Some documents are descriptive - e.g., minutesof meetings. However, they can reflect one Example of how qualitative methods can
person’s view (e.g., the minute-taker or chair of be used to identify intermediate outcomes
the meeting), more than the collective view.
Ways to use qualitative methods in
Let us return to an earlier example34 where an effectiveness evaluation
intervention consisted of a workplace- basedincident investigation team assisting Interviews, questionnaires, observations and supervisors in their investigation of incidents.
documents are used alone or in combination towards several purposes in safety intervention measures (frequency and severity of injuries) evaluations. Here, we elaborate on five ways in which they can contribute to an effectiveness report and percentage of incidents generatingcorrective action proposals). Interviews Ways to use qualitative methods in
helped fill in the picture further of how the effectiveness evaluation
intervention could have led to the observeddecrease in injuries and their severity. The safety representatives found the incidentinvestigation teams helpful and felt that better 2. Verifying and complementing quantitative 3. Eliminating threats to internal validity intermediate outcome - has been identified as and severity of injuries were decreased.
7.3.1 Identifying implementation and
7.3.2 Verifying and complementing
intermediate outcomes
quantitative outcome measures
Qualitative data can help elucidate the steps between the intervention and the final outcome, quantitative measures. Through an approach of including implementation and intermediate “triangulation”, two or more different outcomes. They can identify results not captured methodological approaches can measure the in the quantitative measures. This can be an same thing in order to establish consistency. You Qualitative Methods
Chapter 7
might undertake a broad-based safety initiative 7.3.4 Identifying unintended outcomes
to change the “safety climate” in the workplace.
Certainly, you could use a safety climate Interviews and, possibly, observations are useful questionnaire, which typically consists of close- ways to identify unintended outcomes.
ended questionnaire items, to assess a change in safety climate. Also valuable are open-ended assessed quantitatively, such as an increase in an questionnaire items or interviews completed by untargeted type of injury, others would be better key personnel regarding observed changes in the detected through qualitative inquiry.
workplace atmosphere concerning safety. If themethods are consistent in their portrayal of Interviews are an especially good at gauging the change in safety climate, then a “cross- reactions of intervention participants and others validation” of the methods has been achieved and you can present your conclusions with more supervisors, union leaders and managers. Their reactions and those of others involved with theintervention, are important, since a poor Sometimes the methods are complementary in response by an influential individual or group of that they might measure different aspects of the individuals at a work-site could have a big effect on the program. It might explain the lack of interviews might detect aspects of change missed success of a promising intervention. Unintended by a questionnaire containing only close-ended outcomes can also be more positive. In one evaluation, for example, interviews with workersand foremen showed that several people believed 7.3.3 Eliminating threats to internal validity
that the recent decrease in the number of laborgrievances could be attributed to the improved Interviews with key officials can provide information crucial for addressing potential 7.3.5 Developing quantitative measures
Example of how qualitative information
Data collected using qualitative methods in the helps reduce threats to internal validity
planning stage of the evaluation can provide thebasis for the subsequent development of relevant In the evaluation example just discussed on the quantitative measurement instruments. Here are previous page, interviews and analysis of safety committee minutes revealed the followinginformation which helped eliminate threats tointernal validity. The workplace physical plan,products, production techniques and activities, aswell as the safety-related policies, purchases andactivities (apart from the creation of the incidentinvestigation committee) had remained constantover the six-year evaluation period - suggestingno evidence of history threats. There was also noevidence for an instrumentation or reportingthreat, since there were no changes in the incidentreporting criteria, nor in safety-related policies,purchases and activities (apart from the creation ofthe committee).
Chapter 7
Qualitative Methods
Examples of how qualitative studies can help develop quantitative instruments
1) Interviews, observations and document analysis can lead to the development and inclusion of certain items on questionnaires. For example, say that opinions expressed in interviews had arepeating theme that safety is for sissies. If your intervention is in part designed to change thisattitude, then it would be a good idea to develop a questionnaire that includes questions whichmeasure such safety attitudes.
2) People have used the details of incident records, a qualitative information source, to develop workplace-specific checklists of work practices or work-site conditions used in quantitativeassessment. They review records to find which unsafe practices and conditions are associated withincidents. Interventions are then developed which encourage the practice of safer alternatives. Aswell, checklists of these safe practices and work-site conditions are developed and used inevaluation. Quantitative measurement consists of making (random) observations and recordingwhether the safe or unsafe version of the practice or work-site condition was observed.
3) Menckel and Carter35 described a new safety initiative in which a group assisted workplace supervisors in their investigation of incidents within their division. Preliminary interviews anddocument analysis showed that there was often a long delay between incident occurrence and itsformal reporting. As a result, corrective safety measures were correspondingly delayed in theirimplementation. Thus, one of the ways evaluators chose to measure the effect of a new workplaceincident investigation group was by how long it took for incidents to be reported. 7.4 Selecting a sample for qualitative
occupational safety program. Then compare quotes across groups to see if there are differencesin experiences which might influence the Once you have decided to use qualitative data intended goals of the program. Furthermore, you collection methods as part of the program might separately ask male and female workers evaluation, you need to decide from whom, or about any problems in participating in the about what, data should be collected. This might program. Again, comparisons can be made to include collecting data from specific employee see if both females and males similarly received work groups, manager groups, female or male workers, or different classifications of workers.
Additionally, you might want to collect data We describe eight different purposeful sampling about a particular event, occurrence, or incident.
Rather than surveying the entire work force, use Extreme or deviant case sampling
what is called purposeful sampling. Here, oneselects information-rich cases to study in-depth.
Identify unusual or special cases. It is possible They are purposefully selected so that the investigator can learn, in detail, about issues of conditions (good or bad) rather than the many central importance to the program. For example, possibilities which fall in the middle. For example, survey data collected after a safety workgroups about their experience in a particular program is over might show one or two people Qualitative Methods
Chapter 7
who have made big changes. A follow-up with Criterion sampling
an interview could validate the responses as wellas discover what in the program motivated them to make such big changes. By limiting the focus predetermined important criterion. Even if all to extreme cases, this approach to sampling is employees at the work-site receive the training, you might interview only those most exposed tothe particular hazard targeted by the training.
Heterogeneity sampling/maximum variation
They may reveal major system weaknesses that sampling
Identify cases with differing characteristics (e.g., Politically important case sampling
age, gender, education, job classification) toprovide diverse points of view. Any common Identify, and select (or not) politically sensitive patterns emerging from the variant cases can individuals. You might want to interview a capture the core experiences and shared aspects union steward who supports the program, and thereby can enrich the information obtained.
Homogenous sampling
Convenience sampling
Identify a small group of information-rich cases - similar in terms of background, employment participants for qualitative data collection lies level, experiences, etc. and explore the issue of with picking cases that are easiest to obtain and interest in depth. It might be of interest to those most likely to participate. This is also the separate groups of management and then labor least desirable method. The problem is that in and compare their opinions about a safety the end, it is difficult to know exactly who was interviewed and if their opinions are consistentwith others possibly affected by the program.
Typical case sampling
Identify “typical” individuals to describe the 7.5 Qualitative data management and
benefits of the program. Cases are selected with analysis
the co-operation of key informants such asprogram staff. This information can be used to help “sell” the program to others reluctant to qualitative data. The process is described here in very general terms and appears as a sequenceof steps, which in actual practice can occur Critical case sampling
simultaneously or may even be repeated. First,all raw information, if not already in a written Find individuals who could dramatically make a form, is converted to text. Thus, taped interviews point about the program. They may be identified by asking a number of people involved with the summarized using words, etc. This body of program. A good bet are the leaders in the group textual material is reviewed to identify important who could provide suggestions about how to features and, possibly, summarize them. A coding system of keywords, or some other datareduction technique, is developed to facilitate thisprocess. The data, either in summarized form ornot, is then reviewed to identify patterns. These Chapter 7
Qualitative Methods
patterns are concerned with the following: However, researchers generally try to reduce similarities or differences among groups or their effect on their research by using concepts events; repeated themes; and relationships and methods agreed upon by other researchers.
Ways to guard against bias include the following:outlining explicit methods for data collection and data analyses; adhering to these methods; having more than one researcher collect data; regarding the data. Depending on the scope of having a second, non-biased person summarize and/or draw conclusions from the data; and trustworthiness of these generalizations by letting the data speak for themselves and not testing them with the results of further data forcing them into a framework designed by the collection or comparing them with existing Appropriate sampling
Success at the data analysis stage requires thatgood data management practices are observed Someone reading your evaluation wants to be from the beginning of data collection. Use sure that the right sample has been selected for systematic methods for collecting, storing, the stated purpose. For example, you could not retrieving and analyzing data. People have developed various techniques to help highlight, organize or summarize data. A useful reference intervention, if either management or employee representatives are not represented. Thus, the Huberman 1994]. This reference also reviews the various software developed to assist in both the explicit and justified with respect to the study’s data reduction and pattern recognition stages of Validation by subjects
7.6 Ensuring good quality data
One of the best ways to determine whether or Concerns about reliability and validity apply to not you “got it right” in your study, is to check qualitative data, just as they do to quantitative with the subjects you are studying. This involves confirming the accuracy of the data collected, the qualitative investigation wants to know that the reasonableness of the method used to summarize stated methods have been used consistently it, and the soundness of the conclusions. Of throughout the study (reliability concerns). They course the potential biases of the subjects also want to know that there are no hidden biases consulted must be kept in mind when weighing in the data collection, the data analysis nor the conclusions drawn (validity concerns).
Thorough methods of drawing conclusions
The following contains considerations andsuggestions for ensuring that good quality data is Avoid drawing conclusions too soon. This can be caused by researcher bias or pressure to comeup with answers quickly. In contrast, well- Minimizing evaluator bias
grounded conclusions require time for at leastsome of the following activities: 1) reviewing The product of a study no doubt bears the collected data to identify anything which has personal mark of the people conducting it.
been overlooked; 2) searching for evidence which Qualitative Methods
Chapter 7
contradicts preliminary conclusions, either by 7.7 Summary
reviewing data already collected or by gatheringnew data; 3) confirming important data or conclusions through “triangulation”, i.e., finding gathering qualitative information: interviews; agreement when using a different data source, questionnaires with open-ended questions; methodology or researcher; and 4) exploring observations; and document analysis.
alternative explanations for patterns observed in Qualitative data can be used in several ways to complement quantitative methods: identifyingimplementation and intermediate outcomes; Conduct a pilot study
verifying and complementing quantitativeoutcomes; eliminating threats to internal validity; Conducting a pilot study or trial run with your proposed research methods is often of great developing quantitative measures. In contrast to value. Feedback from those involved in the pilot quantitative methodology, qualitative methods study can be used to refine a sampling strategy, interview guide, other data collection procedures, sampling strategies. We briefly discussed and even procedures for data management.
methods of analysis and methods to ensure goodquality data.
Key points from Chapter 7
• Use interviews and focus groups, questionnaires with open-ended questions, observations, and document analysis to enrich your evaluation.
• Use qualitative methods for one or more purposes: identify implementation and intermediate outcomes verify and complement quantitative measures • Use one of several purposeful sampling strategies.
• Collect and analyze data in ways which enhance their reliability and validity.

Source: http://ssmon.chb.kth.se/safebk/Chp_7.pdf