James Price Dillard*
Department of Communication Arts & Sciences, The Pennsylvania State University, USA
Received date: May 30, 2013; Accepted date: June 17, 2013; Published date: June 20, 2013
Citation: Dillard JP (2013) Hand The Effects of Prior Behavior on Judgments of Perceived Message Effectiveness: Evaluating HPV Vaccine Messages. J Vaccines Vaccin 4:193. doi: 10.4172/2157-7560.1000193
Copyright: © 2013 Dillard JP. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Visit for more related articles at Journal of Vaccines & Vaccination
Although message pretesting is crucial to the success of media campaigns there is very little research into the processes by which individuals make judgments of perceived message effectiveness. A random sample of college women participated in an internet survey in which they were asked to evaluate one of three messages concerned with the vaccine for the human papillomavirus (N=304). Judgments were made in both terms of the attributes of the message (e.g., logical vs. illogical) and its likely impact (e.g., convincing vs. not convincing). This conceptual distinction was borne out by confirmatory factor analysis. Participants also reported on their frequency of prior exposure to the messages, whether or not their physician encouraged them to be vaccinated, and whether or not they had done so. Although message exposure and physician encouragement did not produce observable effects on the judgment process, having obtained the vaccination corresponded with more favorable evaluations of the messages’ attributes. The data also indicated that attribute judgments were causally antecedent to impact judgments. The results contribute to a fledgling theory of perceived message effectiveness.
Evaluating HPV; Human papillomavirus; Perceived effectiveness
An analysis of structure, process, and bias
The ultimate impact of any mass media campaign depends on the success of various judgments that are made prior to campaign implementation, one of which is the estimate of a message’s anticipated effect. When empirical methods are employed to assist with this evaluation, the procedures typically involve asking individuals to assess the perceived effectiveness (PE) of the message. A favorable PE judgment should be considered a necessary, though not a sufficient condition for including the message in the campaign .
In light of the centrality of this process to campaign development, it would be valuable to have a thorough understanding of how PE judgments are made. But, until recently, this need has not been sufficient to engender much research. Little systematic theorizing has been done on PE. The concept itself is not well explicated and the process by which PE judgments are formed remains understudied. Consequently, this paper focuses on three issues: (a) elaborating the notion of PE, (b) suggesting a process relationship between two components of PE, and (c) considering one factors that might bias judgments of PE. For reasons elaborated below, we pursued answers to these questions in the context provided by messages that focus on the vaccination for human papillomavirus.
Theorizing perceived effectiveness: PE may be defined as an estimate of the degree to which a persuasive message will be favorably evaluated by recipients of that message. But, what, exactly, is meant by the phrase favorably evaluated? Most prior studies sidestep this question altogether by treating PE as a conceptual primitive . However, recent efforts to theorize PE suggest that it may be useful to distinguish between two subconcepts. Attribute PE consists of all those judgments that pertain to specific qualities of a message. For example, some messages are informative whereas others are realistic and still others are logical. Apart from the various particular features of messages, one can consider their impact. Indeed, impact PE has been assessed by requesting that research participants evaluate the overall extent to which a message is persuasive, compelling, or convincing.
The rationale for distinguishing between the two forms of PE is based on various theories of judgment all of which posit that general evaluations result from the integration of a series of more specific judgments. For example, McGuire’s probabilogical model proposes the beliefs are the product of a series of conditional probability judgments . Fishbein and Ajzen’s theory of reasoned action specifies beliefs and evaluations as the elements that combine to form attitudes . Similarly, Anderson’s information integration theory identifies the scale value and weight of different pieces of information as the basis for subsequent beliefs and attitudes . Although we take no stance on which of the algorithms proposed by these theorists is correct, we do adopt the assumption that is common to all three: Relatively microscopic judgments (i.e., attributes) provide the basis for relatively macroscopic judgments (i.e., impact). Further, we suggest that attribute judgments are linked to impact judgments through a system of rulebased inference that can be modeled as syllogistic reasoning. To wit: (Major premise) Realistic messages are compelling, (Minor premise) this particular message is realistic, therefore (Conclusion) this particular message is compelling. This general logic allows for derivation of several hypotheses. For one, the minor premise is an element in the syllogism that is logically distinguishable from the conclusion. To the extent that the psychological process of message evaluation is accurately conceived of in terms of a syllogistic reasoning, then:
H1: Judgments of attribute and impact PE are empirically distinct.
Further, and in line with the notion that the premises of the syllogism lead to the conclusion without any additional intervening processes, we suggest that:
H2: Attribute judgments are the proximal cause of impact judgments.
A final implication of the syllogistic model is that change in a conclusion can only be brought about by a change in either the major or minor premise. Hence, any variables that are external to the syllogism should be expected to exert their influence through one or the other of the premises. Given the emphasis here on judgments of the message itself, rather than the general rules of message evaluation, it was anticipated that:
H3: Variables that influence PE judgments will do so by directly affecting attribute judgments and only indirectly by affecting impact judgments (via attribute judgments).
Behavioral influences on the judgment process: When an individual commits an irrevocable behavior, such as vaccination, he or she becomes bound to that act in ways that may influence subsequent judgments . This commitment may predispose individuals to subsequently evaluate evidence that is relevant to the action through consistency-colored lenses. Consistent with this idea, Lord et al. demonstrated a tendency for individuals to evaluate evidence that supports their own beliefs as more convincing than evidence that does not . This form of biased message processing has been documented by several other researchers [8-10] and has produced a literature that indicates that information consistent with a preferred judgment is less likely to stimulate critical analysis than is information inconsistent with that judgment.
Although not specifically designed to provide a test of biased processing, several studies provide data consistent with the idea that prior behavior might well influence PE judgments. For example, drug users judged an anti-drug message as less effective than non-users , smokers evaluated an anti-smoking public service announcement as less effective than did non-smokers, and individuals who had indicated their decision to be an organ donor on their drivers’ license appraised a pro-donation advertisement as more effective than persons who had not made the same declaration . Given a clear theoretical expectation and prior data consistent with it, I anticipated that:
H5: Prior behavior (i.e, vaccination) is positively associated with PE judgments of vaccine-relevant messages.
Context for the study: We addressed the questions raised above in the context of the vaccine for the human papillomavirus (HPV). HPV is the sole known causal antecedent of cervical cancer, a malady that results in the death of over 250,000 deaths annually . In 2006, Merck & Co., Inc. began marketing Gardasil®, a vaccine that protects against the two strains of HPV (16 and 18) most commonly implicated in the development of cervical cancer. And, as it typical of for-profit companies promoting a new product, Merck initiated a large scale advertising campaign designed to encourage uptake of the vaccine. The “One Less” (death due to cervical cancer) campaign relied on both print and video advertising. In light of the clear benefits to women who received the vaccine, non-profit organizations such as university health services began to disseminate pro-vaccine messages concurrently with commercial campaign. In keeping with the aims of such organizations and the resources available to them, these messages tended to possess three features: (a) for reasons of cost they were executed in print, (b) they were fact heavy, and (c) they lacked the vivid presentational features that are typical of professional advertising. Although we were not interested in theorizing these particular aspects of the communication situation, the naturally occurring variation across the Gardasil® video ad, the Gardasil® print ad, and a University Health Service message enabled a test of the extent to which the results generalized across multiple messages. Hence:
RQ1: Will hypotheses 1-5 hold across message types?
Although the decision to utilize multiple, pre-existing messages has some clear benefits, it also introduces a potential confound in the form of prior exposure. Thus, I included a measure of prior exposure for use as a control variable and asked:
RQ2: Will prior message exposure influence PE judgments?
College women were contacted by email and invited to respond to a web-based survey. The three forms of the survey varied only with regard to the message that respondents received: A 60 second video advertisement for Gardasil®, a Gardasil® print advertisement, or a print message from the University Health Service website (hereafter abbreviated UHS). A description of each of the messages is given in the Appendix.
Participants and procedure
With assistance from the Registrar’s Office, we drew a random sample of 1800 undergraduate women enrolled at Penn State University. The sample included equal numbers of women at each of the four undergraduate class levels. Potential participants were contacted by email and invited to take part in an online survey on “women’s vaccination decisions.” No compensation was offered. Screening questions limited participation to women between the ages of 18 and 26. After a follow-up email one week later, 364 respondents had completed all or some portion of the survey for a response rate of 20.2%. Forty cases were eliminated due to excessive amounts of missing data, which reduced the sample size to 304 (video = 104, print = 117, and UHS = 83).
Participants ranged in age from 18 to 24 (M = 19.72, SD = 1.30). Among them, 87% of participants described themselves as White/ Caucasian, 6% as of Asian decent, 3% as of Hispanic descent, 3% as Black/African descent, and 2% as other/unidentified. Freshmen accounted for 26% of the sample, sophomores for 21%, juniors for 28%, seniors for 25%.
Prior exposure: A single item asked How many times in the past have you seen the message you just viewed? The response options were 0, 1-5 times, 6-10 times, 11-15 times, 16-20 times, and More than 20 times.
Prior behavior: Behavior with regard to the HPV vaccine was assessed by asking, Have you started the process of vaccination? Response options were Yes (= 1) and No (= 0).
Physician recommendation: This variable had three options: Physician encouraged obtaining the vaccine (= 3), have not discussed it with your physician (= 2), and physician discouraged obtaining the vaccine (= 1).
Attribute PE: Nine word pairs were used to create a semantic differential scales with 5-point response options: rational/irrational, reasonable/unreasonable, professional/unprofessional, logical/illogical, realistic/unrealistic, biased/unbiased, truthful/not truthful, true to life/ not true to life, and informative/not informative. All of the scales were coded such that higher numerical values reflected the positive pole of the word pair. The items were summed and divided by 3 to return the values to the original metric.
Impact PE: Three word pairs, in semantic differential format, tapped impact PE: convincing/not convincing, persuasive/not persuasive, compelling/not compelling. Higher values reflected more of the property. The items were averaged to produce a scale value.
H1: The structure of PE judgments
For the initial test of structure, confirmatory factor analyses were conducted within messages using the maximum likelihood routine in AMOS17.0. In each case, the first model included one latent variable (i.e., overall PE) and 12 indicators, that is, the combined set of attribute and impact items. The second model was comprised two latent variables, that is, attribute (nine indicators) and impact (three indicators). To enable tests of parallelism , three additional variables, all of which were manifest, were included in both models: previous message exposure, doctor’s encouragement, and prior behavior. All of the exogeneous variables were allowed to correlate with one another.
The analyses were carried out simultaneously in each of the three groups for both models. The results suggested invariance of the measurement weights across samples. For the one-factor model: χ2 (22)=26.6, p=.23, and for the two-factor model, χ2 (20)=25.1, p=.20.
Evaluation of the relative fits of the one- and two-factor models was based on two indices. The χ2 difference test favored the two-factor model: 647.7one-factor - 546.4two-factor = 101.3difference (df = 3), p < .0001. The same held true for the Akaike Information Criterion: 828.3one-factor versus 735.5two-factor. Consequently, we retained the two-factor model.
The overall fit statistics for the two-factor model were satisfactory by some standards (i.e., χ2/df = 2.1, RMSEA = .061 (90% confidence interval = .054/.061), but not by others (i.e., TLI =.87, PCLOSE = .015). Hence, the next step was to examine how the model might be modified to produce improved fit. Bidirectional paths were added, one at a time, to within-factor errors. Allowing correlated errors between logical and realistic, between realistic and informative, between true to life and informative, and between unbiased and truthful produced the following results: χ2 (246) = 401.3, p < .0001, χ2/df = 1.63, TLI = .92, RMSEA = .046 (90% confidence interval = .037/.053), PCLOSE = .83, all of which indicate excellent fit. Hall, et al present evidence that collapsing items into parcels when those items share some secondary influence does not bias parameter estimates . Because the intended use of the attribute items was as a single parcel, within-factor correlated errors caused no concern regarding the conceptual integrity of the model. Accordingly, we concluded that the two-factor model was consistent with the data and that H1 was supported.
H2: Attribute judgments cause impact judgments
A series of structural equation models was constructed to address the remaining hypotheses and research questions. The predicted model, shown in Figure 1, has three exogeneous variables – message exposure, doctor encouragement, and prior behavior – all of which were represented as manifest. The two multi-item PE factors were latent variables with single indicators, which consisted of the average of the summed items. Measurement error was accounted for in the latent variables by setting the error variance at (1 – a) s2. Using multi-group analysis, the model was assessed simultaneously in each of the three data sets. The initial test confirmed that the structural weights were invariant across samples: χ2 (8) = 5.5, p = .70. Global fit statistics for the model were quite good: χ2 (9) = 11.4 p = .24 , χ2/df = 1.26, TLI = .97, RMSEA = .030 (90% confidence interval = .000/.075), PCLOSE = .71.
The path coefficient between Attribute PEand impact PE was .76, p < .001. This was consistent with H2, which asserted that Attribute PEis best conceived of as the causal antecedent of impact PE. To evaluate the possibility of reverse causation an alternative model was constructed in which the location of Attribute PEand impact PE were exchanged. In other words, the same exogeneous variables had direct causal paths to impact PE, which, in turn, had a causal path to Attribute PE. Although the attribute-impact coefficient changed only slightly (to .77), the model showed poor fit to the data: χ2(15) = 161.6, p = .0001, χ2/df = 10.7, TLI = .03, RMSEA = .180, (90% confidence interval = .156/.206), PCLOSE = .0001. Thus, it was reasonable to claim full support for H2.
H3: Exogeneous variables influence impact judgments only via attribute judgments
To evaluate this hypothesis, three additional models were run, each of which contained a path that corresponded to a direct influence of prior behavior, physician recommendation, or message exposure on impact PE. impact PE was cast as causally prior to Attribute PE. Unexpectedly, the path coefficient was significant. However, the fit indices indicated serious problems with the model: χ2/df > 9.0, negative TLI values, and RMSEAs in excess of 17. Thus, despite the significant coefficient between impact and attribute, the models were clearly untenable. The data supported H3.
H4 and H5: Social and behavioral influences on the judgment process
The relationship between physician recommendation and PE was nonsignificant (.05, p > .50). Hence, there was no support for H4. Consistent with H5, the path coefficient for prior behavior was positive and significant (.26, p < .001).
RQ1 and RQ2: Generalizing across messages and controlling for prior exposure
RQ1 asked whether the results would hold across the three samples and their corresponding messages. This question was addressed by the multiple groups analyses conducted at each step of the measurement and theoretical analyses. In every case, the results indicated that invariance of the structural coefficients could be assumed.
RQ2 was concerned with the potential for prior message exposure to bias PE judgments.
With a path coefficient of .03, p > .50, from message exposure to Attribute PE, there was no evidence of bias.
Making perceived effectiveness judgments
The data were compatible with the distinction drawn between attribute and impact PE insofar as the two-factor model showed good fit to the data and the one-factor model did not. These findings replicate the results of the one other study (known to us) to have explicitly examined the structure of PE judgments. However, there are significant qualifications to our conclusions. First, as with previous work, the overall number of items used to assess Attribute PEis rather modest. The nine word pairs used in this study is a noticeable improvement over the four used previously, but there is considerable distance to travel before the conceptual domain of Attribute PEis fully tapped. In addition, whereas the current data clearly distinguished between Attribute PEand impact PE, there is nothing in our conceptual model that requires that there be no more than two factors. Indeed, writers such as Morley’s subjective message constructs theory specifies three distinct attribute-type judgments -- importance, plausibility, and novelty – each of which is a necessary to bring about persuasion . Similarly, Palmgreen et al. contend that public service announcements are judged in terms of three attributes: emotional arousal, dramatic impact, and novelty . Although additional research is needed to move toward a more definitive understanding of the possible structures of Attribute PE, the results presented here and elsewhere argue for maintaining a clear theoretical distinction between attribute and impact PE.
The data were also consistent with the claim that attribute judgments are causally antecedent to impact judgments. Thus, the results are compatible with the claim that making PE judgments is a process that depends on evaluating specific message features against general rules for judging message persuasiveness. Although it is important to appreciate the limits of cross-sectional data when making a causal claim, it is also valuable to recognize that the model which reversed the predicted causal flow was rejected. In this respect, support for the hypothesized process was clear. Further support must await the conduct of an experiment.
One additional piece to a fledgling theory of PE is the idea that variables that are likely to influence PE judgments exert their effects by altering attribute judgments, which, in turn, produce changes in impact judgments. Here again the data were consistent with expectations, although not as unequivocally at might be wished. One substantial limitation is the small number of “other variables.” Of the three tested – prior message exposure, doctor encouragement, and prior behavior – only the latter manifested a biasing effect on the judgment process. Research is needed that tracks the effects of other biasing variables through the PE judgment process and beyond.
It is worth noting that the multi-group analyses allowed us examine the three hypotheses simultaneously in three message-differentiated samples. The fact that the measurement and structural models held across the three samples is a highly desirable manifestation of consistency across messages. This strengthens our confidence in the generalizability of the results and suggests that the structure and process propositions embodied in H1-H3 possess some degree of durability.
Influences on the judgment process
We tested two variables that had the potential to bias the PE judgment process, one social and one behavioral. With regard to the former, we anticipated that a physician’s recommendation might exert an influence on women’s evaluations of the HPV messages. However, no such effect was observed. Despite our expectations to the contrary, it is certainly conceivable that women did not see a physican’s recommendation to obtain the vaccine as relevant to messages that promoted that action (the advertisements) or simply provided information about it (the UHS message). This finding may be juxtaposed with previous experimental research in which subjects were informed that a distal group had evaluated a public service announcement as effective or ineffective. That study did find a substantial effect for social information. But, the crucial difference may be that the information was specific to the messages under scrutiny. Hence, one possible conclusion is that social information that is topicrelevant, but not message specific, is insufficient to bias PE ratings. Given very large body of research documenting the effects of social information to other aspects of persuasion [18-20], it would certainly premature to conclude that the process of PE judgment is impervious to such effects. Rather, additional testing is needed.
In contrast to the results for physician recommendation, the data provided good evidence that prior behavior can bias estimates of PE. Specifically, women who had started the 3-shot sequence of injections that comprises the HPV vaccination judged all three of the messages under consideration as higher in Attribute PEthan women who had not begun the vaccine sequence. To gain some appreciation for the size of this bias, we conducted an analysis of variance using message exposure and physician encouragement as covariates. The estimated means for Attribute PEfor vaccinated women was 4.42 versus 4.09 for unvaccinated women (η = .27 on a 5-point scale). Although the mean difference of .33 appears rather modest, it is important to consider its implications. Were such a difference to locate itself around the midpoint of the scale there would be the possibility of a categorically incorrect conclusion. In other words, if the sample on which a message was pretested was composed of large numbers of persons who had or had not performed the behavior in question, the results could be biased such that the mean PE judgment was biased in one direction or the other. This could, in turn, lead to faulty conclusions concerning categorical effectiveness of the message and it subsequent value to any given media campaign. What are small differences initially may become larger as they move through decision processes intended to judge the potential potency of a message. This possibility underscores the more general point, which is that theorizing and testing for potential sources of bias in PE judgments is crucial to message pretesting.
This project contributes to our knowledge of PE in several ways. First, it provides evidence for a structural distinction between attribute and impact PE. Second, it sheds light on a causal process that places Attribute PEand the immediate antecedent of impact PE. Third, it illustrates generally how external variables may bias the judgment process and describes the operation of one in particular, that is, prior behavior. Although each of these contributions is modest in its own right, jointly they help to move us toward the development of the theoretically grounded technology of message evaluation that will ultimately enable more successful media campaigns.