Ad Watchers season two is underway. In this second episode of the season, hosts La Toya and Eric are joined by a special guest, Professor Joel Steckel. Professor Steckel is a Marketing Professor at the Stern School of Business at New York University and the Vice Dean for Doctoral Education. He joins in on a dynamic conversation centered on consumer perception surveys. In season one’s episode, “What Does it Mean to Have a “Reasonable Basis” Standard,” your hosts promised loyal listeners a conversation addressing reliable consumer surveys.
La Toya and Eric fulfill this promise by beginning their discussions with a definition and intended purpose of consumer perception surveys. La Toya mentions that companies often decide to conduct this particular survey for ad development purposes. Advertisers seek to identify the appropriate messages conveyed by proposed ads via consumer perception survey results. A word of advice shared by La Toya, “It’s a good thing to do even before you get caught up in a NAD challenge or litigation .”
Eric then moves the conversation into survey design basics. He warns that the National Advertising Divison (NAD) examines the survey design with a critical eye as it is considered evidence. Flaws found in the design deem the survey unreliable. Therefore NAD won’t credit it.
Moving on from design, the hosts provide a rundown of other items NAD frequently looks for when reviewing cases involving consumer perception surveys. They begin with the survey should represent an appropriate universe sample. The questionnaire should also be well designed to ensure instructions and questions are not leading. NAD also reviews for the proper control cell and data analysis. The control is used to mimic a test ad that excludes the hypothesized message conveyed. Also, how the survey data is analyzed matters, and responses must be coded appropriately.
Professor Steckel adds to the intricate conversation with three vital criteria that constitutes a consumer perception survey—construct validity, internal validity, and external validity. Professor Steckel notes that construct validity refers to whether or not the survey measures what it is intended to measure. The internal validity looks to understand if the survey tests what it was developed to test. The external validity asks, does the behavior exhibited in the survey reflect marketplace conditions?
Professor Steckel adds that if one of these criteria could be sacrificed, it is external validity. “If a measure doesn’t measure what it’s supposed to measure or a test doesn’t test what it’s supposed to test, at least in the context of a survey, then the researcher has nothing,” Professor Steckel defines.
He also provides further insights on La Toya and Eric’s points shared at the beginning of the episode. Professor Steckel contributes the following points on each:
- A sample used in a survey must represent a target universe. It doesn’t have to mirror the target universe, but it must represent it.
- The benefits of internet-based surveys outweigh the challenges compared to the more traditional intercept survey.
- Designing a survey control is both an art and a science.
- When designing a control, the fewer elements you can change, the better, but the elements you change have to be consistent with the allegations of deception.
- When reviewing the verbatim responses of a survey, remove the experts from the process.
- The most common mistake advertisers make in designing a consumer perception survey is eliminating pretests to cut costs. Not pretesting the survey produces a significant unknown. Advertisers won’t learn how respondents will interpret the questions, whether they can guess the survey’s purpose, who the client is, et cetera.
La Toya leaves listeners with this final note, “Every part of your survey needs to be backed up by objective evidence. This is the correct way to test the hypothesis. It’s not just what one expert thinks is correct. It’s what the field would believe is appropriate.”
Don’t forget to tune in next month when Ad Watchers tackles cosmetic advertising.
Visit to Learn More: NAD FAQs
Contact Information: nad@bbbnp.org
Tune into the full episode here.