Researching Market Research: Google Scholar

One of the challenges facing market researchers is how to properly write a questionnaire. How many points should we include in a scale…and should we label each point or not? Is there a cutoff for how many attributes to include in a grid question? Should I arrange my scale horizontally or vertically? How long should my survey be?

I’ve been in the industry for 10+ years . You’ll find that if you pose these kinds of questions to others in the field, you’ll get lots of opinions. Some methods are just standard practice or what you might call conventional wisdom. I worked at one firm that recommended no more than 20 attributes on a grid question. No one knew where the number “20” came from, and why not “15” or “25” attributes?

Turns out there’s also a whole body of scientific literature you can turn to….research on research studies that examine best practices for survey design. We’re not the only one’s fielding surveys. Government agencies, psychologist, and economist use questionnaires to predict behavior, forecast election outcomes, and set policy decisions. One survey research project even comes from a mandate by the US Constitution: the US census. It’s serious business that has to be done with accuracy and precision, and we can learn from the work of others.

Google has a search engine called “Google Scholar” that indexes a large archive of scientific literature. Type in an area of interest like “survey research” and you’ll discover hundreds of academic articles devoted to the subject.  You can even refine your search to “questionnaire length” if that’s what you want to know more about.   And because Google Scholar is devoted to primary research studies, you can feel pretty confident that the information you’re gathering from these searches is grounded in real data, not opinions. These are all researchers that have performed controlled, experimental studies to answer tough questions about how we should be conducting questionnaire-based research.

As with any kind of internet search, Google Scholar involves a bit of sifting through the volume of information that’s out there. However, the great thing about “Google Scholar” is that it searches scientific articles which conform to a specific writing style to help you quickly find what you are looking for. Article titles clearly state the topic of a paper. For example:  “A comparison of web and mail survey response rates” or “Effects of personal salutations in e-mail invitations to participate in a web survey.”  Those are pretty dry titles, but they directly tell you what to expect. Click on any article that comes up in “Google Scholar” and most of the times you’ll be directed to the article’s abstract. This is usually a one or two paragraph summary describing the purpose of the study, results, and conclusions. Often times, there’s no need to read any further than that.

There’s no limit to what you’ll find. I discovered one of my favorite scholars on the subject of survey design by using Google Scholar. Jon Krosnick is a professor at Stanford University and investigates survey design and sampling practices for a living.  Many of his studies are published on his website. There, you’ll find experimental studies devoted to respondent fatigue, writing scale question, response bias, measuring attitudes, you name it. As market researchers, we are accustomed to spending a lot of time researching consumer preferences for products to help us make decision on how to market to them. We must also remember to take time to research how to conduct research.