In preparation for writing a questionnaire to gather feedback on my ARP intervention, I read two articles:
Writing Survey Questions
Pew Research Center
https://www.pewresearch.org/writing-survey-questions/
and
The Tools at Hand
Jean M. Converse & Stanley Presser
https://methods-sagepub-com.arts.idm.oclc.org/book/survey-questions/n3.xml
In Writing Survey Questions, the author presents various aspects of questionnaire writing and how these aspects should not only be considered but also how question order, context, types of answers can affect the answers and data from these surveys.
Creating questions
Questions should be used to accurately measure the opinions, experiences and behaviours of the group. Questions shouldn’t be ambiguous or biased. Essentially that there is a science that underpins good survey questionnaires.
The author talks about using the same questions at different points in the process, to measure change over time. Something I hadn’t thought about and perhaps too late to add to my process now but I could ask questions that require the respondent to look at how their work and ideas have changed over the course of the intervention.
“Asking the same questions at different points in time allows us to report on changes in the overall views of the general public (or a subset of the public, such as registered voters, men or Black Americans), or what we call “trending the data”.” (Pew Research Center, 2021)
“When measuring change over time, it is important to use the same question wording and to be sensitive to where the question is asked in the questionnaire to maintain a similar context as when the question was asked previously” (Pew Research Center, 2021)
The use of open- and closed-ended questions is a key component of questionnaires and “One of the most significant decisions that can affect how people answer questions” (Pew Research Center, 2021). Open-ended questions are where the respondents provide a response in their own words, whereas closed-ended questions are where respondents are asked to choose from a list of answer choices. As with questions, the order of answers can influence how people respond to them, “Answers to questions are sometimes affected by questions that precede them.” (Pew Research Center, 2021)
Question wording
Part of the design of the questions, “The choice of words and phrases in a question is critical in expressing the meaning and intent” (Pew Research Center, 2021). Questions should be clear and specific, and written in such a way that each respondent is able to answer, using simple and concrete language and avoiding unfamiliar abbreviations or jargon.
One aspect that might be important to my particular survey is something called ‘Social desirability bias’ which is about how people want to be accepted and liked, and may provide inaccurate answers to questions that deal with sensitive subjects and may understate / overstate answers to fit in. As my project is about identity, which can be a sensitive subject for some people, I should be mindful of this.
Question order
Once the questions have been designed, I will need to develop an order for the questions.
“Surveyors must be attentive to how questions early in a questionnaire may have unintended effects on how respondents answer subsequent questions. Researchers have demonstrated that the order in which questions are asked can influence how people respond; earlier questions can unintentionally provide context for the questions that follow (these effects are called “order effects”).” (Pew Research Center, 2021).
Respondents are much more likely to mention concepts or considerations mentioned in earlier closed-ended questions, so again, this is something to be mindful of.
“A questionnaire, like a conversation, should be grouped by topic and unfold in a logical order. It is often helpful to begin the survey with simple questions that respondents will find interesting and engaging” (Pew Research Center, 2021).
In the article ‘The Tools at Hand’, the authors state that “Every questionnaire must, finally, be handcrafted” (Converse, J. M., & Presser, S. (1986)) and that the designer must “cut and try, […] see how people react to it, […] and try again” (Converse, J. M., & Presser, S. (1986))
Like any piece of design, I need to allow time to test, get feedback and make changes to the questionnaire.
Exploration
– Have a clear set of research purposes
– Knowledge of work on the problem
– How a survey could shed some new light
Exploratory study “should take investigators out beyond their own academic or industrial subculture, to new “experts”” (Converse, J. M., & Presser, S. (1986))
“Survey questions, finally, must seem fair to people of widely different viewpoints” (Converse, J. M., & Presser, S. (1986))
The article talks about how inquiry might involve interviews with members of the target population rather than consulting ‘the literature’ to gain a more authentic understanding of members of the group.
“At this exploratory stage, there is little prospect of formally sampling the target population, but interviewing even a few individuals can enrich the researcher’s perspective. Another useful procedure is to assemble somewhat more formally the insiders of a given subculture in a “focused discussion group.” This can be of special value when a target population is likely to have special perceptions, problems, and idioms that may be relatively foreign to the investigator—youth culture, gambling, drugs, prisons, and so on.”
Does this tie in to what I’m doing? As essentially I’m trying to understand the identities of people fro different backgrounds to my own?
Bibliography
Converse, J. M., & Presser, S. (1986). The tools at hand. In Survey Questions (pp. 48-75). SAGE Publications, Inc., https://doi.org/10.4135/9781412986045
Pew Research Center (2021). Writing Survey Questions. https://www.pewresearch.org/writing-survey-questions/
