웹 설문조사를 위한 Scalar 질문 설계(Sociological Methods & Research, 2009)

웹 설문조사를 위한 Scalar 질문 설계(Sociological Methods & Research, 2009)

Designing Scalar Questions for Web Surveys

Leah Melani Christian

Pew Research Center for the People & the Press, Washington, DC

Nicholas L. Parsons

Eastern Connecticut State University, Willimantic, CT

Don A. Dillman

Washington State University, Pullman, WA

고찰 및 결론

Discussion and Conclusion

지난 연구와 우리 실험의 결과  응답자들은 문항의 워딩이나 보기에 붙은 label 그 이상의 것에 영향을 받는다. 특히, 응답자는 척도의 전반적인 레이아웃에서 정보를 얻는데, 여기에는 다른 보기와의 물리적 위치관계, 숫자 라벨의 활용 여부 등이 있다. 시각적 설계 효과visual design effect에 관한 다른 연구 결과와 유사하게, 우리 연구에서도 약간의 visual manipulation이 크고 유의한 차이를 냈음을 보여준다.

Results from past research and our experiments indicate that respon- dents often rely on more than just the question wording and verbal cate- gory labels when answering scalar questions. Specifically, respondents gain information from the overall layout of the scale, each category’s phy- sical position in relation to other responses on the scale, and any numeric labels that may be used. Similar to other research on visual design effects, our results indicate that some visual manipulations of the response scale produce large and significant differences and others more minor differ- ences (see Table 7 for a summary of our results).

일관된 순서(긍정적->부정적 순서 와 부정적->긍정적 순서 를 섞거나 안 섞거나)로 보기를 배치한 것은 응답 성향에는 영향이 없었으나 일관된 순서로 한게 더 대답을 빠르게 하는 결과가 있었다. 보기의 순서가 일관되었을 때, 즉 "높은 번호가 긍정적인 것up means good" 휴리스틱이 발동했다고 볼 수 있으며, 척도를 인지하고 이해하는데 시간이 덜 걸려서 더 빠르게 응답했을 것이다. 추가로, 보기를 한 칼럼으로 배치한 것은 응답자가 척도를 처리하는 속도를 빠르게 하여 선택을 내리기 더 쉽게 만든다. 반면, 응답을 여러 칼럼으로 배치하는 것은 답을 선택하는데 더 오래 걸리게 한다. 중간점midpoint는 중요한 anchor가 된다. '모르겠다' 응답으로서 midpoint를 분리하는 것(시각적으로, 개념적으로 midpoint가 align되어 있기만 하면) 은 응답에 영향을 주지 않는다.

Overall, we find that consistently presenting the positive end of the scale first for all questions did not affect people’s responses, but it did result in respondents providing a response more quickly. When the options are pre- sented consistently with the ‘‘up means good’’ heuristic (Tourangeau et al. 2004), it seems to take less time for respondents to perceive and comprehend the scale, so they can provide a response faster. In addition, presenting the options linearly in one column facilitates respondents’ processing of the scale and encourages them to process the categories in the same order, making it easier to provide a response. In contrast, presenting the categories in multiple columns different responses based on whether results in respondents process the options horizontally or vertically and in respondents taking longer to pro- vide an answer. Our findings confirm that the midpoint is an important anchor for respondents when answering scalar questions. Separating the midpoint, don’t know option, or endpoints through the use of space does not seem to affect responses as long as the visual and conceptual midpoints of the scale are aligned.

추가로, graphic layout을 완전히 없애고, scale에서 각 척도가 몇 점인지만 설명해주고 직접 점수를 쓰게 했을 때는 응답에 크게 영향을 주었고, 응답 시간도 늘어났다. 그러나 이전 연구 결과를 보면, 점수가 응답자의 expectation과 일관되게 배정되어 있으면 (높은 점수가 긍정적 응답) graphic display를 없애는 것이 응답에 영향을 주지 않았다. 유사하게, 과거 연구에서 음수negative number를 사용하는 것이 양수positive number만 사용하는 것에 비해서 응답에 크게 영향을 주었다. 마지막으로 우리 연구결과에서 응답자들은 numerical label의 양 끝에만 설명이 붙어있을 경우 응답에 더 오랜 시간이 걸렸다(응답 결과에 차이는 없었다)

Additionally, we find that removing the graphic layout of the scale entirely and providing only a verbal description of the scale in the question stem significantly influence respondents’ answers and increase response time when lower numbers indicate more positive categories. However, pre- vious research has shown that when the numbers are assigned consistently with respondents’ expectations, with higher number assigned to positive categories, removing the graphic display of the scale does not significantly affect responses (Christian et al. 2007a). Similarly, past research has shown that the use of negative numbers dramatically affects responses compared to the use of only positive numbers (Schwarz et al. 1991; Tourangeau et al. 2007). Finally, our results demonstrate that respondents take longer to respond to polar point scales with numeric labels because of the additional information they must cognitively process; however, there are no statisti- cally significant differences in responses.

전통적으로 설문 방법론자들은 워딩에만 초점을 두었으며, 시각적으로 어떻게 보기를 배치하는가에는 관심이 별로 없었다. 이는 종종 설문을 하는 사람들이 (비교를 목적으로) 다른 설문에서 워딩을 그대로 배끼면서 (공간 문제로) 시각적 레이아웃은 바꾸는 결과를 가져왔다.

Traditionally, survey methodologists have focused on wording alone when constructing response scales, and the visual presentation of response scales has received little attention (Krosnick and Fabrigar 1997). This has sometimes led to the unfortunate practice of surveyors replicating the wording of questions from one study to another with the intent of compar- ing responses across studies, but changing the visual layout to fit with the style of a particular questionnaire design or because of space limitations.

이번 연구와 예전 연구에서, 모든 시각적 제시 방법의 차이가 응답에 영향을 가져오는 것은 아님을 보여주었다. 추가로, verbal label이나 numeric label이 (개개의 질문에 대한 답에 초점을 둔다면) visual cue에 비해서 더 영향력이 클 수 있다. 그러나 어떻게 visual/numeric/symbolic 정보가 verbal 정보보다 더 강력할 수 있는지도 보여진 바 있다. 특히 응답자가 정보를 처음 처리할 때에는 언어 정보보다 시각 정보가 더 우선권을 갖는다.

As shown here and in previous research, not all differences in the visual presentation of scalar questions significantly influence respondents’ answers. In addition, verbal and numeric labels may take precedence over subtle visual cues, particularly when respondents are focused on respond- ing to individual questions (Tourangeau et al. 2007). However, it has also been demonstrated how visual, numeric, and symbolic information can be more powerful than verbal information (Christian, Dillman, and Smyth 2007b; Redline et al. 2003). In particular, visual information often takes precedence over verbal information when respondents first process the information presented (Ware 2004). The development of an overall theory of when and why visual layout makes a difference remains in its early stages (see Dillman et al. 2009).

함의점. 종이 또는 웹 설문을 개발할 때, 얼마나 보기를 많이 제시할지 뿐만 아니라, 어떤 언어적 라벨을 붙일지, 그리고 시각적으로 어떻게 제시할지도 고민해야 한다. 또한 과거 설문을 활용하여 결과를 비교하려고 할 때, 워딩 뿐만 아니라 visual quality 에도 신경을 써야 한다. 또 다른 함의는 mixed-model 설문이 점차 많이 사용되고 있는데, 설문을 받는 사람들은 여러 설문 방식간 유사성을 유지하는데 신경써야 한다.

Our overall finding that the visual design of response scales makes a difference has significant practical implications. When designing response scales for paper and Web questionnaires, surveyors must not only consider how many categories to offer and what verbal labels to provide but also how the scale will be presented visually to respondents. It also seems important that when surveyors use questions from earlier surveys with the intent of comparing results, they attempt to maintain the visual qualities of previous questions as well as the question wording. Another implication is that at a time when mixed-mode surveys are being conducted with greater frequency, surveyors must recognize that attention needs to be given to maintaining similarity across survey modes.

마지막으로 응답자의 부담burden에 대한 함의. 전통적으로 응답자의 부담은 문항의 워딩이나 내용에 따른 것이라고 생각했다. 여기에서의 실험 결과는 응답 결과에 영향을 주지 않더라도, 어떤 시각적 레이아웃은 정보를 더 빠르게 처리하는데 도움이 된다는 것을 보여준다. (숫자 없이 polar point scale만 제시하는 것). 또 어떤 경우에서는 시각적 디자인에 따라 결과가 달라질 뿐만 아니라 응답 시간에도 영향을 준다(polar point vs 숫자 기입하기).  응답시간은 응답자가 정보처리가 어려울 경우 더 길어진다. 따라서 paradata를 통해서 어떤 문항이 특히 burdensome했고, 혹은 응답자가 잘 못 이해했고, 어떤 문항을 먼저 응답했는지 등을 명확하게 보여준다.

Finally, the results presented here provide insight into respondent bur- den. Traditionally, respondent burden has been thought of as an attribute of a question that was dependent on words and substance. The experimental data reported here show that even when people’s responses are not affected, some visual layouts are more quickly processed than others (e.g., polar point scales without numbers). In other instances, not only does the visual design influence responses, but also it can affect the time taken to provide a response (e.g., polar point vs. number box when the more positive answers are assigned lower numbers). Response time is longer for formats that are difficult for respondents to process. It is increasingly clear that paradata have multiple uses for survey methodologists, by helping them to under- stand when questions are especially burdensome or misunderstood by respondents as well as the order in which respondents complete questions and select individual answers.

---

Designing Scalar Questions for Web Surveys

1. Leah Melani Christian
   1. Pew Research Center for the People & the Press, Washington, DC

1. Nicholas L. Parsons
   1. Eastern Connecticut State University, Willimantic, CT

1. Don A. Dillman
   1. Washington State University, Pullman, WA

Abstract

This paper explores how the visual design of scalar questions influences responses in web surveys. We present the results of five experiments embedded in two web surveys of university students. We find that consistently presenting the positive end of the scale first did not impact responses but increases response times. Displaying the categories in multiple columns influence how respondents process the scale and increase response times. Separating the midpoint, ``don't know'' option, or endpoints spatially does not impact responses when the visual and conceptual midpoint align. Removing the graphical layout of the scale influences responses when lower numbers indicate more positive categories and increases response time. Finally, response times are longer for polar point scales with numeric labels, but there are no differences in responses. Overall, our results suggest that the visual design of response scales impacts measurement, but that some manipulations produce larger and more significant differences than others.