설문 시행, 분석, 보고 (BMJ, 2004)

설문 시행, 분석, 보고 (BMJ, 2004)

Administering, analysing, and reporting your questionnaire

Petra M Boynton

파일럿 시행

Piloting

파일럿 검사를 시행할 때 일반적 형식에 대해서, 그리고 구체적 질문에 대해서 설문응답자들이 어떻게 반응하는가를 자세히 기록해야 한다.

During piloting, take detailed notes on how partici- pants react to both the general format of your instrument and the specific questions.

• 얼마나 오래 걸리나 How long do people take to complete it?

• 질문을 반복 또는 설명해줘야 하나 Do any questions need to be repeated or explained?

• 어떻게 답을 골랐는가 How do participants indicate that they have arrived at an answer?

• 특정 질문에 혼란 또는 놀랐는가 Do they show confusion or surprise at a particular response—if so, why? Short, abrupt questions may unintentionally provoke short, abrupt answers.

자료 수집 계획

Planning data collection

데이터 보호와 관련된 법률, 그리고 기관 차원의 규약code를 따라야 한다.

You should be aware of the relevant data pro- tection legislation and ensure that you follow internal codes of practice for your institution—

파일럿팅 단계에서는 설문을 어떻게 배포하고 다시 받을 것인지를 점검해야 한다.

The piloting phase should include planning and testing a strategy for getting your questionnaire out and back—for example,

• 누구에게 응답을 요청할 것인가 who you have invited to complete it (the sampling frame),

• 누가 응답해 주었는가 who has agreed to do so (the response rate),

• 응답은 활용 가능한 형태로 왔는가 who you’ve had usable returns from (the completion rate), and

• 언제 reminder를 줄 것인가 whether and when you needed to send a reminder letter.

행정적 실수도 연구 프로세스 진행을 방해hamper할 수 있다.

Administrative errors can hamper the progress of your research.

 

 

설문 시행

Administering your questionnaire

과학적, 윤리적 고려에는 다음을 포함해야 한다.

Scientific and ethical considerations should include:

• 설문응답자의 니즈와 선호,
  The needs and preferences of participants,

• (응답자는) 자신에게 요구되는게 무엇인지 알아야 함
  who x should understand what is required of them;

• (응답자는) 설문 진행과정에서 흥미를 보이고 협조적이어야 함
  remain interested and cooperative throughout completion;

• (응답자는) 적절한 질문을 받아야 하며, 응답이 잘 기록되게 해야 함
  be asked the right questions and have their responses recorded accurately; and

• (응답자는) 설문 작성 중, 작성 후에 적절한 서포트를 받아야 함
  receive appropriate support during and after completing the questionnaire

Table A  Pros and cons of different options for administering a questionnaire  

Method of delivery	Pros	Cons	Practical notes
By post Participants are sent a copy of the questionnaire by post and asked to complete it and return it to the researcher.w5	Quick and easy to distribute.   Relatively inexpensive.	They are not useful for the study of very personal issues (without first giving the participant the option of taking part in the study) and have a notoriously low response rate since you are relying on the goodwill and co-operation of individuals.	Remember to enclose a detailed introductory letter, a complete, contact address, and a stamped addressed envelope so the participant does not have to pay postage.   You may need to send reminder letters and questionnaires to slow/non-responders.
By telephone The researcher calls participants and completes the questionnaire over the phone, with the researcher reading out the questions and recording the answers.w6	Quick and easy to complete.   Relatively inexpensive.	Due to ethical constraints and sample bias these are used less within health research. You cannot control for participant refusal, which is often high. Not suitable for those with hearing problems. Can become laborious if calling someone who is lonely and wants to talk.	Remember to contact participants by letter in advance of your call – and offer them a chance to opt-out of your study (and avoid your phone call).   Many ethics committees won’t permit a study where cold calling is the main design.
By email Questionnaires are sent to participants via email for completion.w7	Easy to design and send out.   Can keep track on who has responded and who hasn’t, and send reminders.	Only suitable for participants with email access, and who can download a questionnaire.   Can lead to confusion, where participants print out questionnaire and answer it by hand, rather than on the computer.	See telephone interview above.  Participants need an introductory email announcing the research and an opt-out option.  Follow data protection legislation, and check sending emails don't breach confidentiality.
By a website The questionnaire is placed within a website and participants are directed to this and invited to complete it. w8	A simple questionnaire can be easily designed and placed within a website.  Since sites offer more space, it’s possible to have more opportunities for qualitative feedback using this measure.	Participants are only those with access to the Internet.   You may find they are a non-representative sample since they’ll have a special interest for visiting your site (e.g. your site is about testicular cancer and they have a particular view about it, or experience of illness).  It is difficult to stop the same person answering the questionnaire a number of times over.	Check your site regularly to ensure you can access the questionnaire and that there aren’t any ‘bugs’ in it.    Encourage participants to report problems with accessing the questionnaire online.
Participant completion with researcher present The researcher can answer questions the participants may have, but the participant answers the questions.w2	The researcher is on-hand to offer support and explain any questions participants might not understand.    They can also be sure that questionnaires are completed and collected.	Participants can inadvertently be ‘led’ by asking the researcher for advice on how to answer the questions.	Ensure your staff have training and support in how to deliver and code questionnaires and manage participants.16
Researcher Administered The researcher asks the question and fills in the appropriate answers as directed by the participant.w9	The researcher can be certain the questionnaires are fully and accurately completed, and collected.	The researcher may ‘lead’ participants by their tone of voice or phrasing of questions.   Participants may not understand what is required of them and not answer in a ‘standardised way’.	As above.   If you are using standardised measures researchers have to read these out in exactly the same order as they appear written in the questionnaire.

응답률 높이기

Maximising your response rate

적은 숫자의 좋은 퀄리티의 응답이 많은 숫자의 낮은 퀄리티의 응답보다 낫다.

Note, however, that it is better to collect fewer questionnaires with good quality responses than high numbers of questionnaires that are inaccurate or incomplete. The third article in this series discusses how to maximise response rates from groups that are hard to research.15

응답을 거부한 사람

Accounting for those who refuse to participate

설문연구는 종종 응답을 수행한 사람에 초점을 둔다. 그러나 참여하지 않은 사람도 마찬가지로 동등하게 중요하며, 이들의 특성을 기록해두어야 한다.

Survey research tends to focus on people who have completed the study. Yet those who don’t participate are equally important scientifically, and their details should also be recorded (remember to seek ethical approval for this).41617

응답률을 높이는 요인 

Box 2: Factors shown to increase response rates

• The questionnaire is clearly designed and has a simple layout5

 • It offers participants incentives or prizes in return for completion6

 • It has been thoroughly piloted and tested5

 • Participants are notified about the study in advance with a personalised invitation7

 • The aim of study and means of completing the questionnaire are clearly explained89

 • A researcher is available to answer questions and collect the completed questionnaire10

 • If using a postal questionnaire, a stamped addressed envelope is included7

 • The participant feels they are a stakeholder in the study11

 • Questions are phrased in a way that holds the participant’s attention11

 • Questionnaire has clear focus and purpose and is kept concise7811

 • The questionnaire is appealing to look at,12 as is the researcher13

 • If appropriate, the questionnaire is delivered electronically14

비-완료 비율을 줄이기 위한 방법은 시작부터 엄격한 exclusion criteria를 세우는 것이다. 그러나 접촉하기 힘든 연구대상 그룹을 systematically 배제하는 것은 비윤리적이며, 추가적인 전략이 필요할 것이다.

One way of reducing refusal and non-completion rates is to set strict exclusion criteria at the start of your research. However, research that systematically excludes hard to reach groups is increasingly seen as unethical, and you may need to build additional strate- gies and resources into your study protocol at the outset.15

비-참여자 대상 자료 수집은 연구 프로세스 모니터에도 도움이 된다.

Collecting data on non-participants will also allow you to monitor the research process.

초반의 거절률이 이례적으로 높으면, 전반적 접근법을 다시 생각해봐야 한다.

In addition, if early refusals are found to be unusually high, you might need to rethink your overall approach.10

 

자료 입력, 확인, 클리닝

Entering, checking, and cleaning data

상당히 일이 많다.

In reality, entering, checking, and cleaning the data account for much of the workload. Some principles for keeping quantitative data clean are listed on bmj.com.

연구가 진행될 때 같이 정리해야지, 다 끝나고 해야겠다고 생각해서는 안된다.

It is good practice to enter data into an electronic database as the study progresses rather than face a mountain of processing at the end.

양적자료 다듬기 

Ways of keeping quantitative data clean10

 

1. Be as careful and accurate as possible when entering data into a database.

2. Take breaks. Fatigue causes mistakes.

3. Double check your questionnaire against the data you have entered. You can either do this with every questionnaire, or by picking a random sample and checking how accurately the data in the database matches the answers on the questionnaire.

4. If possible, get a colleague to work with you (eg read out answers from questionnaires to put into a database, or double-enter the data). Some statistical packages include a warning message or tone for double data entry.

5. Run statistical frequencies on all items and scan the results for obvious anomalies. Are any data missing? Are there numbers that don’t seem right (for example, on a questionnaire when participants can only give answers in the range of 1-5, are there any numbers outside that group in your database?). Go back to your coding sheets and check all anomalous data.

6. Create codes for missing data. This allows you to locate errors quickly. For example, if participants refused to answer, or couldn’t answer a question, or only answered part of the study, you can build in codes to account for this. Remember to make each code distinct, so they cannot be confused with numbers elsewhere in database.

7. For questionnaires that have sections that should add up, ensure the answers tally.

8. When answers are missing or don’t add up, take time to locate incorrect answers. Check your database regularly as you add data, so that when errors arise it doesn’t take too much work to find them.

9. Make a note each time you clean your data, and flag where you’ve got up to each time you work on your database. That way, when you find errors, you will have a more precise idea where to locate the problem.

See also www.uiowa.edu/~soc/datarespect/data_training_frm.html

자료 분석

Analysing your data

어떤 분석을 할 것인지를 미리 생각해야 함.

You should be able to predict the type of analysis required for your different questionnaire items at theplanning stage of your study by considering the struc-ture of each item and the likely distribution of responses (box 3).1 Table B on bmj.com shows some examples of data analysis methods for different typesof responses.18 19 w1 

Table B: Analysis options18 19 w1  w10

Type of response required from participant on the questionnaire	You can analyse this data using…
Binary or yes/no answers	c2 (chi squared), Spearmans, Wilcoxon, Mann Whitney, Kruskal Wallis etc.
Rating or visual scales	Pearsons, t test, analysis of variance (ANOVA) etc.
Open-ended (free text) replies	Thematic content or discourse analysis.

논문 작성과 보고

Writing up and reporting

연구방법 부분

Methods section

연구방법 부분은 배제 기준을 구체적으로 기술해야 하며, 이것이 transferability에 주는 영향을 기술해야 한다. 거절refusal 혹은 적합하지 않은unsuitable 참여자가 있다면 이들을 제시하고 논의해야 하며, 참여자 모집 다이어그램을 활용하는 것이 좋다 통계적, 양적 분석법을 기술하고 justify하라.

The methods section should give details of your exclu- sion criteria and discuss their implications for the transferability of your findings. Data on refusals and unsuitable participants should also be presented and discussed, preferably using a recruitment diagram.w2 Finally, state and justify the statistical or qualitative analyses used.18 19 w2

연구결과 부분

Results section

 

연구결과 부분을 정리할 때, 원래의 연구질문으로 돌아가서 이것을 설명해주는 관찰결과를 정리set out해야 한다. 다른 말로는, 결과가 가설로부터 유도된 것이어야 한다hypothesis driven. 유의하지 않게 나온 결과를 보고하는 것을 두려워하지 않아도 되며, 실제로 유의한 결과만큼 중요할 수 있다. 모든 질문에 대해서 다 분석하고 보고하지 말아라

When compiling the results section you should return to your original research question and set out the find- ings that addressed this. In other words, make sure your results are hypothesis driven. Do not be afraid to report non-significant results, which in reality are often as important as significant results—for example, if partici- pants did not experience anxiety in a particular situation (box 4). Don’t analyse and report on every question within your questionnaire

가장 통계적으로 적합하면서 시각적으로 매력적인 형식의 그래프나 표를 사용하라. 그래프의 각각의 축에 대해서 라벨을 붙이고, 의미가 잇는 제목을 달아라. 표를 refer하고 주요 결과를 강조하라.

Choose the most statistically appropriate and visu- ally appealing format for graphs (table). w3 Label graphs and their axes adequately and include meaningful titles for tables and diagrams. Refer your reader to any tables or graphs within your text, and highlight the main findings.

개방형 질문을 했다면 결과 부분에서 cherry pick하지 말아라. 어떤 주제가 등장했는지를 outline해야 하며, 그 주제와 질적 관찰결과를 보충할 수 있는 quote를 사용해야 한다.

If you have used open ended questions within your questionnaire, do not cherry pick quotes for your results section. You need to outline what main themes emerged, and use quotes as necessary to illustrate the themes and supplement your quantitative findings.

고찰 부분

Discussion section

고찰 부분은 결과 부분으로 돌아와서 주요 결과가 무슨 의미인지를 제시해야 한다. 연구의 한계를 인정하고, 이 관점에서 couch the discussion해야 한다. 만약 설문 응답률이 낮으면, 이 preliminary result를 확인하기 위한 추가 연구를 제안해야 한다. 연구의 conclusion은 연구의 scope을 벗어나서는 안된다. 소규모의 지역에 국한된parochial 연구를 했다면, 국가적 정책을 제안해서는 안된다. 지속적으로 응답을 거절한 참여자, 혹은 기대했던 방향이 아닌 방식으로 응답한 것에 대해서 discussion해서는 안된다.

The discussion should refer back to the results section and suggest what the main findings mean. You should acknowledge the limitations of your study and couch the discussion in the light of these. For example, if your response rate was low, you may need to recommend further studies to confirm your preliminary results. Your conclusions must not go beyond the scope of your study—for example, if you have done a small, parochial study do not suggest changes in national policy. You should also discuss any questions your par- ticipants persistently refused to answer or answered in a way you didn’t expect.

 

 

심리적, 사회적 영향력을 고려하기

Taking account of psychological and social influences

설문 연구는 절대로 완벽하게 객관적일 수 없다. 연구자와 참여자는 인간이고 심리적/감정적/사회적 니즈가 있다. 너무 자주 우리는 이 요인을 연구의 계획/수행/분석에서 놓치곤 한다. 설문은 참여자와 응답자에게 서로 다른 식으로 해석될 수 있다. 연구자는 자료를 원한다. 잘 훈련되어있지 않은 상태에서 초보junior 연구진은 압박이 심할 경우 파일럿팅, 샘플링, 배포/수집/코딩에 실수를 저지를 수 있다.

Questionnaire research (and indeed science in general) can never be completely objective. Researchers and participants are all human beings with psychological, emotional, and social needs. Too often, we fail to take these factors into account when planning, undertaking, and analysing our work. A questionnaire means something different to participants and researchers.w4 Researchers want data (with a view to publications, promotion, academic recognition, and further grant income). Junior research staff and administrators, especially if poorly trained and supervised, may be put under pressure, leading to critical errors in

• piloting (for example, piloting on friends rather than the target group),

• sampling (for example, drifting towards convenience rather than random samples) and

• in the distribution, collection, and coding of questionnaires.15

 

설문 시행을 위해서 고용된 연구원은 연구에 익숙하지 않을 수도 있고, 자신의 무지를 덮기 위해서 한 행동이 연구 전체를 망칠 수도 있다.

Staff employed to assist with a questionnaire study may not be familiar with all the tasks required to make it a suc-cess and may be unaware that covering up their igno- rance or skill deficits will make the entire study unsound.

연구 참여자들은 설문에 응답할 때 흥미가 있을 수도 있고, 지루할 수도 있고, 다른 사람을 돕고자 할 수도 있고, 해야 할 것 같아서 할 수도 있고...등등. 이것들이 모두 잠재적인 bias의 요인이다.

Research participants, on the other hand, may be motivated to complete a questionnaire through interest, boredom, a desire to help others (particularly true in health studies), because they feel pressurised to do so, through loneliness, or for an unconscious ulterior motive (“pleasing the doctor”). All of these introduce potential biases into the recruitment and data collection process.

15 Boynton PM., Wood GW, Greenhalgh T. Hands-on guide to questionnaire research: reaching beyond the white middle classes. BMJ (in press).

---

Ambiguity identified in a pilot study Rena is a family planning trainee interested in sexual dysfunction. She wishes to explore this in a sample of young women. In a pilot study, she finds that participants interpret the question “Did you get wet during sex?” differently. Some have answered in terms of arousal (which is what Rena wanted to find out about), while others thought wet referred to sweat or urinary incontinence. Each participant answered consistently, but what they thought they were talking about differed between participants. In the final questionnaire, Rena included a short explanation of what wet meant.

 

Documenting data on refusals Phuong is leading a study on new parents’ experiences of maternity services. She invites people to participate, and notes the age, sex, ethnicity, and reasons given for refusing to join the study. Most commonly, participants refuse because they are too busy, not interested in the research, or dislike the subject matter of the questionnaire. By keeping tabs on participants who refuse, Phuong can check that no particular subgroup of people is being excluded, and she can later formally compare the characteristics of responders with non-responders. An example of the form used for collecting exclusion/refusal data in Phuong’s study is shown below: Age Gender Ethnicity Maternity ward Reason given for non-participation Researcher’s name

---

BMJ. 2004 Jun 5;328(7452):1372-5.

Administering, analysing, and reporting your questionnaire.

Boynton PM1.

Author information

• 1Department of Primary Care and Population Sciences, University College London, London N19 5LW. p.boynton@pcps.ucl.ac.uk

Erratum in

• BMJ. 2004 Aug 7;329(7461):323.

PMID:

 

15178620

 

[PubMed - indexed for MEDLINE] 

PMCID:

 

PMC420299

 

Free PMC Article