The Psychology of Quality and More

A Toolbook for Quality Improvement and Problem Solving (contents)

Surveys: How to do it

The Quality Toolbook > Surveys > How to do it

When to use it | How to understand it | Example | How to do it | Practical variations

<-- Previous | Next -->

How to do it

1. Identify the objective of the survey. For example, 'To gain ideas for a new machine shop layout'. This can be helped by determining what decisions or actions might occur as result of the survey. It can also be useful for identifying non-objectives, such as 'Not to select machines'. Also ensure objectives are not ambiguous. For example, if investigating employee satisfaction, be clear about the meaning of both 'employee' (All employees? Including managers?) and 'satisfaction' (What does it mean? What does it not mean?).
    
2. Identify the target group of people to be surveyed. This will be one or more groups of people who have information or opinions which will contribute to achieving the survey objective. Note that sometimes it is polite or politic to survey people who will not directly contribute.
    
3. Select the type of survey to use. This will be an interview or questionnaire.

When using an interview, consider the strategy that might be used. This may include one or several of the following approaches:

• A structured interview follows a strictly defined format, aiming to answer a predefined question set. It may be little more than an assisted questionnaire.
• In an informal interview, there will still be predefined questions, but these are used only as a guide for the interviewer, who may decide whether or not to use them. This approach is more flexible, but is more difficult to handle as it is easier to become side-tracked.
• A telephone survey enables the interviewing of people who are geographically remote, such as customers or suppliers. It is usually fairly short and therefore needs a more structured format to capture the requisite information.
• A tandem interview uses two interviewers, typically with one person questioning and the other recording. This can make interviews quicker and capture more useful information.
• A group interview or focus group involves multiple interviewees. It is useful for gaining group consensus and using synergy to create new ideas, but may need careful control as it can degenerate into a free-for-all.

4. Plan the detail of the survey, and ensure that sufficient resources and people are available to perform it. This includes designing and distributing questionnaires, conducting interviews and collating and analyzing results.

Be aware of the time that will be required. For example, a phone survey may require two to three weeks, whilst a mail survey is likely to take at least two months before all responses are returned and collated.

Where appropriate, get permission to conduct the survey, indicating the expected start time and duration required. Also find out if the people have been recently surveyed on other topics as an otherwise helpful person could turn out to be irritated by 'yet more questions'.
 

5. Select the questions that are to be asked. This is particularly important for a questionnaire, as the questions cannot be interactively explained.

Use the objectives from step 1 to identify the key questions that must be answered, then design actual questions that will elicit useful responses. This may require several questions to get a complete answer. In an interview this can be done by starting with an open question, then following up with probing questions to capture requisite detail. The reverse is applicable to a questionnaire, which may start with several closed questions and follow up with an open question that captures other points the respondent may wish to make.

For example, if aiming to determine the effectiveness of a suggestion scheme, an interviewer may start with, 'What do you think of the new suggestion scheme?' and then follow up responses with such as, 'Why do you believe that?' and, 'What feedback did you get to your suggestion?'. By contrast, a questionnaire might start with several closed questions, such as, 'How many suggestions have you made during the last six months?'. and close with, 'Do you have any further comments you would like to make about the suggestion scheme'".

Ensure interviewees know about a subject before questioning more deeply. Thus in the previous paragraph, you might start with, 'Did you attend the presentation on the new suggestion scheme?'.

Be careful with sensitive questions. One way of handling this is to depersonalize the question, for example, 'Do you think the evaluation process is fair?'.

Questions may be identified and organized by using tools such as Brainstorming and the Affinity Diagram. It can be useful to put the final question set into a Tree Diagram to help check that a complete and coherent question set has been found. Make use of available expertise in the question topic during design and review of the question set.

Multiple-choice questions are particularly useful in questionnaires, as they are quick and easy to answer and give specific data that is easy to analyze. Types of multiple choice questions are illustrated below.

Avoid any questions which are either likely to give false information or which may upset or annoy the interviewee. These include:

• Ambiguous questions that may be interpreted differently by different people. The vagueness of many words makes this an easy trap into which many fall. For example, 'Are you often satisfied?' ('often' and 'satisfied' are ambiguous, and thus require more definition). Slang and jargon words are another way to confuse.
• Asking for details that people are unlikely to know or which may have been forgotten. 'Which brand of cereal did you eat this time last year?'.
• Complex or negative questions, such as, 'If these people are not found guilty, should they be prevented from visiting people who do not want to see them?'.
• Leading or coercing questions, such as, 'We find most people like it. You do like it don't you?'. Question sequencing can also lead, for example where two questions about the benefits of a particular product are followed by a query about 'Your preferred product'.
• Multiple questions, such as, "Which do you like, how, when and why do you like it, and how did you come to this opinion?'
• Discrimination or antagonism. 'Aha, so you're the little woman. You probably won't understand this question ...'.
• Emotionally charged words, such as 'stealing' or 'redundancy'.

Table 1. Question types
 

6. Build the questions into a complete questionnaire or interview guide that can be used in the survey.

Start by thanking the respondent for helping and then give a brief explanation of the purpose of the survey. This will help the respondent feel comfortable with the questions and may assist them in giving appropriate answers.

Give instructions on how to complete the survey, either for the interviewer or questionnaire respondent. Instructions may include:

• How to fill in answers (writing, ticking, circling, etc.), possibly with examples. Repeat this, at least in summary, on all pages where it applies.
• Where to go if questions may be skipped.
• The date by when the survey should be completed.
• The person to whom the completed survey should be sent.
   

Lay out the questions, making sure there is enough room for answers. Make sure that each question and its answer space fits completely on one page.

7. Review the final question set against the objectives, checking that responses will allow the objectives to be fully met. Look for missing, incorrect or unnecessary questions.

 
8. Perform a pilot test with volunteers to ensure it is understandable, that it elicits the right sort of response and can be completed within an acceptable time period. This can also be used to try out different questions or options, for example to determine how consistently they are answered.

 
9. Execute the survey, as planned. Consideration of the following points will help it to be successful.

A questionnaire simply requires sending to the distribution list. If appropriate, include a stamped, addressed envelope for the reply (never expect respondents to pay for the return postage!). If delivering them to known people, a personal request can help to encourage replies.

Interviews require significant work at this stage, and effort should be put into making them as consistent as possible. Use a standard approach, such as:

• Ensure the person is relaxed enough to answer honestly and freely. If possible, use a quiet, comfortable and confidential room. It can help if the interviewer is socially similar to the interviewee, for example having similar clothing and hairstyle.
• Introduce yourself and explain why they are being surveyed. This can use closed, factual statements and questions that are easy to answer. For example, 'We have been working on ways to ensure people can work well together. Have you read about the new team management policy?'.
• Start exploring of the area of interest with open questions, such as 'How do you feel about this policy?'. Use body language (nodding, grunts, etc.) to encourage the person to keep talking. Watch their body language, spotting signals and conflicts (e.g. saying yes, but avoiding eye contact).
• Spot specific points of value and probe for more details: 'Can you tell me more about ... '. Differentiate between facts, opinions and feelings. Be careful when talking about areas of personal sensitivity, and only probe there if it is essential.
• Summarize and test understanding with tentative statements: 'It seems as if your prefer ... '.
• Conclude with a closed question: 'So you definitely prefer a structured approach?'.
• Repeat the above steps for each of the areas of interest.

10. Analyze the survey data to give a usable set of information from which decisions can be made and the objective from step 1 be met.

When doing this analysis, if there are gaps in the information, avoid guessing what the respondent might have said. Treat gaps simply as missing information or go back and ask more questions.

 

 

<-- Previous | Next -->