SURVEYS QUESTIONNAIRES
INTRODUCTION
Surveys are the most commonly used method of data collection in the social sciences; so common, that they quite often are mistakenly taken to be the method of social research.
In general, surveys are methods of data collection in which information is gathered through oral or written questioning. Oral questioning is known as interviewing; written questioning is accomplished through questionnaires, which are administered to the respondents by mail or handed to them personally by the researcher in their homes, at work, at school or any other place; they are returned to the researcher after completion. These are also known as self-administered or self-completion questionnaires.
Since interviews also employ a form of questionnaire, in the format of an interview guide. Here we will study questionnaires in a more general manner, concentrating primarily on questioning rather than strictly on questionnaires.
1 QUESTIONNAIRE STRUCTURE
Questionnaires are diverse; they vary according to the way they are administered as well as according to their nature. This diversity results in the following types of questionnaire.
Structured or Standardized questionnaires:
· highly rigid with a high degree of standardization
· no flexibility in answering the questions
· answers are limited to those set in the questionnaire and no other ideas
· propositions or alternative answers are allowed
· mostly employed in quantitative research
Un-structured or Un-standardized questionnaires:
· structure of these questionnaires is less rigid
· degree of standardization fairly low
· usually small and the questions well defined but open
· the responses are unstructured
· allowing respondents to formulate their answers the way they want
· employed in qualitative and feminist research
Semi-structured or semi-standardized questionnaires:
· These questionnaires can logically be placed between the two other types.
· Their structure may include a combination of structured and unstructured questions.
Questionnaires are either handed to the respondents personally, or are sent to them by mail. They must include three main parts; these are the cover letter, the instructions and the main body.
The cover letter
The aim of the cover letter is to introduce the respondents to the research topic and research team, to neutralize any doubt or mistrust respondents might have about the study, to motivate them to participate and answer the questions, and to ensure anonymity and confidentiality. More specifically, the minimum points it must cover are those shown in Box 11.1.
| The content of the cover letter Cover letters should at least: · describe the main objectives and social significance of the study · identify the research team and its sponsors · give reasons why the respondent should complete the questionnaire · guarantee anonymity, privacy and confidentiality · outline requirements for completion such as maximum time, conditions etc · give information about possible risks associated with the project · cover issues related to ethics |
The way the cover letter addresses the respondent (for example, Dear Sir; Dear Sir/Madam; Dear Mr. Jones; Dear Householder) and even the colour of the paper used, the form of letterhead and the style and format of the letter are very significant, at least with regard to the response rate.
Instructions
Instructions about how to fill in the questionnaire are mentioned only briefly in the cover letter (e.g. 'It shouldn't take more than 30 minutes of your time' or ‘you only need to tick the box in front of the questions'). Instructions will be given on the questionnaire and/or on a separate sheet. As well as giving details as to how to state their answer or preference (e.g. in pre-coded questions) the instructions usually remind the respondents that they should not try to please the researcher, that there are no right or wrong answers and that all questions should be attempted, and instructs them about what to do with the completed questionnaire (for example, that it should be returned to the project director in the self-addressed envelope by a certain date). For obvious reasons the instructions are expected to offer as much information as possible and must be written in a simple language. Inadequate instructions are one of the major sources of non-responses and should be avoided.
The main body
The main body of the questionnaire includes the questions to be answered. In order to be effective, this part of the document must be worked out very carefully, especially with regard to content, structure, wording, flow, format and so on, and adhere to the basic rules of questionnaire construction.
2 QUESTIONNAIRE FORMAT
Major issues regarding questionnaire construction are:
· how to formulate the questionnaire
· what types of questions should be considered
· what they should contain
· how long the questionnaire should be
· how it should be presented so that it is clear, easy to read and attractive to the respondent and, most importantly, so that it achieves its purpose
The questionnaire format refers to the order in which questions are organized within the context of the questionnaire. A common requirement of all models is that the questions have to be listed in a logical order, allowing for transition and flow, that is, for a smooth passage from one topic to the next, and to avoid distortions and problems. The following six formats:
· Funnel format: The questioning moves from general to specific, from impersonal to personal, and from non-sensitive to sensitive questions.
· Inverted funnel format: The questioning progresses from specific to general from personal to impersonal, and from sensitive to non- sensitive.
· Diamond format: A combination of the inverted funnel format and the funnel format, where questions progress from specific to general and back to specific, from personal to impersonal and back to personal.
· X-format: The first part of the questionnaire employs a funnel format and the second part an inverted funnel format. The questions here change from general to specific and back to general, from impersonal to personal and again to impersonal and so on.
· Box format: Questions are uniform throughout the questionnaire, with all questions being kept at the same level.
· Mixed format: Here questions appear according to the logic of the project, shifting from general to specific and so on as required. Mixed format questionnaires may also contain sections, each adopting one of the above formats; for example, the first section may employ the funnel format, the second the box format, and the last tile inverted runnel format.
The questionnaire format is chosen to suit:
· the nature of the survey
· the type of respondents
· length of questionnaire
· nature of administering the questionnaire
· the findings of a pilot study
It is important that the format serves the purpose of the study and not the personal preference of the researcher. Factors such as those controlling the soundness of questions must be taken into account when determining the format. It is also important that the questions are related to each other logically and are interesting and relevant to the topic; above all, the presentation and structure of the questionnaire should make the respondents feel at ease and appreciated, rather than the subject of a strict interrogation.
3 QUESTIONNAIRE SIZE
The size of the questionnaire depends on factors such as:
· the research objective
· the type of respondents
· the methods of analysis
· the availability of resources
The number of questions can range from only a few to several hundred. However, the golden rule with respect to questionnaire size is that one should include as many questions as necessary and as few as possible.
Questionnaire, regardless of whether it is offered to the respondent in a written form or as an interview, should be a translation of the central elements of the research topic. Length of a questionnaire depends on the number of variables and indicators considered in the study, as well as the number of questions required to address the indicators fully.
4 TYPES OF QUESTIONS
The questions that are considered vary with respect to a number of criteria, especially regarding their purpose and relevance to the research topic, their structure and approach to the issue they address, their content and wording, and the type of response they require. Some writers distinguish between essential questions (relating directly to the substance of the project), extra questions (set to check the reliability of the answers), and throw-away questions (used at the beginning to encourage rapport with the respondent); but there are many other types, as we shall see.
Primary questions: Primary questions elicit information directly related to the research topic. Each question provides information about a specific aspect of the topic, that is, an indicator of a particular variable. In a study of marital power, the question 'Who controls the relationship?' may be a primary question.
Secondary questions: Secondary questions are ones that do not relate directly to the research topic. They are of secondary importance in that they provide information on secondary issues such as consistency of opinions or the reliability of the instrument used. They do not add new information about the research topic, but they ensure methodological soundness, the integrity of the questionnaire or the truthfulness of the respondents.
Tertiary questions: Tertiary questions have neither primary nor methodological significance. They help to establish a framework that allows convenient data collection and sufficient information without exhausting or biasing the respondent. Two examples are padding questions and probes.
· Padding questions: These questions are not central to the research but are of interest to the respondent. Acting as a 'breather', they are usually placed before or after sensitive questions.
· Probes: These questions are used in interviewing to encourage the respondent to complete, amplify or expand an answer, stimulating and guiding the discussion and establishing a friendly atmosphere, free of bias. There are several techniques of probing, with interviewers developing and applying them according to need; two techniques often referred to by writers are the summary technique and controlled non-directive probing (to be discussed later).
Direct and indirect questions: Direct or personal questions ask the respondent to offer information about himself or herself. An example of a direct question is: 'Do you believe in God?' Indirect questions ask the respondent to offer information about other people, assuming that in this way the respondent will indirectly tell something about himself or herself. An example of an indirect question is: 'Do you think that people of your age believe in God nowadays?’ Indirect questioning is mainly used when it is suspected that respondents will not feel comfortable about offering direct information on the research question.
Suggestive questions: Suggestive questions contain an implied attempt to tempt the respondent to confirm a view. For example, if we were to test the views of students to examinations, a direct question could have been: Do you think that examinations should be abolished?' In a suggestive mode this question could read as follows: 'Don't you also think that examinations should be abolished?' Suggestive questions lend the respondent, and bias the direction of the findings.
Filter and contingency questions: Filter questions aim at eliciting (obtain) for the first time in the study, information related to a general aspect of the research topic, and are usually followed by another more specific question (i.e. a contingency question). An example of such a question is: 'Do you smoke?'
Contingency questions follow the filter question and aim to elicit additional and more specific information on an issue already addressed by a filler question. In our example, the contingency question may be: 'How many cigarettes do you smoke each day?' Asking contingency questions without filter questions is methodologically incorrect.
Fixed-alternative (pre-coded) and open-ended questions: According to their response format, questions can be divided into two categories: open-ended and closed (pre-coded or fixed-alternative) questions. These formats have a strong impact on the nature and usefulness of questionnaires, and are very common in social research. We shall discuss them in more detail below.
5 OPEN-ENDED AND PRE-CODED QUESTIONS
Open-ended (free-answer) questions allow respondents to state their answers in the way they see appropriate, in their own way and in their own words. Fixed-alternative questions (also known as pre-coded or closed questions) offer a set of responses from which the respondent has to choose.
For example, the question 'What do you think about the proposal of the student union to abolish examinations?' followed by an empty writing space, is an open-ended question. If the above question was followed by the response categories: 'Strongly agree (1); Agree (2); Not sure (3); Disagree (4); Strongly disagree (5); it would be a fixed-alternative question. In this case the respondents are expected to indicate their response by placing a tick next to the relevant category or by circling the number that corresponds to their personal view, as instructed.
Open-ended or pre-coded questions
| Open-ended questions are advisable if: · the researcher is interested in ample information · the attitudes, ability to communicate and motivation of the respondents are not known · the respondents cannot communicate : · they are not well informed and have not yet structured an opinion Pre-coded questions can be employed if: · the researcher is interested in classifying responses or respondents · the situation of the respondents is known · they can communicate · they are well informed and have formed an opinion |
In more general terms, whether to use open-ended questions or not depends on their strengths and weaknesses. Some central strengths and weaknesses of open-ended questions are shown below.
Strengths
· They allow freedom to express feelings and thoughts, especially when complex issues are being studied.
· They offer more details than pre-coded questions, especially qualifications and justifications.
· They offer information in areas that might not have been foreseen by the researcher.
· They allow conclusions about the respondents' way of thinking and logic
· They allow respondents to show creativity, self-expression and initiative
Weaknesses
· They are not very suitable for sensitive questions.
· They produce large amounts of information which, require extensive time and effort to code and/or evaluate.
· They are time consuming.
· They do not allow accurate comparisons.
· They can offer useless or irrelevant information.
· They are not suitable if the respondents have problems articulating (clear) well.
· They require additional processing if statistical analysis is intended.
The strengths and weaknesses of pre-coded questions are exactly the opposite of those listed above for open-ended questions.
6 RESPONSE FORMAT
6.1 Introduction
A characteristic of open-ended and fixed-alternative questions is that the former are easy to construct but difficult to process and the latter difficult to construct but easy to process. Constructing response sets for fixed-alternative questions is a complex and quite demanding task.
It is a strict methodological requirement that response sets to fixed-alternative questions adhere to certain standards and principles, of which the most important is that response categories should be accurate, exhaustive, mutually exclusive and unidimensional. A brief description of such options is given below.
Accurate sets: Response sets are expected to be accurate. In other words, they should, first, address the central point of the question; second, be relevant; and third, be related to the essence of the question. A response set developed to provide responses to the question 'How successful has your progress been on this course?' containing the response categories 'very satisfied', 'satisfied', 'unsatisfied' and 'very unsatisfied' is not accurate because it relates to satisfaction and not to success. These two issues may be interrelated but are nonetheless different: a business manager may be 'very satisfied' with the progress of a branch, but also rate its degree of success as 'moderate'.
Exhaustive sets: Response sets must cover all possible options. When answering the question 'Number of children?', followed by '(A) 1 or 2 children; (B) 3 or more children', the set is obviously not exhaustive: it leaves out families with no children. In other cases, options such as 'other', 'no opinion', 'I don't know' must be considered in order for sets to be exhaustive. However, care must be taken that these responses do not become 'soft options', attracting large numbers of responses. In such cases, pro-tests must be employed to detect such cases and call for relevant action.
Mutually exclusive sets: A set of categories is expected to include items that are clearly distinguishable from each other and mutually exclusive. The respondents should only be able to choose one true response, without confusion and ambiguity. The responses 'Single', 'Married', 'Living together', 'Divorced', 'Separated', 'Widowed' are (strictly speaking) not mutually exclusive, because the set allows a certain group of respondents to choose more than one option. For instance a divorced person who is cohabiting can equally tick the third or the fifth option.
Unidimensional sets: This refers to the requirement that a set of categories should refer to and measure only one construct, in only one dimension. A response category that includes the items 'very reliable', 'reliable', 'unreliable', 'very unreliable' is unidimensional because it measures reliability. The response set 'very happy', 'happy', 'unsatisfied', 'very unsatisfied' is not unidimensional because it relates to two dimensions, namely happiness and satisfaction.
6.2 Response sets
The following are possible ways to form response sets in the context of questionnaires; while some are more common than others, all have been and are currently being used by social researchers.
Numerical responses
This response category includes a continuum, with two opposite adjectives at each end and a range of numbers in between, one of which must be circled or otherwise marked by the respondent.
Example A: The response of the Prime Minister regarding material employment was: (Please circle the appropriate number.)
Very satisfactory 5 4 3 2 1 Very unsatisfactory
Verbal scales
In many cases the expected response to a question is formulated in words. The respondent in such cases is expected to tick one of the words in the space provided for that purpose.
Example B: The support provided by the Union was (please tick one):
Very high [---]
High [---]
Moderate [---]
Low [---]
Very low [---]
Scales of increasing strength
Some researchers opt for response categories indicated simply by a descriptor, and are followed by a set of numbers ranging from low to high, from which the respondent is expected to choose. The meaning of the numbers (e.g. 1 standing for very low and 10 for very high) will be explained in the general instructions for the questionnaire.
Example C. In this country the condition of women in the areas shown below can be characterized as: (Please circle the appropriate number,)
Human rights 1 2 3 4 5 6 7 8 9 10
Employment 1 2 3 4 5 6 7 8 9 10
Health 1 2 3 4 5 6 7 8 9 10
Graphic responses
The use of graphic responses is not new in social research. In its simplest form, a response contains a continuum whose extremes are defined by two opposite descriptors connected by dots or a line. The respondent is expected to mark the line at a point that expresses the strength of his or her view. The researcher will then evaluate the answer according to the position of the mark by means of a standard pattern.
Example D: Last Sunday's elections of the municipal officers were:
Fair --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- unfair
Graphic-numerical responses
A combination of graphic symbols and numerals is quite often used by social investigators. The direction of choice and evaluation is based on the selected position of the tick, which is not defined in words.
Example E: My wife's reaction to last week's rise in taxes was: (please tick)
[---] +3
[---] +2
[---] +1
[---] -1
[---] -2
[---] -3
Thermometer scales
In these scales, the responses are set in the form of a thermometer, presenting a continuum that displays the reading range of a thermometer, the extremes of which are described by opposite adjectives, for example 'Very high', 'Very low'. The divisions given on the thermometer are used to reflect the respondent's level of response.
Face scales
Another graphic scale employed to record answers to pre-coded questions in a simple manner is the use effaces. Here, usually five to seven faces (or sets of faces) of equal size and structure are ordered on a line. The faces are identical, except for the shape of the mouth, which at one end is shaped in a U-form giving the impression of happiness, and progressively changes through a neutral position (straight line) to an inverted U at the other end describing unhappiness. There are no explaining adjectives here as it is assumed that the faces offer a clear indication of the implied feeling. The respondents are asked to indicate their feelings to the question by marking the appropriate face. A five-point response containing a combination of faces is shown below.
Example F: How happy are you with the High Court's Decision? (please circle one set of faces)
Very happy ©© ©© ©© Very unhappy;
Ladder scales
In a response set that employs a ladder scale, the responses are given on a ladder presenting a continuum of five or more steps, whose extremes are defined by two opposite adjectives (e.g. "high', 'low' or 'strong', 'weak'). The question could be, for example, 'Whereabouts do you stand on the social ladder?', advising the respondents to pk.cc an X on the point of their choice.
Constant-sum scales
These scales ask respondents to score two or more objects or concepts so that they together add up to a given amount (e.g. 100). This relative measure is most suitable to ascertain, for instance, the psychological distance between stimuli. The respondents may be asked to allocate 100 points to the Labour Party or to the Communist Party, according to their handling of taxation issues. The rating may be 100 to 0 or vice versa, or 20 to 80, 60 to 40 and so on. Scores allocated by the respondents can be further computed and evaluated, for instance, by constructing relations of the pattern A/B = 60/40 = 6/4 = 1.5, or vice versa.
Likert-scale questions
Likert scales are widely used, particularly as a means for studying attitudes. The response categories range between two extreme positions divided into five points corresponding to a verbal-numerical scale.
Example G. The reaction of the nurses to the recent salary proposal is, in my opinion: (Please circle the number corresponding to your choice.)
Very positive 5
Positive 4
Neutral 3
Negative 2
Very negative 1
Ranking scales
Unlike many scales in which respondents are asked to tick one response only, ranking scales require that all responses be answered, for example by ranking them from highest to the lowest. In such cases there are as many ranks as there are items.
Example H: My ranking of the performance of the five political parties from 1(very low) to 5 (very high) is as follows: (Please enter, numbers in the, brackets)
Democrats [---]
Labour Party [---]
Greens [---]
National Party [---]
Liberal Party [---]
Semantic differential scales
These scales are employed to evaluate social units such as teachers, parents, friends or politicians. They contain a set of opposites (up to 70) relating to three major dimensions; these are (a) general impression, (b) power or potency and (c) activity. The space between the opposites is graded from 0 expressing the lowest evaluation (e.g. very bad, very weak, very low) to 6, representing the highest evaluation (e.g. very good, very strong, very high). When these scales are employed, the respondents specify where exactly in that range, in their opinion, the person in question should be ranked. Examples of indicators for each of the three dimensions are given in Example I. The three groups of opposites correspond to the dimensions listed above.
Example l: How would you rate the role of your instructor in your class?
Good 6 5 4 3 2 1 0 Bad
Harmonious 6 5 4 3 2 1 0 Unharmonious
Sociable 6 5 4 3 2 1 0 Unsociable
Hard 6 5 4 3 2 1 0 Soft
Difficult 6 5 4 3 2 1 0 Easy
Unyielding 6 5 4 3 2 1 0 Lenient
Irritable 6 5 4 3 2 1 0 Calm
Active 6 5 4 3 2 1 0 Passive
Sharp 6 5 4 3 2 1 0 Dull
The structure, nature and size of the set of responses depend on many factors, but mainly on the nature of the study, the nature of the respondents and the extent to which statistical analysis will be used. For an accurate and effective construction of response categories of this kind, several techniques have been developed and are frequently used by researchers. Those introduced by Likert Thurstone and Guttman are three examples.
7 QUESTION CONTENT
The content of the questions is obviously the most important element of a questionnaire or an interview guide. While form and order of questions may influence accessibility to information, the content of the questions will tease out the essence of the answer sought in the study. In order for the questionnaire to achieve its purpose, the content of questions must be organized according to the criteria listed below.
Focus: Each question is expected to address one point only. Double-barreled questions, where the question focuses on two points, should be avoided. For instance, the question 'Are your parents caring and supportive?' addresses more than one issue. First, it does not differentiate between father and mother (the mother may be caring and supportive while the father is not); it is not possible here for the respondent to describe the parents separately. Second, it asks about care and support. In such cases, one answer cannot express the respondent’s response to the issues covered. Such questions should be avoided. If information for all these issues is required, separate questions should be asked.
Relevance: The content of each question must be related to the research topic. Questions not directly related to the topic may be asked only if they are well Justified and serve a specific purpose.
Symmetry: The questions should address a specific element of the research topic and be symmetrical: there should not be many questions on one aspect and few on the others unless there are reasons for this. Unless there is good cause to treat the various parts of the questionnaire differently, each part should be given the same value and space.
Clarity and simplicity: The content of the questions must be clear and simple in language and in content. Questions that are too general, ambiguous vague or embarrassing should be avoided. Personal questions should be employed very carefully.
Language: Questions should be formulated in the language of the respondent.
Attitude: Questions should convey a positive attitude towards the respondent and the study, in general, should display friendliness and collegiality.
Presuming questions: Questions that rest on presumptions or presuppositions should be avoided. It is also unethical to ask a student, for instance, 'When did you stop cheating in the examinations?' without first introducing a filter question about cheating.
Suggestive questioning: 'Leading' or suggestive questions should be avoided. These are ones that encourage the respondent to give a certain answer (e.g. 'Don't you also think that examinations should be abolished?'), usually preferred by the researcher.
Prestige bias: This too should be avoided. It occurs when the respondent is encouraged to follow sonic generalized views of important people (e.g. 'Gerontologists believe that progressive age causes alienation and hostility among older males; what is your view on this issue?').
As we shall see later, the nature of questioning and of presenting questions to respondents varies. Qualitative researchers may disagree on a number of points mage earlier in this section; however, there is little doubt as to the validity and acceptance of the points made above regarding question content.
8 RULES FOR QUESTIONNAIRE CONSTRUCTION
There are several rules for questionnaire construction. Some are more important than others, but a set of such rules should be considered during the construction of the questionnaires or interview schedules. The views of many writers on this point may be summed up as shown below (Mahr, 1995).
Rules for layout
· Questions must be easy to read and easy to follow.
· Questions and response categories must be easy to identify and distinguishable from other questions and response categories. Sufficient space should be provided between the questions.
· Clear instructions about how to answer the questions must be given; for example, 'Circle the appropriate number' or 'Tick the right box'. Nothing should be taken for granted.
· Sufficient space should be left for the respondent to make relevant remarks, where required.
Rules for question content
· Every question must be relevant to one or more aspects of the study.
· Ambiguous, non-specific and hypothetical questions are to be avoided.
· Leading, double-barreled and presuming questions should not be employed.
· Embarrassing, personal or threatening questions should be avoided.
· Vague words and academic jargon should not be used.
· The language of the respondent should be employed. If that is not possible, simple language should be used, without jargon, slang, double negatives or complicated expressions.
· The easy flow and logical progression in the questionnaire should be ensured.
· Each question should ask what it is supposed to ask.
Rules for questionnaire format
· The questionnaire must have a professional appearance and give the impression of a document that deserves respect and invokes feelings of responsibility.
· The questionnaire should be presented in a way that encourages the respondent to complete and return it.
· Writing on one side of the page only is preferable to writing on both sides.
· Print and colour of paper and ink must give the questionnaire an attractive appearance.
· The questionnaire should be presented as a complete document, with an inviting and reassuring introductory cover letter and a concluding note containing instructions about returning the questionnaire.
· The questionnaire size should be kept to a minimum, with as many questions as necessary, and as few as possible.
· Sufficient instructions and probes should be provided where necessary.
· Pre-coded questions should offer adequate response categories.
· All questions should be checked for possible bias and ethical adequacy.
These rules relate to the methodological, technical and practical aspects of surveys. Questionnaires are constructed within parameters that take into account professional standards and responsibilities. The respondents are also an important factor in this context. Well-constructed questionnaires with a friendly and inviting appearance encourage a high response rate, and this is a factor researchers have to consider very carefully.
9 STEPS IN QUESTIONNAIRE CONSTRUCTION
Questionnaires are constructed in a very focused and systematic manner. The process of construction goes through a number of interrelated steps, and offers a basis for the research stage to follow. The following are the most common steps in questionnaire construction (see Pfeifer, 2000; Puris, 1995):
Step 1: Preparation
The researcher first decides what the most suitable type of questionnaire is, and determines the way it will be administered. There should also be a search for relevant questionnaires that might already have been developed by other investigators. If suitable questionnaires are found, they can either be adopted for the study or used as guides in preparing the new one. If the search is unsuccessful, the researcher will proceed with the construction of a new questionnaire.
Step 2: Constructing the first draft
The investigator formulates a number of questions, usually a few more than required, including questions of substance (directly related to aspects of the research topic), questions of method (those testing reliability and wording), and secondary as well as tertiary questions.
Step 3: Self-critique
The questions are tested for relevance, symmetry, clarity and simplicity, among other criteria, as well as for compliance with the basic rules of questionnaire construction presented above.
Step 4: External scrutiny
The first draft is then given to experts for scrutiny and suggestions. It is anticipated that some questions might be changed or eliminated, while new questions might be suggested.
Step 5: Re-examination and revision
The critique offered by the experts and group leaders will be considered and eventual change;; implemented. If the revision is not significant, the investigator proceeds to the next step. If the revision is substantial, the questionnaire is presented again to experts and later re-examined and revised until it is considered satisfactory. The investigator then proceeds to the next step.
Step 6: Pre-test or pilot study
In most cases a pilot study or a pre-test is undertaken to check the suitability of the questionnaire as a whole (pilot study) or of some aspects of it (pre-test). A small sample is selected for this purpose, and the respondents are requested to respond to all or part of the questionnaire; thie results are then analyzed and interpreted.
Step 7: Revision
The pre-test and pilot study usually result in some minor or major changes. If the changes are relatively insignificant, the investigators will proceed to Step 8. If the changes are substantial they will return to Step 4.
Step 8: Second pre-test
The revised questionnaire may then be subjected to a second test, mainly with regard to the revised questions. This depends on the extent of revision and the complexity of the issue in question. Usually one pre-test is sufficient. The response is considered and adjustments and revisions follow.
Step 9: Formulation of the final draft
In this final step, apart from implementing the suggestions derived from the pre-tests, the investigator concentrates on editorial work: checking for spelling mistakes, legibility, instructions, layout, space for responses, pre-coding, scaling issues and the general presentation of the questionnaire. This copy will finally be sent to the printer.
10 PRE-TESTS AND PILOT STUDIES
Pre-tests and pilot studies are two instruments employed by quantitative researchers before the actual data collection commences. They are trial studies ('miniature preparatory studies') employed to ensure that the planning of the main study and its study tools are correct:, suitable, reliable and valid. They serve similar purposes, but differ in certain ways, as we shall see.
10.1 Pre-tests
Pre-tests are small tests of single elements of a research instrument that are predominantly used to check its 'mechanical' structure. The response categories to a particular question might need to be tested, or the question might be found to be unclear or misleading. The response categories, for example, might not be sufficient to cover all possible options, and researchers need to avoid overloading the option 'Other, please specify'.
A questionnaire can be pre-tested, and the responses will demonstrate whether there is a need to re-arrange the response categories to a particular question, if the proportion of respondents opting for 'Other, please specify' is relatively small, the question will remain as it is. If, however, the number of the responses resorting to this item is relatively large, the response set will need to be adjusted.
A similar procedure will be employed if the researcher is doubtful about whether the subjects will in fact answer a sensitive question directly or opt for the neutral category 'Undecided', 'No opinion' or 'I don't know'.
10.2 Pilot studies
A pilot study is a small-scale replica and a rehearsal of the main study. While pre-tests help to solve isolated mechanical problems in an instrument, pilot studies are concerned with administrative and organizational problems related to the whole study and the respondents.
Pilot studies serve many goals, but those considered by most writers (e.g. Oppenheim, 1992; Sproull), 1988) to be the most important are shown in Box 11.3. To a certain degree, these points are relevant to both qualitative and quantitative research. In qualitative research, however, pilot studies aim to establish whether respondents are accessible, whether the site is convenient, whether the techniques of data collection generate enough information (neither too little nor too much), whether the plan is well constructed and whether any changes or adjustments are needed.
10.3 Evaluation
The results of the pre-tests and pilot studies will be evaluated in the context of the aims described above. The findings will offer straightforward answers to basic questions asked, and will indicate whether there are any problems that require attention. If for instance, the researcher cannot establish the expected rapport with the respondents, if there are difficulties in convincing the majority of the respondents to take part in the study, if the respondents refuse to answer certain questions or give high proportions of 'I don't know' answers, if answers are given but with small notes or qualifications, and if the observed subjects tend unexpectedly to be distracted by the video camera or other environmental factors, then certain adjustments must be made (Becker, 1989). Pre-tests and pilot studies present a trial run of the study and offer an opportunity for adjustments and 'fine tuning' before the real work begins.
| Goals of pilot studies · to estimate the costs and duration of the main study · to test the effectiveness of the study's organization · to test the suitability of the research methods and instruments · to ensure that the sampling frame is adequate · to estimate the level of response and type of drop-outs · to ascertain the degree of diversity of the survey population · to familiarize researchers with the research environment · to offer an opportunity to practice using the research instruments before the main study begins · to test the response of the subjects to the overall research design · to discover possible weaknesses, inadequacies, ambiguities and problems in all aspects of the research, so that they can be corrected before actual data collection takes place |
11 REVIEWING THE QUESTIONNAIRE
The questionnaire review often involves a large number of points, many of which relate to the nature of the particular research topic. However, many writers (Berger et al., 1989; Beninin, 2000; Puris, 1995; Selltiz et al., 1976) stress that: the following points should be considered in the review:
· Size of the questionnaire: Is the questionnaire too large or too small? The rule here is that every question should have a specific purpose; if not, it has no place in the questionnaire.
· Relevance of procedure: Which point of the topic is the question related to? Is it strictly relevant? Does it ask what it is meant to ask? Questions must be tuned to one specific point in a clear and unambiguous way.
· Necessity: Is every question essential? Could some questions' be omitted? Is there any repetition in the questions? Is more than one question needed for each item? Questions will be retained only if they have a certain purpose and are really necessary.
· Clarity: Are the questions easy to understand, clear and unambiguous?
· Tone and content: Is the tone of the questions acceptable? Are the questions unethical, threatening, insulting, patronizing or otherwise biasing? Such questions must be changed or omitted.
· Layout of the questionnaire: Is sufficient space provided for recording answers given to open-ended questions? Are the layout of the questionnaire, the colour of the paper and the print size adequate and acceptable?
· Pre-coded questions: Are the response categories to pre-coded questions easy to understand, exhaustive, unidimensional and mutually exclusive? If not, they need to be restructured.
· Adequacy: Are all aspects of the topic adequately covered? If not, new questions should be added.
· Instructions: Are sufficient instructions given for filling out the questionnaire and for proper use of probes? Are these instructions adequate to guide the respondent through the questions?
· Level of pitching: Is the wording of the questions appropriate for the respondents' linguistic ability, education, interest and intellectual capacity'
· Cover letter: Is the cover letter constructed adequately? Does it offer the required information? Are any points missing? Is it too long or too short? Are the respondents properly addressed in the cover letter?
· Pre-coding: Is pre-coding (where required) recorded adequately and in accordance with the computer program used?
· Demographic data: Are all demographic data of the respondent (age, education, occupation etc.) required? Are they sufficient? Are they positioned in the right: place on the questionnaire?
· Principles: Have the methodological principles regarding the questionnaire construction been adhered to?
· Legal responsibilities: Are any questions likely to affect the rights of the respondents or third parries?
· Ethical considerations: Is the questionnaire ethically sound?
· Overall impression: Is the questionnaire easy to read and pleasant to follow overall?
This list concentrates on the most important issues. Individual projects will raise additional and more specific requirements that will need to be addressed by the investigator.
12 RELEVANCE OF THE QUESTIONNAIRE
As stated earlier, questions must be relevant to the research topic. Each question will concern one or more aspects of the topic, and all the questions together will cover all aspects of the topic. To ensure that questions are relevant to the topic and evenly distributed among the different aspects of the research question, and that no parts of the research issue are omitted from the questionnaire, researchers usually follow a number of steps. These were mentioned earlier in this chapter and are summarized below:
Step 1 The concept(s) to be studied is (are) identified.
Step 2 The dimensions of the concept(s) are ascertained.
Step 3 Indicators are identified.
Seep 4 Indicators are translated into a number of questions.
Step 5 These questions are put in a questionnaire following the procedures introduced above.
Thus, developing questions for a questionnaire is a process of translating research topics into variables, variables into indicators and indicators into questions. This process ensures that each question has a definite purpose and elicits information related to a specific aspect of the research object.
Let us see how this works in practice. Assume that we wished to study the 'Effects of religiousness on the scholastic achievement of teenagers'. Following the suggestion contained in Step 1 above we identify the concepts, which in this example are 'religiousness' and 'scholastic achievement'. In Step 2 we ascertain their dimensions (respectively, religious beliefs and practice, and attainment in maths, English, science and social studies). Following Step 3, we translate the dimensions into a number of indicators. For example, indicators for the first concept might be belief in God, church attendance, bible reading and participation in religions activities; for the second concept they might be grades in English, mathematics, science and social studies.
The chain of translation
| Concepts | Dimensions | Indicators | Questions |
| Religiousness | Beliefs Activities | Belief in God Belief in saints Church attendance Religious activities | Do you believe in God? Do you believe in saints? Do you go to church? Do you attend Sunday school? |
| Scholastic achievement | Maths English Science Soc- studies | Grade Grade Grade Grade | Your grade in maths? Your grade in English? Your grade in science? Your grade in social studies? |
These indicators are then translated into questions (Step 4) such as whether the respondents believe in God; whether they attend church; whether they read the bible; whether they participate in religious activities; and what grades they achieved in English, in mathematics, science and social studies. While some indicators might be translated into one or two questions, others might require more. One might wish to know, for instance, about how often the respondents attend church, and also about their sibling(s), parent(s) or other relatives at the present time, last year or two years ago. The important point is that questions translate the meaning of a topic or variable and provide information for a particular aspect of the research topic.
If this chain of translation is followed carefully, all questions will be relevant to the research topic and the researcher will know exactly the purpose of every question in the questionnaire. As a result, every point of the research issue will have one or more corresponding questions, and every question will have one or more corresponding points in the research problem for which they provide information.
13 NON-RESPONSE IN MAIL QUESTIONNAIRES
13.1 Introduction
In survey research, particularly when mail questionnaires are employed, it is common that many people included in the sample will not take part in the research. More specifically, most of those who receive a questionnaire will not return it to the researcher by the due date. The most common reasons for this are thought: to be those shown below (see Becker, 1989; Puris, 1995). For instance:
· They are unwilling to do so (for example, they disapprove of the research study, do not like the sponsor, do not like answering questionnaires, or the spouse disapproves of it).
· They are unable to take part in tile study, due to physical, mental or linguistic problems (e.g. handicapped people or migrants).
· They are not accessible (e.g. have moved away, died or were not: at home when visited).
· They did not have time to respond.
· They did not receive the questionnaire.
· They found the questionnaire too complex.
Given that this loss of respondents can often be very high, non-response is an issue that deserves serious attention. It is therefore understandable that investigators make a concerted effort to keep the non-response rate as low as possible.
This is accomplished through various measures, ranging from social or psychological to economic measures, chat fall into two main kinds. The first aims to prevent non-response in the first place, and employs techniques that encourage the return of the questionnaire. The other is to follow up non-responders and encourage them to return the questionnaire (see Box 11.5). In the latter approach, sending reminders to all those who did not return their questionnaire and urging them to do so is a standard procedure. Two reminders may be sufficient. After the second reminder, it is very unlikely that any further attempt will be successful. It is also tempting to offer incentives to respondents so they return the questionnaires. However, this can bias the sample by attracting particular people to the study.
13.2 Preventing non-response
In general, planning the study very carefully, taking appropriate measures and being vigilant can improve the response rate. There is also a lot more that can be done to deal with non-responses. The following suggestions (see Lamnek, 1993; Mahr, 1995; Puris, 1995; Selltiz ctal., 1976) can be pointers in the right direction:
· Approach: The more serious, trustworthy and friendly the cover letter, the more likely it is that the questionnaire will be accepted and completed.
· Explanation: The more clearly the purpose, nature, usefulness and sponsorship of the study are explained, the more likely it is that the questionnaire will be completed.
· Honesty: The more honc.sr, direct and concise the cover letter, the more likely it is that the questionnaire will be answered.
· Length of cover letter: The shorter the cover-letter the better.
· Principles: The more convinced the respondent is of the anonymity and confidentiality of the study, the more likely he or she is to answer the questionnaire.
· Reminders: The more convinced the respondent is about the use of reminders and their relationship with anonymity, the inure crust is generated and the more likely it is chat the questionnaire will be answered.
· Rationale: The more convinced the respondent is of the rationale of the study and the reasons for caking pare in it, the more likely it is that the questionnaire will be completed.
· Time required: The more specific the indication of the time required for the completion of the questionnaire, the more likely it is that the respondent will complete it.
· Size: The smaller the size of the questionnaire, the more likely it is to be completed.
· Degree of difficulty: The easier it is for the questionnaire to be completed, the more likely it is for the respondent to complete it.
· Sensitivity: The less sensitive the question content, the more likely it is that the questionnaire will be completed and returned.
· Method of return: The easier it is for the questionnaire to be returned (e.g. enclosing a self-addressed, pre-stamped envelope), the more likely it is that it will be returned.
· Time of completion: The more convenient the rime for the respondent to complete the questionnaire, the more likely to be returned.
· Administration: The more friendly and personal the delivery of the questionnaire, the more likely it is to be returned by the set date.
· Rewards: The greater the rewards for returning the questionnaire, the more likely that it will be returned to the researcher.
· Return dates: The less appropriate the return date (too short or too long), the more likely it is that the questionnaire will not be returned.
· Appeal: The more appealing and attractive the cover letter, the more likely it is that the questionnaire will be completed and returned.
· Layout and format: The more appealing and attractive the layout and format of the questionnaire, and the more pleasant it is to answer the questions, the more likely it is that it will be completed and returned.
· Appearance: The more impressive the colour of paper, type of print and type of mailing, the more likely it is that the questionnaire will be completed. Trust: The more trustworthy the questionnaire in terms of confidentiality and adherence to clear standards, the more likely it is to be completed and returned.
| Fifteen ways of improving response rate · Make an effort to screen the sampling frame to eliminate drop-outs (e.g. people who have died or moved away). · Make the return time not too short and not too long. · Make the questionnaire brief; in general, long questionnaires are less likely to be returned. · Make the questionnaire look good; appearance pays. · Make it easy to read: e.g. good print, dear language, good instructions. · Make it easy to answer: e.g. dear options, and tick boxes. · Make it appear trustworthy: stress anonymity, confidentiality and ethical standards. · Make it respectable and impressive, in terms of the envelope, cover letter etc. · Make an effort to avoid being intrusive; do not offend the respondents. · Make an appeal to morals or other principles, but be truthful and honest. · Make the respondents feel important by addressing them appropriately. · Make its status evident: highlight associations with sponsors, universities etc. · Make it easy to respond: include a 'return paid' envelope. · Make some promises, but be honest; don't overdo it; and don't beg! · Make good use of one or two follow-ups, using honest and ethical standards. |
14 QUESTIONNAIRES IN FEMINIST RESEARCH
It has often been argued that questionnaires are a research instrument of quantitative rather than qualitative investigators. This is particularly so for standardized questionnaires. Supporting this is the notion that questionnaires are mostly used by quantitative researchers and also that most criticisms of this instrument come from feminist and qualitative researchers.
Although it is correct that large-scale surveys employing standardized questionnaires are the monopoly of quantitative researchers, this does not mean that feminist and qualitative researchers abstain from using this research method. The latter employ un-standardized and unstructured questionnaires rather than standardized questionnaires, and use them more often than quantitative researchers do. Un-standardized and unstructured questionnaires, containing open-ended questions and allowing subjectivity and flexibility in the way questions are constructed and answered, are in accord nor only with epistemological principles but also with common practice.
Feminist and qualitative researchers do not criticize questionnaires as such, but rather highly standardized formats and the objectivity and value neutrality which underpin their structure and presentation. Feminists and qualitative researchers in general propose; ways of improving questionnaires but do not reject them out of hand. Finally, feminists are critical of the way in which researchers are set against the subjects, that is, of the hierarchy of roles and the distance that is systematically expected between them. According to these arguments, however, questionnaires are neither inconsistent with basic aspects of qualitative research, nor ignored by qualitative researchers.
This point is further supported by the fact that feminist empiricists consider questionnaires as important as quantitative researchers do. As with other feminist and qualitative researchers, so for empiricists the use of ‘soft questionnaires' – ones that are brief and un-standardized, with open-ended questions and a close and an egalitarian relationship between the researcher and the researched – is an accepted method of data collection. Moreover, they show an interest in standardize questionnaires, and the implied compilation of data of a higher level of quantification and comparability, adequately adjusted to meet the specific requirements of their underlying paradigm.
Briefly, the distinction here for feminist and qualitative research is not associated with the research instrument per se but rather with the way in which it is conceptualized and applied in the various contexts; and as we already know, most methods - including questionnaires - leave a lot of scope for adjustment and negotiation.
15 STRENGTHS AND WEAKNESSES OF QUESTIONNAIRES
Questionnaires, as methods of data collection, have strengths and weaknesses, and hence advantages and limitations, that the researcher must be aware of. A list of both is given below.
Strengths
· Questionnaires are less expensive than other methods; in the words of Selltiz et al. (1976), 'questionnaires can be sent through the mail; interviewers cannot'.
· They produce quick results.
· They can be completed at the respondent's convenience.
· They offer greater assurance of anonymity.
· They offer less opportunity for bias or errors caused by the presence or attitudes of the interviewer.
· They are a stable, consistent and uniform measure, free of variation.
· They offer a considered and objective view of the issue, since respondents can consult their files and since many subjects prefer to write rather than talk about certain issues.
· The use of questionnaires allows a wider coverage, since researchers can approach respondents more easily than other methods.
· They are not affected by problems of 'non-contacts' (i.e. of respondents who are not available at the time of the study; a problem common in interviewing).
Weaknesses
· Questionnaires do not allow probing, prompting and clarification of questions.
· They do not provide opportunities for motivating the respondent to participate in the survey or to answer the questions.
· The identity of the respondent and the conditions under which the questionnaire is answered are not known. Researchers are not sure whether the right person has answered the questions.
· It is not possible to check whether the question order – where required – was followed.
· Questionnaires do not provide an opportunity to collect additional information (e.g. through observation) while they are being completed.
· Due to lack of supervision, partial response is quite possible.
In questionnaires, as in other methods in general, the focus of evaluation should be not only on the quality of the method but on its suitability. The nature of the research context should be given due attention: questionnaires may be very useful in one context but not in another,
16 QUESTIONNAIRES IN THE COMPUTER AGE
Due to their structure, questionnaires - especially those with a standardized structure – lend themselves to computer assistance. To a certain extent, the use of computers goes further than Just assisting the researcher in data collection. In some cases the computer takes over questioning, and collection of data occurs between the computer and the respondent without the involvement of the researcher.
This is obvious in the area of interviewing, which will be discussed more extensively in the next section. The packages CAPI, CATI, CODSCI and CLSUR allow interviews to be run through a computer, with the respondent reading and answering questions on the screen (following instructions from the computer). Interviews are administered like questionnaires; interviews become electronic questionnaires, often without an interviewer being present.
Apart from this, there are computer packages that take over the administration and completion of the questionnaires, replacing the traditional research assistant. The program 'Computerized self-administered questionnaire' (CSAQ) is one example. The respondent is given no questionnaires to fill in but is asked instead to sit in front of a computer terminal; the computer displays the questions on the screen, giving the necessary instructions and offering advice on technical aspects of the questionnaire when required. The respondent enters the responses in the computer and the computer saves the data in its memory and adds the responses to the research data, gradually preparing them for analysis.
Other programs operate in a similar way. The computer has, obviously extended its function from its traditional role as assistant in statistical analysis to data preparation and collection, functions that were once performed predominantly by research assistants.
Beyond this, computers are the bridge to more advanced developments in the domain of electronic media, allowing the researcher easier and faster access not only to data but also to respondents. More specifically, the computer connects the researcher with the Internet and through this to email and web pages, where researchers contact prospective respondents who may also fill in questionnaire. This opens the door lo a significantly larger number of people and also to forms of questionnaire administration that almost eliminate the human factor on the side of the researcher.
As noted earlier, computers link researchers with the Internet, and so facilitate Internet sampling and .surveys. Internet users may be attracted to the research and those who finally agree to participate are guided to a site where the questionnaire is located. From then on, the procedure is the same as in other computer-assisted questionnaires. Questionnaires are completed and the results filed in databanks. Internet surveys have become popular, and many researchers as well as research organizations take advantage of them. An example of a systematic employment of Internet surveys is the construction of groups of volunteers who makes themselves available via the internet to complete such surveys. (Set fees are in place for each member taking part.)
The Internet is also used in combination with other, more traditional methods. For instance, in order to control representativeness, researchers construct their samples using conventional methods, and respondents are then guided to Internet sites where they complete the questionnaires at their own convenience. This makes contact with researchers and travel to the research site unnecessary.
Main Points
· Questionnaires are a form of survey: a written survey
· The main elements of a questionnaire are the cover letter, instructions and main body.
· The questionnaire format can be one of the following: funnel format, inverted funnel format, diamond format, X-format, box format and mixed format.
· A questionnaire usually contains primary, secondary and/or tertiary questions. They may be fixed-alternative or open questions.
· Response sets in fixed-alternative questions must be accurate, exhaustive and uni-dimensional, and must include mutually exclusive categories.
· Particular attention should be given to the question content, especially to composition, relevance, clarity and simplicity, level and type of language, and to the attitude conveyed through the questions.
· Questions are the last step in a series of translations, leading from the definition of the research topic to indicators and to question wording,
· Pro-tests are small tests of single elements of the research questionnaires, aiming to check their soundness and relevance.
· A pilot study is a small-scale replica of the main study involving a fraction of the sample.
· Pre-tests mainly address research instruments; pilot studies mainly address research process and outcomes.
· Non-response is a serious issue that requires serious consideration.
No comments:
Post a Comment