12. Survey Design

As the number of users of MIS resources expand, gaining information about the users and customers has become more difficult. As the interested population grows, surveying samples have become the efficient method to collect status and opinion information. However, it must be noted that survey research approaches are fraught with difficulties, bias and error.

12.1 Purpose for a survey

Before undertaking any survey, it is critical that the purpose for the study be defined. Surveys have a tendency to grow in size and extent when its purpose is unclear. The purpose should be as specific as possible; preferably tied directly to an identified decision. No survey should be undertaken if its results will not be used.

12.1.1 Systems Development

Identifying the user needs and trying to establish a consensus of needs for large systems is a typical use of surveys. The objective of the survey, in this case, would be to identify the specific system requirements.

12.1.2 Service Development

The MIS functions within organization may serve a broad collection of users. Typically, only the opinions of the most vocal of users are heard. In order to verify that these voices speak for the population of users, a survey is often conducted.

12.1.3 Service Tracking

It is good management practice (and required for Total Quality Management) that the performance of the organization be measured and monitored on a regular basis. Surveys can provide this type of monitoring. However, it should be noted that the objective is to verify the need and performance of services. The goal would be to offer new services, change services presently offered, or modify the services to be more effective.

12.2 Confidentiality

Traditionally, market surveys are undertaken on a confidential basis. The identities of the respondents are not revealed to the initiator of the study. The purpose of this is to encourage truthful responses. Confidentiality may also serve this purpose for internal surveys and should be considered where revealing the identity of the respondent may bias the results.

However, in most cases, internal surveys are not conducted on a confidential basis. The identity of the respondent is available to the initiator of the study for follow-up. In these cases, it is important the respondent is not led to believe that the study is confidential.

12.3 Types of Surveys

For MIS management, there are a number of type of surveys that are typically conducted. If they are done is a regular basis, it would be worthwhile to develop standard forms.

12.3.1 User Needs & Skills

User needs and skill studies are usually undertaken for the development of training and support programs. However, as more attention is being paid to the design of the user interface, this user needs and skill studies should become more common.

Future needs studies is of particular importance to MIS. This type of study includes technology assessment issues as well as internal evaluation.

12.3.2 Customer Satisfaction

Customer satisfaction studies have become the centerpiece of Total Quality Management measurement. Traditionally, in the marketing area, these studies are combined with competitive positioning. As organizations move toward out-sourcing of MIS resources, more attention should be paid to both customer satisfaction and from an internal perspective, to competitive positioning.

12.3.3 Customer Use

Customer use studies focus on how users interact with a system or service. This type of study is more common in the design of service centers, hot lines and communication packages. The focus of the study is on how much and in what way a system is used.

12.3.4 Product Design

Product design studies are usually undertaken only in cases where there is anticipated to be a large number of potential users. This is becoming more common with the development of organization-wide database systems. It is also common in the design of commercial packages for either general use or vertical markets.

12.3.5 Longitudinal Studies

Longitudinal studies focuses on the changing needs, skills and opinions of individuals. These studies consist of multiple phases conducted over time. While they are not typically conducted by MIS group today, they should become more important as the issues of changing organizations become more apparent.

12.4 Inaccuracy and Imprecision

No measure is without error. That error comes from both an imprecision or disparity in the measurement and inaccuracy or uncertainty as to its meaning.

The key to conduct a successful study is to ask the right questions of the right people. Failure in either respect results in inaccuracy.

12.4.1 Types of Errors

Error is derived from a multiple set of sources. The total error is a combination of errors resulting in imprecision and inaccuracy. Sampling Error

Surveys are almost always conducted based on a sample of the population. Because not all respondents are included, there is an expected error around averages. This is the traditional reported imprecision or confidence level of surveys. Unfortunately, it only represents one source of error and one that can be controlled. "Non-Random" Sampling Error

While many samples are reported as randomly selected, few are. Samples are taken from lists, which are themselves, usually incomplete and often biased. Furthermore, many samples are taken based on convenience of a location or easy access to respondents. This may further introduce error. Fortunately, this is usually not a significant problem. Respondent Error

Often we don't ask the right people. A survey targeted to the decision-maker might be sent to the wrong department and answered by a secretary. With phone studies, we try to remove this problem by screening respondents. Measurement Error

Asking the right question in a manner that the respondent understands and can answer in an unambiguous fashion is the second key element of survey research. Measurement error reflects our inability to ask questions appropriately. 1 Inaccuracy

Inaccuracy focuses on our inability to know what the answer to the question means. When undertaking the survey, we have specific issues in mind and potential actions to take. The questions in the survey should be designed to address those issues. However, sometime the mindset of the respondent is not aligned with the decision frame of reference. The questions may have a different meaning to the respondent than the researcher or the user of the survey data. Confusion

Questions and potential answers are often worded in a way hat confuses the respondent. Sometimes the terms are unfamiliar or the question phrased to that multiple answers are possible. Confounding

Questions or results whose meaning have multiple interpretations are confounded. Many times, how we word questions, the meaning may be taken several ways. Response Errors

Response error focuses on what the respondent answers or refuses to answer. Refusal

Refusals introduce non-random sample error. As long as the people who refuse come from the same population as those that do respond to the questionnaire, there is no problem. However, we normally do not know that nor do we have any way of knowing it unless we encourage "refuseniks" to respond. Missing Data

Similar to outright refusals are those cases where the respondent refuses to answer specific questions. While in many cases, this is do to inappropriate design, where specific questions are not appropriate to a specific respondent's situation, it often reduces the use of the data. Fortunately, tests of completed answers can be used to determine internal error due to these missing items. Respondent Fatigue

Long questionnaires are likely to fatigue respondents. Some studies may involve more than a hour to execute. This can lead to a systematic error in the study. An effort should be made to make the questionnaire as easy to execute and as short as is possible. Respondent Irritation

If the respondent is irritated by the wording or being too intimate the responses may be unreliable.

12.4.2 Bias

Bias is a type of error that results in a systematically shift in one or another direction. All errors may also result in a bias as well as a decrease in precision and accuracy. Some degree of bias is inherent in all studies. Semantics tell us that all words contain social implications that go beyond their simple definitions. It is, therefore, impossible to reduce all bias. Our major effort in questionnaire design is to "control and identify" bias. We endeavor to design studies where the bias will only minimally effect its conclusions. Instrument Design

The nature of the questions asked is the most obvious source of bias. Asking loaded questions will give biased answers. Wording

The wording of questions can clearly result in biased results. It is usually not surprising in political polling that results tend to favor the organization and candidate that is undertaking the poll. This is usually due to the wording of the instrument. Order

The order of the questions can influence the responses. A series of questions on morality that followed by a question on abortion, for example, will change the response on the abortion question. Referencing

If the sponsor of the study is identified, explicitly or implicit, the responses may be differ from an truly "blind" study. Allowed Responses

The range of allowed responses can bias the results. Using a 1 to 10 scale for example tends to systematically move values to the high side of the scale. Similarly, if a mid-value is offered it will be taken more often than expected. Interviewer

In phone and in-depth interviews, the interviewer can affect the responses. People wish to please other people, and will try to do so on a survey as in other ways. Interviewers can non-verbally hint at answers consciously or unconsciously. It is, therefore, necessary not to use interviewers involved in the results of the study. Analysis

Analysis can introduce bias. Some techniques emphasize difference in values, whether that difference exists or not. Standardization and normalization sometimes shifts data in systematic ways.

12.5 Process steps

The steps for developing and executing a survey have much in common with those used to develop a MIS system. It should focus on identifying the key value elements, assuring that they are covered, and reducing the non-value adding elements.

12.5.1 Definition

The most important step of the process is the definition of the survey. It is usually useful to split the responsibility for a survey between two individuals, the initiator and the researcher. This provides, at least, two individuals to agree on the definition, design and analysis of the study. During the definitional phase the following should be explicitly identified:

  • The Goals and Objectives;
  • Key questions that need to be answered;
  • The target respondents; and
  • Determining how the respondents are to be identified. Compiling Existing Data

Existing data should next be identified. This data includes:

  • Lists of potential respondents;
  • Operational data on the number of respondents; and
  • Use level.

This information is often available from accounting data. In addition, past surveys of this type are useful to determine possible forms and questions. Furthermore, if previous studies exist their results can be used as reference points. Qualitative Research

The purpose of qualitative research is to obtain an statistical view of the issues being investigated. In many cases, this is sufficient for the project and statistical or quantitative phase is unnecessary. Qualitative research usually uses open-ended questions and discussion to ferret out issues. If the results of the qualitative research are overwhelming, a survey would not tell you much more. However, in most cases, the qualitative phase identifies the issues and the quantitative phase quantifies their importance and significance.

Initial testing of the questionnaire is usually done during the latter part of the qualitative research phase. The purpose is to obtain an in-depth reading of the respondents' understanding of the instrument. In-depth Interviews

In-depth interviews are a typical method of qualitative research. These are conducted both in person and by telephone. Personal interviews are preferred if there are things to be shown as well as discussed. Viewing how a system is used can reveal much more that having the respondent try to explain it. Discussion Outlines

The interview should follow in some form a discussion outline. The outline shows the questions and tasks that need to be covered. There are many styles of interviews. These generally follow the same outline as covered in the earlier section of these notes. Focus Groups

If we need to explore differences of opinion or collective behavior for the survey, we might use focus groups. These are group discussions on a specific set of issues. A discussion outline is used for these as well as for in-depth interviews. Successful focus groups can be done using telephone (teleconferencing) and using video systems. These techniques have not been traditionally used for MIS projects in the past, but have strong potential.

12.5.2 Design

Concurrent with the qualitative research phase is the design of the survey. This mainly focuses on the development of the questionnaire. Survey Design

The sample size, method of sampling, the list of the respondents, type of execution, and all other protocols for the study need to be determined. Since the cost of the survey usually rests on these factors, they tend to be determine early, usually before the questionnaire is designed. It is particularly necessary to determine the type of execution since that will determine the nature of the questionnaire. Questionnaire Development

Writing a effective questionnaire is a long process, typically involving many (~10) revisions before final testing. The more complex the task the greater the number of revisions. Usually questionnaires need to be reviewed by several analysts as well as the initiator of the study to get multiple perspectives on the instrument.

Note: if possible utilize a professional in the preparation of the questionnaire Experimental Tests

Some measurement techniques require giving the respondents a set of objects or stimuli for evaluation. This involves an experimental statistical test procedures. Here again, it is best to use a professional in setting up these protocols. Stepping Through

Before final testing of the instrument, it is necessary to "step through" the process of fielding, tabulating, and analysis of the data. Often features of the questionnaire make it easy or hard to undertake these steps. Preparing a questionnaire that is easy to execute, to enter the data, and to analyze as well as obtain the desired information is a difficult task.

At this point of the study, final approval of the questionnaire is sought from the initiator of the study.

12.5.3 Pre-testing

After a questionnaire is viewed as workable in respect to substance and form, it needs to be tested. Pre-testing is critical. This involves executing the questionnaire with a small sample usually less than 12 respondents.

The initial test may be done during in-depth interviews during the qualitative phase of the study, even if the survey will eventually be done by mail. This allows for testing the understandability of the instrument. However, final testing should be done under actual field conditions.

12.5.4 Fielding

After successful pre-testing, a final approval for the questionnaire should be sought of the initiator. The next step is the fielding the instrument. The time required for the study depends on the type of execution, the importance of high completion rates, and the nature of the study. Telephone survey can be completed in a manner of days for even large samples while mail surveys may take weeks and personal interviews may take months.

As previously noted, poor fielding can result in unreliable results. Fieldwork is usually careful monitored, particularly if phone interviews are being used.

12.5.5 Code and Tab

After fielding, the data needs to be coded into a computer for analysis and the results tabulated. Either this can be a simple task or fairly complex if open ended questions need to be coded. In any case, care needs to be taken that coding and tabulation has been carefully done.

12.5.6 Analysis

The extent of analysis varies widely among studies. In most cases, the analysis involves reviewing the frequencies of responses and cross comparisons between frequencies.

12.5.7 Reporting

Reports are either as full written management reports or as presentations or both. Surveys, and in particular internal surveys, are expensive and time consuming. If the survey is worth doing, it's worth fully analyzing and reporting the results.

12.6 Respondent Selection

The selection of the respondents is a key element for any successful survey. This involves both the identification of who qualifies but also of the size of the sample.

12.6.1 The Target Universe

One can not overly emphasize the importance of proper selection of the respondents. This is referred to as the Target Universe. It consists of the sub-population of people in the "know" from whom we wish to consider the "market". Depending on the nature of the survey, this may be the decision-makers, influencers, users, or management.

12.6.2 A Census

If the Target Universe is fairly small, the survey may involve all of it, or a census. Unfortunately, when that is done, the missing and refusal issue tends to become critical. Results of a census is often less reliable than using a sample because of these problems.

12.6.3 Sampling

Usually, a sampling approach is preferred over a census, at least, based on cost. This involves selecting a sub-population whose characteristics is representative of the total target universe. The sample should represent the total. Sample Size Error

As noted earlier the precision of the survey results usually depend on the sample size. The larger the size the smaller the expected error of the estimate of average values. We usually select the sample size based on tolerable errors around anticipated results. This requires us to return to the original purpose of the study and the decisions that need to be made. Sample and Population Size

Fortunately, the size of the target universe only influences the anticipated error for small populations. This is sometimes counter intuitive and initiators ask for a set fraction of the Target Universe to be covered by the sample. 100 responses or less often is adequate to sample populations of tens of thousands.

However, it should be noted that in internal studies, the target universe is often small and consideration of the total size may need to be considered. Deviation Error Bound

We can estimate the standard deviation around the average by assuming that deviations will be "Normally Distributed"

For a small target universe, N, the standard deviation around the average is:

                                sx = (s/ n) (N-n)/(N-1)

Where sx is the standard deviation of the average, n is the sample size, s is the standard deviation of the target universe. Note that if n is much smaller than N the relationship becomes a much simpler form for a large target universe:

sx = (s/ n)

For large target universes (in the order of a couple of thousand) the sample size can be calculated as:

n = (s/sx)2 Percentage Error Bound

In most cases, we don't compute an average score but the percentage of respondents that indicated a specific way. Under these conditions, we use a standard error estimate rather than the standard deviation. This is estimated assuming the error follows a binomial distribution.

For a small target universe, N, the standard error around the percentage of the sample is:

Sp = p (1-p)/n (N-n)/(N-1)

Where Sp is the standard error around the percent of the sample, p. Here again if n is much smaller than N the relationship becomes a much simpler form for a large target universe:

Sp = p (1-p)/n

Similarly to the above case the sample size for large target universes is then:

n = [p (1-p)/Sp]2

12.6.4 Selection

The selection of the sample can be critical. The objective of sampling is to obtain a group of respondents that represents the total population. In many cases, not all respondents are equally important. In these cases, selection is not random in this regard. Specific respondents are selected and handled as a separate group. Random Samples

To obtain a truly random sample every member of the Target Universe should have a equal chance of being selected. Often random number generators are used for the selection. Note that if refusals or the inability to locate a member of the sample is related to any characteristic of the respondents, the resulting sample will not be random.

A random sample is not necessarily a "good" representation of the population. There is a finite probability that any random selection will result in a sample that has no relationship to the population as a whole. For example, there is a finite probable that a random sample of students will turn up to contain only women students. Sample selection must be reviewed to verify that the sample is representative. Convenience Sample

In many cases, the sampling is done by convenience. For example, in "Mall Intercepts" consumers are stopped "randomly" at a mall and interviewed. The fact that the interception takes place in a specific location makes the sampling not strictly random. It should be noted again that our objective is to obtain a representative sample rather than a strictly random sample. If the sampling is effective in producing a representative sample, such techniques are acceptable. Stratified Samples

Often we wish to examine specific sub-populations. For example, we might wish to examine "power users" of a system and regular users. There is likely to be a large number of regular users but relatively smaller number of power users. If we sample as if dealing with a single group, we may only get a few power users. This would make the expected error around the power users very high. In order to accommodate this analysis, we sample the sub-populations separately. This is referred to as a stratified sampling.

It should be noted, however, that if a sample is stratified, that characteristics used for stratification can not be reported as population information. For example, if we stratify on power and regular users of a system, the fraction in the sample does not represent the fraction in the target universe. Quota Samples

In many cases, we can not identify who belongs to which sub-population without interviewing them. Under these conditions, we select the respondents during execution. A "quota sample" is used for this purpose where the interviewer screens respondents and selects based on a quota that is preset for each interviewer.

12.7 Execution

The execution of the questionnaire consists of all activities to get respondents to respond.

12.7.1 Modes of Execution

The methods of executing questions have been expanding with communications technology. This process is likely to continue. The selection of the mode of execution should be based on the nature of the study, the type of information needed, the effectiveness to get to the respondents and the likelihood of respondents completing the survey. Single Modes

Often more than one method is used in a survey. Several methods can be combined to take advantage of each one's benefits and to limited their individual deficiencies. Mail

Mailed questionnaires are among the widest used method of execution for MIS studies. Its advantages include: (1) fairly complex tasks can be handled in writing (as opposed to telephone), (2) distribution is fairly inexpensive, (3) coding can be well automated, and (4) good control of the execution.

Its disadvantages are: (1) usually there is a high or very high refusal rate, (2) only simple structures can be used (its not easy to include special questions targeted to groups of respondents), (3) its hard to control the order by which the questions are answered and (4) the respondents can not be screened. E-Mail

E-Mail and Fax are new modes of survey execution. Electronic communications have alleviates some of the difficulties of mail questions. We expect E-Mail should increase both the response rate and the speed of execution. Data entry can be automated using E-Mail and Fax, however, special care needs to be taken with automation of these modes. Phone

Telephone surveys are commonly used for marketing research both for consumer and industrial products in North America and now in Western Europe. However, it has not be used widely for MIS projects. Its advantages include: (1) getting to the correct respondent, (2) speed of execution, (3) until recently a high response rate, (4) the respondents can be screened, (5) control of the order of questions, and (6) the capability of complex structures. The introduction of CRT - Computer display prompted telephone interviewing systems allows very complex branching structures. This allows surveys to branch based on previously answered questions.

Its disadvantages include: (1) respondent fatigue can be high, (2) more expensive than mail, (3) only a small number of response choices can be given, (4) only simple (non-trade-off) responses can used, (5) no visual stimuli can be presented, and (6) potentially a reduced level of field control. Personal Interview

Personal interviews are the traditional method of execution outside North America and Western Europe. Until recently, it was the preferred method for executive surveys. However, other methods including teleconferencing have gain in use. Its advantages are: (1) lower fatigue rate among respondents, (2) use of complex visual and audio stimuli (video also), and (3) with trained interviewers, you can collect non-verbal information.

Its disadvantages include: (1) high cost, this is usually the most expensive mode of research unless the respondents are available at a location such as Malls or Airport Intercepts and Trade Shows, and (2) it is prone to interviewer bias. Group

Group techniques have become popular since the 1930's, particularly, the focus group procedures. However, there are a number of other procedures that may also be of value including: (1) creative problem solving workshops, and (2) the "Delphi" method of forecasting. These methods are generally among the most expensive of the modes of execution even if conducted electronically. The major disadvantage of the group techniques is that the resulting information may not be representative of the individuals or the decision making process. Multi-modes

Often multi-modes are used to take advantage of the single mode benefits and to eliminate their disadvantages. However, this usually comes with higher costs. Telephone and mail methods are highly compatible and are most often used together in marketing research. Phone-Mail

Phone-mail is usually used to both screen respondents and to improve the response rate. Respondents are first screened and then told that they will receive a questionnaire by mail. This is heavily used for narrowly targeted studies. Phone-Mail-Phone

Phone-mail-phone is a widely used method when visual information needs to be shown to the respondent. In this method, a screen phone interview is used to arrange a time for the main part of the survey. Information is then mailed to the respondent (often by express mail) and the interview conducted later. The format for this structure is principally a phone survey but with the ability to use many more opinions and complex stimuli. In general, the data is collected over the phone.

12.7.2 Fielding a Study

Control is the major issue in fielding studies. Many industrial firms insist that on using Full Service Suppliers in order to assure better control of fielding operations. It is always advisable to conduct some on-site monitoring of telephone interviews. All professional facilities have the ability to monitor the telephone interviewers. Some monitoring of personal interviewers is also called for. However, control in this mode is much poorer than for telephone surveys.

12.8 Questions

The centerpiece of survey research is, of course, the questions being asked. Their selection, design, and wording are critical for the success of the study.

12.8.1 Purpose

Questions that do not add value to the study should not appear. Every question increases the chances of respondent fatigue, non-response, irritation, and cost. Only questions that are needed should be included. Each question should be tested as to it use to the study. The same is true of the range of responses. If the response category will not be used it should be dropped. Use

The core of the questionnaire should be those questions that are needed to meet the objectives of the study. Mock Tables

A useful exercise is to prepare tables displaying what the data should look like. With such tables, it becomes more obvious what data will be useful and what is likely not to be. This technique, of developing mock tables, is also useful to clarify how the data will be analyzed after coding. Screening

Some questions are used to screen respondents. This is particular important for stratified samples. These qualifying questions should read directly on the characteristics of the target universe and the stratification structure. Verification

Some questions are intended to verify that the sample is representative of the Target Universe. These are often characteristics whose average values are known. Usually only a very few of these questions are necessary, if any. Identification

Some, demographic, information is often collected to identify the population by sub-group. These classifications should work with the direct use data to give operational results. If the demographic variable is not actionable, it should not be included. Surrogate Questions

Beware of surrogate questions. There is a tendency to ask for information in one area to imply characteristics in another. For example, we may ask about job title to imply social-economic characteristics. It is always far better to ask the question that you need information than to use surrogates. In most cases, the information desired can be obtained while surrogate information may not be meaningful.

12.8.2 Preparation

In preparing questions, we need to be careful in regards to wording and the type of acceptable response. Simulating the Decision/Use Process

An overall objective in preparing questions is that they are in a context of the decision that we wish to examine. Even simple word changes can change the meaning and conditions that will not reflect the issues being examined. Acceptable Language

Language and culture are intertwined. The words and phrases that we use reflect our culture, training, and industry. Unfortunately, in the construction of questionnaires, the respondents' background is likely to be different from the designers. This is particularly true in the area of MIS where jargon tends to be raised to an art form. Single Meanings

Terms in questions and responses should have only one recognized meaning. In marketing research, for example, using the term "income" may have different meanings and, therefore, the responses may not be comparable. In systems, development the term "report" refers to any computer output. The same term may have a widely different meaning to users and managers. Jargon

Jargon should be avoided wherever possible particularly if it is merely a combination of letters such as ASCII, or JCL. While these terms may be widely understood in the MIS community, they may be totally misunderstood by respondents. Understandable

Similar to jargon, phrases may not be understandable to users in the same way as to MIS organizations. Care must be taken to verify that the questions are understood by the respondent in the way they are meant to be. During testing, it is insufficient to merely ask the respondent if he understands a question. That understanding may be totally different than the intent. Biased Wording

Responses to questions can be biased by poor selection of words. Double negatives in questions tend to be confusing and lead to just the opposite result of the intent. Loaded Words

Loaded words are those which evoke an emotional response. Such words tend to generate a bias in responses to that question and other near it. For example, the question "Should we continue to support lazy welfare swindlers?" will not get the same response as a more neutral rendition of the question. Irritating Descriptions

Irritating descriptions or options can lead to refusal to complete the questionnaire. Race questions, for example, have in the past tended to raise this problem. At different periods in time, different terms were irritating. This is particularly a problem in international surveys where the instrument is translated into several languages. In some languages, the grammatical form may be the basis of irritation as well as misinterpretation. Single Question

Each question should have only a single response. Questions like, "Should we increase or decrease the property tax? is inappropriate since two questions are being asked: (1) to increase or (2) to decrease the tax. For the question to be clear, it should request an unambiguous response. Unique Response

Usually a multiple-choice option is given to respondents. It is important that these choices are not overlapping. For example, a question of "How old are you?" may have the following options:

o < 20 yrs. o 20 - 30 o 30 - 40 o > 40 yrs.

A respondent that is 30 years old can choose two options. Using separations of "20 - 29" and "30 - 40" will correct this problem. All Inclusive Responses

Options should cover all possibilities. Usually this is handled by using an option "Other". However, this may introduce a large uncertainty as to what this refers to. A large number of "Other" responses usually reflect an incomplete understanding of the respondents' options and probably inadequate qualitative research.

12.9 Questionnaire Design

Even the best of questions can be combined to produce a poor questionnaire. The questionnaire layout and design should reduce bias and encourage completion.

12.9.1 Question Ordering

The order of the questions can both effect the responses and the completion rate. Demographic Questions

Demographic questions tend to be boring and can be irritating. Respondents often do not wish to give out financial information for example. In general, unless the specific demographic questions will be used for the screen, they are usually handled at the end of the questionnaire. The only damage, is if the questionnaire is long, they may not get answered. It is important, therefore, to test the questionnaire to determine the likely missing demographic data rate and how this will effect analysis. Order Bias

In some questions the order of appearance of items and options can effect they evaluation. Significant research has been done to document this effect. To Rotate

One method of handling order bias is to rotate the questions, items or options. This needs to be done within sub-populations that will be compared during analysis. The major disadvantage of this procedure is the increased difficulty in coding. The coding process needs to reverse the rotation in order to get a consistent database. Professional data entry and CRT-data entry system provide this ability. Split Sample

Another approach is to split the sample, giving only different questions to different groups. In this case, only the composite results should be reported. This also reduces the precision of the study since only a sub-group sees each set. This is undertaken usually when one expects a severe order effect such as in the case new product evaluation.

12.9.2 Design Considerations

The layout of the questionnaire can influence completions. It is important that the instrument does not look excessively long or complex if the respondent is to see it. Typically, phone questionnaires (scripts) look much more complex than their corresponding mail questionnaire. Open Space

Using open space tends to make a questionnaire less imposing. Typically, we use fairly large margins and open space. The only exception, is the need to keep the questionnaire short. It is, therefore, a balance between shortness and open space. Font

A clear font should be used. Typically a Times Roman, Arial or Helv (Universal), Letter Gothic and Courier are used. The proportional block or Letter Gothic is recommended for clarity. Very small type, less than 10 point, should be avoided. Typically, 12 point type is used. Auto-coding

Questionnaires may be prepared with coding information on them. Typically, this involves small numbers to indicate how each response is the entered and other small numbers indicating the column. These auto-coding facilities are usually printed with the question (either in black or a second color). However, overlays may also be used, particularly if complex methodologies are to be employed.

12.9.3 Cover Letters and Introductions

The preparation of the cover letter and introduction can be critical. It at least informs the respondent of some reason why the questionnaire is being executed. In some cases, the cover letter can be signed by a person of sufficient prestige to influence the questionnaire's completion.

12.10 Measurement

There are additional considerations regarding measurement. What kind of scales and forms of the response effect the ease of answering, the amount of information, and the extent of the allowable analysis.

12.10.1 The Nature of the Scale

For most questions, the coded responses will be numbers. However, these numbers do not have the same meaning. We divide the meaning of numbers into four classes or scales. In some cases, responses may not fall clearly into a specific class but will take on some attributes of a couple. In some cases, we have choice in designing the questionnaire as to what scaling we want. These choices are balances of the difficulty of the respondent to answer the question and the necessity for analysis. Nominal (Categorical)

The most restrictive type of number or scale is Categorical or Nominal data. These numbers are only names of variables. For example, we might code a question with a 1 for NO and 2 for a YES. The fact that we have selected these values is totally arbitrary. The problem, of course, is that there is no meaning to an average or a median of this scale. While this is the most restrictive type of scale, it is the most widely used in questionnaires. Ordinal

Ordinal scales are a series of options, which increase or decrease consistently from top to bottom. For example, the classic rating from 0 to 10, is an ordinal scale. We know that 10 is better then 9 but we don't know if the difference between 9 and 10 is the same as the difference between 8 and 9. In this case again, averages are meaningless, but median and percentiles are useful. Interval

We could compute a meaningful average if the differences between values were all the same (or at least known). Such scales are interval in nature. Significant work is done with anchoring scales to make them as interval in nature as possible. Ratio (Cardinal)

With interval data, we can not say that a result is twice the size of another. Because there is no starting or reference point, one can not analyze for relative size. To capture relative size a natural zero must exist. Such data are referred to as Ratio or Cardinal scaled data. It should be noted that percentage responses on a questionnaire are ratio scaled.

12.10.2 Response Forms

The desired response to a question takes a number of forms. Open-Ended Response

Open-end responses require the respondent to fill in an answer. This is used both for numerical and qualitative information. It allows the maximum flexibility, but is a "bear" for coding and analysis. This data may be of any of the above types of scales but usually it is categorical in nature. Multiple Choice

Multiple choice is the preferred approach for large-scale analysis. This requires an all-inclusive set of options for the respondent. Single and multiple responses can be allowed. However, multiple responses are also more difficult to code and to analyze. This data is almost always consider categorical. Ranking

Rank order of items gives ordinal data. This type of response requires a comparison among options. This comparison can be a difficult task for respondents and to analyze but may give results that are more meaningful. Rating Scales

Rating scales are widely used and are well recognized. These are sometimes considered interval if well anchored. These scales are non-comparative and tend to have unique characteristics and biases. Respondents tend to group results toward the upper or lower ends of these scales. This limits the value of item comparisons. The other major problem is how to effectively anchor these scales. Semantic Differentials

A traditional way of anchoring rating scales is to use extreme words or phrase on the scale. For example, we may wish to rate a system as on a scale from "Easy to Use" to "Impossible to Use" with a couple of intermediate values. This forms a "Semantic Differential". The major problem is that some items do not have natural opposites let alone intermediate anchor values. Monopolar Differentials

Another alternative approach is the agree-disagree scale. This is monopolar in that there is only one concept being evaluated. This is usually preferred if there are no clear semantic differentials available for all items. Constant Sum

Another approach is to request the respondent to distribute points among a set of alternatives. This method combines the comparative advantages of ranking with the scale advantages of rating systems. Furthermore, the resulting data is ratio in nature. This allows direct comparisons of values. The major problem with constant sum is the same with ranking; it is a difficult task, which should not be applied to large numbers of items.

12.11 Coding

After completion of the fieldwork, the data needs to be entered and compiled into a database. Traditionally this activity had been divided into coding, data entry, and data verification. We now generally consider this in one step. However, the procedures for coding should be prepared before execution.

12.11.1 Code Assignments

All possible responses on the questionnaire should be assigned a specific code and data location. This is often on the questionnaire as part of the auto-code features or in the CRT system. Missing Data

Missing data produces unique problems in that there may be several reasons for missing data. Several codes may be used to differentiate simple refusal, termination, and inappropriate questions.

12.11.2 Open Ended Questions

Coding open-ended questions is particularly difficult. Usually the coding is selected after the fielding of the study and based on a sub-sample of responses. Coding these questions is labor intensive and somewhat imprecise.

12.12 Analysis

Any study that was worth undertaking is worth analyzing. Unfortunately, too many studies not analyzed adequately.

12.12.1 Tabulation

The most basic and useful form of survey data analysis is tabulation. This involves compiling the types of response by question and between questions. Tabulations are usually done by field and "Full" Service suppliers. Most coding packages also generate tabulations or tables. Frequencies

Most tabulations are the simple listings of responses for each question for the total populations. Often sub-populations are also examined. Groups of customers are compared such as users and non-users of a system. Banners

These frequencies are often combined in to larger tables. The questions and responses to be combined are referred to as a banner since they represent title row on the table. When tabulations are requested, the banner structure and the breakdown need to be specified. Cross Tables

In a similar way cross tables, comparing the results of specific questions can also be generated. These are similar to the simple frequency and banners, however they usually are handled as only two specific questions in the questionnaire.

The number of cross tables that can be generated is enormous. If M is the number of questions, one can have M(M-1)/2 cross tables. For example, with only 20 questions, one can get 190 cross tables generated. This can be an expensive process. In general, only those cross table necessary for analysis should be generated.