How to... design a survey
A survey is a very commonly used method in business research (both academic and commercial) which gathers data from people (note, other disciplines also collect non-human data) using a (generally) structured instrument in the form of a questionnaire.
Here are some definitions of surveys:
You carry out a survey in order to establish people's views of what they think, believe, value or feel, in order to ...support an argument..., sampling a population of potential respondents in order to generalize conclusions more widely.
A.D. Jankowicz, Business Research Methods, 2000
...the word "survey" is used most often to describe a method of gathering information from a sample of individuals. This "sample" is usually just a fraction of the population being studied.
American Statistical Society, What is a Survey?
The broad area of survey research encompasses any measurement procedures that involve asking questions of respondents.
William Trochim, Research Methods Knowledge Base,
Some basic facts
A survey is the most used form of descriptive research, and is concerned with present phenomena (as opposed to historical research, which is concerned with past events).
"A survey or questionnaire will be at its best when getting a snapshot of the current state of affairs in a given group or population, what researchers call descriptive work."
Joseph Janes, Survey research design,
Library Hi Tech, Vol. 19 No. 4
It is particularly useful for collecting a large amount of data, but it must be based on a sample which has been scientifically chosen to represent the larger population.
What types of situations lend themselves to surveys?
A great many! Here are some:
- Determine people's views or explore attitudes – e.g. to a new service, or attitudes of employees – both examining the relative frequency with which a view is held, and also exploring different points of view:
In Coming to the table with Acas: from conflict to co-operation (Employee Relations, Vol. 26 No. 5), Gill Dix and Sarah Oxenbridge explore attitudes to the British conflict resolution agency Acas and in particular perceptions of it in its role as helping employee performance generally.
A particularly common type is the user survey, which is explored in User survey at Waterford Institute of Technology Libraries: how a traditional approach to surveys can inform library service delivery (Helen Hayden, Terry O'Brien, and Maoilíosa Ó Rathaille, New Library World, Vol. 106 No. 1).
- conduct an exploratory study, prior to doing further research
- do a a confirmatory study, testing a concept
- investigate a phenomenon:
In The Influence of External Pressure Groups on Corporate Social Disclosure: Some Empirical Evidence (Accounting, Auditing & Accountability Journal, Vol. 7 No. 4), Carol Ann Tilt conducts a survey into the relationship between corporate disclosure and pressure groups.
- investigate behaviour – e.g. frequency with which a particular item is bought, or shopping centre is visited
- investigate the potential market for a new product or determine attitudes towards an existing product
- explore perceptions of, for example, a company brand
- obtain facts – spending patterns, transport habits, buying habits, number of days off sick, etc.
Surveys can explore:
- both the frequency with which an attitude, event, etc. occurs and different attitudes, perspectives, etc.
- issues both at a particular point in time (cross sectional) and over a particular time period (longitudinal):
Anthony J. Onwuegbuzie, Qun G. Jiao and Sharon L. Bostick conducted a cross sectional study to determine the proportion of students with debilitating levels of library anxiety and a longitudinal study to monitor library anxiety over time. (Library Anxiety: Theory, Research and Application, The Scarecrow Press Inc., 2004, ISBN 0-810-84955-0).
- phenomena that are both fixed, for example attitudes at a particular point in time as in the example above, and dynamic, for example to do with an organization that is in the process of change, for example a supply chain that is changing as a result of use of the Internet.
Advantages and disadvantages of surveys
|Good at gathering a large amount of data from different sources||Unless a survey is administered by a researcher, non-response can be a big problem and you may end up with an unrepresentative response rate|
|Can be used to access a wide range of information, such as attitudes, beliefs, values, and past behaviours||You only get answers to the questions that you ask|
|Easy to develop a standard instrument, thus avoiding errors||You depend on respondents having the motivation to respond and responding honestly, that is, either giving responses they think you want to hear, or show them in a good light, or simply failing to remember the right information|
|(Generally) fairly easy to administer a survey – a lot of the work is done up front, the administrative work is usually routine and repetitive||It is not good for affective issues, or for exploring issues in depth, because the questions have to be so focused|
|Possible to ask very specific, focused questions, and to home in on very specific areas||It is not good for complex social phenomenon, because of the above|
|There is a certain amount of self-selection in the respondents (some may choose not to respond) which means that there is an element of non-probability in the sample|
Factors that increase the chances of success
The big disadvantages of surveys as described above are:
- they are inappropriate for some issues which are complex and affective; and
- response rate may be poor (different people have different ideas on what constitutes a poor response rate, but lower than 50 per cent for a consumer survey is not considered good).
You can avoid the former drawback by developing a particular type of instrument (the semi-structured interview, which will be explored in the next section), and the latter by incentivizing people to fill in the questionnaire. The following factors will increase the success rate of your survey and therefore your research.
- Ensure that your measures are appropriate, which means that you need to ask yourself very carefully what it is you are measuring and how, and whether you are doing it in the most appropriate fashion. For example, if your research population is relatively small, and your research question relatively complex, you might be better off conducting an interview rather than getting people to fill in a questionnaire.
- Increase your response rate by motivating people to fill in the survey:
- KISS – keep it short and simple.
- Make the job of filling it in easy, use good presentation with tick boxes etc., and provide a pre-paid envelope.
- Make the questions closed (i.e. with a set number of responses) rather than open (i.e. the respondent has to decide what to say).
- Trial the survey, to see how easy it is to fill in.
- Provide an incentive:
- in the case of a commercial survey, it may be possible to provide some payment, or some giveaway, such as a pen or coupon;
- in the case of a non-commercial survey, say for what purpose, e.g. part of an academic project, research degree.
- Address any concerns that the respondents may have – privacy etc. – in a covering letter.
- Follow up non-respondents – send out further letters:
In Leadership style and market orientation: an empirical study (European Journal of Marketing, Vol. 35 No. 5/6), Lloyd C. Harris and Emmanuel Ogbonna examine the role of leadership in market orientation, and describe the measures they took to ensure that their survey achieved a high response rate.
In Mail survey response behavior: A conceptualization of motivating factors and an empirical study (European Journal of Marketing, Vol. 32 No. 11/12), S. Tamer Cavusgil and Lisa A. Elvey-Kirk look at response rates.
Different methods and instruments
The basic definition of surveys is that they involve asking questions and collecting data from people. The commonest method for doing this is by a series of highly structured questions, with responses pre-selected. There are two main methods of data collection in surveys:
- structured techniques
- semi-structured techniques.
And a number of ways in which surveys can be administered.
As their name suggests, these arrange for data to be collected in a highly structured way. All the variables will have been identified, and the interview structure, and the way the data will be coded, will be predefined. The advantage of this sort of data is that they are very easy to code and analyse, because the type of answers which the respondent can give is set out in advance.
The instrument (the means whereby the data are collected) is highly structured and specific, and generally comprises either:
- a written questionnaire; or
- a verbal interview, in which case the interviewer becomes part of the instrument along with the interview schedule, although the high degree of structure maintains the objectivity.
The type of questions are generally closed-ended, meaning there are a set number of responses. Open-ended questions require the user to supply the response.
Questionnaires are said to be self-administered when they are filled in by the respondent on his or her own, and researcher administered when administered by a researcher in the form of an interview. The following are some advantages of both types according to Wikipedia:
|Cheaper||Can clarify questions|
|Don't require interviewers||Ensure completion of questionnaire|
|Can be used for large numbers||Higher response rate|
|Avoid interviewer bias||Greater control of environment|
|Quick and easy to code and analyse|
These are a highly effective method of data collection, in that they require less time to administer and are therefore less expensive, and permit data collection from a larger sample. The advice generally given is to use closed-ended questions, which are easier to code, store and analyse.
Here, the interviewer has a number of structured questions, which he or she goes through with the respondent. The responses are either articulated by the interviewer, or else are present in the coding structure which the interviewer fills in as the respondent talks. The advantage is that the interviewer can develop a rapport with the respondent, which increases the propensity for an honest, and accurate response, and can clarify any misunderstood questions.
In Survey research design by Joseph Janes (Library Hi Tech, vol. 29 no. 4), Janes has this to say about interviews:
"Most authors agree that the face-to-face interview method can get you the best, highest-quality data. You can ask more questions, and more specific questions. You can know when people are in trouble and not understanding you, and get a higher response and completion rate. You can also decrease the number of 'don't know/no opinion' type responses. With a trained interviewer, you can also get non-verbal behaviors, but that takes work. A face-to-face interview should last about 15-25 minutes; anything over about 45 minutes gets tedious and people can become restive. Interviewers need to be trained to be familiar with the questions, and to ask them as neutrally and consistently as possible, in order to eliminate potential sources of bias and noise in responses. In fact, in some situations you may want to use many interviewers of different ages, genders, races, or backgrounds, to mix up any such possible sources of error. Responses should be recorded exactly (or even tape or video recorded, though this has to be agreed to by subjects in advance). It is an expensive proposition, but sometimes the benefits are worth the effort."
In Scenario planning: strategic interviews and conversations (foresight, Vol. 4 No. 1), John Ratcliffe describes both structured and unstructured interviews in his research on the role of conversation in planning strategy.
These arrange for the collection of data which do not necessarily correspond to a pattern. The advantage is that because data do not need to correspond to pre-set variables, it is easier to explore dynamic and changing situations.
Here, the interviewer has a number of questions on the schedule, but can depart from these as appropriate should other relevant issues crop up. There is no set pattern for responses and the respondent is free to respond as he or she sees fit, while the interviewer is at liberty to probe as interesting new issues emerge.
Whereas with structured techniques, all variables are known in advance, with semi-structured techniques it is possible to start with an incomplete knowledge of variables which is a good way of obtaining more in-depth data and further exploring a particular situation.
Are women better at organisational learning? An SME perspective (Lynn Martin, Women in Management Review, Vol. 16 No. 6) uses semi-structured interview techniques.
By mail – self-administered
This is an excellent way of sending the same instrument to a large number of people, but the response rate may be low as you are relying on people having the time and the motivation to fill it in. In addition, you are dependent on having identified all your variables, and it is not the best way of handling data about a complex situation.
Survey research design (Joseph Janes, Library Hi Tech, Vol. 29 No. 4), has a section on self-administered mail surveys and the main issues to be aware of.
The reliability of mail surveys is reported on in Mail survey reliability through follow-up mailings: the case of auditor changes (Kimberly A. Dunn and H. Fenwick Huss, Managerial Auditing Journal, Vol. 19 No. 8).
Here, people are gathered into a group and handed the instrument which they fill in in situ. This allows for collection and administration at one point in time, but it needs to be set up. An example would be the end-of semester student satisfaction questionnaire, where students are often requested to fill in a questionnaire as part of a lecture, as a way of ensuring response.
Personally administered, in the mall/street or at home
Here interviews are conducted with people who are intercepted in the street or visited at home. This ensures a reasonable response rate, but the cost is high and people are likely to walk quickly past if they are in a hurry! An advantage is the ease with which you are able to obtain a response to something that requires the senses – sight, touch, smell.
The advantage is that these can be cheaper than the mall/street interview, but the big drawback is that most of us resent the interruption of being phoned up for a "market research" interview when we are having a meal, working, watching television, etc., and telemarketing has given telephone interviews a bad name. In addition, it can be difficult to take people through multiple response items over the phone as there is a danger that the first item is forgotten by the time the last one is reached.
Online or e-mail
The use of these has been growing, and they are very quick and cheap to deliver, particularly online interactive forms. The drawbacks are: computer glitches, hostile attitude to spam (arguably we are more likely to delete spam than we are to fail to open unsolicited mail), and (although with the huge growth in Internet access this is becoming less of one) you limit your population to those with Internet access.
In A comparison of online and postal data collection methods in marketing research, (Marketing Intelligence & Planning, Vol. 21 No. 2), Heath McDonald and Stewart Adam report the results of their studies into the relative effectiveness of online and mail surveys.
In Developing automated e-survey and control tools: an application in industrial management (Eusebio Scornavacca Jr, Joao Luiz Becker and Stuart J. Barnes, Industrial Management & Data Systems, Vol. 104 No. 3), the authors look at a particular example of an e-survey tool.
The following screenshot shows an example of an online survey, and is a particularly good example of one that is very easy to fill in:
Different ways of administering surveys are explored in User survey at Waterford Institute of Technology Libraries: how a traditional approach to surveys can inform library service delivery (Helen Hayden, Terry O'Brien and Maoilíosa Ó Rathaille, New Library World, Vol. 106 No. 1).
The first issue to be concerned with when planning a survey is, what do I need to do to answer my research question? How will the instrument which I develop answer the needs of the basic question(s) I am trying to answer? And what is the population that I need to obtain the data from?
As a related issue, you need to decide if you wish to test a related theory, and hypothesis.
From this basic issue arise a whole range of other issues:
- Who has done a similar or related survey?
- From whom do I need to get the data – what is my population and how should I sample it?
- What is the nature of the data that I need to obtain, and what should the mode of data collection be?
- Will there be problems in obtaining the data from the respondents?
- How much will it cost and how long will it take?
Who has done a similar or related survey?
As in any research exercise, you need to start off doing a literature survey. This will help uncover whether anyone has done exactly the same thing, or something similar, in which case you may be able to borrow ideas relating to their sampling technique, questionnaire design, method of analysis, etc.
From whom do I need to get the data?
The key here is to locate all members of the population you are researching and ensure that they have an equal chance of being covered.
Who is the research population? Over what geographical area is it located? You should try and be as accurate as possible about whom you target – perhaps addressing the key spender in the household, or if you are looking at a firm, finding the likely title of the person whom you will be targetting.
What sample (i.e. the actual group, the subset) of the population will you be targetting? How large should this sample be to be representative? This decision will also partly depend on how easy people are going to be to locate, and what the response rate is likely to be (the lower the response rate is likely to be, the bigger the sample you should try and obtain), but also on the types of statistical tests that you will want to perform.
Generally speaking, you will be better off getting a probability sample, that is, every member of the population has an equal chance of being chosen.
What is your sampling frame (the place where you will identify and locate your sample, e.g. a directory of members of a marketing association)?
In an article, Organisational culture in the public sector: evidence from six organisations (International Journal of Public Sector Management, Vol. 13 No. 2), Rachel Parker and Lisa Bradley surveyed the public sector in Queensland and adopted the following approach to sampling:
"A mailing list was developed from the Public Sector Directory – executive level, which was accessed over the Internet. This is a list of Queensland public sector employees with managerial responsibilities, listed by their department. The researchers chose six of the largest departments that as a group carried out a range of activities undertaken in the public sector including central co-ordination, infrastructure provision and social services. A sampling frame of 530 names was compiled from the six departments, which included all executive level names listed in this directory."
In an article, Product-market positioning and prospector strategy: an analysis of strategic patterns from the resource-based perspective (Robert E. Morgan, Carolyn A. Strong and Tony McGuinness, European Journal of Marketing, Vol. 37 No. 10), the authors use a particular sampling frame in order to exclude smaller firms, and within the sample targetted a particular individual. For full details, look under the heading "Research method".
What is the nature of the data I need to obtain?
- Can I identify all the variables at the outset, in such a way that I can obtain all the information by asking closed-ended questions?
- Or, are the data less easy to capture and would it be more appropriate to ask open-ended questions?
If you are attempting to survey a large population, then a written questionnaire is probably more appropriate, particularly if the population is dispersed over a wide geographical area. On the other hand, if you are wanting to explore more in depth issues, and you have a smaller population, then an oral, interview survey would be better.
All these questions will help determine your choice of research instrument, as in the example below:
In Marketing and company performance of Chinese small firms in Mainland China: a preliminary study (Journal of Small Business and Enterprise Development, Vol. 7 No. 2), Wai-sum Siu uses the interview technique despite what they perceived as its drawbacks, because this technique had been shown to be effective in Mainland China. A questionnaire instrument was devised and administered by a team of professors in Mainland China.
Will there be problems in obtaining the data from the respondents?
Will there be a problem of bias amongst the respondents – for example, will they tell you what they think you want to hear, will they be afraid to answer questions on certain issues, will they just not tell the truth?
Nearly every opinion poll in the run up to the 1992 UK general election failed to predict the eventual victory by the Conservative Party. A subsequent investigation found some technical reasons for this, but considered that just as important was the reluctance of many respondents to admit to voting Conservative on the grounds of fear of tax increases under Labour, not wishing to appear motivated by self-interest.
Will respondents just simply fail to respond, thus creating a small sample with an inbuilt bias?
A survey was conducted of non-UK students studying on a UK-based distance learning degree. Two-thousand students studying in various centres were sent a questionnaire. The response rate was in the region of 10 per cent, and while the responses were positive, ability to deduce anything from this is affected by the low response – were people only motivated to respond if they had something positive to say?
Will the respondent actually be who it says?
This is particularly important in surveys in companies, where you may have deliberately targetted e.g. the marketing director, who then passes it off to a more junior colleague.
How much will it cost and how long will it take?
Budgeting and scheduling your survey is of fundamental importance. Your departure point as always should be your research question, but you should also consider the actual time you have available, which may be framed by academic deadlines, or by your need to find something out in time to realize a business project. Likewise, you may be working to a limited budget. There will always be a trade-off between the research question and the available resources, and the former needs to be revised if the latter are not sufficient to produce the necessary data.
To determine how long the survey will take, you need to break it down into its stages, as in the following example of a questionnaire schedule (note, some stages, notably 4 and 5, can be done simultaneously):
- Decide the research questions that need to be answered, and the major objectives.
- Read around to see what other surveys have been done.
- Decide the most appropriate method, and the what instrument.
- Determine the research population and the sample and locate the sampling frame.
- Prepare the questions.
- Design the questionnaire.
- Trial with a small sample.
- Finalize the questions.
- Prepare and duplicate the instrument.
- Send out the questionnaire.
- Response time – allow sufficient time, but not too long.
- Send out reminders.
- Collect in responses.
- Transfer responses to a computer file.
- Analyse responses.
- Produce report.
The following need to be taken into consideration:
- Staff time for planning.
- Cost of directories etc. for sampling frame.
- Pre-testing the questionnaire.
- Hiring interviewers.
- Designing and printing the questionnaire.
- Cleaning the data so that they can be put into an Excel file.
- Computer analysis.
Some articles from Emerald journals
There is a wealth of information on surveys from the journal articles in the Emerald Fulltext database, as well as many examples of research using surveys. Some examples of the latter are cited in other parts of the section to illustrate particular types or uses of surveys. Below is a list with links to some of the many article which discuss surveys as a methodology.
Articles looking at surveys from a particular discipline standpoint
Survey research in operations management: a process-based perspective
International Journal of Operations & Production Management, Vol. 22 No. 2
Provides a brief scholarly overview.
A review of survey research in knowledge management: 1997-2001
Daniele Chauvel and Charles Despres
Journal of Knowledge Management, Vol. 6 No. 3
Survey methodology issues in manufacturing strategy and practice research
Robert S. Collins and Carlos Cordon
International Journal of Operations & Production Management, Vol. 17 No. 7
Patient Satisfaction Studies: Methodology, Management and Consumer Evaluation
Claire Batchelor, David J. Owens, Martin Read and Michael Bloor
International Journal of Health Care Quality Assurance, Vol. 7 No. 7
Survey research design
Library Hi Tech, Vol. 29 No. 4
Library Hi Tech, Vol. 17 No. 3
Emphasis is on the design of the actual questionnaire.
Different survey methods
A comparison of online and postal data collection methods in marketing research
Heath McDonald and Stewart Adam
Marketing Intelligence & Planning, Vol. 21 No. 2
Mail survey reliability through follow-up mailings: the case of auditor changes
Kimberly A. Dunn and H. Fenwick Huss
Managerial Auditing Journal, Vol. 19 No. 8
User survey at Waterford Institute of Technology Libraries: How a traditional approach to surveys can inform library service delivery
Helen Hayden, Terry O'Brien, and Maoilíosa Ó Rathaille
New Library World, Vol. 106 No. 1
Mail survey response behavior: A conceptualization of motivating factors and an empirical study
S. Tamer Cavusgil and Lisa A. Elvey-Kirk
European Journal of Marketing, Vol. 32 No. 11/12
Developing automated e-survey and control tools: an application in industrial management
Eusebio Scornavacca Jr, Joao Luiz Becker and Stuart J. Barnes
Industrial Management & Data Systems, Vol. 104 No. 3
Some online resources
There is a great deal of information on surveys on the Web, much of it linked with commercial use. Some of the better sites are listed below.
American Statistical Organization (Survey Research Method Section)
An excellent summary, easy to read.
Wikipedia on surveys
Simple easy to digest overview.
A Canadian publication on the latest survey techniques.
World Advertising Research Centre
Use it to search articles on surveys.
A new agenda for interdisciplinary research methods
From the Center for Disease Control and Prevention, National Center for Health Statistics (PDF file)
From the Food and Nutrition Technical Assistance Project, Washington DC. (PDF file)
Introduction to Survey Research Design
From the Survey Research Laboratory, University of Illinois at Chicago
Is it safe to combine methodologies in survey research?
Article on Mori's website
University of Plymouth Department of Psychology skills package – how to put questionnaires on the Internet
A useful site looking at how to set up a web questionnaire, including the coding.
Public opinion polls
International Conference on Questionnaire Development, Evaluation and Testing Methods
Site hosts some papers on questionnaire development.