Building Surveys

Once you determine how you want to collect data, the next step is to develop the tools you will use to do it. This section describes how to develop surveys, the most common method for collecting data. Make sure you develop all tools before you start collecting any data, especially if your plan includes both pre- and post-program data collection activities.

Choosing a Survey Tool

There are many survey tools that help you collect—and even analyze—your data. If you are planning to collect large amounts of data in regular intervals, it may be wise to invest in a paid tool, which usually offers more functionality than free options. Consider these questions:

  • How much money do you have to invest in a tool?
  • How often will you collect data?
  • How many people will receive your surveys over time?
  • Do you need data storage capabilities?
  • What kind of analytical capabilities do you need?

Most surveys will be simple enough for free or moderately priced survey tools. Free versions typically impose limitations on the number of questions and responses, as well as survey functionality. Most survey tools provide only basic reporting, so always make sure the tool you choose allows you to export all of your data in an easy-to-use format. Many survey tools, like the ones listed below, also integrate with database and contact relationship management (CRM) systems.

For more advanced survey functionality, you may need to invest in a premium version of one of the tools above or in a tool like Qualtrics Research Suite.

Many tools offer discounts for nonprofits, so make sure to talk to a representative before purchasing!

Developing Your Survey

Once you have selected your tool, you should:

  • Define your questions
  • Choose the question type
  • Choose the measurement scale
  • Review and refine the survey

(1) Define your questions

The first step is to decide which questions to ask. All of your survey questions should help you to answer your organization’s guiding questions. For example, if you are interested in how satisfaction with a program varies depending on an age cohort, you should make sure to ask survey respondents to input their ages.

You can create your own questions from scratch or you can start with questions that have been developed by others:

Take a look at this survey-design guide, which includes a comprehensive glossary of terms.

(2) Choose the question type

For most organizations, a good survey has a mix of open- and closed-ended questions to obtain both quantitative and qualitative data.

Closed-Ended Questions

This type of question requires the individual to select a finite set of responses, such as yes/no or a 1 to 5 scale. Closed-ended questions provide you with quantitative data to which you can apply simple and complex quantitative analysis.

Open-Ended Questions

Also called free response, these questions ask individuals to respond in their own words. These questions provide qualitative data—the stories and context you need to get a well-rounded view of your programming. However, there are many ways to analyze qualitative data by transforming the responses into quantitative data through rubrics and other formats. For example, you could count the number of times people used positive (awesome!), neutral (ok) and negative (awful) descriptions of an experience to measure satisfaction.

While quantitative data is easier to analyze, qualitative data provides important context for your organization’s story. Storytelling with data is about marrying the quantitative and qualitative to form a complete picture of the impact your organization’s work is making in the lives of others.

(3) Choose the measurement scale

When collecting quantitative data, you need to choose a measurement scale. Ideally, you use the same measurement scale throughout your survey and over time so you can compare results within and between groups.

The most common type of measurement scale is a Likert scale. Characteristics of a Likert scale include:

  • An ordered continuum of responses (e.g., strongly disagree to strongly agree)
  • A balanced number of positive and negative response options
  • Numeric values assigned to each response (although those numeric values may or may not appear to respondents)

If you have taken a survey before, you may have noticed that some scales include an odd number of choices, while others include an even number of choices. The original Likert scale used 5 response options, including a neutral option (e.g., neither agree nor disagree). There is much debate in the research world about whether a Likert scale with a neutral option is better or yields more valid results than one without, but there are clear advantages to both.

Even Scale (no neutral option) Odd Scale (neutral option)

With no neutral option, individuals are likely to be more thoughtful about their responses, as there is no “easy out”

With a perceived “easy out,” individuals may be less likely to skip a question, especially if the topic is sensitive

If you are concerned that you will not receive accurate information if individuals perceive they must make a positive or negative selection—when in reality they have no opinion or feel neutral—use an odd scale. If you think your respondents can be thoughtful and deliberate in choosing positive or negative answers, use an even scale.

Likert scales are particularly useful if you have a set of statements that individuals can respond to using the same scale. A matrix approach makes good use of survey real estate.

View an example of a matrix survey question
Example

How did you feel about each of the following program elements?



Tips for creating an easy user experience:
  • Use the same rating scale for as many of your questions as you can
  • Try and keep language (e.g., tense) consistent. If you are asking about past, present and future experiences or attitudes, arrange the survey in chronological order
  • Use simple language. The harder it is for a user to understand the question, the more likely they will fail to complete the survey
  • Test your survey with a small focus group to make sure your questions are being interpreted the way you intend

(4) Test and refine

Once you have developed your survey, review it using these questions to make sure the survey will achieve your goals:

  • Will my questions give me the data I need?
  • Are my questions clear and easy to understand?
  • Does my survey flow in a logical order?
  • Is the time to complete the survey appropriate for the audience?