p58 a DesignSurvey

Some Notes on Designing Research Survey Forms

Michael Jaffarian

MichaelDawna@earthlink.net

Missionary Researcher, WorldVenture

Senior Research Associate, Operation World

Published in Global Missiology, Research Methodologies, April 2006, www.globalmissiology.org

A Christian researcher in Europe asked me about how to design a survey form. I offered these general comments and pointers:

1. Remember that survey design is as much an art as a science. A survey form can be technically correct but still difficult for respondents to make sense of, or for other reasons not get good data. Yes, ask a professional sociologist or the like to evaluate your data-gathering instrument, but don't leave it at that. Also get feedback from ordinary people, including from likely respondents, and from those whom you hope will read the report. Get help from people who have plain old common sense. Get help from people whom you know to be good, effective communicators, in speaking or writing. Since this task is as much an art as a science, get help from 'artists' as well as 'scientists'.

2. It is important is to know well your survey respondents and how they think. What categories and terminology do they themselves know, understand, and use? What words and ideas guide their own thinking? The wording has to make sense to the respondents. Also, the form must be written in a way that gets to the exact questions you are trying to answer in the research. This can be tricky.

3. Think about the final report and the kinds of statements you hope to make there. Here you need to think not only about the survey respondents, but about the readers of your report. For example, in your report you may want to say something like, "x% of this group have not shared their faith with anyone in the past 5 years". (Well, for starters, you'll have to say, 'BY THEIR OWN ADMISSION x% have not ....', or 'x% SAY they have not ....'). On your survey form, you might ask, "How many people have you shared your faith with in the past 5 years?" But then - think about it. Your likely respondents - what would they understand by the phrase "share your faith"? Would it be the same understanding that you carry about it? Would it be the same understanding that your readers would carry about it?

4. Also very important: a good question is one where respondents, when reading it, will know how to answer. A bad question is one where respondents, even just many, when reading it, will say, "Gosh, I have no idea," or "Wow, I really don't know how to answer that."

5. Remember that there might actually be no realistic way to answer the research questions you would like answered. Be honest about this. For example: you may want to know how many people in France are truly saved; that is, if the world ended tonight how many would be in heaven tomorrow? Well, good luck coming up with a question that will give you a confident answer on that one - no matter how badly you wish you knew the answer! Only Jesus knows the answer to that one. Another example: you might survey a group of pastors and ask, "How many of your people have daily devotions?" It may be (a) that these pastors just don't know, and most of them will reply by saying, "I have no idea", or (b) that these pastors think they know but are way off the mark (which could be tested by surveying all the members of some sample congregations, if you had the resources to do that, and comparing it to their pastors' answers). Again, no matter how much you may wish you could get a good answer to this question, maybe you just can't. So do the research and find out what you can, but be humble and realistic enough to admit what you cannot.

6. Know the standard questions and methodologies in your field. If you are doing a survey of a denomination, find out about other denominational surveys in your country or region, or, better yet, all over the world, so the data from your denomination can be compared to the data from other denominations. This can be extremely valuable and much more can be learned. For myself, as a missionary researcher with a mission agency in the USA, I would like to ask our missionaries the same questions that other American mission agencies are asking their missionaries. The North American Mission Handbook questions are important ones to use in my case, since that is the definitive, on-going research report in this field.

7. To put it another way: do your research in such a way that your data can feed into regional or global research. Think about the data needed for Operation World, or the World Christian Database, or the Joshua Project peoples list, or the Ethnologue, or other global research. Have a Kingdom perspective. Don’t just serve your own immediate needs if you can also, at the same time, serve the global Body of Christ.

8. Write questions that can be used for many years to come, in a wide range of situations. This is not always easy to predict, but it is certainly something worth thinking about. Ask yourself, "Am I phrasing this in just the jargon of the day, or is this a question likely to be relevant for a long time?" Ask old people, and young people, what they think of your questions, and how they understand them. In the USA, major religious researchers, like George Barna and George Gallup, have been asking some of the same questions for many years, with the same wording. So also with the Mission Handbook. There are some modifications along the way, of course, as things change. The Mission Handbook, for example, didn't used to ask questions about tentmakers or even about short-termers, but they do now. Still, consider that the Gallup organization has been surveying the national population since the 1940s (at least), and asking some of the same questions all these years. That provides extremely valuable data, tracking changes in religious beliefs, attitudes, and practices in the country over these years and decades.

9. On that, let me remind you that the best research keeps at it for many years, even decades, continually doing later editions, thus tracking changes and trends, as well as continuing to improve the research year by year. Think long-term. This requires researchers that accept this ministry as a life-long vocation.