1. Be very nice to the data people. Let them help you without
dominating the process.
2. Use sampling, not 100% reporting to collect new data
3. Don't burden workers by collecting data you don't need. Use the 4 quadrants
to see what's most important.
4. Design data collection from the final report backwards.
Advice from:
Organizational Resources
1. The Evaluation Forum has created a useful guide to
data gathering methods at www.evaluationforum.com
References
The Short Answer
1. Start with the data you already have. Most programs collect
far more data than they use. Do not start the process by collecting new data.
2. Create new data where necessary. Be creative. Use common
sense sampling techniques. Some data is better than no data.
Full Answer
Existing sources and creation of new data (link to data sources)
(1)
Program
people will often say, "We don't have any good data." And sometimes
they're right. But more often than not programs collect a great deal of data.
The problem is not the availability of data but the fact that it is not used
for anything. Programs collect far more data than they use.
An interesting
corollary here is that programs enter more data into management information
systems than they use. Another use of the 4 quadrants is to evaluate requests for data elements in
MIS design to focus on those that are most important and most likely to be used.
(2)
So
the first place you look is at the data that's already available. The technique
described in the selection process in step 4 identifies the
data elements in the upper right and lower right quadrants for which there is
currently good data.
(3)
Where
data does not exist, it is possible to create new data. And the most important
thing to remember here is that this does not have to involve 100% reporting. It
is possible to gather important performance data using sampling techniques. And
these processes can be put in place quickly - in some cases in a matter of days,
not years.
(4)
Story: Let
me tell you a story about a community mental health office in a small state. I
went to visit these people last year. And the conversation more or less started
like this: "We don't have any data. We can't do this." I asked, "If
we took 10% of your caseload every month and asked just two questions, do you
think you could do that?" They thought they could. And we fashioned two
customer satisfaction questions, one in the upper right and one in the lower
right quadrants, in the most simple, plain language we could think of:
"Did we treat you well?" (a proxy for courtesy, timeliness and cultural
competence etc.)
"Did we help you with your problems?" (a proxy for making a difference in their
life or the life of their children.)
12
words total!!
(5)
The
difference between one time and ongoing data collection. This is one of the most
important things to think about with regard to evaluation. Often evaluations are
thought of and structured as one time events. Partly this is because evaluations
involve intensive data collection and while you're at it you might as well
collect a lot of data, often too much data, so much that they burden the
program. Performance measurement data to be useful must be collected on a
regular basis. This does not mean 100% reporting or even continuous reporting.
It could be a sample every 3 months
For more information on data
collection methodologies see the work of The Evaluation Forum www.evaluationforum.com
