Home    Index of Questions    Index of Tools    Index of Topics    Case Studies
Search   Active Schematic   Complete Guide   Resources   Comments

2.9	What do we do if we don't have any good data at all?

---

Tools 1.  Exercise: The Whole Distance from Results to What Works 2. Story Analysis Exercise for Results-Based Accountability   3. How to create working baselines to drive the what works thinking process. 4. Creating Baselines from Group Knowledge and Consensus 5. Prototype progress assessment format 6. Data Resources and Tools

Stories

Tips 1. Remember to focus on the results. Indicators are just tools to assess progress.

Advice from:

Organizational Resources

References

The Short Answer

1. Use the results and the experiential version of results to drive the process. 

  Instead of asking “How are we doing on the baseline?” we ask “How are we doing producing the result we want?” How prevalent are the experiences which tell us about this result?” “Are things
 getting better or worse? 

Instead of asking “What is the story behind the baseline?” we ask “What are the causes of our our current conditions on this result?”

Instead of asking “What works to turn the curve?” we ask “What will it take to do better on this result? What will it take to produce the experiences that best describe this result?"

2. Or use one of 2 other methods to create working baselines or a sense of current status and direction on indicators.

3. Don't give up on the idea of using data to assess progress. See 2.8 Where do we get the data for indicators? How do we get better data?

 

Full Answer

(1) Don't give up on the idea of results-based decision making. The most important idea behind results based decision making is that we start with ends and work backwards to means. Data in the form of indicators is a method for being clear about ends. If ends can be described in measurable terms then there is less ambiguity about what progress looks like. And there can be more discipline to the business of getting there. 

(2) Data is not the end-all and be-all of this work. You can have other ways of describing desired results or end conditions of well-being which can serve as proxies for data until data can be obtained and used. 

Here are several ways you can run the process if you really have no indicator data and no prospect of getting any any time soon. 

(3) Method #1:  Using results and indicators without baselines: It is possible to steer the thinking process with the results, experience and desired data at the top of the page. The planning process can then frame the questions not in terms of indicator baselines but in terms of these three elements.

  Instead of asking “How are we doing on the baseline?” we ask “How are we doing producing the result we want?” How prevalent are the experiences which tell us about this result?” “Are things getting better or worse? 

Instead of asking “What is the story behind the baseline?” we ask “What are the causes of our our current conditions on this result?”

Instead of asking “What works to turn the curve?” we ask “What will it take to do better on this result? What will it take to produce the experiences that best describe this result?" 

This is an admittedly weaker way to plan, because the definition of success becomes more clouded and less objective. But it is a way to get started while data is being improved.  

(4)Method #2 Working Baselines: Consider using one or more working baselines, of the type used in the turn the curve exercises. 

First ask the question: "If we could have data to measure these conditions of well-being in our community, what would we want?" Then for each of these answers use the following technique:

"Create a working baseline for purposes of the exercise, by asking the following questions: Is this indicator getting better or worse? Has it been going in this direction for a few years? Has it been getting (better or worse) fast or slow (steepness of baseline)? Do you think it will continue in this direction for the next several years if we stay on our current course (i.e. don't change what we do)?"

While this will not take the place of the need for real data, it will allow people to engage in real discussions of the story behind the curve and what works, the essence of the talk to action process. If this approach is not followed up at some point with real data we will be left wondering if our impressions of improvement or decline are real.

(5) Method #3: Good, Bad or Indifferent: Go down the list of potential indicators (those with data and those without) and ask "How are we doing on this indicator now - Good or Bad?" Label the answers next to the indicator "G" or "B". Then ask: "Are things getting better or worse or about the same?" Mark these answers next to the G/B letters with an up arrow, a down arrow or a horizontal line. This picture can also be used to drive the rest of the process of story, partners, what works and action. This is one technique which can be used in the "Whole Distance" wall exercise.

(6) Don't give up on getting indicator data. See the answer to 2.8 Where do we get the data for indicators? How do we get better data? for more details. Developing new data is not as daunting as it seems. This is because planning and budgeting data does not have to meet research quality standards. It is OK if it is credible and usable. So community groups can sponsor informal surveys of residents. Many communities have in fact done just this on the matter of safety, asking questions of whether residents “feel” safe. Often these numbers conflict with the crime statistics and this allows the community to go deeper on solutions that address both real and perceived safety. Other kinds of surveys are possible. Monthly surveys of graffiti and vacant housing could be accomplished by a walkthrough of the neighborhood, as could many other observable conditions of the community. These could be compiled, and used to create a baseline against which to judge progress. Even simpler surveys might be used by collaboratives – such as a questions at the monthly meeting “How many people here saw children riding bikes since the last meeting. How many of you saw children riding without bike helmets?” – a simple survey that could be charted on the wall and turned over time.