Tip #2: consider replacing your pre/post survey with a REtrospective survey...but only under certain conditions

Do you use traditional pre/post surveys to assess programmatic impact on participants?  If so, you might have encountered the following known problems:

1. Participants have difficulty responding to the “pre” survey items because they have little knowledge of the program content and choose to leave many items blank.
2. Participants feel overburdened with the “post” survey because they answered similar items on the “pre” survey and do not fill-out the “post” survey.
3. A participant is not present for either the “pre” or “post” survey, resulting in an incomplete data set for that individual.
4. Participants gain insights into program content and see it differently than at the beginning. Known as the Response Shift Bias (Howard, 1980), participants may overestimate their initial attitudes due to lack of knowledge at baseline; after the program, their deeper understanding affects their responses on the “post” survey.​

​Retrospective Surveys to the Rescue!

Retrospective surveys, by contrast, ask participants to compare their attitudes before the program to after.  Because a participant completes a retrospective survey in one sitting, responses are more complete.  Not only is there a higher completion percentage with this method, but it also has been found to reduce the Response Shift Bias in participants. Also, isn't it A LOT easier to administer one survey instead of two? You bet! Evaluators love the simplicity of one survey...and instructors and program directors do not need to sacrifice as much time to administer the survey. Less burden all around!
​
Major Advantages...and more Accurate?

In several of my projects, the retrospective survey had clear advantages over the pre/post survey. It yielded more complete datasets and higher response rates. Also, the data from retrospective surveys seem to align more "accurately" to qualitative data.

• For example, in one evaluation project several years ago, participants were telling me, via focus groups, that the summer workshop helped strengthen their confidence and content knowledge in computing. They loved it and are eager to dive deeper into computing!
• But, the traditional pre/post survey did not show this. The pre/post survey actually told me that participants' computing skills were decreasing from before the workshop to after!
• Why did this happen? Well, we can thank the Response Shift Bias for that one! Humans have a tendency to overestimate their knowledge/confidence in a field because they are largely ignorant of the vastness and complexity of that field.
• To use a personal example, if you had asked me 10 years ago how good I was at Excel, I would have said "very good." Now, I would rate myself as "average" even though I currently teach courses in Excel and design some pretty intricate dashboards in Excel.
• Basically, my understanding of the possibilities has fundamentally changed. I know what's achievable in Excel, and I've been exposed to true experts in the field. Only a retrospective survey would "accurately" capture this shift in mindset. A traditional pre/post survey would give evaluators a false sense that my skills and confidence in Excel went from "very good" to "average" over a 10 year period. I was, like, a different person back then, and my understanding of the rating scale (1, poor to 5, very good) shifted. Makes sense?

Any Disadvantages?

Are retrospective surveys the answer to all of our problems? Well...not exactly! There are certain conditions that make the retrospective survey unusable and frankly inaccurate:

1. Younger children have difficulty with retrospective surveys. Their memories fade quicker and they forget how they felt before an event occurred. What can I say: Kids live in the moment! That's a good thing, but bad for retrospective surveys, I suppose. Younger children also get confused by a retorspective survey. They need major 'handholding' as they complete the survey.  
2. Memory is less and less accurate the further removed we are from an event. Our memories are faulty and we are subject to false memories (Loftus & Ketchum, 1994). The further we step away from an event, the more hazy our memories get. Retrospective surveys are best to use for short-ish interventions. A 1 week workshop is an ideal set-up for a retrospective survey. A 3 month summer workshop is also pretty good. A semester course? Yeah...that usually works. A one year course? Maybe not.
3. Retrospective Surveys require reflection...and some people have an easier time with this than others: A retrospective survey forces a participant to reflect on their experiences over time. Some people are eager to contemplate how they've changed over time. (Psychology majors live for this stuff!) But, for others, thinking about the past is more difficult and can be off-putting. A painful memory or an unpleasant experience might deter people from looking back. We don't want to change for the worse, right? There might be a subtle positivity bias that emerges when responding to a retrospective survey. That is, it could be that some participants are motivated to show growth from before to after. This "growth" might, in fact, accurately reflect how much they've changed as a result of the intervention. (That's usually the case, I've found!) Alternatively, it might not be accurate; the positivity bias made them do it! It's really hard to tell. Even retrospective surveys have their faults, people! 

More research is clearly needed on this topic. But, there are so many advantages to using retrospective surveys over traditional pre/post surveys;  the good might outweigh the bad.
​What do you think? ​