Three Inherent Risks Of The Observation Technique

The observation technique is particularly useful when interviewing stakeholders or studying existing documentation does not produce requirements in sufficient detail to facilitate analysis. To further understand the context in which stakeholders carry out their responsibilities, observation is often employed to get a feel for how things work.

With the ever-increasing volume of data and information in circulation, we have a natural inclination to unconsciously filter the information we let in. What we choose to keep and what we discard depends heavily on what psychologists call a "schema". Differing schemas is the fundamental reason why two people can observe the same phenomenon and come away with two completely different interpretations. 

Before using the observation technique to elicit requirements, it’s important to be aware of the inherent risks associated with the technique so that when it is used, it is with knowledge of potential pitfalls and how they can be addressed.

This piece discusses 3 inherent risks of the observation technique that can affect the schema, should they occur.

1. Confirmation Bias

This is the tendency to see only what you expect to see. If you already have at the back of your mind an assumption or hold a view that you believe to be true, you will unconsciously and in some cases consciously, look for evidence to confirm that assumption, discarding other facts. 

Confirmation bias is said to have occurred when a person seeks out information that conforms to their existing beliefs, rejecting contrary evidence. Once what a person is doing during the observation session does not conform to what you believe, you tune out. Confirmation bias also happens when you observe with the objective of proving someone wrong instead of observing with the objective of understanding the situation. 

9 times out of 10, you’ll find that confirmation bias is related to the desire to be right, stay in one’s knowledge comfort zone and possibly avoid the potential embarrassment that comes with withdrawing from a previously advocated belief.

Address this by: Documenting your assumptions and beliefs. Be careful not to discard whatever does not conform to your belief system by being receptive to what others have to say even if it holds no relevance to you. 

2. Cargo Cult Science

The term, cargo cult science, was used by Scientist, Richard Feynman to refer to situations where experiments and in this case, observations, are made without the understanding, expertise and transparency that is required. 

Observation that seems accurate, but lacks the integrity of complete transparency and repeatability qualifies as "cargo cult". For example, not admitting the conditions under which an observation was carried out and the factors with the potential to affect the validity of your results.

Cargo cult science occurs when observers "see what they want to see". People who are passionate about a certain outcome for example, can easily misinterpret what they observe without any concrete basis or evidence for doing so.

Address this by: Ensuring that you conduct observation sessions with other professionals or organising some form of peer review so that the results of your observation can be confirmed by another party. It is also useful to organise more than one observation session to double check your findings. Practise good science by being transparent and honest in reporting your observations.

3. Hawthorne Effect

The Hawthorne Effect is a fallout of an experiment carried out in the 80s at a company known as Hawthorne Works, Chicago. Worker productivity improved regardless of the changes made during observation. It refers to the tendency for a person's actions to change when they are being observed. Read more about the Hawthorne effect here.

Address this by: Ensuring that observation is done over a considerable period of time. This will help fend off short-term improvements observed as a result of any changes made to working conditions. Also, communicate the objectives of the observation exercise clearly so that all stakeholders understand what’s in it for them. If for example, employees think the objective of the exercise is to reduce costs by downsizing the workforce, they will alter their behaviour to protect themselves.

What other pitfalls can one experience with the observation technique?

Picture Attribution: “Risk Concept” by suphakit73/Freedigitalphotos.net