On Signals & Noise

And my many failures to correctly distinguish between the two...

Most days, while I’m fresh, and with a jug of freshly brewed black coffee to hand, I try to fit in three hours or so of deep work. 

Before I can do that though, I need to check my emails and messages. Not necessarily with a view to responding to all of them, but trying to figure out whether there’s something urgent that requires my immediate attention and/or some kind of action. 

I scan email titles to see whether anything jumps out. I look to see who sent the email, or who they sent it to. If the email is from, or to, someone important (including me of course!), I might open it up and take a peek, to figure out whether it’s going to require some amount of effort to respond to, or whether I can write a quick reply and draw a line under it there and then. 

I take a similar kind of approach to messages in Slack or Teams, applying the following heuristics: 

• What’s the message about? 

• Who is it from? 

• Does it require immediate attention, or can it wait until later?

The does it require immediate attention point is important. It speaks to prioritisation, which is a critical facet of any knowledge worker type role. And has particular significance in the product space, since the product manager role revolves around being a master of prioritisation. I have to think about how the request fits into my plan for how I’m going to go about my work that day. As a PM, I may also need to think about how it fits into a bigger picture of business priorities, stakeholder requirements, company politics, customer needs, and other dimensions of priority.

Having duly pondered some or all of the above - I ultimately have to make some kind of decision as to whether the email or message actually requires my attention, how quickly, and what  to actually do about it. 

What this amounts to, in case you hadn’t already realised, is signal detection. I’m seeking to optimise my attention for signals, and trying to filter out noise. In the general case, that boils down to 3 steps:

1. Filtering: In an age of information overload, you, dear reader, are constantly having to decide which pieces of information to attend to and which to ignore. Determining, for example, which email is critical to read now versus which can wait (or be discarded) can be framed in terms of Signal Detection Theory: Hits, misses, false alarms, and correct rejections.

2. Prioritisation: When selecting which actions or tasks to pursue, your decision will often revolve around incomplete or uncertain information. Your ability to discriminate between what's genuinely impactful versus what only seems so, can determine the success of an activity or project. At the macro level, it could affect the success of your team, organisation or business.

3. Decision Making: When you’re faced with ambiguous data, you have to decide on a course of action (including taking no action). For instance, you may have to decide if a change in a data trend is a genuine signal of an underlying issue, or if it’s just random noise.

A quick primer on Signal Detection Theory (SDT)

Signal Detection Theory (SDT) is a framework for understanding how decisions are made in the presence of uncertainty, particularly when detecting faint or ambiguous stimuli. The concept emerged from the field of electrical engineering and was then applied to radar signal processing during World War II. Later on, the principles were adopted and extended in psychology to explain how humans and other animals make decisions under conditions of uncertainty. The theory differentiates between the actual state of the world (whether a signal is present or absent) and your decision about that state. 

The key concepts you need to know for the purposes of this article are that a signal detection activity has four possible outcomes: Hits, Misses, False Alarms, and Correct Rejections.

1. Hit: Signal is present, and you correctly detect it.

2. Miss: Signal is present, but you fail to detect it.

3. False alarm: Signal is absent, but you believe it's present.

4. Correct rejection: Signal is absent, and you correctly identify it as absent.

The concepts of signal detection are applied in lots of areas, including psychology, medicine and economics, to name a few. They’re generally applicable in day-to-day life (trying not to get run over when crossing the road for example, or literally paying attention to signals if you’re driving a car), in knowledge work and for the purposes of this article, product management.

How signal detection fits into product management

Signal Detection Theory (SDT) isn't considered a product management framework in itself; nor is it explicitly integrated into any widely recognised product management frameworks. However, I wouldn’t be writing this article if I didn’t think that the principles of signal detection couldn’t be incorporated into some of your existing product management practices, and help you to make better decisions in the midst of uncertainty, ambiguity, and change. Areas where I think it could be used include:

• Prioritisation: Tools like RICE (Reach, Impact, Confidence, and Effort) or WSJF (Weighted Shortest Job First) require decisions based on uncertain or incomplete information. Applying signal detection principles can enhance these tools by helping you to differentiate between genuine insights and random fluctuations or biases.

• Roadmapping: When you’re planning a product roadmap, distinguishing genuine needs (signals) from transient or less impactful items (noise) is essential. Isolating signals from noise helps to make the distinction more clearly.

• Research and User Feedback: When generating or analysing user feedback, product managers are aiming to separate genuine patterns and needs (signals) from the variability and noise of less useful feedback.

• Data Analysis: Metrics sometimes show fluctuations due to a myriad of reasons. For example,  our NPS score went up 10 points last week. Why?  Is there some important insight (signal) to be drawn from the sudden spike, or is it just noise? 

• Stakeholder Communication: Feedback from stakeholders can be conflicting or reflect organisational politics and biases. Applying heuristics to evaluate which pieces of feedback are strategically important signals and which might be noise, will help ensure that product decisions align with strategic goals.

Failure modes

There’s lots of ways to fail at product management. Many of them are related to misclassifying signals as noise and vice-versa. Most of them, I’ve made myself at some point:

During feature development:

• I’ve been swayed by loud or influential customers without verifying the broader applicability of their request.

• I’ve implemented features just because competitors have (or didn’t have them) them without properly assessing their relevance.

• I’ve massively underestimated technical complexity, leading to maintenance issues and technical debt.

While gathering user feedback:

• I’ve prioritised feedback based on volume, rather than quality or relevance.

• I’ve disregarded feedback from less vocal user groups.

• I’ve acted on feedback without validating its broader applicability.

During analysis and research:

• I’ve wasted time chasing new trends without assessing its longevity or relevance.

• I’ve neglected to reevaluate past decisions in light of new information.

• I’ve been too rigid and slow in adapting to significant market shifts.

When interpreting data:

• I’ve misinterpreted data without considering external influences.

• I’ve focused too much on short-term metrics without considering long-term impacts.

• I’ve implemented changes based on inconclusive results.

When managing projects:

• I’ve spread resources too thinly across too many projects.

• I’ve neglected to revisit and adjust resource allocation as situations have changed.

• I’ve been influenced by politics or internal pressures rather than objective product needs.

We’ve had a new PM join us recently, and perhaps because she’s just new to the team, or maybe because of some personality traits or perhaps because she’s just more skilled in some areas than I am, she treats a lot of things that as signals, that I would otherwise disregard as noise.  Sometimes she hits, sometimes she misses; but it’s interesting to observe the process because it forces me to reassess my own responses and skills in this area.

I’m a boots on the ground kinda guy. I love speaking to customers, and I work hard at being responsive to what I hear from them. Equally I listen to what my team and colleagues are saying, and am very responsive to signals that they may be blocked or need support in some way. I have good intuition about problems that are likely to explode and therefore require immediate or escalated attention before they do so.

I’m not always very good at identifying political signals from the wider business and stakeholders, or at spotting patterns in data that may indicate a trend which requires attention. I sometimes miss important signals or confuse them with noise. 

Given that I’ve identified some weaknesses in my signal to noise detection capabilities, what are some ways I can make some improvements?

Getting better at signal detection

Here’s my ideas for helping to develop those skills, if you feel you may be falling short of the mark (as I often do!)

1. Implement feedback loops: Seek to regularly collect feedback from stakeholders, users, and teams. Reflect on your capabilities and hone in on areas of improvement based on this feedback.

2. Diversify your Information Sources: Don’t rely on a single source of data. Triangulate your insights from different channels to paint a more accurate picture.

3. Seek external perspectives: Try to gather viewpoints from outside of your immediate circle. Utilise friends, mentors and peers to gain fresh insights.

4. Stay user or customer centred: Making sure you’re interacting with users and customers regularly will ground decision-making in the realities of their needs and desires.

5. Cultivate critical thinking: Question your assumptions regularly. Develop a habit of playing devil's advocate to challenge prevailing ideas (something I find alarmingly easy!)

6. Utilise appropriate frameworks & models: Employing decision-making frameworks like RICE (Reach, Impact, Confidence, Effort) or cost-benefit analyses can help to structure thinking.

7. Practice reflective listening: Make sure you truly understand the feedback or data before making decisions based on it. I literally repeat to people what I think I heard them say and ask for confirmation we’re on the same page.

1. Accommodate Uncertainty: The reality is that not every decision will have a clear signal associated with it. Sometimes you just have to make the best decision with information  available at the time, and be open to iterating based on the outcome. Jeff Bezos coined the one-way versus two-way door analogy, which I think is useful in this regard.

2. Manage your time: Set aside time for deep work without distractions. Give yourself room for more focused analysis and applied discernment.

3. Learn continuously: Regularly engage in professional development, courses, and workshops and the like to refine your professional skills. 

There shouldn’t actually be many surprises here, since our brains have basically evolved to help us pay attention to useful information and disregard the detritus. What’s important, particularly in the product realm, and really anywhere your decisions are likely to have a lasting impact, is knowing when you can trust your instincts and think fast, and when you need to slow down and take more measured approach.  

Keeping in mind the hit, miss, false alarm and correct rejection Signal Detection Theory concepts from above, it seems to me that you can probably train yourself to hit more frequently than you miss, avoid false alarms and, critically for the slightly overwhelmed PM’s, know when to make a defensively correct rejection. Hopefully some of the ideas above help you (and me!) to do that. 

If you enjoyed this article, or got some useful ideas from it, I’d appreciate it if you could hit the subscribe button or leave a comment, just to let me know. If you have some ideas for improvements or want me to write about something specific, I’d be happy to hear about that too.

Thanks, and see you for the next one!

Sources:

• "An Introduction to Signal Detection and Estimation" by H. Vincent Poor

• "Signal Detection Theory and ROC Analysis in Psychology and Diagnostics: Collected Papers" by John A. Swets 

• “Awakening from the Meaning Crisis, Episode 39: The Religion of No Religion” by John Vervaeke [YouTube]