Patrick McKendry

These past weeks, LK-99 captured our attention and sparked our imagination. The only problem, it didn’t really do much else. The polycrystalline - made up of lead, oxygen, phosphorus, and infused-copper - was believed to be the first room-temperature, ambient-pressure superconductor. That type of material, which transfers electricity with zero resistance, would be transformative to the power grid, computing, transportation, medical technology, and more. While thousands of superconducting materials exist today, most of them are only operable at extremely low temperatures and only a handful can be mass-produced. Per experts, a more workable material would easily win a Nobel prize and, economically, be worth so much more.¹ Researchers on LK-99 shared the findings directly to pre-print, bypassing the standard process of scientific peer review and immediately drawing concerns. Further, two papers were published from the team with two different sets of authors, conflicting data, and disagreements on the structure of the work.¹ Upon publication, researchers around the world took to the task of validating the initial study. After dozens of replication efforts in the weeks since, the general consensus is clear. LK-99 is not a superconductor.²

At the surface, the story of this false superconductor offers clear reminders that need no explanation regarding how we should deliver work. However, if we continue to pick this story apart we can also find lessons of common logical traps that regularly trap our teams. We can see how the researchers fell prey to confirmation bias and will talk about effective task prioritization for projects in our teams. In the materials selection for LK-99, we see aspects of the not invented here bias leading the research team to pursue a bizarre production path and will talk about how to properly decide when and why to deviate from existing infrastructure. In the end, this story is a gentle perspective reminder that success is not only found in the outcome but in the process for how we get there.

‍

Probabilistic Prioritization

As industry experts began to review the initial studies on LK-99, they immediately identified some blatant shortcomings of research design. Of particular note was the heat anomaly study on the material. Or, rather, the lack of one. In superconductor research, it is common practice to produce and publish these anomaly results. Per Chris Grovenor, director of the Centre for Applied Superconductivity, “All superconductors that have ever been proven to be superconductors show this specific heat anomaly. If there is no specific heat anomaly, it ain’t a superconductor.”¹

In this case, we see a textbook example of congruence bias. Congruence bias is a special kind of confirmation bias that describes our preference for directly testing our hypothesis rather than pursuing alternative hypotheses that could cement our understanding. Said differently, we tend to avoid tests that could disprove our initial thinking and prefer to test again and again for ideas that could support our view.³ LK-99’s research team could have easily determined that their proposed material was not a superconductor with this standard test. Instead, they published tests highlighting low electrical resistance using suspect measurement techniques.¹ Similarly, the research team shared a now viral video of LK-99 levitating as evidence of its superconductor characteristics, a feat that could also be explained with alternative physical reasons.² Ideally, it would have been better had the research team put as much effort into disproving its findings as it did into providing evidence supporting it.

In the scientific community, diligent replication and potential dispute of the originating hypothesis is key to the field’s ethos. That same spirit is less core to many business organizations and can create problems. In product management, new features are often built upon a hypothesis of what users want and will pay for in a product. Congruence bias can drive product managers to design and conduct user research studies that support what they believe to be true rather than asking questions to disprove their ideas. For example, a PM who believes that a community-lead support forum is a good addition might ask questions like “Would you find value in a discussion forum feature that lets you learn from the use cases and problems of other users?” The default answer to that question is clearly affirmative. However, a question like “How often do you find that the problem statements of other users precisely match your own?” would be more clarifying. More often than not, users will find their experiences to be unique and significantly limit the value and frequency of use for a shared forum. By asking different questions, our PM can create answers that either confirm or deny their hypothesis.

The danger of congruence bias comes from the fact that time and people resources are limited on all teams. That means that every project we pursue carries opportunity cost. We need tactics that diminish time spent pursuing the wrong projects. One which I frequently recommend is what I call probabilistic prioritization. Most tasks are prioritized with two factors in mind: time to deliver and value realized. This impact prioritization has one critical flaw. It assumes that our hypothesis of value is correct. Yet, we know that congruence bias might have blinded us to a project's realistic potential.

Probabilistic prioritization also looks at time to delivery and value realized but adds a third factor to consideration. We also look at each piece of the project for how probable it is to work as expected and we then bump up priority on the least likely to succeed tasks. Said differently, we prefer to focus on something that is 10% likely to work as expected over something that is 70% likely. The goal for this is to confirm the viability of a project as early as possible so that we can cut the project short and reallocate resources if needed. Ultimately, a project’s likelihood to succeed is a function of the success probabilities of its composite parts. When we start by tackling the lower probability tasks, we give ourselves the best early indication of whether the project as a whole will work.

We can look again at feature building in product management. If we were tasked with building the community support forum, we could expect the following pieces of work for that project:

The common behavior I see is that teams will pursue what is a nearly guaranteed success in order to show progress and feel that the project is moving forward. In this case, we might get the rationale that “The user profile will be a foundational piece that is a prerequisite to other pieces of work, so we’ll start there. We can’t do push notifications until we have a concept of a user to push them to.”

Probabilistic prioritization would argue differently. That approach would suggest that we first start with answer validation because we are not very optimistic that we will be able to effectively solve the initial user question with this forum. In this case, we’ll need to take on some tech debt and build within the existing constraints because we do not yet have a forum to use. We might simply start with highlighting our existing support documentation and prompting users with a simple Thumbs-Up/Thumbs-Down response for the question “Did this article answer your question?” After collecting a sample of data, we’re simply going to look to get a sense of what percentage of users found the standard documentation helpful. If that rate is reasonably high, it tells us that existing documentation is helpful and that users are likely going to be able to find relevant answers for their individual problems through the forum. However, if that approval rate is low, it is unlikely that we’ll be able to make up the needed difference via a forum. In that case, we should abandon the project altogether. By accounting for congruence bias and seeking disconfirming information early, we are able to avoid wasting effort and resources.

‍

If It’s Not Yet a Thing, There Might Be a Reason

A second eyebrow-raising observation from experts was the material selection for LK-99. Most superconductors start with metals as their base material. Here, researchers selected a mineral base. As bluntly stated by Michael Norman, former director of materials sciences at Argonne National Laboratory, “When you start with a rock, chances are you will end with a rock.”¹ As such, the research and its conclusion was a complete surprise. It varied meaningfully and suddenly from all previous research which had, at no point, suggested a reason to pursue an alternative starting point for materials. Per David Larbalestier, chief materials scientist of the National High Magnetic Field Laboratory, “This discovery is completely out of the blue. I have no idea what the idea, frankly, behind doping this [mineral] with copper was.”¹

On one hand, diverging from existing solutions is sometimes required for breakthrough innovation. For example, it was Blue Origin’s split from existing standards that made it the first organization to land a rocket for re-use in space exploration.⁴ It is fair to think that researchers on LK-99 felt that the existing metals-based approach for superconductors had yet to produce a viable room-temperature material and that a paradigm shift was needed. However, the not-invented-here bias could also be at play. This bias describes the tendency to discredit ideas from outside of our own organizations and may have lead the researchers to devalue a plethora of scientific evidence suggesting from the outset that the design they were pursuing was the wrong one.⁵

For our teams, this can also be a problem. The business environment is a competitive one where companies struggle constantly to find a competitive edge through innovative and proprietary approaches. Seeking to stand out from the crowd can drive teams to lean even more heavily into the not-invented-here bias, sometimes even to a fault. For example, particularly in software product teams, nuanced and precise use cases may lead to an overly favorable preference for building custom software solutions when off-the-shelf services may be able to meet the vast majority of primary requirements. More broadly speaking, the not-invented-here bias can lead to insulated thinking that either disregards or completely misses on widely known and accepted market insights. Highly operational direct response companies may discredit the sales benefits of brand marketing or consulting firms may continue to promote outdated best practice playbooks that ignore evolving trends, as examples. Further, teams can miss out on opportunities to chip away at leading competitor market share by failing to monitor and copy winning strategies that the leader is implementing.

To account for this, teams should make a habit out of getting outside of their own walls. This can manifest in a number of ways, ranging from introducing a current events review to weekly performance meetings to a quarterly competitor insights deep dive. For their part, leaders should be regularly coached on striking the right balance for their teams between optimizing existing operations with testing against new trends and competitor tactics. When done well, this can help any team to disrupt itself and its market on their own terms.

‍

Conclusion

The creation of a production-ready, room-temperature superconductor would be an incredible and historic scientific achievement. So much so that it is fairly understandable why the LK-99 research team took such an optimistic view of their findings. In a similar way, we can see this same influence in start-up founders who ambitiously pitch their vision for “unicorn status” and end up over-leveraging their company. Here we see yet another reminder that, while the accolades of such achievement are desirable, pursuing the outcome first and foremost is a recipe for disappointment. Rather, we must see success not as the destination but as the journey that takes us there. When we find wins and fulfillment in the progress, we energize our efforts, take greater value out of what we learn along the way, and through compounding accelerate ourselves towards that destination.

‍

References

‍

LK-99 & Lessons in Product Delivery