I keep thinking about the phrase 'broken safety culture' and what it actually looks like when your office is trying to build a digital god. It’s one thing for a startup to have a broken culture because the snacks are stale or the middle management is toxic, but it’s another thing entirely when the 'product' is Artificial General Intelligence. When Jan Leike walked away from his role leading the Superalignment team at OpenAI in May 2024, he didn't just leave a job; he left a philosophy. He basically told the world that the shiny, competitive race to ship new features has finally outpaced the boring, quiet work of making sure those features don't ruin us.
What fascinates me is the silence that follows these exits. We see a tweet thread, a polite LinkedIn update, and then a massive void where the safety research used to be. I wonder if we are witnessing the birth of a new kind of corporate secrecy, one where 'safety' isn't a shared human goal anymore, but a proprietary trade secret. If you find a way to stop an AI from lying to its users, do you share that with the world to keep everyone safe, or do you keep it in a vault because it gives your chatbot a competitive edge?
The Price of a Proprietary Conscience
OpenAI started as a non-profit with a mission to benefit humanity, but it’s currently sitting on billions of dollars in investment from Microsoft. That kind of money changes the gravity of a room. I’m curious about the exact moment the shift happens—the Tuesday afternoon meeting where someone realizes that a safety check will delay a product launch by three months and cost the company a lead in the market. Leike’s departure suggests that these moments are no longer rare; they are the new standard operating procedure.
When safety teams lose their resources—Leike claimed his team was 'sailing against the wind'—it makes me wonder if safety was ever the destination or if it was just the life jacket they wore to get through the initial regulatory storm. If the people hired to be the brakes are complaining that the brake lines have been cut, we should probably stop admiring the car's top speed for a second. It feels like we're watching a live experiment in whether a for-profit entity can actually prioritize a risk that might not manifest for a decade over a quarterly earnings report that is due in three weeks.
Photo by Faruk Tokluoğlu on Pexels
The Great Silo of Alignment
There is a growing chasm between the 'Open Source' believers and the 'Closed Gate' corporations. The open-source crowd thinks that if we all have the code, we can all help fix the bugs and spot the dangers. The closed-gate crowd—the OpenAIs and Anthropics of the world—argues that the tech is too dangerous to let out into the wild. But here’s the puzzle: if the tech is too dangerous to share, but the internal teams tasked with managing that danger are quitting because they feel ignored, where does that leave the rest of us?
I’m genuinely trying to map out a world where safety research is a competitive advantage rather than a public good. In medicine, we have peer-reviewed journals. In aviation, when a plane goes down, every airline in the world gets the data so it never happens again. But in AI, if a model starts exhibiting 'deceptive alignment'—the tech version of playing along until it’s strong enough to stop listening—that discovery stays behind a corporate firewall. We are essentially trusting a handful of CEOs to be the sole arbiters of what is 'safe' for eight billion people.
- If safety research is siloed, we lose the 'citizen scientist' layer of protection.
- Corporate safety benchmarks often focus on 'vibes' and PR-friendly safeguards rather than deep architectural robustness.
- The talent drain from big labs to smaller, safety-focused non-profits suggests a massive ideological rift in the industry.
The Missing Regulatory Middle
Where is the referee in this game? Right now, it feels like the players are also the ones writing the rulebook and officiating the matches. Every time a high-profile researcher leaves and points back at the building with a look of concern, I look to the regulators, and I see them still trying to figure out what a LLM actually is. There is a massive knowledge gap between the people building GPT-5 and the people tasked with governing it. By the time a law is passed, the technology has usually moved three iterations down the line.
I wonder if we need a 'CERN for AI'—a place where the smartest minds can work on alignment without the pressure to produce a 'Pro' version for $20 a month. Without a neutral ground for safety research, we are essentially betting the future on the hope that corporate interests and human survival will always perfectly overlap. History suggests that’s a very risky bet. If the internal 'conscience' of these companies is quitting, maybe it's because they've realized that the overlap is shrinking.
What This Actually Means
This isn't just about one guy quitting a high-paying job. It's about the erosion of the idea that AI safety is a collective human problem. When the smartest people in the field start moving toward decentralized, open-source, or non-profit models, they are telling us that the corporate structure is currently incompatible with the level of caution required for AGI. They are voting with their feet because they no longer believe the 'move fast and break things' mantra works when the thing you might break is the social fabric.
We are moving into an era of 'Black Box Safety,' where we are told everything is fine, but we aren't allowed to see the math. I’m left wondering if the 'broken culture' Leike described is a bug or a feature of the current AI boom. If you want to be first to the finish line, you don't pack a heavy medical kit. You run light. You run fast. And you hope you don't trip.
Ultimately, the departure of the Superalignment team suggests that the 'Super' part was the priority, and the 'Alignment' part was the PR. We should be paying very close attention to who fills those empty seats—and whether they are researchers or just safety-themed marketers.
Quick Answers
Why did Jan Leike leave OpenAI?
He stated that safety culture and processes have taken a backseat to shiny products, and that his team struggled for the compute resources promised to them.
What is 'Alignment' in AI?
It is the process of ensuring an AI's goals and behaviors match human values and intentions, preventing it from causing unintended harm.
Is open-source AI safer than closed-source?
It’s a debate: open-source allows for global scrutiny of flaws, while closed-source prevents bad actors from easily weaponizing the underlying code.
