---
title: "What the All-In Podcast Got Right About AI Data Leakage (and What It Means for Your Models)"
url: "https://bitrefinery.com/blog/what-the-all-in-podcast-got-right-about-ai-data-leakage"
description: "On the September 11 All-In Podcast, the hosts explained why zero data retention falls short and why companies with valuable IP are moving AI onto infrastructure they control."
author: "Bit Refinery Team"
date: "2026-09-13"
lastmod: "2026-09-13"
tags: ["private ai", "data sovereignty", "ai security", "zero data retention", "open weight models", "gpu hosting"]
source: "blog CMS"
---

# What the All-In Podcast Got Right About AI Data Leakage (and What It Means for Your Models)


If you listen to the All-In Podcast, the September 11 episode will probably be remembered for its argument about AI safety. The segment that deserves more attention from anyone running AI inside a company came about an hour in, when the four hosts spent 15 minutes on a narrower question: when your team types proprietary work into a frontier model, where does that work end up?

Their answers were more candid than most vendor documentation, and they match what we hear from security and compliance teams. Here is what this post covers:

- What happened with OpenAI's Navier-Stokes announcement, and the one sentence in OpenAI's response that matters most
- Why zero data retention (ZDR) is weaker than many buyers assume
- Why the question has reached boards and audit committees
- Why a workstation under a desk does not solve the problem for a company
- What a practical setup looks like, with a short checklist

## 1. A math result, a customer's question, and one careful sentence

On September 8, OpenAI announced that an unreleased model had proven a result on the Navier-Stokes equations, the fluid dynamics problem that has been open since the 1800s and is one of the Millennium Prize Problems. The work reportedly used about 10,000 agents and roughly 130 billion output tokens.

Within hours, NYU mathematician Tristan Buckmaster raised a question. He and Levent Alpöge had been working on a closely related problem, and they had used OpenAI's Codex as an assistant along the way. Buckmaster was careful to say he did not know whether their data had been used, and that he was not accusing anyone.

OpenAI's researchers denied it directly. Noam Brown wrote that nobody looked at the pair's prompts, and Chief Research Officer Mark Chen said no people or AI systems searched user data to solve the problem. OpenAI later said the pair's prompts from the two months before the announcement could not have influenced the system. Its statement also included this sentence:

> "We cannot rule out that de-identified data derived from their usage of our products helped improve our models."

We are inclined to take OpenAI at its word that nobody went looking through a customer's prompts (as David Sacks said on the show, timing alone proves nothing). But the denial is not the part a CIO should focus on. The sentence above describes how the product normally works. If one of the most sophisticated AI labs in the world cannot rule out that a customer's work shaped its models, then no customer using the standard service can rule it out either.

## 2. Zero data retention is a promise, not a control

Chamath Palihapitiya took the point past this one case. Zero data retention, the setting most enterprises rely on when they sign an API agreement, is offered on a commercial best efforts basis. The provider will try not to keep your data. It does not guarantee the outcome.

He also noted that ZDR has side doors. A user clicking the like button in a chat window, for example, may send that exchange down a different path than the API traffic the agreement was written for. In most organizations we have seen, nobody has mapped all of these paths (and the terms of service change more often than anyone rereads them).

The deeper issue is what "deidentified" means. Deidentification removes names, account details, and anything that points to a specific person or company. It does not remove the idea. David Friedberg made this point clearly: in research, the approach to a problem is the intellectual property. A model can learn a method from a conversation without keeping a single identifying detail.

Friedberg, who runs the plant breeding company Ohalo Genetics, described something that happened to his own team. They worked through a novel scientific question with a model, and the model flagged the idea as new. Later, using a different account and a newer model version, they asked a similar question and got that same idea back as a suggestion. He was clear that these were a handful of anecdotes. He also knows his field well enough to say that no newly published research could explain it.

## 3. The question has reached the board

The part of the discussion most relevant to IT leaders was about who is now asking. Chamath described how awareness of AI data leakage is moving up through the risk and audit committees of public company boards, often with help from the large audit and consulting firms. From there it reaches the CEO, who turns to the CIO and asks what the real exposure is.

His prediction was blunt: some CIOs will lose their jobs over this within the next year, because they signed a standard API agreement and assumed ZDR covered them. We would put it more gently, but the direction seems right. Once a company has to tell shareholders it cannot say whether critical IP ended up in a vendor's model, the question moves out of IT and into the boardroom, where the answers tend to be less forgiving.

Sacks added two points that make the risk harder to wave away:

- **Legal protection is thin.** In his view, AI chat data currently has less legal protection than email. The government usually needs a search warrant to read your email, but AI conversation logs can often be obtained with a subpoena or court order. A question you put to your lawyer is privileged, while the same question put to an AI model may not be.
- **Your vendor may become your competitor.** The frontier labs have made clear they intend to build their own applications, and some of those launches have already landed on top of products built by their own API customers. As Sacks put it, it is hard to trust a vendor with your data when that vendor has reserved the right to enter your market.

## 4. A workstation under the desk is not the answer either

Jason Calacanis described his own response. He bought two Mac Studios, loaded open source models onto them, and expects to move most of his team's sensitive work off hosted models. For one person or a small team, that approach works well.

Chamath's reply is the one to remember. A company needs more than a model running on a box. It needs a shared service that people across the organization can reach, with a knowledge base, memory, access controls, and room to grow. A machine on a desk handles none of that well, and a few hundred employees would need a rack of them (at which point you are running a data center, just a badly cooled one).

That leaves three realistic options:

| | Public API with ZDR | Local workstation | Private GPU cloud |
|---|---|---|---|
| Prompts leave your control | Yes | No | No |
| Can influence a shared model | Cannot be ruled out | No | No |
| Shared access, memory, knowledge base | Yes | Limited | Yes |
| Scales past a small team | Yes | No | Yes |
| What protects your data | Terms of service | Physical possession | Dedicated hardware and your own network boundary |

![Comparison of a public API with zero data retention, a local workstation, and a private GPU cloud across data control, exposure to shared model training, team access, and scale](/api/storage/files/blog-images/blog-1789312800000-allin-leakage-infographic.jpg)

## 5. What "your own terms" looks like in practice

Chamath's advice was to go "straight to the bare metal on your own terms." In practice that usually means dedicated GPU servers that no other customer shares, open weight models that you host and control, and a network boundary you define. The model weights change only when you change them, so nothing your team types makes its way into anyone else's product.

Serious companies are already doing this. Harvey, the legal AI company, introduced its own model in August, called Tenet. It is built on the open weight Kimi K3 model and was further trained for legal work on roughly 150 GPUs over about two months. For a company whose customers are law firms, owning the model also helps answer the confidentiality question.

If you are evaluating your own setup, these are the questions we would ask first:

1. **Who can see prompts and outputs?** Name the people and the systems, including logging and monitoring tools.
2. **Where do logs live, and for how long?** Ideally on storage you control, with a retention period you set.
3. **Can anything you send change the model?** With self hosted open weight models, the answer should be a simple no.
4. **Is the hardware shared?** Multi tenant GPUs add a layer of trust you have to verify.
5. **Where does fine tuning data go?** It should stay on storage you own, under the same controls as the rest of your IP.
6. **What can a subpoena reach?** Ask your counsel whether keeping data on infrastructure you control changes the answer.

For more detail, see our posts on [data sovereignty for regulated teams](/blog/data-sovereignty-private-gpu-infrastructure-regulated-teams) and [what to look for in private AI infrastructure](/blog/private-ai-infrastructure-what-to-actually-look-for).

## Where this leaves most teams

None of this means frontier models are off limits. For public information and routine work, hosted APIs are often the right tool. The line worth drawing is around the work that makes your company valuable: research, source code, deal terms, patient records, and anything you would not want a competitor to learn. For that work, the safest assumption is the one OpenAI's own statement points to. Once it goes into a shared model, you cannot be certain it stays yours.

Bit Refinery runs [private GPU cloud](/services/private-gpu-cloud) infrastructure for teams in this position: dedicated GPU servers in our own data centers, with open weight models hosted and managed for you, so your prompts and data stay out of shared models. If you want to see how that works for a regulated workload, our [healthcare use case](/private-gpu-use-cases/hipaa-gpu-hosting) is a good place to start.