Voice cloning without the consent horror story

Voice cloning is the feature a voice AI company can get in the most trouble over. It's also the feature customers ask for most.

The tension is obvious. The customer who asks for it is almost always a reasonable one, a studio cloning its own talent for dubs, a telco cloning a founder for an internal announcement system, an accessibility product giving someone back a voice they lost to illness. The outcome if it goes wrong, a synthetic someone-else convincingly saying something they didn't say, is, in the strict engineering sense of the word, catastrophic.

So we had to build it. We had to build it carefully. Here's what "carefully" ended up meaning.

The four-way problem

Most cloning products treat voice cloning as a one-axis feature: quality. The better the clone, the better the product. We think that's wrong. Voice cloning is a four-way problem:

1. Quality. The clone has to sound like the target voice, in the target language, at the target register. No argument there.
2. Consent. The voice being cloned has to have given explicit, traceable, revocable consent. If it can't be proven, the clone can't be issued.
3. Traceability. Every clip the model produces needs to be verifiable, after the fact, as a synthetic clip produced against a specific consent artifact.
4. Revocation. Consent can end. When it does, every currently-active API key that targets the voice has to stop working, retroactively, not just for new calls.

Picking just (1) is how you end up in a news article about deepfakes. Picking (1) + (2) is how you end up with a policy against deepfakes that nobody can verify you're actually following. We think you need all four, and most of the engineering we've done on cloning is in the three that aren't about quality.

The consent artifact

The consent layer is the one customers notice first, because it's the friction. When you submit a voice for cloning, we require:

• A signed consent document from the voice's owner, with identifying information sufficient to match the recorded audio to the person. This is a PDF we provide a template for; the customer fills it with the subject's details, and the subject signs it.
• A short verbal confirmation recorded by the subject, saying a short unambiguous phrase that includes the date and their name. This gets attached to the consent document.
• A customer attestation that the customer has legal authority to use the voice in the intended context. This is a single checkbox in the studio with meaningful contract language underneath.

None of this is novel, standards bodies and industry peers have converged on roughly this set. What matters is that without all three artifacts on file, the model won't synthesize. It's not a policy, it's an enforcement. The inference pod reads a signed consent token from the API gateway, and if the token is missing or revoked, the call returns 403 before a single sample is produced.

We chose friction on purpose. It's cheap to make cloning easy. It's expensive, and important, to make cloning traceable.

The watermark

Every audio clip the model produces is watermarked at inference time. The watermark is:

• Inaudible to a human listener under normal playback conditions (we've verified this in blinded listener panels).
• Robust to common audio transformations. MP3 compression, voice-call narrowband filtering, format conversion, moderate noise, mild pitch shifting.
• Not robust to deliberate destruction, a motivated attacker with a spectrograph and enough time can remove a watermark. We don't pretend otherwise. The watermark is a detection layer, not a DRM scheme.
• Cryptographically linked to the originating consent artifact. Anyone with access to our /detect endpoint (including law enforcement, trust-and-safety teams, and any customer checking their own output) can query a clip and get back: is this a Hakim-synthesized clip? against which consent artifact? on what date?

The goal isn't to make cloning un-misusable. That's impossible, and anyone claiming otherwise is either lying or hasn't thought about it. The goal is to make misuse detectable at scale, so that if a clone of a public figure shows up on a social platform tomorrow, there's a public, queryable way to determine whether it came from our stack.

Revocation

Consent is revocable. That's not a nice-to-have; it's the whole thing.

The way revocation works in practice:

1. The subject (or the customer on the subject's behalf) hits a revoke endpoint, identified by the consent artifact's ID.
2. Within minutes, the revocation propagates to every region. Every subsequent synthesis call that targets the voice returns 403.
3. Existing API keys that have a cached "voice is valid" token refresh the token and see the revocation; the keys themselves keep working for other voices.
4. Clips the customer has already downloaded and distributed are not recalled, that's physically impossible, but are still detectable via the watermark + the /detect endpoint, which will now report the clip as "synthesized against a revoked consent."

This last bit matters. A revocation doesn't just stop future calls; it taints the history. If a clip was legitimate at the moment of synthesis, and the consent was subsequently revoked, the clip's provenance record reflects both facts, the consent existed, and it ended.

What we won't do

A few things we've been asked for and declined.

"Can we clone a public figure without their consent, for parody?" No. Our tooling is the wrong tool for that. There are other ways to produce parody; we don't want ours to be one of them.

"Can we skip the verbal confirmation for an enterprise customer?" No. The verbal confirmation is the thing that's hardest to fake and easiest to audit. Removing it to save five minutes of onboarding friction turns a defensible system into an undefensible one.

"Can we train the model to produce a particular celebrity's voice out of the box?" No. The model doesn't learn celebrity voices as first-class entities. Every clone is built against a specific consent artifact; there's no "global voice library" backchannel.

The default

The cloning product is deliberately harder to use than it needs to be for the happy-path customer, because every shortcut we carve is a shortcut an adversary will find. The engineering we've put into the un-sexy side, consent plumbing, watermark verification, revocation propagation, is most of the work.

The quality is good. The clones sound like the people. That part we figured out in the first quarter. The next eighteen months of engineering on the cloning feature have been the other three axes, and that's why the feature's still in beta.

If you have a legitimate use case and the patience to go through the consent flow, we'd love to talk. Everyone else can keep waiting; the trade-off we've made is deliberate.