read
5 min read

Your AI Evaluator Holds Keys Worth Stealing

At Comet, an idle account was enough to get 29 of them.

Share Link copied! Message
Facebook LinkedIn Bluesky X Create PDF
Cover generated with Midjourney, edited in Photoshop.

Companies buy evaluation tools because they've stopped taking their AI on faith, and they're right to. A second model grades the first, flags the answer that breaks policy, and gives the release team something firmer than a good demo.

But the grader has to call models, so it needs API keys. A tool brought in to check the work now has its own authority.

Comet, the company behind the open-source evaluation platform Opik, just showed what that authority is worth to someone else. Between October 24 and 26, an intruder used a user API key from an inactive comet.com account to get into a few of Comet's internal test workspaces. It was the same kind of key any Opik user has, and Comet says it has no evidence its own systems were involved in how the key got out. The account had no administrator rights and no access to customer workspaces. It didn't need either. A flaw in Opik's AI provider settings handed the intruder 29 of the API keys Comet uses for OpenAI and Anthropic, and the intruder tried the same flaw on other Comet systems.

Comet heard about it from an AI provider. That's the line in Comet's disclosure I'd underline.

A company whose product exists to show you what your AI is doing learned about its own breach from someone else.

New monitoring for provider settings changes and unusual outbound traffic appears on its list of fixes, next to the revoked keys, the removal of the inactive account from every workspace, an audit of all workspaces for stale members, and rotated internal credentials. Comet says no customer data, customer accounts, or customers' own provider keys were touched. It bases that on a review of access logs, including the logs for every account the attacker created.


A settings change can move a secret

Suppose you want to know whether a support assistant follows your refund policy. You can ask a second model to score the assistant's answers. Opik lets teams connect model providers for exactly that kind of scoring, in its Playground and its online evaluations. When the connection uses an API key, the evaluator makes requests on the key owner's behalf.

The key is masked in the dashboard once you save it, and the software underneath keeps using it. If someone allowed to edit the settings can change the destination while the saved key stays in place, the service could present that key to a server the editor controls. Comet hasn't described exactly how the intruder pulled the keys out. Its fix points one way, though. Comet says Opik now requires the API key to be re-entered whenever a provider address changes, and restricts which addresses provider requests can reach. The matching open-source change, merged into Opik's code on October 26, rejects any settings update that swaps the address while keeping the stored key. A commit note on that change calls the move "the credential-exfiltration shape the rule exists to prevent."

This is also why "no administrator rights" isn't the comfort it sounds like. In its guidance for self-hosted users, Comet says exploiting the flaw requires an account with permission to edit AI provider settings. That was the permission that mattered. Comet doesn't say whether the inactive account held it in those test workspaces, or whether the intruder worked through accounts it created.


Running it yourself moves the gate

Comet says self-hosted Opik wasn't affected, and for this incident that's true, since it happened on comet.com. Customers on comet.com and on Comet-managed deployments don't need to do anything. The flaw did exist in earlier open-source releases, and Comet recommends upgrading. For the future, it points to two safeguards: exploiting the flaw requires an account allowed to edit provider settings, and self-hosted Opik doesn't allow public sign-ups by default.

Now read Opik's security policy. Open-source Opik ships with authentication turned off. A stock install through Docker Compose or Helm, the two standard ways to set it up, treats every request as a fully authorized user of the default workspace.

The policy tells you to treat anyone who can reach the API as an administrator. There are no sign-ups to restrict on that install, because there's no login. Whoever reaches the API can edit the provider settings.

Upgrading gets you the key re-entry rule. On those same installs, it doesn't get you the destination check by default, and that check covers the other thing the intruder tried at Comet: using the evaluator to reach other systems. The new setting defaults to strict, but Compose and Helm default to relaxed, and Opik's documentation says relaxed turns off the protection against server-side request forgery on provider URLs. That flexibility is useful when your models run inside your own network. It also means that if you run Opik yourself, your network boundary is your authentication, and the destination check is a setting you have to turn on.


Ask the vendor to show you the handoff

A useful procurement demo starts with an account nobody uses anymore. Ask the vendor how it notices that a member has stopped logging in, whether user API keys expire, and who gets told when one sits idle. Comet's own cleanup included an audit of every workspace for stale members. That's the check to see if it's working before you sign.

Then run the departure. Remove a test user from a workspace and try reaching it with that user's API key. How long does the old access last, and who can verify that it's gone? Then try changing a provider address with the role your evaluators will use. Find out who can reauthorize the connection, where destination restrictions are enforced, and whether the change sets off an alert your team can see. Watching a change get refused will tell you more than a tour of the permissions screen.

I'd pay particular attention to the handoff between the team that depends on the evaluations and the team that controls the provider account. If the evaluator shares a key with the production application, revoking it can interrupt both. Ask whether the integration can use a dedicated key your team can revoke at the provider, and whether the vendor can show when the last successful evaluation ran.

A release team deciding whether to ship deserves to know when the checks it relies on have stopped running.


Share Link copied! Message
Facebook LinkedIn Bluesky X Create PDF