The Prompting Company

Benchmark report

Blocked by a prerequisite

9m 22s elapsed

An external prerequisite prevented the documented task from completing. Benchmark again

Experience score

35/ 100

Critical Gating
  • AInstall
  • CAuth
  • DExecute
  • SDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth C, Execute D, Docs S, Current SAInstallCAuthDExecuteSDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
The credential handoff: the first secret returned a 404-ing URL instead of a key, costing several shape-inspection probes, and the clarifying re-request timed out.

Where the time went

5:26 on the owner · 3:57 agent time
  • Starting0:01
  • Research0:19
  • Attempt2:43
  • Verify0:54
9m 22sTotal elapsed
—Time to first success
5:26Waited on the owner

Judge reason

Setup is proven: system pip hit PEP 668, so the agent built /tmp/benchmark-home/.venv and installed exa-py 2.22.2 (verified by importlib.metadata output), and wrote both the documented SDK script (exa.search(..., contents={'highlights': True})) and the REST equivalent (POST https://api.exa.ai/search with Authorization: Bearer). Neither ever reached HTTP 200: sdk_stderr.txt shows exa_py raising 'status code 401 {"error":"Invalid API key","tag":"INVALID_API_KEY"}' (requestId 1a7099114086e9530dd95aaee35b975c) and rest_output.txt shows 'HTTP 401 ... requestId 63257db07f810ec3717ab4dbe241b9d9'; sdk_output.txt is 0 bytes and no exa_search_results.json/exa_rest_results.json exist. No titles, URLs, or highlight text were ever returned, so the success criteria are unmet. benchmark-first-success.json is absent and the agent explicitly did not call benchmark-success, so firstSuccessVerified is false. The 401s from api.exa.ai confirm network reachability and correct request shape, isolating the failure to the missing credential.

Attempt record

  1. 2:40Asked the owner for Exa API key string (EXA_API_KEY)no response within the window
  2. 8:29Blocked by a prerequisiteNo valid Exa API key: the supplied secret was an http(s) URL on a non-exa.ai host (404 when fetched), and a second clarifying request timed out; Exa's /search requires a key.

Attempt artifacts saved.