Troubleshooting
Find the message you see. agent-secrets prints a broker refusal as <message> (<CODE>) and a
helper refusal as <CODE>: <message>; the error reference,
generated from the code, lists every code.
The session has no broker identity
Section titled “The session has no broker identity”| Message | Cause | Fix |
|---|---|---|
AGENT_SECRETS_URL is required (exit 2) |
The process does not know the broker’s address. | Set AGENT_SECRETS_URL to the broker’s public URL. A command run by agent-secrets NAME -- … keeps it. |
no session identity: the key dir … has no key.pem … and no helper socket is set or at … |
The process is neither in a container with a key nor on a machine with a helper. | On a machine, run the agent under agent-secrets register --exec; in a container, its launcher must enroll it. |
NOT_A_SESSION: pid … is not inside a registered machine session; a session root is started with agent-secrets register --exec -- <agent argv> |
The helper is running, but this process does not descend from a registered session. | Start the agent with agent-secrets register --wait 10 --exec -- <agent>. |
NOT_ENROLLED: this session is not enrolled with the broker yet |
The session registered, but the helper has not enrolled it yet: it was started without --wait, or the broker was unreachable. |
Start sessions with --wait 10. If it persists, check the helper’s log and that AGENT_SECRETS_URL reaches the broker. |
this machine is not logged in to the secrets broker; not an agent session (run: agent-secrets machine login) |
The helper holds no machine credential: it restarted, or the credential expired. | Log the machine in. |
agent-secrets: helper unreachable at … |
The helper is not running, or the socket path is wrong. | Start agent-secrets-helper, or point AGENT_SECRETS_HELPER_SOCK at its socket. |
agent-secrets register: AGENT_SECRETS_HELPER_SOCK is unset and no helper socket is at … (machine sessions only; an agent box has a key dir instead) (exit 2) |
No helper listens on the default socket, $XDG_RUNTIME_DIR/agent-secrets/helper.sock, and AGENT_SECRETS_HELPER_SOCK names none. |
Start agent-secrets-helper, or set AGENT_SECRETS_HELPER_SOCK to the socket it listens on. |
the launcher credential reached its expiry; run: agent-secrets machine login, or the broker refused the launcher credential (expired or revoked, …); run: agent-secrets machine login (from machine login-status) |
The credential expired or was refused: past its lifetime, revoked by the person who approved it, a clock far off the broker’s, or an AGENT_SECRETS_URL that is not exactly the broker’s public URL. |
Fix the clock or the URL if either is wrong, then log in again. |
ENROLLMENT_NOT_LIVE from agent-secrets renew (exit 76) |
The session ended: its lease lapsed, its launcher unenrolled it, or its machine login was revoked. The renewer names the enrollment, HTTP status and code; the pod shim stops rather than retrying it. | Start a new session (a box: a new key and enrollment, run an agent in a container). |
PROOF_INVALID |
The broker refused the proof itself: its iat is outside BROKER_PROOF_SKEW_SECONDS, its key does not match the enrollment, or AGENT_SECRETS_URL differs from the broker’s public URL. A renewer keeps retrying this code because a clock or URL correction can make the next proof valid. |
Check the clock and URL; for a key mismatch, start a new session. |
A pod’s renewer cannot read its token
Section titled “A pod’s renewer cannot read its token”| Message | Cause | Fix |
|---|---|---|
agent-secrets renew: ERROR: enrollment … cannot read its pod token (…); nothing was sent, so its lease lapses unless the file becomes readable |
The pod’s renewer (agent-secrets renew --pod-token-file) could not read its projected service-account token, or read it blank, so it sent the broker nothing. The broker never hears of it, so this line in the pod’s container log is the only record. The renewer does not exit 76 for it: at start it exits 2 and the worker shim starts it again 5 s later, and after a renewal it waits and reads the file again. If the file stays unreadable the enrollment’s lease lapses, and its grants end with it. |
Check the projected-token volume is mounted read-only at /var/run/legion/agent-secrets-token in that role’s container, with the file token present and non-empty, and that the container’s user can read it: the volume’s mode is 0440, readable through the pod’s fsGroup. Once the file is readable, the next renewal succeeds. |
The request was refused
Section titled “The request was refused”| Message | Cause | Fix |
|---|---|---|
request … was denied (exit 77) |
The approver denied it, or one of the names is a service’s secret, which only that service’s sessions get. Nothing ran. | Ask the approver why, or ask for the names you may have. |
request … is still waiting for approval; nothing was run. Check it with: agent-secrets status … (exit 75) |
Nobody decided within --wait (30 minutes by default). |
Ask the approver the waiting for … line named (agent-secrets status prints it again): the person the broker recorded when the request was made, which can differ from whoever the secret’s description or your memory names. Rerun the command once it is granted, or with a longer --wait, a duration such as 1h. If that person’s approval is refused NOT_APPROVER, something changed after the request was made (Approvals says what each change does). Usually the secret’s owner changed: run agent-secrets cancel <request id> and ask again, and the new request goes to the current owner. If instead the session’s operator withheld one of the names, the request waits on anyone, and the broker still serves that name to the session, the operator can approve it, so ask them. Otherwise no one can approve it: cancel it and request the names separately, one request per approver, leaving out any name the broker no longer serves (approving a request lists every NOT_APPROVER case). |
invalid value "…" for flag -wait: parse error (exit 2) |
--wait takes a duration with its unit (90s, 5m, 1h), not a bare number. register --wait is the exception: it takes whole seconds. |
Add the unit. |
request … was expired or request … was cancelled |
Nobody decided it before it expired, or the session cancelled it (agent-secrets cancel). A session that ended reads none of its requests: its calls are refused PROOF_INVALID. |
Ask again. |
no agent secret has this name (UNKNOWN_SECRET) |
No secret the broker serves has the name: none exists under the namespace in that name (DEEL_API_KEY is <namespace>deel-api-key), or the broker leaves it out because a tag is missing or malformed, it is on the wrong key, or it has no value yet, which the broker logs by name (agent secret policy refused, Operating the broker). |
Check the spelling (agent-secrets secret list lists every agent secret); ask the secret’s owner, or whoever runs the broker, to fix its tags or key or to put its value (manage a secret). Ask again once it is fixed: the broker reads a name it does not serve again before refusing it (up to ten such rereads per session, refilled one every ten seconds), so the next request is served. |
the requested secrets need different approvers; request them separately (MIXED_APPROVERS) |
One command asked for secrets that different people approve. | Request them in separate commands. |
a request for one or more of these secrets is already pending: request … is still pending (REQUEST_PENDING) |
This session already has a request waiting for approval that names one of these secrets, with another set of names or another --ttl. A session has one pending request per secret; this holds for requests without --ttl too, so asking for A B while a request for A waits is refused. Only the identical request (the same names and the same --ttl, or none) joins the pending one. |
Wait for the named request to be decided (agent-secrets status <request id>), or end it with agent-secrets cancel <request id> and ask again with the names and --ttl you want. |
the grant released no value for …; nothing was run |
The broker granted a name but returned no value for it. | Tell whoever runs the broker. |
grant is expired, revoked, or its session ended (GRANT_NOT_LIVE) |
The grant ended between the decision and the read, one of its secrets has no value yet, or the secret was deleted or its tags changed since it was granted: the secret is now left out or denied, one granted automatically now needs approval, or it now belongs to someone who did not approve the grant. | For a secret with no value, its owner puts the value: agent-secrets secret set has the broker serve it at once, and a value put any other way resumes the grant within about ten minutes, as the broker rereads the namespace every five and Secrets Manager’s listing can lag a change by up to five more. Otherwise, run the command again to ask anew. |
secret is not in the secrets store (SECRET_NOT_IN_STORE) |
The secret was deleted from the secret store by something that did not ask the broker to reread it (the AWS console or the AWS CLI), after the broker last read the namespace: outright, or with a recovery window (their default, which keeps the secret until the window passes and refuses every read of its value meanwhile). A delete through agent-secrets secret delete is reread at once, so its grants end GRANT_NOT_LIVE instead. |
Ask again within about ten minutes: the broker rereads the namespace every five, and Secrets Manager’s listing can lag a change by up to five more. The request is then refused UNKNOWN_SECRET unless the secret is back (a secret within its recovery window is restorable: agent-secrets secret restore). |
request object invalid (REQUEST_INVALID) |
The broker could not verify the signed request: usually a clock far off, or an AGENT_SECRETS_URL that is not the broker’s public URL. |
Check the clock and the URL. |
Managing a secret
Section titled “Managing a secret”These are the messages of agent-secrets secret
(manage a secret). A usage error exits 2 and writes
nothing. SIGHUP, Ctrl-C, Ctrl-\ or SIGTERM at the value prompt ends the CLI by that signal
(a shell reports 129, 130, 131 or 143) and writes nothing; every other failure exits 1.
| Message | Cause | Fix |
|---|---|---|
read the broker's settings: … |
AGENT_SECRETS_URL does not reach the broker. |
Check the address; the CLI reads the namespace, key, account and region from it. |
load your AWS sign-in: failed to get shared config profile, … |
--profile or AWS_PROFILE names a profile your AWS config does not have. |
Name a profile from ~/.aws/config. |
read your AWS sign-in: … get credentials: failed to refresh cached credentials, … |
The AWS SDK found no usable credentials. For an IAM Identity Center profile the rest of the message says which: failed to read cached SSO token file when you have not signed in with that profile; when the sign-in has expired, the SSO session has expired or is invalid for a profile that names its own sso_start_url, or refresh cached SSO token failed for one that names an sso_session. |
Run aws sso login --profile <profile>, and name that profile with --profile or AWS_PROFILE. |
your AWS sign-in … is in account …, but the agent secrets are in account … |
The sign-in is in another account than the agent secrets: every form, a read included, refuses it. | Sign in to the account the message names. |
… is not a person's Identity Center sign-in: a secret is written under your own sign-in … |
A write ran under a machine’s role, an assumed service role or an IAM user. | Sign in as yourself through IAM Identity Center in the broker’s account. |
load your Google Cloud Application Default Credentials (run `gcloud auth application-default login`): … |
GCP: no Application Default Credentials are set up, or they are expired or malformed. | Run gcloud auth application-default login. |
your Google Cloud Application Default Credentials are not a person's sign-in (run `gcloud auth application-default login` instead of using a service account): a secret is written under your own sign-in, never a machine's |
GCP: a write ran under a service account key, or a GCE/GKE metadata-server or workload identity. | Run gcloud auth application-default login as yourself. |
"…": not an agent secret name: … (exit 2) |
The name is not uppercase letters, digits and single underscores starting with a letter. | Write the name as an agent asks for it: DEMO_DEPLOY_TOKEN, not demo-deploy-token. |
the value is read from standard input, which was empty or no value was entered at the prompt (exit 2) |
create or set got an empty value: empty standard input, or Enter or Ctrl-D alone at the prompt. Nothing was written. |
Pipe the value in, or type it at the prompt. |
the value must be valid UTF-8 (exit 2) |
The prompt or standard input contained bytes that Secrets Manager’s text value cannot preserve. Nothing was written. | Supply a UTF-8 text value. |
your sign-in's email … is not an email the broker accepts as an owner (exit 2) |
--owner me would write an owner tag the broker refuses: your AWS Identity Center session name, or your Google sign-in’s email, is not one. Nothing was written. |
AWS: sign in with an Identity Center user whose user name is your email. GCP: sign in (gcloud auth application-default login) with a Google Account whose email the broker accepts. |
the control byte 0x… cannot be typed at the prompt (exit 2) |
An unhandled control byte was typed or pasted. The prompt discards input through the line ending and any bracketed paste’s end before reporting the error. | Finish with Enter or Ctrl-D, then type the value again or pipe it in: agent-secrets secret set NAME < value.txt. |
nothing was stored: the terminal stopped while reading the value; run … again and type the whole value (exit 2) |
Ctrl-Z (or another stop) came while the value was being typed. The prompt discards what the terminal held unread when it stopped, so nothing that arrived with or before Ctrl-Z reaches your shell, and refuses the whole entry, even if some of it had already been read. Keys typed after the stop took effect went to your shell. | After fg, press Enter to finish discarding the entry, then run the command in the message and enter the whole value again. |
no shell can bring the value prompt to the foreground of this terminal; pipe the value in: … < FILE (exit 2) |
The command was started in the background (&) from a shell that has since exited, such as a subshell or the terminal’s own shell, so no shell can give it the terminal with fg. It showed no prompt and wrote nothing. |
Run the command in the message with the value piped in, or run the command again in the foreground. |
no value prompt is available on this platform; pipe the value in: … < FILE (exit 2) |
This build of agent-secrets has no hidden prompt: the prompt runs on Linux and macOS for amd64 and arm64. Nothing was written. |
Run the command in the message with the value piped in. |
read the value at the terminal: the terminal hung up before the value ended |
The terminal closed, or its ssh session dropped, while a value was still being typed outside a paste. Nothing was written. | Run the command again and enter the whole value, or pipe it in: agent-secrets secret set NAME < value.txt. |
a value of more than one line must be piped in: … (exit 2) |
The prompt found more than one line and wrote nothing. A bracketed paste is discarded through its closing mark. Without brackets, draining ends after 200 ms of quiet, with no total limit, and Ctrl-C ends it; a paste-start mark split before the line ends holds the drain for one second for the rest of the mark: later bytes can reach the shell, and a second line delayed past the quiet window can leave the first already stored without a refusal. | Pipe multi-line values in: agent-secrets secret set NAME < key.pem. This also replaces a first line a split paste stored. |
read the value at the terminal: the paste was cut short before its closing mark |
The terminal closed, or its ssh session dropped, while a paste was still arriving at the prompt, or a paste fell quiet for 10 seconds before its closing mark came. The bound counts the quiet since the last input, so a paste still arriving is read on, and input that arrived after the paste was given up reached your shell. Nothing was written. | Run the command again, and paste the whole value, or pipe it in: agent-secrets secret set NAME < key.pem. |
… CreateSecret, … ResourceExistsException: …, or InvalidRequestException from a create |
A secret of that name exists, perhaps scheduled for deletion. | secret set gives an existing secret a new value; secret restore brings back a deleted one. |
InvalidRequestException from set or retag |
The secret is scheduled for deletion, which Secrets Manager refuses to change. | secret restore it first. |
… AccessDeniedException: User: … is not authorized to perform: secretsmanager:… |
IAM refuses that call to your sign-in. Two causes: the secret is another person’s, and your access does not cover it; or your access does not include the permission the message names, which can happen on your own secret too. delete or restore refused on your own secret means your access is not granted secretsmanager:DeleteSecret or secretsmanager:RestoreSecret. Operating the broker lists each form’s permissions. |
For another person’s secret, ask its owner. For your own, ask whoever administers IAM in the broker’s account to grant the permission the message names. |
rpc error: code = PermissionDenied desc = … |
GCP: IAM refuses that call to your ADC identity. Same two causes as AWS’s AccessDeniedException: another person’s secret, or your own access missing the permission the call needs. Operating the broker lists each form’s GCP permissions. |
For another person’s secret, ask its owner. For your own, ask whoever administers IAM on the project to grant the permission the call needs. |
a shared secret's owner and tier are an administrator's to change, after AccessDeniedException (AWS) or PermissionDenied (GCP) from a retag |
The secret is shared, whose owner and tier only an administrator may change; or you asked to make your own secret shared (--owner shared) and your access does not include that retag. |
For a shared secret, ask an administrator. For your own, ask whoever administers IAM (AWS) or GCP IAM (GCP) to grant the retag. |
the broker refuses … after the write (reason=…), after broker: refusing … (reason=…) |
The write stands in the secret store, but the broker leaves the secret out for that reason: absent, or a refusal Operating the broker lists, such as no-current-value. |
Fix what the reason names (for no-current-value, secret set gives the secret a value) rather than repeating the write. |
the broker still serves … after its delete |
The delete stands in the secret store, but the broker’s reread found the secret served. | secret show it; once it shows deleted:, ask for the reread again (curl -X POST "$AGENT_SECRETS_URL/v1/secrets/<NAME>/reread"). |
the write to the secret store stands, but the broker could not be asked to reread … |
The broker was unreachable or rate-limited the reread (RATE_LIMITED). |
Do not repeat the write: what happens next depends on the write, and asking for the reread later serves it at once. |
… has no owner tag or no tier tag, and a retag sends both: name the missing one with --owner or --tier (exit 2) |
The secret is missing a tag, and retag sends both. |
Name the missing one too. |
Listing and revoking machine logins and grants
Section titled “Listing and revoking machine logins and grants”These are the messages of agent-secrets machine list|revoke and agent-secrets grant list|revoke
(revoke a session or a grant),
which act under this machine’s login. A usage error exits 2; every other failure exits 1.
| Message | Cause | Fix |
|---|---|---|
IN_SESSION: pid … is inside a registered machine session, which acts on itself alone; run machine and grant commands from your own shell |
The command ran inside an agent session’s process tree. The helper signs these commands with the machine’s login only for a process outside every session’s tree. The check covers that tree only: a process a session sends out of it (( cmd & ), setsid -f, a tmux server it started) passes. Through these commands it can list and revoke your own machine logins, never a service’s, and the operator’s grants. |
Run it from your own shell. A session ends its own grants with agent-secrets revoke. |
IN_SESSION: pid … may be inside a registered machine session: …; run machine and grant commands from your own shell |
The helper could not follow the command’s ancestry to init: it could not read a parent’s /proc/<pid>/stat, the walk looped, or it ran past 64 processes. The message names which. It refuses a caller it cannot show is outside every session. |
Run it from your own shell, started outside every session. |
this machine is not logged in to the secrets broker; run: agent-secrets machine login |
The helper holds no machine credential, so there is no login to act under. | Log the machine in. |
the launcher credential is not valid (LAUNCHER_INVALID); this machine's login is expired or revoked (run: agent-secrets machine login) |
The broker refused the machine’s credential: it expired, someone revoked it, or the clock or AGENT_SECRETS_URL is off. |
Log the machine in again. |
no agent-secrets-helper at … (…): machine and grant commands act under this machine's login, which its helper holds (exit 2) |
No helper listens on the socket. These commands run on a machine whose helper holds a login. | Start agent-secrets-helper and log it in, or point AGENT_SECRETS_HELPER_SOCK at its socket. |
this machine's agent-secrets-helper is older than this client and cannot sign for machine and grant commands; restart it on this release |
The helper running predates these commands. | Restart the helper on the same release as agent-secrets, then log the machine in again: the restart discards its credential. |
only the person who approved a person's machine login may revoke it or only the grant's approver or its enrollment's operator may revoke it (NOT_APPROVER) |
The login or grant is someone else’s. | Ask the person who approved it, or the session’s operator. |
no such machine login (NOT_FOUND) or no such grant (NOT_FOUND) |
No login or grant has that id, or the login is a service’s, which machine revoke neither lists nor ends. |
Copy the id from machine list or grant list. Revoke a service’s login on Dispatch’s machine-login page. |
launcher credential ids are UUIDs (CREDENTIAL_ID_INPUT) or grant ids are UUIDs (GRANT_ID_INPUT) |
The id is not a UUID. | Copy the id from machine list or grant list. |
Approving and logging in, in Dispatch
Section titled “Approving and logging in, in Dispatch”| Message | Cause | Fix |
|---|---|---|
only the record's approver may decide it (NOT_APPROVER) |
You are not the approver the request names, or you approved it after the secret’s owner changed while it waited (what an owner change does). A person’s machine login is that person’s alone to decide. | The named approver decides it. After an owner change, the new owner approves it if the request waited on anyone, and the approver it names can still deny it. |
only the person who approved a person's machine login may revoke it (NOT_APPROVER) |
You did not approve that person’s machine login. Machine logins lists your own machines’ and every service’s login, which anyone signed in may revoke. | Ask whoever approved it to revoke it. |
a service logs in with its own Kubernetes ServiceAccount token … (SERVICE_LOGS_IN_WITH_TOKEN, when starting it or approving it) |
A service’s login opened through a code, which the broker no longer accepts or approves: the Legion daemon logs in with its own Kubernetes token. | Deny it or let it expire. Upgrade a daemon that still opens one. |
request is already decided or this machine login has already been decided (RECORD_TERMINAL) |
It was decided already, or the session that asked has ended. | Nothing to do. |
… expired before its approver acted on it (RECORD_TERMINAL) |
It waited past its expiry: a request’s or a machine login’s. | The agent or the machine asks again. |
no pending machine login has this code (NO_SUCH_CODE) |
The code is mistyped, already decided, or expired. | Check the code on the machine’s terminal; start a new login if it expired. |
confirmation code does not match (CODE_MISMATCH) |
The decision carried a different code from the login it names. | Look the code up again and decide from that page. |
this machine login's key already holds a live launcher credential (KEY_HOLDS_LIVE_CREDENTIAL) |
The machine signed two logins with one key, and one is already approved. | Deny this one; the machine is already logged in. |
| No Credential requests section and no Live grants | Dispatch has no broker connected (DISPATCH_AGENT_SECRETS_URL is unset). |
See Operating the broker. |