> ## Documentation Index > Fetch the complete documentation index at: https://docs.bytebase.com/llms.txt > Use this file to discover all available pages before exploring further. # Recover Administrator Access > [!NOTE] > > Available only for: > > * Self-hosted Bytebase 3.22.0 and later. The steps on this page need 3.24.0 or later. For 3.22.0 to 3.23.x, see [Earlier versions](#earlier-versions). > * Shell access to the host, or `kubectl` access to the cluster, that runs Bytebase If nobody can sign in to Bytebase, run `bytebase recovery` on the server to turn password sign-in back on or to set a new password. **Recovery needs no downtime. Wait one minute before you sign in.** Bytebase keeps running while you recover, and it picks up the change within one minute, without a restart. If a **Workspace Admin** can still sign in, reset passwords, roles, and 2FA from the console instead. ## Choose what to run | Situation | Run | | - | - | | An administrator knows their password, but **Disallow signin with email & password** is on | `enable-password-signin` | | A user forgot their password, or is locked out after too many failed attempts | `reset-password` | | **Disallow signin with email & password** is on, and no administrator knows a password, for example because they all use SSO | `reset-password` for an administrator, then `enable-password-signin` | | A user was removed from **Members** and has no workspace role | The [interactive menu](#use-the-interactive-menu), to reset the password and add a role | Recovery can't: * **Create or reactivate a user.** The user must already exist and be active. Service accounts and workload identities don't work. * **Change a role that a user already has.** The menu adds a role only to a user with no workspace role at all: not directly, not through a group, and not through `allUsers`. When such a user tries to sign in, Bytebase gives them the **Workspace Member** role, even if the sign-in fails, so add the role first. If no **Workspace Admin** is left and every user already has a role, recovery can't restore administrator access. * **Reset two-factor authentication.** Only another **Workspace Admin** can [reset a user's 2FA](https://docs.bytebase.com/administration/2fa.md#reset-a-users-two-factor-authentication). * **Bypass the workspace domain.** When **Members restriction** is on, the email must belong to an allowed [workspace domain](https://docs.bytebase.com/administration/sign-in-restriction.md#sign-in-from-email-domains). If none of these fits your situation, [contact us](https://www.bytebase.com/contact-us/). ## Run recovery **No downtime, then wait one minute.** Leave Bytebase running: don't stop it before recovery, and don't restart it afterward. After the command succeeds, wait one minute for Bytebase to pick up the change before you sign in. Run the command inside the container that runs Bytebase. That container already knows how to reach the metadata database, where Bytebase stores its users and settings, so you don't pass any connection details: no `PG_URL`, no volume, and no network flags. Back up the metadata database first. A snapshot from a managed PostgreSQL service needs no downtime. The procedures in [Back up and restore](https://docs.bytebase.com/get-started/self-host/upgrade.md#back-up-and-restore) stop Bytebase while they copy. **Docker** The commands use the container name `bytebase`. Replace it with yours. 1. Check the version: ```bash theme={null} docker exec bytebase bytebase version ``` If it's older than 3.24.0, follow [Earlier versions](#earlier-versions) instead. 2. Run the command for your situation: ```bash theme={null} docker exec bytebase bytebase recovery enable-password-signin ``` ```bash theme={null} docker exec bytebase bytebase recovery reset-password --email --password '' ``` If you start Bytebase with your own `--data` or `--port`, add the same flags at the end of the command. `docker exec` doesn't pass them on. **Kubernetes** The commands work for both the Helm chart and the manifest from [Deploy with Kubernetes](https://docs.bytebase.com/get-started/self-host/deploy-with-kubernetes.md). Both create a Pod named `bytebase-0` with a container named `bytebase`. With several replicas, use any one of them. 1. Check the version: ```bash theme={null} kubectl -n exec bytebase-0 -c bytebase -- bytebase version ``` If it's older than 3.24.0, follow [Earlier versions](#earlier-versions) instead. 2. Run the command for your situation: ```bash theme={null} kubectl -n exec bytebase-0 -c bytebase -- bytebase recovery enable-password-signin ``` ```bash theme={null} kubectl -n exec bytebase-0 -c bytebase -- bytebase recovery reset-password --email --password '' ``` > **The chart's escapePassword option is on** > > With the experimental `escapePassword` option, the server's start script builds `PG_URL`, so a command started with `kubectl exec` doesn't have it. Build it the same way before you run recovery: > > ```bash theme={null} > kubectl -n exec bytebase-0 -c bytebase -- /bin/sh -c 'export PG_URL=postgres://:$(cat /mnt/bytebase-shared-volume/pg_escaped_password)@:/; exec bytebase recovery enable-password-signin' > ``` If you can't use `kubectl exec`, for example because you change the cluster only through manifests or a pipeline, [run recovery as a Job](#run-recovery-as-a-kubernetes-job). **Binary** For a `bytebase` binary that runs directly on a host. Unlike a container, the command doesn't share the server's settings, so you pass them yourself. 1. Check the version: ```bash theme={null} bytebase version ``` If it's older than 3.24.0, follow [Earlier versions](#earlier-versions) instead. 2. Run the command with the same binary, as the same OS user that runs the server. For embedded PostgreSQL, pass the server's `--data` as an absolute path, and its `--port` if it sets one: ```bash theme={null} bytebase recovery enable-password-signin --data ``` For external PostgreSQL, pass the server's `PG_URL`: ```bash theme={null} PG_URL='' bytebase recovery reset-password --email --password '' ``` If the server isn't running, recovery starts the embedded PostgreSQL itself. Don't start Bytebase until the command exits. Each command does one thing and exits: * `enable-password-signin` switches off **Disallow signin with email & password**. It works only when an active **Workspace Admin** has a password, so that someone can sign in afterward. * `reset-password` sets a new password that meets the workspace [password restriction](https://docs.bytebase.com/administration/password.md), and clears the lockout from failed attempts. It doesn't change the user's role, 2FA, or SSO sign-in. When it works, the command prints one line that ends with `Live within one minute.`, for example: ```text theme={null} Password updated for user admin@example.com. Live within one minute. ``` * If `reset-password` also prints `User has no workspace role`, add a role with the [interactive menu](#use-the-interactive-menu) before the user signs in. * If the command prints `Error:`, read the reason in [Troubleshooting](#troubleshooting). When the success line appears above the error, your change was saved, and only a follow-up step failed, such as writing the audit log. * The command exits with status 0 only when everything succeeded. Running it again is safe: the second run changes nothing. > [!WARNING] > > The password you pass with `--password` is saved in your shell history, and other users on the host can see it while the command runs. Change it after you sign in. To keep it hidden, use the [interactive menu](#use-the-interactive-menu) instead. ## After recovery 1. **Wait one minute, then sign in** with the email and password. Bytebase applies the change within one minute. If you sign in earlier, the new password can be rejected, and five rejected attempts within 10 minutes lock the account for 10 minutes. If that happens, run the same `reset-password` command again to clear the lock, then wait one minute. 2. If you passed the password with `--password`, change it. 3. Repair and test the identity provider. 4. Switch **Disallow signin with email & password** back on in **Settings** > **General** > **Account** if it was on before. 5. Review the recovery events in the [audit log](https://docs.bytebase.com/security/audit-log.md). Each action records a warning-level event whose method starts with `/bytebase.cli.Recovery/`. ## Run recovery as a Kubernetes Job Use a Job when you can't run `kubectl exec`. The Job's Pod runs the command once, beside the server. It must use the same image as the server and connect to the metadata database the same way. A Job only works with external PostgreSQL, which is what the chart and the manifest use. 1. Get the exact image the server runs: ```bash theme={null} kubectl -n get pod bytebase-0 -o jsonpath='{.status.containerStatuses[?(@.name=="bytebase")].imageID}' ``` Use the printed value as ``, without the `docker-pullable://` prefix if there is one. Don't copy the tag from the StatefulSet: a tag such as `latest` can pull a newer version than the server runs. 2. For `reset-password`, store the new password in a Secret, so it isn't written in the manifest: ```bash theme={null} kubectl -n create secret generic bytebase-recovery --from-literal=password='' ``` 3. Create the Job. This example is for a server that reads `PG_URL` from a Secret, which is what the chart's `existingPgURLSecret` option sets up: ```yaml theme={null} apiVersion: batch/v1 kind: Job metadata: name: bytebase-recovery namespace: spec: backoffLimit: 0 template: spec: restartPolicy: Never serviceAccountName: containers: - name: recovery image: args: ["recovery", "reset-password", "--email", "", "--password", "$(NEW_PASSWORD)"] env: - name: PG_URL valueFrom: secretKeyRef: name: key: - name: NEW_PASSWORD valueFrom: secretKeyRef: name: bytebase-recovery key: password ``` * `` is the service account of the server Pod. * Keep each argument as its own item, as shown. If you write the whole command as one item, it fails before recovery runs, and the error message shows the password. * Kubernetes fills in `$(NEW_PASSWORD)` from the Secret. If the variable isn't defined, the command receives `$(NEW_PASSWORD)` as is and refuses it. * For `enable-password-signin`, use `args: ["recovery", "enable-password-signin"]` and remove `NEW_PASSWORD`. * `backoffLimit: 0` stops Kubernetes from retrying a failed run. Never run recovery as a Deployment or StatefulSet, which would repeat it every time it exits. 4. Read the result. The Job succeeds when the command exits with status 0: ```bash theme={null} kubectl -n logs job/bytebase-recovery ``` 5. After you can sign in, delete the Job and the Secret: ```bash theme={null} kubectl -n delete job/bytebase-recovery secret/bytebase-recovery --ignore-not-found ``` Then follow [After recovery](#after-recovery). > **The server doesn't read PG\_URL from a Secret** > > Copy what the Job needs from the server Pod's spec, which `kubectl -n get pod bytebase-0 -o yaml` prints: > > | How the server connects | What to copy into the Job | > | - | - | > | `PG_URL` written as a value, as in the manifest from [Deploy with Kubernetes](https://docs.bytebase.com/get-started/self-host/deploy-with-kubernetes.md), or the chart's `url` or `pgHost`, `pgUsername`, and `pgPassword` options | The server container's `PG_URL` entry, as it is | > | The chart's `existingPgPasswordSecret` option | The server container's `PG_PASSWORD` and `PG_URL` entries, in that order, because `PG_URL` refers to `$(PG_PASSWORD)` | > | The chart's `awsRdsIam` or `gcpCloudSqlIam` option | The server container's `PG_URL` entry, and the server Pod's service account, which carries the cloud identity | > | A `PG_URL` that is a file path, such as a mounted Secret | The volume and the mount that hold the file, at the same path | > | Mounted TLS certificates or other files | The same volumes and mounts | > | The chart's experimental `escapePassword` option | The server Pod's init container, its `bytebase-shared-volume` volume, and a script that builds `PG_URL`, as below | > > With `escapePassword`, the init container writes the escaped password to the shared volume. Keep the init container and the volume, and give the recovery container the mount and this script: > > ```yaml theme={null} > command: ["/bin/sh", "-c"] > args: > - | > export PG_URL=postgres://:$(cat /mnt/bytebase-shared-volume/pg_escaped_password)@:/ > exec bytebase recovery reset-password --email '' --password "$NEW_PASSWORD" > volumeMounts: > - name: bytebase-shared-volume > mountPath: /mnt/bytebase-shared-volume > ``` > > Inside the script, write `"$NEW_PASSWORD"` without parentheses, so the shell doesn't treat any character of the password as part of the script. ## Use the interactive menu Use the menu to add a role to a user, or to type the password where nobody can see it. Run `bytebase recovery` with no action, in the same place as the commands above, and add `-it` to `docker exec` or `kubectl exec` to get a terminal. Over SSH, connect with `ssh -t`. ```bash theme={null} docker exec -it bytebase bytebase recovery ``` The menu stays open until you enter `q`, so you can run several actions in a row. This example resets a password: ```text theme={null} Select a recovery action: 1. Enable password sign-in 2. Reset user password q. Exit > 2 User email (blank to cancel): admin@example.com Password requirements: - Minimum length: 8 characters New password: Confirm new password: Resetting user password... Password updated for user admin@example.com. Live within one minute. ``` * **Enable password sign-in** asks you to confirm with `y`, then does what `enable-password-signin` does. * **Reset user password** does what `reset-password` does. If the user has no workspace role, it then offers to add one: confirm, pick a role from the list, and confirm again. If you decline, the new password stays. Each action is saved as soon as it finishes, so you can exit at any time. Then follow [After recovery](#after-recovery). ## Troubleshooting ### Terminal required ```text theme={null} Error: recovery requires terminal input and output; without a terminal, run "bytebase recovery enable-password-signin" or "bytebase recovery reset-password --email --password " ``` The interactive menu needs a terminal, so that the password stays hidden. Add `-it` to `docker exec` or `kubectl exec`, or run one of the two commands, which need no terminal. ### Embedded metadata database may be in use ```text theme={null} Error: failed to open recovery metadata store: cannot connect to the embedded metadata database (...), and its data directory "/var/opt/bytebase/pgdata" may be in use: run recovery inside the Bytebase server's container, with the server's --data and --port; if no Bytebase server is running, start it once, then run recovery that way ``` Recovery couldn't reach the server's embedded PostgreSQL. Usually it runs in a second container that mounts the data volume, or with a different `--port` than the server. Run it in the server's container, with the server's `--data` and `--port`. If no server is running, the lock file is left over from a crash: start Bytebase once, then run recovery in its container. ### Data directory not accessible ```text theme={null} Error: failed to open recovery metadata store: unable to access --data directory /srv/bytebase: stat /srv/bytebase: no such file or directory ``` The `--data` path doesn't exist where recovery runs. Pass the same `--data` as the server, as an absolute path. ### Embedded metadata database not initialized ```text theme={null} Error: failed to open recovery metadata store: embedded metadata database is not initialized at "/var/opt/bytebase/pgdata" ``` This error appears only when `PG_URL` isn't set. The `--data` directory isn't the one the server uses. Pass the server's `--data`. If the server uses external PostgreSQL, pass its `PG_URL` instead. ### Embedded metadata database not readable ```text theme={null} Error: failed to open recovery metadata store: failed to inspect embedded metadata database: stat /srv/bytebase/pgdata/PG_VERSION: permission denied ``` Recovery runs as a different OS user than the server. Run it as the user that owns the data directory. ### External metadata database not connected ```text theme={null} Error: failed to open recovery metadata store: failed to connect to external metadata database: ... ``` This error appears only when `PG_URL` is set. The text after the last colon names the cause. Recovery needs everything the server uses to reach PostgreSQL: the network, the file when `PG_URL` is a file path (`failed to read database URL from file`), TLS files, or IAM credentials. Running in the server's container provides all of them. For a Job, compare it with the server Pod's spec. ### Unknown command ```text theme={null} Error: unknown command "recovery" for "bytebase" ``` The version is older than 3.22.0. [Upgrade](https://docs.bytebase.com/get-started/self-host/upgrade.md) to 3.24.0 or later, then run recovery. ### Unknown subcommand ```text theme={null} Error: unknown command "enable-password-signin" for "bytebase recovery" ``` The version is 3.22.0 to 3.23.x, which has only the interactive menu. `reset-password` fails the same way, with `Error: unknown flag: --email`. Follow [Earlier versions](#earlier-versions). ### No usable workspace administrator ```text theme={null} Error: no usable workspace administrator has an active end-user identity with a password credential ``` `enable-password-signin` needs an active **Workspace Admin** with a password. Run `reset-password` for an administrator, then try again. If no administrator is left, use the [interactive menu](#use-the-interactive-menu) to reset the password of a user with no workspace role and add them as **Workspace Admin**. If every user already has a role, recovery can't restore administrator access. ### Change not applied after one minute Recovery saved the change to a different metadata database than the servers use. Make sure recovery uses the same `PG_URL` as the servers, including the database name. ## Earlier versions On 3.22.0 to 3.23.x, `bytebase recovery` has only the interactive menu, and the servers don't see its change until they restart. So recovery needs downtime. Alternatively, [upgrade](https://docs.bytebase.com/get-started/self-host/upgrade.md) to 3.24.0 or later first, then follow [Run recovery](#run-recovery). > **Recover on 3.22.0 to 3.23.x** > > Because the server is stopped, recovery runs in a new, one-off container. Unlike `docker exec`, that container starts with nothing, so you pass it the same image and the same metadata database settings as the server. > > 1. Write down the exact version and image the server runs. Recovery must use the same version, and the restart must not upgrade Bytebase. > > 2. Stop every Bytebase server that uses the metadata database, and keep them stopped until recovery exits. On Kubernetes, scale the StatefulSet to `0`, and pause `helm upgrade` and any GitOps sync, which would restore the replicas. > > 3. Back up the metadata: the [data directory](https://docs.bytebase.com/get-started/self-host/upgrade.md#back-up-the-data-directory) for embedded PostgreSQL, or the [metadata database](https://docs.bytebase.com/get-started/self-host/upgrade.md#back-up-the-metadata-database) for external PostgreSQL. > > 4. Run `bytebase recovery` in a one-off container, with a terminal. For embedded PostgreSQL with Docker, mount the same data volume as the server: > > ```bash theme={null} > docker run --rm -it --volume :/var/opt/bytebase recovery > ``` > > For external PostgreSQL with Docker, pass `-e 'PG_URL='` instead of the volume. On Kubernetes, pass the same `PG_URL`, and press Enter if the menu doesn't appear: > > ```bash theme={null} > kubectl -n run bytebase-recovery --rm -it --restart=Never --image= --env='PG_URL=' --command -- bytebase recovery > ``` > > Also pass anything else the server uses to reach PostgreSQL: the network, a mounted connection-string file or TLS certificate, and IAM credentials. On Kubernetes, when `PG_URL` comes from a Secret, a file, or IAM authentication, create a Pod with the same `env`, `volumes`, `volumeMounts`, and `serviceAccountName` as the StatefulSet's Pod template, then `kubectl exec -it` into it. With a binary on a host, run the same binary as the same OS user, with the same `--data` or `PG_URL`. > > 5. Use the menu as described in [Use the interactive menu](#use-the-interactive-menu). On these versions, nothing takes effect until you restart Bytebase. > > 6. Start Bytebase again on the image you wrote down, without a floating tag such as `latest` and without `--pull always`. On Kubernetes, pin the tag first, then restore the replica count. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.