Add installation/backup/restore to docs

docs/docsite/rst/administration/backup_restore.rst

@@ -0,0 +1,208 @@
+.. _ag_backup_restore:
+
+************************
+Backup and Restore
+************************
+
+.. index::
+   single: backup
+   single: restore
+   pair: backup; database
+   pair: restore; secret key
+
+Ascender is backed up through the Ascender Operator, which captures the database, the secrets that encrypt it, and the deployment configuration. `ascender-install <https://github.com/ctrliq/ascender-install>`_ drives that process and copies the result off the cluster for you, so use it if you installed with it.
+
+What a backup contains
+=======================
+
+A backup is three files:
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 1
+
+   * - File
+     - Contents
+   * - ``tower.db``
+     - The PostgreSQL database, as a ``pg_dump`` custom-format archive
+   * - ``secrets.yml``
+     - The secret key, admin password, database credentials, and any TLS, custom CA, and receptor secrets
+   * - ``awx_object``
+     - The Ascender resource specification
+
+Project files are not included. Projects that sync from source control are fetched again after a restore, so they need nothing. Manual projects, whose files exist only on the volume, need backing up separately.
+
+The secret key in ``secrets.yml`` is what makes the database readable. Restore it alongside the database and stored credentials work; restore the database on its own against a new deployment and every credential in it is unreadable, with no way to recover them.
+
+Backing up with the installer
+==============================
+
+Run the backup from the ``ascender-install`` directory, with the ``custom.config.yml`` you installed with::
+
+	./setup.sh -b
+
+The files land in ``ascender_install_artifacts/backups/``, in a timestamped directory, with a ``current`` symlink pointing at the newest. Copy that directory somewhere off the machine, because nothing does it for you, and treat it as credentials when you do.
+
+See `Backup and restore <https://github.com/ctrliq/ascender-install/blob/main/docs/configuration/backup_restore.md>`_ in the installer repository for the full procedure.
+
+.. note::
+
+	A backup needs a running deployment, because it executes commands inside a live task pod. Take them routinely rather than reaching for one after something has already broken.
+
+Restoring with the installer
+=============================
+
+Restore reads from ``ascender_install_artifacts/backups/current``, so put the backup you want there first::
+
+	./setup.sh -r
+
+.. warning::
+
+	Restoring is destructive. It scales the deployment down, deletes the PostgreSQL volume, and rebuilds the database from the backup. It also replaces the current administrator password with the one from the backup. There is no confirmation prompt.
+
+After a restore, confirm the pods come back and sign in with the administrator credentials from the backup, not the ones you were using before. If you are restoring into a disaster recovery deployment that should not run jobs, disable schedules before it starts reconciling.
+
+To restore onto a rebuilt or replacement machine, see `Restoring onto a rebuilt k3s node <https://github.com/ctrliq/ascender-install/blob/main/docs/configuration/backup_restore.md>`_ in the installer repository, or `Restoring onto a new cluster`_ below.
+
+Backing up and restoring manually
+===================================
+
+To back up and restore by hand, create the operator's ``AWXBackup`` and ``AWXRestore`` resources yourself. This is what ``ascender-install`` does on your behalf, with the addition of copying the files off the cluster.
+
+Taking a backup
+----------------
+
+Create an ``AWXBackup`` naming the deployment to back up::
+
+	apiVersion: awx.ansible.com/v1beta1
+	kind: AWXBackup
+	metadata:
+	  name: ascender-backup-2026-08-28
+	  namespace: ascender
+	spec:
+	  deployment_name: ascender-app
+	  backup_storage_requirements:
+	    requests:
+	      storage: 20Gi
+
+Give each backup a distinct name, with the date in it. Re-applying an existing ``AWXBackup`` updates that resource rather than producing a new backup, and the resources stay in the namespace as the record of what you have.
+
+``deployment_name`` is the only required field. The operator creates a persistent volume claim named ``<deployment_name>-backup-claim`` unless you point ``backup_pvc`` at one you made yourself. Set ``backup_storage_requirements``, because the default is empty and leaves the size to whatever your storage class does with an unset value.
+
+Apply it and wait for the operator to finish::
+
+	kubectl apply -f backup.yml
+	kubectl get awxbackup ascender-backup-2026-08-28 -n ascender -o yaml
+
+The backup is complete when ``status.backupDirectory`` and ``status.backupClaim`` are populated. The directory is timestamped, so the claim accumulates one per run.
+
+Copying a backup off the cluster
+---------------------------------
+
+The operator writes to the claim and stops there. To get the files out, run a pod that mounts the claim and copy from it::
+
+	kubectl run backup-access -n ascender --image=busybox:stable --restart=Never \
+	  --overrides='{"spec":{"containers":[{"name":"backup-access","image":"busybox:stable","command":["sleep","3600"],"volumeMounts":[{"name":"backup","mountPath":"/backups"}]}],"volumes":[{"name":"backup","persistentVolumeClaim":{"claimName":"ascender-app-backup-claim"}}]}}'
+
+	BACKUP_DIR=$(kubectl get awxbackup ascender-backup-2026-08-28 -n ascender \
+	  -o jsonpath='{.status.backupDirectory}')
+
+	kubectl cp ascender/backup-access:$BACKUP_DIR ./ascender-backup
+	kubectl delete pod backup-access -n ascender
+
+.. note::
+
+	A backup is a set of credentials. ``secrets.yml`` holds the secret key, the administrator password, the database credentials, and any TLS private keys, all recoverable in plain text. Once it is off the cluster it has none of the protection a Kubernetes secret gave it, so store it where you would store those credentials, restrict who can read it, and keep it out of source control and shared drives.
+
+Restoring from a backup resource
+---------------------------------
+
+Restore rebuilds the deployment from the backup, so remove the old one first. Confirm you have a copy of the backup off the cluster before you start.
+
+Find the database volume claim. Its name is derived from the PostgreSQL version and the deployment name, so read it rather than assuming it::
+
+	kubectl get pvc -n ascender
+
+Then delete the Ascender resource and that claim::
+
+	kubectl delete awx ascender-app -n ascender
+	kubectl delete pvc postgres-15-ascender-app-postgres-15-0 -n ascender
+
+.. warning::
+
+	Do not delete the namespace. The backup claim lives in it, so removing the namespace destroys the backup you are about to restore from.
+
+Then create an ``AWXRestore`` pointing at the backup::
+
+	apiVersion: awx.ansible.com/v1beta1
+	kind: AWXRestore
+	metadata:
+	  name: ascender-restore
+	  namespace: ascender
+	spec:
+	  deployment_name: ascender-app
+	  backup_name: ascender-backup-2026-08-28
+
+Naming ``backup_name`` works when the original ``AWXBackup`` resource is still in the cluster, because the operator reads the claim and directory from its status.
+
+Apply it and watch::
+
+	kubectl apply -f restore.yml
+	kubectl get pods -n ascender -w
+
+The operator applies the secrets first, then recreates the Ascender resource from the backed-up specification, then loads the database. Applying the secrets before the resource is what preserves the secret key, so the restored deployment can still read its own credentials.
+
+Restoring onto a new cluster
+-----------------------------
+
+After losing a cluster, there is no ``AWXBackup`` resource to name, so point the restore at the files directly.
+
+Install the operator and create the namespace first, as described in :ref:`in_manual_install`.
+
+Create a claim to hold the backup. On a new cluster nothing has made one yet, so this is the same shape the operator would have created::
+
+	kubectl apply -n ascender -f - <<'EOF'
+	apiVersion: v1
+	kind: PersistentVolumeClaim
+	metadata:
+	  name: ascender-app-backup-claim
+	spec:
+	  accessModes:
+	    - ReadWriteOnce
+	  resources:
+	    requests:
+	      storage: 20Gi
+	EOF
+
+Add ``storageClassName`` if the cluster has no default. Size it for the database dump, not the deployment.
+
+Copy the files onto the claim, which is the reverse of getting them off::
+
+	kubectl run backup-access -n ascender --image=busybox:stable --restart=Never \
+	  --overrides='{"spec":{"containers":[{"name":"backup-access","image":"busybox:stable","command":["sleep","3600"],"volumeMounts":[{"name":"backup","mountPath":"/backups"}]}],"volumes":[{"name":"backup","persistentVolumeClaim":{"claimName":"ascender-app-backup-claim"}}]}}'
+
+	kubectl cp ./ascender-backup ascender/backup-access:/backups/ascender-restore
+	kubectl delete pod backup-access -n ascender
+
+Then name the claim and the directory instead of a backup::
+
+	apiVersion: awx.ansible.com/v1beta1
+	kind: AWXRestore
+	metadata:
+	  name: ascender-restore
+	  namespace: ascender
+	spec:
+	  deployment_name: ascender-app
+	  backup_pvc: ascender-app-backup-claim
+	  backup_dir: /backups/ascender-restore
+
+``backup_dir`` is an absolute path, because the operator mounts the claim at ``/backups`` and checks that the directory exists before it starts. The three files must sit directly inside it.
+
+Apply it the same way. The restored deployment takes its name from ``deployment_name`` and its configuration from the ``awx_object`` file in the backup, so it comes back as it was rather than as a fresh install.
+
+External databases
+===================
+
+.. warning::
+
+	Backup and restore cover the database the operator manages. If you set ``ASCENDER_PGSQL_HOST`` to use your own PostgreSQL server, restore stops without doing anything, and backup runs without capturing your database. Back up an external database with your own tooling, and keep ``secrets.yml`` from an Ascender backup alongside it so the secret key survives.

docs/docsite/rst/administration/index.rst

@@ -31,6 +31,7 @@ Likewise content in this guide can be removed or replaced if it applies to funct
   metrics
   performance
   secret_handling
+  backup_restore
   security_best_practices
   awx-manage
   configure_ascender

docs/docsite/rst/administration/init_script.rst

@@ -3,15 +3,15 @@
 Starting, Stopping, and Restarting Ascender
 ---------------------------------------------
 
-To install Ascender: https://github.com/ansible/awx-operator/tree/devel/docs/installation
+To install Ascender, see :ref:`in_start`.
 
 .. these instructions will be ported over to here in the near future (TBD)
 
-To migrate from an old Ascender to a new Ascender instance: https://github.com/ansible/awx-operator/blob/devel/docs/migration/migration.md
+To migrate from an old Ascender to a new Ascender instance: https://github.com/ctrliq/ascender-operator/blob/devel/docs/migration/migration.md
 
 .. these instructions will be ported over to here in the near future (TBD)
 
-To upgrade you Ascender instance: https://github.com/ansible/awx-operator/blob/devel/docs/upgrade/upgrading.md
+To upgrade your Ascender instance, see :ref:`in_install`.
 
 .. these instructions will be ported over to here in the near future (TBD)
 
@@ -21,7 +21,7 @@ To restart an Ascender instance, you must first kill the container and restart i
 .. these instructions will need to be fleshed out (TBD)
 
 
-To uninstall you Ascender instance: https://github.com/ansible/awx-operator/blob/devel/docs/uninstall/uninstall.md
+To uninstall your Ascender instance: https://github.com/ctrliq/ascender-operator/blob/devel/docs/uninstall/uninstall.md
 
 .. these instructions will be ported over to here in the near future (TBD)
 

docs/docsite/rst/administration/instances.rst

@@ -149,7 +149,7 @@ Below are sample values used to configure each node in a mesh ingress topology:
      - [hop node]  
 
 
-In order to create a mesh ingress for Ascender, see the `Mesh Ingress <https://ansible.readthedocs.io/projects/awx-operator/en/latest/user-guide/advanced-configuration/mesh-ingress.html>`_ chapter of the AWX Operator Documentation for information on setting up this type of topology. The last step is to create a remote execution node and add the execution node to an instance group in order for it to be used in your job execution. Whatever execution environment image used to run a playbook needs to be accessible for your remote execution node. Everything you are using in your playbook also needs to be accessible from this remote execution node.
+In order to create a mesh ingress for Ascender, see the `Mesh Ingress <https://github.com/ctrliq/ascender-operator/blob/devel/docs/user-guide/advanced-configuration/mesh-ingress.md>`_ chapter of the Ascender Operator documentation for information on setting up this type of topology. The last step is to create a remote execution node and add the execution node to an instance group in order for it to be used in your job execution. Whatever execution environment image used to run a playbook needs to be accessible for your remote execution node. Everything you are using in your playbook also needs to be accessible from this remote execution node.
 
 .. image:: ../common/images/instances-job-template-using-remote-execution-ig.png
     :alt: Job template using the instance group with the execution node to run jobs.
@@ -334,13 +334,13 @@ The example health check shows the status updates with an error on node 'one':
 Using a custom Receptor CA
 ---------------------------
 
-Refer to the AWX Operator Documentation, `Custom Receptor CA <https://ansible.readthedocs.io/projects/awx-operator/en/latest/user-guide/advanced-configuration/custom-receptor-certs.html>`_ for detail.
+Refer to the Ascender Operator documentation, `Custom Receptor CA <https://github.com/ctrliq/ascender-operator/blob/devel/docs/user-guide/advanced-configuration/custom-receptor-certs.md>`_ for detail.
 
 
 Using a private image for the default EE
 ------------------------------------------
 
-Refer to the AWX Operator Documentation on `Default execution environments from private registries <https://ansible.readthedocs.io/projects/awx-operator/en/latest/user-guide/advanced-configuration/default-execution-environments-from-private-registries.html>`_ for detail.
+Refer to the Ascender Operator documentation on `Default execution environments from private registries <https://github.com/ctrliq/ascender-operator/blob/devel/docs/user-guide/advanced-configuration/default-execution-environments-from-private-registries.md>`_ for detail.
 
 
 Troubleshooting

docs/docsite/rst/administration/performance.rst

@@ -206,7 +206,7 @@ Capacity planning for Operator based Deployments
 .. index::
    pair: Operator; deployment
 
-For Operator based deployments, refer to `Ansible AWX Operator documentation <https://ansible.readthedocs.io/projects/awx-operator>`_.
+For Operator based deployments, refer to `Ascender Operator documentation <https://github.com/ctrliq/ascender-operator>`_.
 
 
 Monitoring Ascender

docs/docsite/rst/administration/secret_handling.rst

@@ -37,38 +37,31 @@ Secret handling for operational use
 
 Ascender contains the following secrets used operationally:
 
--  ``/etc/awx/SECRET_KEY``
+-  ``<deployment>-secret-key``
 
    -  A secret key used for encrypting automation secrets in the
-      database (see below). If the ``SECRET_KEY`` changes or is unknown,
+      database (see below). If the secret key changes or is unknown,
       no encrypted fields in the database will be accessible.
 
--  ``/etc/awx/awx.{cert,key}``
+-  The ingress TLS secret
 
-   -  SSL certificate and key for the Ascender web service. A
-      self-signed cert/key is installed by default; the customer can
-      provide a locally appropriate certificate and key.
+   -  SSL certificate and key for the Ascender web service, supplied
+      when the deployment is created.
 
--  Database password in ``/etc/awx/conf.d/postgres.py`` and message bus
-   password in ``/etc/awx/conf.d/channels.py``
+-  ``<deployment>-postgres-configuration``
 
-   -  Passwords for connecting to Ascender component services
+   -  Host, database name, and password for connecting to PostgreSQL
 
-These secrets are all stored unencrypted on the Ascender server, as they are all needed to be read by the Ascender service at startup
-in an automated fashion. All secrets are protected by Unix permissions, and restricted to root and the Ascender service user awx.
+These are Kubernetes secrets in the namespace Ascender is deployed to, readable by the Ascender service account so the service can read them at startup. Restrict who can read secrets in that namespace to control access to them.
 
-If hiding of these secrets is required, the files that these secrets are read from are interpreted Python. These files can be adjusted to retrieve these secrets via some other mechanism anytime a service restarts.
+List them with ``kubectl get secrets -n <namespace>``.
 
 .. note::
 
     If the secrets system is down, Ascender will be unable to get the information and may fail in a way that would be recoverable once the service is restored. Using some redundancy on that system is highly recommended.
 
 
-If, for any reason you believe the ``SECRET_KEY`` Ascender generated for you has been compromised and needs to be regenerated, you can run a tool from the installer that behaves much like Ascender backup and restore tool.
-
-To generate a new secret key, run ``setup.sh -k`` using the inventory from your install.
-
-A backup copy of the prior key is saved in ``/etc/awx/``.
+If you believe the secret key has been compromised, replacing it is not a routine operation. Everything already encrypted with it, meaning every stored credential, becomes unreadable the moment it changes. Plan to re-enter those credentials, and keep a copy of the old secret alongside a database backup taken before the change. See :ref:`ag_backup_restore`.
 
 
 Secret handling for automation use

docs/docsite/rst/administration/troubleshooting.rst

@@ -33,7 +33,7 @@ Error logging and extra settings
 
 Ascender server errors are streamed and not logged, however you may be able to pass them in on the Ascender spec file.
 
-With ``extra_settings``, you can pass multiple custom settings via the ``awx-operator``. The parameter ``extra_settings``  will be appended to the ``/etc/tower/settings.py`` file and can be an alternative to the ``extra_volumes`` parameter.
+With ``extra_settings``, you can pass multiple custom settings via the Ascender Operator. The parameter ``extra_settings``  will be appended to the ``/etc/tower/settings.py`` file and can be an alternative to the ``extra_volumes`` parameter.
 
 +----------------+----------------+---------+
 | Name           | Description    | Default |

docs/docsite/rst/index.rst

@@ -3,6 +3,12 @@ Ascender Documentation
 
 Ascender helps teams manage complex multi-tier deployments by adding control, knowledge, and delegation to Ansible-powered environments.
 
+.. toctree::
+   :maxdepth: 2
+   :caption: Install
+
+   installation/index
+
 .. toctree::
    :maxdepth: 2
    :caption: Get started