Skip to content

Update install instructions from guided-setup #489

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
HappyTetrahedron merged 1 commit into from
Jul 31, 2026
Merged

Update install instructions from guided-setup #489

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 16 additions & 14 deletions docs/modules/ROOT/partials/guided-setup/cloudscale-decommission.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -533,7 +533,7 @@ kubectl delete machineautoscaler -A --all
[[step-12]]
=== And I delete all persistent volumes

This step deletes all persistent volumes on the cluster, so that the corresponding Cloudscale resources can be decommissioned by the controller.
This step deletes all persistent volumes on the cluster, so that the corresponding CSP resources can be decommissioned by the controller.

By cordoning all non-master nodes and deleting all their pods (except the csi driver pods) we ensure that no new PVs are created, while the existing ones can be cleaned up. Deleting all pods has the additional benefit that we don’t have to deal with PDBs when deleting the machinesets in the next step.

Expand Down Expand Up @@ -659,6 +659,11 @@ GIT_AUTHOR_EMAIL=$(git config --global user.email)
HIERADATA_REPO_TOKEN=${INPUT_gitlab_api_token}
EOF

if [[ ! -d catalog ]]
then
commodore catalog compile "$INPUT_commodore_cluster_id"
fi

tf_image=$(\
yq eval ".parameters.openshift4_terraform.images.terraform.image" \
dependencies/openshift4-terraform/class/defaults.yml)
Expand Down Expand Up @@ -838,7 +843,10 @@ terraform init \
"-backend-config=unlock_method=DELETE" \
"-backend-config=retry_wait_min=5"

terraform state rm "module.cluster.module.lb.module.hiera[0].gitfile_checkout.appuio_hieradata"
if terraform state list | grep "gitfile_checkout.appuio_hieradata" > /dev/null
then
terraform state rm "module.cluster.module.lb.module.hiera[0].gitfile_checkout.appuio_hieradata"
fi

# Suppress errors on the first run; it is expected to fail
terraform destroy --auto-approve || true
Expand Down Expand Up @@ -1120,6 +1128,7 @@ In this step you need to remove the LoadBalancer servers from control.vshn.net

* `lb_fqdn_1`
* `lb_fqdn_2`
* `control_vshn_api_token`

==== Script

Expand All @@ -1129,20 +1138,13 @@ OUTPUT=$(mktemp)

# export INPUT_lb_fqdn_1=
# export INPUT_lb_fqdn_2=
# export INPUT_control_vshn_api_token=

set -euo pipefail
echo '###################################################################################'
echo '# #'
echo "# Please manually delete the cluster's LoadBalancer servers before proceeding. #"
echo '# #'
echo '###################################################################################'
echo
echo You can go to:
echo https://control.vshn.net/servers/definitions/appuio/${INPUT_lb_fqdn_1}/delete
echo https://control.vshn.net/servers/definitions/appuio/${INPUT_lb_fqdn_2}/delete
sleep 2
# NOTE(aa): This step is currently annoying to automate, but once ticket PORTAL-253 is resolved,
# it should be easy.
echo "Deleting server definitions ..."
curl -XDELETE -H"$AUTH" https://control.vshn.net/api/servers/1/appuio/${INPUT_lb_fqdn_1}
curl -XDELETE -H"$AUTH" https://control.vshn.net/api/servers/1/appuio/${INPUT_lb_fqdn_2}
echo "Server definitions deleted."


# echo "# Outputs"
Expand Down
129 changes: 89 additions & 40 deletions docs/modules/ROOT/partials/guided-setup/exoscale-decommission.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,20 +10,21 @@
* <<step-8,Then I disable the OpsGenie heartbeat>>
* &nbsp;&nbsp;<<step-9,And I disable Project Syn>>
* &nbsp;&nbsp;<<step-10,And I delete all Load Balancer services>>
* &nbsp;&nbsp;<<step-11,And I prepare the cluster repository>>
* <<step-12,Then I save the loadbalancer metadata>>
* &nbsp;&nbsp;<<step-13,And I downtime the loadbalancers in icinga>>
* &nbsp;&nbsp;<<step-14,And I decommission Terraform resources>>
* &nbsp;&nbsp;<<step-15,And I delete all S3 buckets>>
* &nbsp;&nbsp;<<step-16,And I delete the cluster backup>>
* &nbsp;&nbsp;<<step-17,And I delete the cluster's API tokens>>
* &nbsp;&nbsp;<<step-18,And I remove the LoadBalancers from control.vshn.net>>
* &nbsp;&nbsp;<<step-19,And I decommission the LoadBalancers>>
* &nbsp;&nbsp;<<step-20,And I remove the cluster's DNS entries>>
* <<step-21,Then I delete the cluster's Vault secrets>>
* &nbsp;&nbsp;<<step-22,And I delete the cluster's OpsGenie heartbeat>>
* &nbsp;&nbsp;<<step-23,And I delete the cluster from Lieutenant>>
* &nbsp;&nbsp;<<step-24,And I delete the Keycloak service>>
* &nbsp;&nbsp;<<step-11,And I delete all persistent volumes>>
* &nbsp;&nbsp;<<step-12,And I prepare the cluster repository>>
* <<step-13,Then I save the loadbalancer metadata>>
* &nbsp;&nbsp;<<step-14,And I downtime the loadbalancers in icinga>>
* &nbsp;&nbsp;<<step-15,And I decommission Terraform resources>>
* &nbsp;&nbsp;<<step-16,And I delete all S3 buckets>>
* &nbsp;&nbsp;<<step-17,And I delete the cluster backup>>
* &nbsp;&nbsp;<<step-18,And I delete the cluster's API tokens>>
* &nbsp;&nbsp;<<step-19,And I remove the LoadBalancers from control.vshn.net>>
* &nbsp;&nbsp;<<step-20,And I decommission the LoadBalancers>>
* &nbsp;&nbsp;<<step-21,And I remove the cluster's DNS entries>>
* <<step-22,Then I delete the cluster's Vault secrets>>
* &nbsp;&nbsp;<<step-23,And I delete the cluster's OpsGenie heartbeat>>
* &nbsp;&nbsp;<<step-24,And I delete the cluster from Lieutenant>>
* &nbsp;&nbsp;<<step-25,And I delete the Keycloak service>>

[[step-1]]
=== Given I have all prerequisites installed
Expand Down Expand Up @@ -503,6 +504,52 @@ echo '# Deleted Services. #'
----

[[step-11]]
=== And I delete all persistent volumes

This step deletes all persistent volumes on the cluster, so that the corresponding CSP resources can be decommissioned by the controller.

By cordoning all non-master nodes and deleting all their pods (except the csi driver pods) we ensure that no new PVs are created, while the existing ones can be cleaned up. Deleting all pods has the additional benefit that we don’t have to deal with PDBs when deleting the machinesets in the next step.

==== Inputs

* `kubeconfig_path`

==== Script

[source,bash]
----
OUTPUT=$(mktemp)

# export INPUT_kubeconfig_path=

set -euo pipefail
export KUBECONFIG="${INPUT_kubeconfig_path}"
kubectl cordon -l node-role.kubernetes.io/worker
kubectl get po -A -oyaml | yq '.items = [.items[] |
select(.spec.nodeName | test("master-") | not) |
select(.metadata.namespace != "syn-csi-cloudscale")]' |\
kubectl delete --wait=false -f-
kubectl delete pvc -A --all --wait=false
kubectl wait --for=delete pv --all --timeout=120s

echo '########################################################'
echo '# #'
echo '# Please verify that all PVs were deleted properly. #'
echo '# #'
echo '########################################################'
echo
echo If the cluster still has PVs, please manually run the following:
echo " "kubectl delete pv --all
sleep 2


# echo "# Outputs"
# cat "$OUTPUT"
# rm -f "$OUTPUT"

----

[[step-12]]
=== And I prepare the cluster repository

This step prepares the local cluster repository by cloning the Commodore hieradata repository
Expand Down Expand Up @@ -582,7 +629,7 @@ commodore catalog compile ${INPUT_commodore_cluster_id} --push \

----

[[step-12]]
[[step-13]]
=== Then I save the loadbalancer metadata

This step gathers metadata on the LoadBalancer instances (such as their icinga zone and backup server), such that they can be properly decommissioned down the line.
Expand Down Expand Up @@ -639,6 +686,11 @@ GIT_AUTHOR_EMAIL=$(git config --global user.email)
HIERADATA_REPO_TOKEN=${INPUT_gitlab_api_token}
EOF

if [[ ! -d catalog ]]
then
commodore catalog compile "$INPUT_commodore_cluster_id"
fi

tf_image=$(\
yq eval ".parameters.openshift4_terraform.images.terraform.image" \
dependencies/openshift4-terraform/class/defaults.yml)
Expand Down Expand Up @@ -705,7 +757,7 @@ popd

----

[[step-13]]
[[step-14]]
=== And I downtime the loadbalancers in icinga

In this step you have to configure downtimes in Icinga for the cluster's load balancers.
Expand Down Expand Up @@ -744,7 +796,7 @@ sleep 2

----

[[step-14]]
[[step-15]]
=== And I decommission Terraform resources

This step decommissions all Terraform resources for the cluster.
Expand Down Expand Up @@ -824,7 +876,10 @@ terraform init \
"-backend-config=unlock_method=DELETE" \
"-backend-config=retry_wait_min=5"

terraform state rm "module.cluster.module.lb.module.hiera[0].gitfile_checkout.appuio_hieradata"
if terraform state list | grep "gitfile_checkout.appuio_hieradata" > /dev/null
then
terraform state rm "module.cluster.module.lb.module.hiera[0].gitfile_checkout.appuio_hieradata"
fi

# Suppress errors on the first run; it is expected to fail
terraform destroy --auto-approve || true
Expand All @@ -839,7 +894,7 @@ popd

----

[[step-15]]
[[step-16]]
=== And I delete all S3 buckets

This step deletes the cluster's associated S3 buckets from Exoscale.
Expand Down Expand Up @@ -878,7 +933,7 @@ exo storage rb -r -f "${INPUT_commodore_cluster_id}-logstore" || true

----

[[step-16]]
[[step-17]]
=== And I delete the cluster backup

This step deletes the cluster's associated backup bucket from Exoscale.
Expand Down Expand Up @@ -990,7 +1045,7 @@ exo storage rb -r -f "${INPUT_commodore_cluster_id}-backup" || true

----

[[step-17]]
[[step-18]]
=== And I delete the cluster's API tokens

This step deletes the cluster's associated Exoscale API tokens from Exoscale.
Expand Down Expand Up @@ -1031,7 +1086,7 @@ exo iam api-key delete -f "${INPUT_commodore_cluster_id}" || true

----

[[step-18]]
[[step-19]]
=== And I remove the LoadBalancers from control.vshn.net

In this step you need to remove the LoadBalancer servers from control.vshn.net
Expand All @@ -1040,6 +1095,7 @@ In this step you need to remove the LoadBalancer servers from control.vshn.net

* `lb_fqdn_1`
* `lb_fqdn_2`
* `control_vshn_api_token`

==== Script

Expand All @@ -1049,20 +1105,13 @@ OUTPUT=$(mktemp)

# export INPUT_lb_fqdn_1=
# export INPUT_lb_fqdn_2=
# export INPUT_control_vshn_api_token=

set -euo pipefail
echo '###################################################################################'
echo '# #'
echo "# Please manually delete the cluster's LoadBalancer servers before proceeding. #"
echo '# #'
echo '###################################################################################'
echo
echo You can go to:
echo https://control.vshn.net/servers/definitions/appuio/${INPUT_lb_fqdn_1}/delete
echo https://control.vshn.net/servers/definitions/appuio/${INPUT_lb_fqdn_2}/delete
sleep 2
# NOTE(aa): This step is currently annoying to automate, but once ticket PORTAL-253 is resolved,
# it should be easy.
echo "Deleting server definitions ..."
curl -XDELETE -H"$AUTH" https://control.vshn.net/api/servers/1/appuio/${INPUT_lb_fqdn_1}
curl -XDELETE -H"$AUTH" https://control.vshn.net/api/servers/1/appuio/${INPUT_lb_fqdn_2}
echo "Server definitions deleted."


# echo "# Outputs"
Expand All @@ -1071,7 +1120,7 @@ sleep 2

----

[[step-19]]
[[step-20]]
=== And I decommission the LoadBalancers

This step decommissions resources associated with the Puppet managed LoadBalancers.
Expand Down Expand Up @@ -1195,7 +1244,7 @@ echo "# Deleted backups from Burp server. #"

----

[[step-20]]
[[step-21]]
=== And I remove the cluster's DNS entries

In this step, you must manually remove any DNS entries associated with the cluster from https://git.vshn.net/vshn/vshn_zonefiles.
Expand All @@ -1222,7 +1271,7 @@ sleep 2

----

[[step-21]]
[[step-22]]
=== Then I delete the cluster's Vault secrets

This step cleans up all the cluster's Vault secrets.
Expand Down Expand Up @@ -1294,7 +1343,7 @@ done

----

[[step-22]]
[[step-23]]
=== And I delete the cluster's OpsGenie heartbeat

This step deletes the cluster's OpsGenie heartbeat.
Expand Down Expand Up @@ -1333,7 +1382,7 @@ curl "https://api.opsgenie.com/v2/heartbeats/${INPUT_commodore_cluster_id}" \

----

[[step-23]]
[[step-24]]
=== And I delete the cluster from Lieutenant

This step deletes the cluster from Lieutenant
Expand Down Expand Up @@ -1365,7 +1414,7 @@ curl -H "Authorization: Bearer $(commodore fetch-token)" -XDELETE \

----

[[step-24]]
[[step-25]]
=== And I delete the Keycloak service

This step deletes the cluster's keycloak service from control.vshn.net
Expand Down