feat(oci): multi-cloud provider seam with a working Oracle Cloud target - #21
Open
MikaAK wants to merge 30 commits into
Open
feat(oci): multi-cloud provider seam with a working Oracle Cloud target#21MikaAK wants to merge 30 commits into
MikaAK wants to merge 30 commits into
Conversation
Adds a provider seam to deploy_ex and implements Oracle Cloud behind it, without
changing a byte of what an AWS user renders. The full OCI chain is verified live
against a real tenancy:
mix terraform.build → init → plan → apply → drop
mix ansible.build → ping → setup → deploy
mix deploy_ex.upload
A release was fetched by the node itself via instance principals, unpacked,
swapped into place, started under systemd, and its release-state written back to
object storage — with no credential on disk.
## The seam
Dispatch is DERIVED from provider descriptor modules rather than written into a
dispatcher: `DeployEx.Cloud` holds no capability module names at all, so adding a
provider costs one registry entry plus one descriptor. Capabilities a provider
has not implemented return `{:error, %ErrorMessage{code: :not_implemented}}`
instead of crashing on a nil module.
Provider-specific templates live under `priv/**/providers/<name>/`, and the AWS
file set is defined by ONE complement rule — everything not under a `providers/`
directory — so gates never need per-provider rewording. The rule tests a path
COMPONENT, not a substring, because an AWS tree legitimately contains a root
`providers.tf`.
## Object storage on OCI
Goes through the `oci` CLI's native API, not OCI's S3-compatibility endpoint.
Three measured reasons: ExAws cannot reach OCI at all (its partition table is
compile-time, so an OCI region can never be registered, and signing with an AWS
region returns 403 because OCI requires the OCI region inside the SigV4
credential scope); the compat endpoint does NOT accept instance principals, so
using it would put a long-lived Customer Secret Key on every node; and no usable
OCI SDK exists for Elixir or Erlang (`ex_oci_sdk` covers only the Queue service,
and "OCI" in the Erlang ecosystem means Oracle Call Interface, a database driver).
`ReleaseUploader.AwsManager` now resolves through `Cloud.capability(:object_store)`,
so `mix deploy_ex.upload` works on whichever provider is configured. Verified in
both directions in one run: 93 keys from an AWS bucket and 11 from an OCI bucket
through the same function.
ClickHouse is the exception that proves the rule — its `s3` disk is a compiled
binary that only speaks S3, so its cold tier must use the compat endpoint with
Customer Secret Keys. Two different auth surfaces on the same storage.
## Silent-failure bugs fixed along the way (these affect AWS too)
Four separate cases of "the work failed, the tool said fine":
- A failed ansible play exited 0. `Task.async_stream` reports task COMPLETION, so
a run_fn returning `{:error, _}` arrived as `{:ok, {:error, _}}` and matched the
reducer's SUCCESS clause. Both `ansible.setup` and `ansible.deploy`, both the
console and TUI paths. In CI: a green pipeline that deployed nothing.
- `mix terraform.drop` exited 0 when the destroy failed, leaving resources alive
and billing while reporting success. `terraform.refresh` had the same shape.
Mix does not fail a task on run/1's return value — only a raise does.
- `mix ansible.build` silently skipped template writes whose contents differed and
still exited 0, so a rebuilt tree kept a stale inventory and setup playbook.
- A timed-out playbook died with a bare "(EXIT) time out" naming nothing, because
`Task.async_stream` defaults to killing the caller rather than yielding.
Plus two paginated list calls that returned partial results as if complete — S3
`list_objects` returning 1000 of 7138 keys, and EC2 `describe_instances` returning
one page of four.
## Ansible fixes, mostly AWS-affecting
- `grafana_alloy` passed `args: warn: false`, removed in ansible-core 2.14 and a
hard error since.
- `grafana_alloy` needs `unzip`, which it never declared — the `awscli` role
happened to install it, so swapping in `oci_cli` on OCI exposed the dependency.
- `ipv6` wrote `~/.aws/config` into a directory nothing created, for the same
reason.
- Setup started right after `terraform.apply` raced cloud-init's
unattended-upgrades for the dpkg lock; `beam_linux_tuning` now waits.
- `save_ami` reads the EC2 metadata endpoint and has no OCI counterpart, so it is
gated on provider like `awscli`/`oci_cli`.
- The OCI release-lookup scripts printed the CLI's error banner to stdout on
failure, and the role took it as the release name — an auth failure surfaced as
a 404 on an object literally named "ServiceError:".
## Terraform
OCI renders per-app instances from the same `<app>_project` map the AWS templates
use, plus a release bucket, and a dynamic group and policy for instance
principals. Identity resources go through a second provider aliased to the HOME
region, because OCI routes every IAM write there regardless of where resources
live — without it an apply fails partway, after the network already exists.
The generated map now emits provider-appropriate keys. It previously shipped
AWS-only fields into OCI trees that the module never read, so
`instance_type = "t3.micro"` sat there looking authoritative and changed nothing.
Terraform also generates the SSH keypair on OCI, matching AWS, because
`mix ansible.build` looks for one under a `<kebab-project-name>*pem` glob and the
failure mode without it is opaque.
## ClickHouse
New role: pinned apt install, config.d/users.d fragments, systemd, bound to
`database_*_clickhouse`. Cold tier off by default and rendered whole-or-not-at-all,
since a fragment naming a dead endpoint is a boot risk and a table naming an
undefined policy fails NO_SUCH_POLICY. Verified live: SELECT 1 over TCP and HTTP,
default user reachable over the configured CIDR, and cold-tier-enabled boot with
the expected tiered/hot/s3_cold policy shape.
Loopback is listed unconditionally in allowed networks: ClickHouse's `<networks>`
check is literal and never implicitly permits 127.0.0.1, so on-box healthchecks
fail AUTHENTICATION_FAILED without it.
## Compatibility
The AWS render is byte-identical to before this branch except three deliberate
shared-role fixes above. Verified by rendering both revisions and diffing, not by
inspection. 699 tests; the 6 failures are pre-existing and unrelated.
…ble generators mix deploy_ex.export_priv produced a wrong tree on any non-AWS project. PrivRenderer copied all of priv/terraform — including a raw, unrendered providers/oci/ — and rendered only top-level *.eex, so an OCI project got the AWS templates plus unrendered .eex files, and an AWS project got providers/oci/*.tf in its terraform root where tofu would try to load them. It also carried its OWN copies of the six variable generators. Those had drifted: terraform.build's were made provider-aware, PrivRenderer's still emitted AWS-only keys, so an OCI export got instance_type/enable_ebs fields the OCI instance module never reads — values that look authoritative and do nothing. Extracted to DeployEx.TerraformVariables so there is one copy and the next change cannot diverge again. Ansible export is provider-aware too: it renders the provider's ansible.cfg and inventory rather than always AWS's, and strips the providers/ subtree so one provider's exclusive files never land in another's export. The OCI inventory is written EMPTY on purpose — it is a point-in-time snapshot of a live compartment that an export cannot know, and mix ansible.build fills it in. Emitting invented hosts would be worse than an obviously empty file. Also passes cloud_provider into the setup-playbook render, which raised "assign @cloud_provider not available" once that template became provider-aware. VERIFIED: exports for both providers contain only their own files with zero unrendered .eex; the OCI variables.tf has 0 AWS-only sizing keys and 16 OCI ones; and the AWS render from terraform.build/ansible.build is byte-identical to before the extraction.
…al works Every oci os call relied on the CLI resolving the namespace internally. That resolution costs a tenancy read, which a correctly-scoped CI credential does not have — MEASURED with a user holding only object permissions: Error: Unable to retrieve namespace internally. Please provide the namespace using the option "--['namespace-name']". Auto-resolution only appears to work when the credential is over-privileged, so the bug is invisible until someone does the right thing and scopes the CI user down. Verified end to end afterwards: mix deploy_ex.upload succeeds under env-var auth as a user that can write release objects and nothing else. Mutation-tested — removing the flag from list_objects fails the new test.
MEASURED in CI against an umbrella whose six releases are role-named (server,
ingestion, service, pipeline, polling, migrate):
** (FunctionClauseError) no function clause matching in IO.chardata_to_string/1
# 1
nil
(elixir) lib/path.ex:671: Path.join/2
(deploy_ex) lib/mix/tasks/deploy_ex.release.ex:223
ProjectContext.app_path/2 looks the name up in apps_paths() and returns nil when
a release is not also an app directory. A release BUNDLES applications and is
commonly named for its role rather than for any one app, so this is normal
project structure, not misconfiguration — but it killed the entire build on the
first release attempted.
Both call sites now skip the asset steps when there is no app directory to search
in. A release whose name does match an app resolves exactly as before, so
projects where the two coincide are unaffected.
… tagging MEASURED in CI: every release on a qa/ branch failed with "oci object storage has no object tagging" AFTER the object had already uploaded, so the job went red with all six artifacts sitting correctly in the bucket. Object tagging is an OPTIONAL ObjectStore capability. OCI has none — its nearest equivalent is user metadata, settable only at put time — so a provider without it must not fail the upload. Nothing is lost by skipping it. The tag is write-only inside deploy_ex: @qa_tag_key appears at its definition and at the single write site and nowhere else. What actually marks a QA release is the key prefix, which drives remote-release lookup and is provider independent — the uploaded keys are already qa/server/..., qa/ingestion/... and so on. The tag exists for external tooling, so its absence is logged rather than swallowed. Not unit-tested: AwsManager.upload/4 and tag_object/4 build their own store opts and drop the caller's, so there is no seam to inject a stub store through, and a test would have to shell out to a real oci CLI. Verified by the live CI run instead. Threading opts through AwsManager would make this testable and is worth doing separately.
…n OCI Adds DeployEx.Cloud.OciSecurityGroup behind DeployEx.Cloud.Security and routes mix deploy_ex.ssh.authorize through Cloud.capability(:security) rather than calling AwsSecurityGroup/AwsIpWhitelister directly. This was a live blocker: the GitHub Actions deploy workflow had to drop its ssh.authorize/-r pair because the task was AWS-only, which meant a runner could not open its own IP and instances would have to leave SSH permanently open instead. Uses network security groups, not security lists. The deciding reason is mutation safety, not blast radius: / act on individual rules, while a security list has no such API — updating one REPLACES the whole rule set, so any implementation on that path risks silently deleting the rules terraform created. On blast radius the two are a wash as wired here, since the same NSG is attached to every instance's VNIC, mirroring both the shared subnet and AWS's single shared security group. Idempotency diverges from AWS deliberately. MEASURED live: is idempotent by rule content — adding twice yields one rule and the same id — so authorize does not pre-check and returns :ok. AWS instead reports a conflict. Revoking a CIDR with no matching rule is :ok on both. Terraform gains an NSG with no default rules plus an nsg_ids wiring through the oci-instance module, so the group exists for the task to target. Live-verified end to end against the real tenancy by driving the compiled task: authorize added the SSH rule and left a pre-existing HTTPS rule untouched, revoke removed exactly that rule, and re-running each was a harmless no-op. Every resource created was torn down and the compartment audited empty with table output. 718 tests, the same 6 pre-existing failures, and the AWS render is byte-identical.
The wizard-parity test asserts every mix task switch is exposed in the TUI. It has been failing on main, which means it caught nothing — including the switches this branch added. The provider flag was invisible in the wizard, so there was no way to pick a cloud from the TUI at all. Adds the missing inputs: provider on terraform.build, ansible.build, ansible.deploy and ansible.setup, the five oci_* options on ansible.build, and the pre-existing gaps this branch did not create (wait/timeout/poll_interval on autoscale.refresh_status, only_local_release on qa.deploy, and aws_region/instance_id/git_branch on ansible.setup). The test now passes, so drift from here on is caught rather than absorbed into an already-red assertion. Suite goes from 6 pre-existing failures to 5; the remaining five are AwsInfrastructureTest calling functions that no longer exist, which is a separate pre-existing issue on main.
…tually exists The suite has been red on main for a while. Both causes were tests that did not follow their code: find_iam_instance_profile/1 accepts a :request_fn seam, but its test passed none, so `mix test` HIT THE REAL AWS IAM API. It failed with a list of actual instance profiles from whatever account was configured — meaning the suite depended on network, credentials, and the contents of someone's AWS account. parse_key_pairs_response/2 was deleted when key-pair parsing moved inline into find_key_pair_name/1, but its four tests stayed behind calling a function that no longer exists. find_key_pair_name/1 had no injection seam at all — it called ExAws.request directly — so the current behaviour could not be tested without a live account. Added :request_fn there, matching find_subnet_ids/1 and find_iam_instance_profile/1. The tests now exercise the current public API against stubbed responses: newest matching key pair wins, a single-element key set (which AWS returns unwrapped) parses, another project's keys are ignored, an empty set is not_found rather than a crash, and the IAM lookup short-circuits on a configured profile and reports what IS available when the default is missing. Suite goes from 6 failures to 0. A green suite is the point — with it red I spent this session unable to tell my own regressions from inherited ones.
Renders a backend "s3" block into the OCI providers.tf pointed at the configured state bucket via the region's compat endpoint. Gated on both terraform_backend :s3 and an :oci release_state_bucket being configured, so existing OCI projects keep local state until they opt in. The default state key follows the oracle/<region>/<stack>/terraform.tfstate layout already in use in the bucket, so deploy_ex state sits alongside states written by other tooling. Auth is a Customer Secret Key surfaced as an AWS credentials profile (state_profile) — the compat endpoint accepts neither OCI signatures nor instance principals. No lock table: OCI has no DynamoDB, so concurrent applies are unguarded. Oci.backend_template/0 flips nil -> :s3 accordingly.
Adds a resource_databases map to the OCI render backed by OCI Database with PostgreSQL — the managed analogue of the AWS RDS path. Regionally durable storage (no AD pinning), E5 flex shape sized via ocpus/memory, admin password generated into (remote, private) state like the AWS random_password approach. Reachability is a dedicated NSG admitting 5432 from the VCN CIDR: the subnet security list only admits SSH and OCI filters intra-subnet traffic too, so subnet membership alone would not connect the nodes. Static .tf rather than .eex — an empty map creates nothing, so the template needs no render-time gating.
…-time captures Module attributes capture DeployEx.Config values when the DEP compiles, so a consuming project changing its config silently renders stale defaults until a forced deps.compile. Mix.env() had the same problem for :env — always :dev in a local shell regardless of `config :deploy_ex, :env`, which rendered an environment="dev" default while the state key (via Config.env) said prod.
Oracle prices E6 identically to E5 ($0.03/OCPU-hr + $0.002/GB-hr, measured from the public price API) with newer silicon, so E5 as a default is strictly dominated. E4 stays available via tfvars for cost-floor cases.
…irst seed The static (non-.eex) copy ran only when ./deploys/terraform did not exist, so an existing tree never received new provider files or module updates while the .eex renders around it DID update — producing a render that passes module variables its stale modules do not declare (measured: instance.tf's nsg_ids against a module missing the variable, and database.tf never arriving at all). Static files now sync through write_file every build: identical contents skip silently, changed contents prompt (or --force), declined overwrites warn loudly instead of vanishing.
…AD region Single-AD regions (ap-seoul-1, ap-chuncheon-1) reject isRegionallyDurable=true with 400-InvalidParameter and require an explicit availabilityDomain. Default flips to AD-pinned with a regionally_durable opt-in key for 3-AD regions. Measured on a live apply against ap-seoul-1.
The psql service rejects a public subnet with 400 "is not a private subnet" (measured on live apply). The DB now gets a dedicated private subnet gated on resource_databases: no public IPs and a route table with no routes, reachable only intra-VCN through the database NSG.
A fresh random pem name every build made each rebuild+apply replace the key file under a new name while ansible.cfg still pointed at the previous one — every node UNREACHABLE with "no such identity" (measured live). The build now reuses whatever name the existing render carries; the random suffix only seeds the first build.
The role was Debian-only in three ways, all measured on the first prod setup against OCI Ubuntu 24.04: redis_version pins a +deb13u1 revision that Ubuntu's archive does not carry; the stack path bolts a bullseye repo on for libssl1.1, which Ubuntu 24.04 does not ship; and packages.redis.io publishes no redis-stack-server for noble at all. Redis 8 folded the stack modules into redis-server itself, so an Ubuntu host with redis_stack_enabled now installs redis-server 8.x from packages.redis.io under its own codename and runs it under the plain redis-server unit (restarted, so the managed conf takes effect over the package's stock start). Every Debian task keeps its exact prior behaviour behind a distribution guard; the stack-binary service dance is keyed off one derived fact, redis_use_stack_binary.
The Debian units run redis as root and /data arrives from the EBS mount role. Ubuntu's package unit runs as User=redis under ProtectSystem, so redis.conf's `dir /data` failed the restart with a bare "No such file or directory" (measured on the prod redis node). Creates /data owned by redis and whitelists it for the unit via a systemd drop-in.
Ubuntu's redis-server unit is Type=notify; with `supervised no` redis comes up (PONG) but never signals ready and systemd times the start out. Debian's units are Type=simple, where `supervised no` is right, so the template branches on distribution and Debian renders unchanged.
Renders a DatabaseKey-tagged clickhouse project entry (E6, 2 OCPU/16GB, 200GB boot) into the OCI project map, which routes the node into the database_*_clickhouse inventory group the existing setup playbook targets. Opt-in rather than opt-out like redis: ClickHouse is not part of the default stack on either provider — an AWS user hand-writes the entry — so an OCI render must not grow a node nobody asked for. AWS rendering is unchanged.
…nstances Unlike AWS, OCI applies the security list to traffic between instances on the same subnet. With only the SSH rule, an app node could not reach redis, clickhouse or any BEAM peer (measured: 6379 refused server → redis on the prod fleet). Adds the analogue of the AWS app security group's self rule: all protocols from the VCN CIDR; the public edge stays gated.
OCI platform images ship an iptables INPUT chain that REJECTs everything but SSH, persisted via netfilter-persistent — so with the VCN wide open an app node still gets "No route to host" from redis, clickhouse and every BEAM peer (measured on the prod fleet). Debian AMIs on AWS have no such default. beam_linux_tuning now flushes the chain to accept-all and rewrites the persisted ruleset, guarded on Ubuntu + rules.v4 present + a REJECT actually in INPUT, so Debian and already-open hosts are untouched. Network policy stays with the cloud.
…S VPC default clickhouse_allowed_network_cidr defaulted to 10.0.0.0/16 (the AWS VPC), so on OCI the server got AUTHENTICATION_FAILED from clickhouse purely because 10.20.x was not in <networks> (measured, same class as the monitoring 10.0.1.x constants). vcn_cidr is now a config :deploy_ex, :oci key that BOTH the terraform variable default and the OCI ansible group_vars render from, so they cannot drift.
Redis 8 enforces protected mode against every non-loopback client when no password is set — the server node's first boot got -DENIED from redis over the VCN (measured). Reachability is the cloud's job (VCN security list, no public 6379 at the edge), so protected mode is off on the Ubuntu path; Debian's stack path keeps the upstream default.
…mq_server role Single node, on purpose: the app declares quorum queues, which are correct on one node (Raft majority of 1); more nodes buy HA and a 2-node layout is the one to avoid (majority of 2 is 2, losing either halts every queue). Install path is GitHub Releases artifacts — Erlang 27.3 from rabbitmq/erlang-debian-package (tarball of debs per Ubuntu codename) and rabbitmq-server 4.1.8 — because RabbitMQ's apt mirrors were unresolvable or 404 from both a laptop and an OCI node, and Ubuntu's archive carries 3.12. Chain verified live on an Ubuntu 24.04 node before being written down: Erlang 27 + 4.1.8 active, management plugin, vhost, users, permissions, HTTP API auth semantics. Provisioning is plain rabbitmqctl (deploy_ex takes no collection deps), guarded so reruns do not churn passwords. The management-API user gets the `management` tag plus read+write on the vhost — BrokerMgmt publishes through the HTTP API — and no configure. Empty passwords fail the play loudly instead of shipping a guessable credential.
… seed Setup playbooks were copied only when ./deploys/ansible was first created, so a role added to deploy_ex later synced its role tree into an existing project but never its setup/<name>.yaml — `mix ansible.setup --only rabbitmq` matched no file and exited 0 having done nothing (measured). Playbooks already on disk are user-owned and stay untouched; only absent ones are seeded.
…ad-timeout Seven parallel ~30MB uploads to OCI from a GitHub runner raced a hard 60s Task.async_stream ceiling: six landed, the seventh's timeout killed the whole stream (`Task.Supervised.stream(60000) ** (EXIT) time out`, measured on a real prod build), so the run went red AFTER most artifacts were in the bucket — a partial release set. Timeout is now generous, per release, kills only the offending task, and is a switch.
…ad_balancer.enable Sprint OCI-LB-1: load_balancer.tf (NSG + rules, NLB with explicit is_private=false + optional reserved_ips, http/https backend sets with TCP-fallback health checks, listeners), module inputs with null-passthrough health knobs, load_balancer_public_ips output, nsg_ids splat concat in main.tf. 14 render-content tests, every assertion mutation-proven.
Sprint OCI-LB-2: instance.tf.eex passes vcn_id + the 8 load_balancer keys + reserved_ip_ocid via try() with contract defaults (neither port nor instance_port — dead on AWS, fixed 80/443 both providers); load_balancer_public_ips root output keyed by app; load_balancer added to the <app>_project recognized keys. Render tests T11-T17.
…guide Sprint OCI-LB-3: commented load_balancer block in the :oci release variables (AWS clause byte-identical, pinned by test); providers/oci README documents the NLB shape, every ignored AWS key with reason, the enable-only gate divergence, NSG posture, and adopted-tree operator rules (never --force, rebuild flags are tree identity, enable-bit persistence); guide drops the dead instance_port example.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Re-opens the OCI multi-cloud work as a PR: #20's squash merge was reverted on main (e06649a) so this lives in PR form until sign-off; the running opgg prod fleet pins the branch tip directly, so nothing deployed changes.
Everything from #20 plus the follow-ups proven live on the opgg prod fleet: provider seam, OCI object storage via the native CLI, S3-compat terraform state backend, managed PostgreSQL, opt-in ClickHouse and RabbitMQ nodes, Ubuntu-aware redis role, OCI host-firewall and intra-VCN fixes, stable pem names, static-file and setup-playbook sync fixes, upload timeout fix. 733 tests.