Skip to content

terraform module docs cleanup #1809

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
DaedalusG wants to merge 8 commits into
base: main
Choose a base branch
from
Open

terraform module docs cleanup #1809

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
d994e04
minor typo fixups
DaedalusG Jul 8, 2026
c2bed34
update versions
DaedalusG Jul 8, 2026
baae788
better example machine types
DaedalusG Jul 8, 2026
8feb599
examples
DaedalusG Jul 8, 2026
1887826
update with new field options
DaedalusG Jul 8, 2026
7fdd6ff
split terraform docs into two seperate docs one for aws one for gcp
DaedalusG Jul 8, 2026
adb9b48
improve linking so the terraform options are more discoverable
DaedalusG Jul 8, 2026
f683ab8
docs validation addition updates
DaedalusG Jul 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
346 changes: 346 additions & 0 deletions docs/self-hosted/executors/deploy-executors-terraform-aws.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,346 @@
# Deploying Sourcegraph executors using Terraform on AWS

A [Terraform module](https://github.com/sourcegraph/terraform-aws-executors) is provided to
provision machines running executors on AWS.

See also: [Deploying on Google Cloud](/self-hosted/executors/deploy-executors-terraform-gcp)

## Basic Definition

The following is the minimum required definition to deploy an executor on AWS.

```terraform
module "executors" {
source = "sourcegraph/executors/aws"

# Find the latest version matching your Sourcegraph version here:
# https://github.com/sourcegraph/terraform-aws-executors/tags
version = "<version>"

availability_zone = "<availability zone to provision resources in>"

executor_sourcegraph_external_url = "<external url>"
executor_sourcegraph_executor_proxy_password = "<shared secret>"

# Either:
executor_queue_name = "<codeintel | batches>"
# Or:
executor_queue_names = ["codeintel", "batches"]

executor_instance_tag = "<tag to filter in monitoring>"
executor_metrics_environment_label = "<label to filter custom metrics>"
executor_use_firecracker = true
}
```

| Variable | Description |
| ---------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `availability_zone` | The AWS availability zone to create the instance in. |
| `executor_sourcegraph_external_url` | The public URL of your Sourcegraph instance. This corresponds to the `externalURL` value in your Sourcegraph instance's site configuration and must be resolvable from the provisioned executor compute resources. |
| `executor_sourcegraph_executor_proxy_password` | The access token corresponding to the `executors.accessToken` in your Sourcegraph instance's site configuration. |
| `executor_queue_name` | The single queue from which the executor should pull jobs - [`codeintel`](/code-navigation/auto-indexing) or [`batches`](/batch-changes/server-side). Either this or `executor_queue_names` must be set. |
| `executor_queue_names` | The multiple queues from which the executor should pull jobs - one or more of [`codeintel`](/code-navigation/auto-indexing) and [`batches`](/batch-changes/server-side). Either this or `executor_queue_name` must be set. |
| `executor_instance_tag` | A label tag to add to all the executors; can be used for filtering out the right instances in monitoring. |
| `executor_metrics_environment_label` | The value for environment by which to filter the custom metrics. |
| `executor_use_firecracker` | Whether to use [Firecracker](/self-hosted/executors/firecracker) sandboxing for job execution. Requires bare metal instances (e.g. `c5n.metal`). Defaults to `true`. |
| `private_networking` | If `true`, the executors and Docker registry mirror will live in a private subnet and communicate with the internet through a NAT Gateway. Defaults to `false`. See the [Private Single Executor](#private-single-executor) example. |
| `randomize_resource_names` | Use randomized names for resources. Defaults to `false`. Existing resources are updated in-place when enabled. |
| `permissions_boundary_arn` | The ARN of an IAM policy to use as the [permissions boundary](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_boundaries.html) for IAM roles and users created by the module. Optional. |
| `private_ca_cert_path` | Path to a private CA certificate file. Use this when executors need to communicate with a Sourcegraph instance that uses a certificate signed by a private/internal CA. Optional. |

See the [AWS Terraform module variables](https://github.com/sourcegraph/terraform-aws-executors/blob/main/modules/executors/variables.tf) for additional configurations.

## Terraform Version

The executor Terraform modules require Terraform `>= 1.1.0, < 2.0.0`.

## Permissions

Access to get and create in the following resources.

- Auto Scaling
- CloudWatch Logs
- EBS (EC2)
- EC2 (Elastic Compute Cloud)
- IAM (Identity & Access Management)
- VPC (Virtual Private Cloud)

## Supported Regions

- `us-east-1`
- `us-east-2`
- `us-west-1`
- `us-west-2`
- `eu-west-1`
- `eu-west-2`
- `eu-west-3`
- `eu-north-1`
- `eu-south-1`
- `eu-central-1`
- `ap-northeast-1`
- `ap-northeast-2`
- `ap-southeast-1`
- `ap-southeast-2`
- `ap-southeast-3`
- `ap-east-1`
- `ap-south-1`
- `sa-east-1`
- `me-south-1`
- `af-south-1`
- `ca-central-1`

## Examples

### Single Executor

Provisions a single executor to pull from the `codeintel` queue.

- [AWS example](https://github.com/sourcegraph/terraform-aws-executors/tree/main/examples/single-executor)

### Multiple Executors

Provisions two executors, one to pull from the `codeintel` queue and the other for the `batches` queue.

- [AWS example](https://github.com/sourcegraph/terraform-aws-executors/tree/main/examples/multiple-executors)

### Private Single Executor

Provisions a single executor in a private subnet (no public IP). A NAT Gateway is used for outbound internet traffic.

- [AWS example](https://github.com/sourcegraph/terraform-aws-executors/tree/main/examples/private-single-executor)

## Step-by-step Guide

The following is a step-by-step guide on provisioning a single `codeintel` executor on AWS.

### Provision

1. [Install Terraform](#terraform-version).
2. Install the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).
3. Run `aws configure` to set up your credentials and default region.
4. Set up your Sourcegraph instance's Site configuration for executors
1. Click on your profile picture in the top right corner
2. Select **Site admin**
3. Expand the **Configuration** section
4. Select **Site configuration**
5. Set the following,
- `"externalURL": "<URL>"`
- A URL that is accessible from the EC2 instance (e.g. a public URL such as `https://sourcegraph.example.com`)
- `"executors.accessToken": "<new long secret>"`
- Can be generated by running `cat /dev/random | base64 | head -c 20`
- The secret will be displayed as `REDACTED` once it's saved.
- `"codeIntelAutoIndexing.enabled": true`
- _This is only for `codeintel` executors._
5. Download the [example files](https://github.com/sourcegraph/terraform-aws-executors/tree/main/examples/single-executor).
6. Change the following in `providers.tf`
- `region` to the AWS region to provision the executor in
7. Change the following in `main.tf`
- `availability_zone` to an availability zone within your chosen region (e.g. `us-west-2a`)
- `executor_sourcegraph_external_url` to the URL configured in your instance's **Site configuration**
- `executor_sourcegraph_executor_proxy_password` to the access token configured in your instance's **Site configuration**
8. Run `terraform init` to download the Sourcegraph executor modules.
9. Run `terraform plan` to preview the changes that will occur to your AWS infrastructure.
10. Run `terraform apply` and enter "yes" after reviewing the proposed changes to create the executor resources.
- Ensure `terraform apply` exited with code 0 and did not print any errors
11. Go back to the site admin page, expand **Executors**, click **Instances**, and check to see if your executor shows up in the list with a green dot 🟢

### Validation

The following can be done to troubleshoot or double-check that the executor has been properly provisioned.

Ensure the executor instance is running. Either go to **EC2 → Instances** in the AWS Console or run the following command.

```shell
$ aws ec2 describe-instances \
--filters "Name=tag:executor_tag,Values=<your executor_instance_tag>" \
"Name=instance-state-name,Values=running" \
--query "Reservations[].Instances[].[InstanceId,InstanceType,State.Name,PrivateIpAddress]" \
--output table
-------------------------------------------------------------------
| DescribeInstances |
+----------------------+-------------+---------+------------------+
| i-0abc123def456789 | c5n.metal | running | 10.0.1.42 |
+----------------------+-------------+---------+------------------+
```

You can connect to the instance using [AWS Systems Manager Session Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html). The module attaches the `AmazonSSMManagedInstanceCore` policy to the executor IAM role, so SSM works out of the box.

```shell
aws ssm start-session --target i-0abc123def456789
```

Then run the following command to check if the service is running.

```shell
ubuntu@ip-10-0-1-42:~$ systemctl status executor
● executor.service - User code executor
Loaded: loaded (/etc/systemd/system/executor.service; enabled; preset: enabled)
Active: active (running) since Thu 2021-11-18 02:28:48 UTC; 19s ago
```

To check the logs, you can either query **CloudWatch Logs** in the AWS Console or run the following command while connected to the instance.

```shell
ubuntu@ip-10-0-1-42:~$ journalctl -u executor | less
```

Ensure the `EXECUTOR_FRONTEND_URL` and `EXECUTOR_FRONTEND_PASSWORD` in `/etc/systemd/system/executor.env` are correct.

```shell
cat /etc/systemd/system/executor.env
```

Ensure the instance can reach your `externalURL`:

```shell
ubuntu@ip-10-0-1-42:~$ curl <your externalURL here>
<a href="/sign-in?returnTo=%2F">Found</a>
```

### Configure Auto-indexing

1. Go to the **Site admin** page
2. Expand **Code graph**,
3. Select **Configuration**
4. Click **Create new policy**, and fill in:
- Name: `TEST`
- Click _add a repository pattern_
- Repository pattern #1: set this to an existing repository on your Sourcegraph instance (
e.g. `github.com/gorilla/mux`)
- Type: `HEAD`
- Retention: Disabled
- Auto-indexing: Enabled
5. Expand **Code graph**
6. Select **Auto-indexing**, and check to see if an indexing job has appeared. If nothing is there:
- Try clicking **Enqueue**
- Try setting a higher update frequency: `PRECISE_CODE_INTEL_AUTO_INDEXING_TASK_INTERVAL=10s`
- Try setting a lower delay: `PRECISE_CODE_INTEL_AUTO_INDEXING_REPOSITORY_PROCESS_DELAY=10s`
7. Once you have a completed indexing job, click **Uploads** and check to see that an index has been uploaded.
8. Once the index has been uploaded, you should see the **`PRECISE`** badge in the hover! 🎉

## Auto-scaling

> NOTE: Auto scaling is currently not supported
> when [downloading and running executor binaries yourself](/self-hosted/executors/deploy-executors-binary),
> and on managed instances when using self-hosted executors, since it requires deployment adjustments.

Auto-scaling of executor instances can help to increase concurrency of jobs, without paying for unused resources. With auto-scaling, you can scale down to 0 instances when no workload exist and scale up as far as you like and your cloud provider can support. Auto-scaling needs to be configured separately.

Auto-scaling makes use of **AutoScalingGroups** on AWS. Sourcegraph's `worker` service publishes a scaling metric (that is, the number of jobs in queue) to CloudWatch. Then, based on that reported value, the auto-scaler adds and removes compute resources to match the required amount of compute. The autoscaler will attempt to hold 1 instance running per each `executor_jobs_per_instance_scaling` items in queue.

For example, if `executor_jobs_per_instance_scaling` is set to `20` and the queue size is currently `400`, then `20` instances would be determined as required to handle the load. You might want to tweak this number based on the `machine_type`, `maximum_num_jobs` (concurrency per machine), and desired processing speed. See the [AWS variable definitions](https://github.com/sourcegraph/terraform-aws-executors/blob/main/modules/executors/variables.tf) for details.

With the Terraform variables `executor_min_replicas` and `executor_max_replicas`, you can configure the minimum and maximum number of compute machines to be run at a given time.

For auto-scaling to work, two things must be true:

1. `executor_min_replicas` must be `>= 0` and `executor_max_replicas` must be `> executor_min_replicas`.
2. The Sourcegraph instance (its `worker` service, specifically) needs to publish scaling metrics to CloudWatch.

For the latter to work, the Sourcegraph instance needs to be configured with the correct credentials that allow it to access AWS.

The `credentials` submodule in the [AWS executor module](https://github.com/sourcegraph/terraform-aws-executors/tree/main/modules/credentials) exists for that purpose. When used, the `credentials` module sets up an IAM user with permission to write CloudWatch metrics and returns the access key in the Terraform outputs.

Here's an example of how one would configure auto-scaling.

```terraform
module "executors" {
source = "sourcegraph/executors/aws"
version = "<version>"

# Basic configuration...

# Auto-scaling
executor_min_replicas = 0 # Spin down when not in use
executor_max_replicas = 30
executor_jobs_per_instance_scaling = 20
}

module "my-credentials" {
source = "sourcegraph/executors/aws//modules/credentials"
version = "<version>"

resource_prefix = "<optional prefix to add to created resources>"
# permissions_boundary_arn = "<ARN of IAM permissions boundary policy>"
}

output "metric_writer_access_key_id" {
value = module.my-credentials.metric_writer_access_key_id
}

output "metric_writer_secret_key" {
value = module.my-credentials.metric_writer_secret_key
sensitive = true
}
```

After running `terraform apply`, retrieve the credentials by running the following commands.

```shell
$ terraform output metric_writer_access_key_id
$ terraform output metric_writer_secret_key
```

### Configuring the Sourcegraph instance

The AWS EC2 auto-scaling groups configured by the Sourcegraph Terraform module respond to changes in metric values written to **CloudWatch**. The target Sourcegraph instance is expected to continuously write these values.

To write the scaling metric to **CloudWatch**, the `worker` service must have defined the following environment variables.

| Environment Variable | Description |
| --------------------------------------- | -------------------------------------------------------- |
| `EXECUTOR_METRIC_ENVIRONMENT_LABEL` | Same value as `executor_metrics_environment_label` |
| `EXECUTOR_METRIC_AWS_NAMESPACE` | Must be set to `sourcegraph-executor` |
| `EXECUTOR_METRIC_AWS_REGION` | The target AWS region |
| `EXECUTOR_METRIC_AWS_ACCESS_KEY_ID` | The value of the output of `metric_writer_access_key_id` |
| `EXECUTOR_METRIC_AWS_SECRET_ACCESS_KEY` | The value of the output of `metric_writer_secret_key` |

### Testing auto scaling

Once the environment variables have been set and the worker service has been restarted, you should be able to find the scaling metrics in **CloudWatch**.

To test if the metric is correctly reported: go to the **CloudWatch** metrics section. Under **All metrics**, select the namespace `sourcegraph-executor` and then the metric `environment`, `queueName`. Make sure there are entries returned.

Next, you can test whether the number of executors rises and shrinks as load spikes occur. Keep in mind that auto-scaling is not a real-time operation and usually takes a short moment and can have some delays between the metric going down and the desired machine count adjusting.

## Upgrading executors

Upgrading executors is relatively uninvolved. Simply follow the instructions below.
Also, check the [changelog](https://sourcegraph.com/changelog) for any Executors related breaking changes or new features or flags that you might want to configure. See [Executors maintenance](/self-hosted/executors/deploy-executors#Maintaining-and-upgrading-executors) for version compatibility.

### **Step 1:** Update the source version of the terraform modules

> NOTE: Keep in mind that only one minor version bumps are guaranteed to be disruption-free.

```diff
module "executors" {
source = "sourcegraph/executors/aws"

# Find the latest version matching your Sourcegraph version here:
# https://github.com/sourcegraph/terraform-aws-executors/tags
- version = "7.3.0"
+ version = "7.4.0"

availability_zone = "<availability zone>"

executor_sourcegraph_external_url = "<external url>"
executor_sourcegraph_executor_proxy_password = "<shared secret>"

# Either:
executor_queue_name = "<codeintel | batches>"
# Or:
executor_queue_names = ["codeintel", "batches"]

executor_instance_tag = "<tag to filter in monitoring>"
executor_metrics_environment_label = "<label to filter custom metrics>"
executor_use_firecracker = true
}
```

### **Step 2:** Reapply the terraform configuration

Simply reapply the terraform configuration and executors will be ready to go again.

```bash
terraform apply
```
Loading