Skip to content

feat(elasticsearch): enhance Elasticsearch output to support multiple endpoints #3451

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Clee2691 wants to merge 1 commit into
base: master
Choose a base branch
from
Open

feat(elasticsearch): enhance Elasticsearch output to support multiple endpoints #3451

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,3 +90,5 @@ tags
# End of https://www.gitignore.io/api/go,vim,emacs,visualstudiocode
.cache/
.target/

.claude/settings.local.json
32 changes: 28 additions & 4 deletions api/observability/v1/output_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -508,7 +508,7 @@ type Cloudwatch struct {
// The 'username@password' part of `url` is ignored.
//
// +kubebuilder:validation:Optional
// +kubebuilder:validation:XValidation:rule="self == '' || isURL(self)", message="invalid URL"
// +kubebuilder:validation:XValidation:rule="self == '' || isURL(self)", message="invalid URL"
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Destination URL",xDescriptors={"urn:alm:descriptor:com.tectonic.ui:text"}
URL string `json:"url,omitempty"`

Expand Down Expand Up @@ -676,8 +676,32 @@ type ElasticsearchTuningSpec struct {
Compression string `json:"compression,omitempty"`
}

// EndpointURL is a URL to an Elasticsearch endpoint.
// +kubebuilder:validation:XValidation:rule="isURL(self)", message="invalid URL"
type EndpointURL string

// Elasticsearch provides optional extra properties for `type: elasticsearch`
//
// +kubebuilder:validation:XValidation:rule="has(self.url) || self.endpoints.size() > 0", message="URL or endpoints required"
Comment thread
Clee2691 marked this conversation as resolved.
type Elasticsearch struct {
URLSpec `json:",inline"`

// URL to send log records to.
// Basic TLS is enabled if the URL scheme requires it (for example 'https' or 'tls').
// The 'username@password' part of `url` is ignored.
//
// +kubebuilder:validation:Optional
// +kubebuilder:validation:XValidation:rule="isURL(self)", message="invalid URL"
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Destination URL",xDescriptors={"urn:alm:descriptor:com.tectonic.ui:text"}
URL string `json:"url,omitempty"`
Comment thread
coderabbitai[bot] marked this conversation as resolved.

// Endpoints is a list of Elasticsearch endpoints to send log records to.
// Vector distributes events across endpoints using load balancing with
// automatic failover. When both URL and Endpoints are provided, URL is
// prepended to the Endpoints list.
//
// +kubebuilder:validation:Optional
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Elasticsearch Endpoints"
Endpoints []EndpointURL `json:"endpoints,omitempty"`

// Authentication sets credentials for authenticating the requests.
//
Expand Down Expand Up @@ -887,7 +911,7 @@ type HTTP struct {
// ProxyURL URL of a HTTP or HTTPS proxy to be used instead of direct connection.
//
// +kubebuilder:validation:Optional
// +kubebuilder:validation:XValidation:rule="self == '' || isURL(self)", message="invalid URL"
// +kubebuilder:validation:XValidation:rule="self == '' || isURL(self)", message="invalid URL"
ProxyURL string `json:"proxyURL,omitempty"`

// Format defines data format used to send data to remote destination.
Expand Down Expand Up @@ -1257,7 +1281,7 @@ type Loki struct {
// ProxyURL URL of a HTTP or HTTPS proxy to be used instead of direct connection.
//
// +kubebuilder:validation:Optional
// +kubebuilder:validation:XValidation:rule="self == '' || isURL(self)", message="invalid URL"
// +kubebuilder:validation:XValidation:rule="self == '' || isURL(self)", message="invalid URL"
ProxyURL string `json:"proxyURL,omitempty"`
}

Expand Down
6 changes: 5 additions & 1 deletion api/observability/v1/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 8 additions & 1 deletion bundle/manifests/cluster-logging.clusterserviceversion.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ metadata:
categories: OpenShift Optional, Logging & Tracing, Observability
certified: "false"
containerImage: quay.io/openshift-logging/cluster-logging-operator:latest
createdAt: "2026-07-15T11:39:59Z"
createdAt: "2026-09-01T18:57:04Z"
description: The Red Hat OpenShift Logging Operator for OCP provides a means for
configuring and managing log collection and forwarding.
features.operators.openshift.io/cnf: "false"
Expand Down Expand Up @@ -934,6 +934,13 @@ spec:
path: outputs[0].elasticsearch.authentication.username.secretName
x-descriptors:
- urn:alm:descriptor:com.tectonic.ui:text
- description: |-
Endpoints is a list of Elasticsearch endpoints to send log records to.
Vector distributes events across endpoints using load balancing with
automatic failover. When both URL and Endpoints are provided, URL is
prepended to the Endpoints list.
displayName: Elasticsearch Endpoints
path: outputs[0].elasticsearch.endpoints
- description: Headers specify optional headers to be sent with the request
displayName: Headers
path: outputs[0].elasticsearch.headers
Expand Down
24 changes: 20 additions & 4 deletions bundle/manifests/observability.openshift.io_clusterlogforwarders.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2393,7 +2393,7 @@ spec:
type: string
x-kubernetes-validations:
- message: invalid URL
rule: self == '' || isURL(self)
rule: self == '' || isURL(self)
required:
- authentication
- groupName
Expand Down Expand Up @@ -2476,6 +2476,20 @@ spec:
- secretName
type: object
type: object
endpoints:
description: |-
Endpoints is a list of Elasticsearch endpoints to send log records to.
Vector distributes events across endpoints using load balancing with
automatic failover. When both URL and Endpoints are provided, URL is
prepended to the Endpoints list.
items:
description: EndpointURL is a URL to an Elasticsearch
endpoint.
type: string
x-kubernetes-validations:
- message: invalid URL
rule: isURL(self)
type: array
headers:
additionalProperties:
type: string
Expand Down Expand Up @@ -2567,9 +2581,11 @@ spec:
type: integer
required:
- index
- url
- version
type: object
x-kubernetes-validations:
- message: URL or endpoints required
rule: has(self.url) || self.endpoints.size() > 0
googleCloudLogging:
description: GoogleCloudLogging configures forwarding log events
to GCP (formally Stackdriver) Operations
Expand Down Expand Up @@ -2822,7 +2838,7 @@ spec:
type: string
x-kubernetes-validations:
- message: invalid URL
rule: self == '' || isURL(self)
rule: self == '' || isURL(self)
timeout:
description: Timeout specifies the Http request timeout
in seconds. If not set, 10secs is used.
Expand Down Expand Up @@ -3134,7 +3150,7 @@ spec:
type: string
x-kubernetes-validations:
- message: invalid URL
rule: self == '' || isURL(self)
rule: self == '' || isURL(self)
tenantKey:
description: |-
TenantKey is the tenant for the logs. This supports vector's template syntax to allow dynamic per-event values.
Expand Down
24 changes: 20 additions & 4 deletions config/crd/bases/observability.openshift.io_clusterlogforwarders.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2393,7 +2393,7 @@ spec:
type: string
x-kubernetes-validations:
- message: invalid URL
rule: self == '' || isURL(self)
rule: self == '' || isURL(self)
required:
- authentication
- groupName
Expand Down Expand Up @@ -2476,6 +2476,20 @@ spec:
- secretName
type: object
type: object
endpoints:
description: |-
Endpoints is a list of Elasticsearch endpoints to send log records to.
Vector distributes events across endpoints using load balancing with
automatic failover. When both URL and Endpoints are provided, URL is
prepended to the Endpoints list.
items:
description: EndpointURL is a URL to an Elasticsearch
endpoint.
type: string
x-kubernetes-validations:
- message: invalid URL
rule: isURL(self)
type: array
headers:
additionalProperties:
type: string
Expand Down Expand Up @@ -2567,9 +2581,11 @@ spec:
type: integer
required:
- index
- url
- version
type: object
x-kubernetes-validations:
- message: URL or endpoints required
rule: has(self.url) || self.endpoints.size() > 0
googleCloudLogging:
description: GoogleCloudLogging configures forwarding log events
to GCP (formally Stackdriver) Operations
Expand Down Expand Up @@ -2822,7 +2838,7 @@ spec:
type: string
x-kubernetes-validations:
- message: invalid URL
rule: self == '' || isURL(self)
rule: self == '' || isURL(self)
timeout:
description: Timeout specifies the Http request timeout
in seconds. If not set, 10secs is used.
Expand Down Expand Up @@ -3134,7 +3150,7 @@ spec:
type: string
x-kubernetes-validations:
- message: invalid URL
rule: self == '' || isURL(self)
rule: self == '' || isURL(self)
tenantKey:
description: |-
TenantKey is the tenant for the logs. This supports vector's template syntax to allow dynamic per-event values.
Expand Down
7 changes: 7 additions & 0 deletions config/manifests/bases/cluster-logging.clusterserviceversion.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -857,6 +857,13 @@ spec:
path: outputs[0].elasticsearch.authentication.username.secretName
x-descriptors:
- urn:alm:descriptor:com.tectonic.ui:text
- description: |-
Endpoints is a list of Elasticsearch endpoints to send log records to.
Vector distributes events across endpoints using load balancing with
automatic failover. When both URL and Endpoints are provided, URL is
prepended to the Endpoints list.
displayName: Elasticsearch Endpoints
path: outputs[0].elasticsearch.endpoints
- description: Headers specify optional headers to be sent with the request
displayName: Headers
path: outputs[0].elasticsearch.headers
Expand Down
Loading
Loading