Skip to content

test(integ-test): exclude suites integTestRemote cannot support - #5709

Open
mengweieric wants to merge 1 commit into
opensearch-project:mainfrom
mengweieric:fix/integtestremote-mirror-integtest-exclusions
Open

test(integ-test): exclude suites integTestRemote cannot support#5709
mengweieric wants to merge 1 commit into
opensearch-project:mainfrom
mengweieric:fix/integtestremote-mirror-integtest-exclusions

Conversation

@mengweieric

@mengweieric mengweieric commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Description

integTestRemote runs against an externally provided cluster (-Dtests.rest.cluster). It cannot install plugins, cannot set node-level settings, and — unlike integTest — does not start Prometheus. Three groups of suites need one of those, so instead of being skipped they are selected and fail during setup.

Group Suites What this task cannot provide
analytics-engine AnalyticsEngineCompatIT, AnalyticsEngineSecurityIT, AnalyticsEngineProfileIT the analytics-engine plugin stack, and the node settings opensearch.experimental.feature.pluggable.dataformat.enabled / ...transport.stream.enabled
Prometheus PrometheusDataSourceCommandsIT, ShowDataSourcesCommandIT, InformationSchemaCommandIT a running Prometheus instance — integTest starts one via dependsOn startPrometheus, this task has no such dependency
datasource DataSourceAPIsIT, DataSourceEnabledIT plugins.query.datasources.encryption.masterkey, which EncryptorImpl validates as "a required config for using create and update datasource APIs" and which must be in opensearch.yml on every node

Each group already has a task that provisions what it needs: the dedicated analyticsEngine*IT tasks, and integTest/yamlRestTest whose testClusters blocks start Prometheus and set the masterkey. Those are untouched, so no coverage is lost.

Change

The analytics-engine group is added as an else on the existing analyticsEnabled gate, so those suites still run when the analytics-engine route is active and all analytics-engine filtering for this task stays in one place.

The Prometheus and datasource groups are excluded unconditionally, as integTest already does for the analytics-engine suites it cannot provision. Class-level exclusion is needed rather than method filtering because these suites also fail in @After / @AfterClass cleanup, which a test filter cannot suppress.

Verified by inspecting the task's resolved filter state at configuration time: with no system properties set all eight suites are excluded and the pre-existing bwc.*IT exclusion is unaffected; with -Dtests.analytics.parquet_indices=true the analytics-engine suites are no longer excluded while the other two groups remain.

Not included: the security package

integTest also excludes org/opensearch/sql/security/** because those suites run in :integTestWithSecurity. That does not carry over: integTestWithSecurity uses its own configureSecurityPlugin() cluster and does not cover the remote case, and integTestRemote forwards https, user and password precisely so it can target a secured cluster — where those suites are meaningful. Whether they can run depends on the target cluster, which Gradle cannot determine at configuration time, so that case needs a runtime check in the tests rather than a static exclusion. Left for separate work.

Related Issues

No separate tracking issue.

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • New functionality has javadoc added.
  • New functionality has a user manual doc added.
  • New PPL command checklist all confirmed.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff or -s.
  • Public documentation issue/PR created.

Unchecked items are not applicable: this changes a test task's exclusion list and adds no functionality, API surface or user-facing behaviour.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@mengweieric mengweieric added the infrastructure Changes to infrastructure, testing, CI/CD, pipelines, etc. label Aug 20, 2026
@mengweieric
mengweieric force-pushed the fix/integtestremote-mirror-integtest-exclusions branch from 54f7af5 to 0210c53 Compare August 20, 2026 04:01
@mengweieric mengweieric changed the title Exclude analytics-engine suites from integTestRemote Mirror integTest's test exclusions in integTestRemote Aug 20, 2026
@mengweieric
mengweieric force-pushed the fix/integtestremote-mirror-integtest-exclusions branch from 0210c53 to 6289059 Compare August 20, 2026 04:05
@mengweieric mengweieric changed the title Mirror integTest's test exclusions in integTestRemote Exclude suites integTestRemote cannot provide a cluster for Aug 20, 2026
@mengweieric
mengweieric force-pushed the fix/integtestremote-mirror-integtest-exclusions branch from 2ce0abe to f8658e8 Compare August 20, 2026 04:15
@mengweieric mengweieric changed the title Exclude suites integTestRemote cannot provide a cluster for test(integ-test): exclude analytics-engine and Prometheus suites from integTestRemote Aug 20, 2026
@mengweieric
mengweieric force-pushed the fix/integtestremote-mirror-integtest-exclusions branch from f8658e8 to b53453a Compare August 20, 2026 04:16
@mengweieric
mengweieric marked this pull request as ready for review August 20, 2026 04:52
@mengweieric mengweieric changed the title test(integ-test): exclude analytics-engine and Prometheus suites from integTestRemote test(integ-test): exclude with gates for analytics-engine and Prometheus suites from integTestRemote Aug 20, 2026
@mengweieric
mengweieric force-pushed the fix/integtestremote-mirror-integtest-exclusions branch from b53453a to e83160d Compare August 23, 2026 22:13
@mengweieric mengweieric changed the title test(integ-test): exclude with gates for analytics-engine and Prometheus suites from integTestRemote test(integ-test): exclude suites integTestRemote cannot support Aug 23, 2026
@github-actions

Copy link
Copy Markdown
Contributor

PR Code Analyzer ❗

AI-powered 'Code-Diff-Analyzer' found issues on commit e83160d.

PathLineSeverityDescription
integ-test/build.gradle1389mediumAnalyticsEngineSecurityIT is excluded from the integTestRemote task. While the comment cites environment limitations (missing plugin stack), excluding a security-focused integration test suite reduces security test coverage for remote cluster runs. The justification should be verified: a dedicated task (analyticsEngineSecurityIT) is claimed to cover this, but if that task is not mandatory in the CI pipeline, security tests may be silently skipped.

The table above displays the top 10 most important findings.

Total: 1 | Critical: 0 | High: 0 | Medium: 1 | Low: 0


Pull Requests Author(s): Please update your Pull Request according to the report above.

Repository Maintainer(s): You can bypass diff analyzer by adding label skip-diff-analyzer after reviewing the changes carefully, then re-run failed actions. To re-enable the analyzer, remove the label, then re-run all actions.


⚠️ Note: The Code-Diff-Analyzer helps protect against potentially harmful code patterns. Please ensure you have thoroughly reviewed the changes beforehand.

Thanks.

integTestRemote runs against an externally provided cluster. It cannot
install plugins, cannot set node-level settings, and unlike integTest it
does not start Prometheus. Three groups of suites depend on one of those
and are still selected by it, so they fail during setup instead of being
skipped.

Analytics-engine suites. AnalyticsEngineCompatIT, AnalyticsEngineSecurityIT
and AnalyticsEngineProfileIT create composite (parquet-backed) indices and
depend on opensearch.experimental.feature.pluggable.dataformat.enabled and
opensearch.experimental.feature.transport.stream.enabled. Each has a
dedicated task whose testClusters block provisions that stack. Added as an
else branch on the existing analyticsEnabled gate, so all analytics-engine
filtering for this task stays in one place and the suites still run when
the analytics-engine route is active.

AnalyticsEngineSecurityIT is also the slowest suite in the run: its setup
polls the security configuration API 60 times at one second intervals and
only marks itself initialised on success, so every test method repeats the
full wait before failing.

Prometheus suites. integTest starts Prometheus itself (dependsOn
startPrometheus) and so only skips these when the caller passes
-DignorePrometheus. integTestRemote has no such dependency and cannot
supply Prometheus for an external cluster.

Datasource suites. Creating or updating a datasource requires
plugins.query.datasources.encryption.masterkey, which EncryptorImpl
validates as "a required config for using create and update datasource
APIs" and which must be present in opensearch.yml on every node.
testClusters supplies it for integTest and yamlRestTest; integTestRemote
sets no cluster settings at all and cannot add a node config to an
externally provided cluster.

The last two groups are excluded unconditionally, the same way integTest
unconditionally excludes the analytics-engine suites it cannot provision.
integTest continues to cover all of them, so no coverage is lost.

Signed-off-by: Eric Wei <mengwei.eric@gmail.com>
@mengweieric
mengweieric force-pushed the fix/integtestremote-mirror-integtest-exclusions branch from e83160d to a723993 Compare August 23, 2026 22:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

infrastructure Changes to infrastructure, testing, CI/CD, pipelines, etc.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant