feat(roles/graylog_sidecar): add the Graylog Sidecar roles

CHANGELOG.md

@@ -12,6 +12,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+* **role:graylog_sidecar**: New role to install and configure the Graylog Sidecar log collector agent on RHEL-family hosts. It ships logs to a Graylog server and is managed centrally from the Graylog web UI via tags.
+* **role:repo_graylog_sidecar**: New role deploying the Graylog Sidecar package repository (separate from the main Graylog repository), with support for custom mirrors and basic auth like the other `repo_*` roles.
 * **role:files**: The `files:directories`, `files:files` and `files:symlinks` tags manage one kind of file system entity each, so a single directory or symlink can be deployed without touching the rest.
 * **plugin:bitwarden_item**: The lookup can be told not to create secrets, so a lookup that finds no matching item aborts the run instead of silently generating a new password. Set `LFOPS_BITWARDEN_LOOKUP_ITEM_CREATE=false`, or `create = false` in the `[bitwarden_item_lookup]` section of your `ansible.cfg`; the default is the previous behaviour.
 

COMPATIBILITY.md

@@ -49,6 +49,7 @@ Which Ansible role is proven to run on which OS?
 | grav                                  |        |        |   x    |  (x)   |   (x)   |           |           |           |                                              |
 | graylog_datanode                      |   x    |   x    |   x    |   x    |   (x)   |    (x)    |    (x)    |    (x)    |                                              |
 | graylog_server                        |   x    |   x    |   x    |  (x)   |   (x)   |    (x)    |    (x)    |    (x)    |                                              |
+| graylog_sidecar                       |        |        |  (x)   |  (x)   |    x    |           |           |           |                                              |
 | haveged                               |  (x)   |  (x)   |   x    |   x    |   (x)   |    (x)    |    (x)    |    (x)    |                                              |
 | hetzner_vm                            |        |        |        |        |         |           |           |           | Fedora 35+                                   |
 | hostname                              |  (x)   |  (x)   |   x    |   x    |    x    |    (x)    |    (x)    |    (x)    |                                              |
@@ -134,6 +135,7 @@ Which Ansible role is proven to run on which OS?
 | repo_gitlab_runner                    |        |        |   x    |  (x)   |   (x)   |           |           |           |                                              |
 | repo_grafana                          |   x    |   x    |   x    |   x    |   (x)   |    (x)    |    (x)    |    (x)    |                                              |
 | repo_graylog                          |   x    |   x    |   x    |  (x)   |   (x)   |    (x)    |    (x)    |    (x)    |                                              |
+| repo_graylog_sidecar                  |        |        |  (x)   |  (x)   |    x    |           |           |           |                                              |
 | repo_icinga                           |   x    |   x    |   x    |   x    |    x    |     x     |    (x)    |    (x)    |                                              |
 | repo_influxdb                         |   x    |   x    |   x    |   x    |   (x)   |    (x)    |    (x)    |    (x)    |                                              |
 | repo_mariadb                          |   x    |   x    |   x    |   x    |    x    |    (x)    |     x     |    (x)    |                                              |

CONTRIBUTING.md

@@ -672,7 +672,7 @@ Make sure to use the following format when passing multiple injections to avoid
 * Use the target path for the file in the `template` folder, for example: `templates/etc/httpd/sites-available/default.conf.j2`. This makes it clear what the file is for, and avoids name collisions.
 * Always use the `.j2` file extension for files in the `template` folder.
 * If deploying self-written scripts, copy them to `/usr/local/sbin` (due to SELinux). Internal helper scripts that are only ever run by a systemd unit (not invoked by an admin and not exec'd by a confined domain) MAY instead live in `/usr/local/libexec`. Files there get the `usr_t` type, and the targeted policy lets a root `oneshot` service (which runs in `init_t`) execute them in place via `execute_no_trans`, so there is no AVC denial on RHEL/Rocky 8, 9 and 10. Keep admin-invokable commands in `/usr/local/sbin`, and never put a script a confined domain must exec under `/usr/local/libexec`.
-* Keep templates as close to the original file as possible. This makes handling of rpmnew/rpmsave files easier.
+* Keep templates as close to the original file as possible. This makes handling of rpmnew/rpmsave files easier. Where upstream ships a fully-commented default config (with every setting documented and commented out), start the template from that exact file and preserve its comments, ordering and commented-out defaults verbatim. Only uncomment and template the keys the role actually manages; leave everything else as the upstream comment. A package update then drops an `*.rpmnew` whose only differences from the deployed file are the handful of managed lines, so the diff is short and reviewable. See `roles/graylog_sidecar/templates/etc/graylog/sidecar/sidecar.yml.j2`.
 * Add the following task after deploying a file that might get rpmnew or rpmsave files (or their Debian equivalents):
     ```yaml
     - name: 'Remove rpmnew / rpmsave (and Debian equivalents)'

extensions/molecule/graylog_sidecar/converge.yml

@@ -0,0 +1,2 @@
+- name: 'Converge graylog_sidecar playbook'
+  ansible.builtin.import_playbook: 'linuxfabrik.lfops.graylog_sidecar'

extensions/molecule/graylog_sidecar/inventory/group_vars/systems_under_test.yml

@@ -0,0 +1,21 @@
+# Variables the playbook under test needs, applied to every system under test.
+
+# Mandatory: the repository is versioned independently of the Graylog server.
+repo_graylog_sidecar__version: '1.5'
+
+# Mandatory: the sidecar needs a server to talk to. No Graylog server runs in this scenario, so
+# the URL points at a documentation domain (RFC 2606). The sidecar starts anyway and retries,
+# which is what verify.yml uses to prove it read the configured value.
+graylog_sidecar__server_url: 'http://graylog.example.com:9000/api/'
+graylog_sidecar__server_api_token: 'linuxfabrik'
+
+# Optional, set to a non-default so the scenario exercises the templating rather than only the
+# shipped defaults.
+graylog_sidecar__tags:
+  - 'linuxfabrik-molecule'
+graylog_sidecar__update_interval: 30
+
+# Narrowed to the collector the RPM actually ships, which is what a real deployment should do.
+# Also exercises the list rendering with a value other than the shipped default.
+graylog_sidecar__collector_binaries_accesslist:
+  - '/usr/lib/graylog-sidecar/filebeat'

extensions/molecule/graylog_sidecar/inventory/hosts.yml

@@ -0,0 +1,15 @@
+# yamllint disable rule:empty-values
+
+# Scenario inventory, layered on top of the shared inventory
+# (extensions/molecule/inventory) through the two --inventory flags in config.yml.
+
+lfops_graylog_sidecar:
+  children:
+    systems_under_test:
+
+# The Graylog Sidecar repository only ships RPMs, so the role is RHEL-family only.
+systems_under_test:
+  hosts:
+    rocky8-vm:
+    rocky9-vm:
+    rocky10-vm:

extensions/molecule/graylog_sidecar/molecule.yml

@@ -0,0 +1,8 @@
+# Molecule scenario marker. Required even when empty, so that Molecule discovers the scenario.
+# Everything this scenario needs comes from extensions/molecule/config.yml; nothing is overridden
+# here.
+#
+# The scenario converges without a Graylog server in the test setup. That is deliberate: the
+# sidecar keeps running and retries when the server is unreachable, so the whole role is still
+# exercised, and the retry attempts in the journal prove the running process picked up the
+# configured server_url.

extensions/molecule/graylog_sidecar/verify.yml

@@ -0,0 +1,67 @@
+# verify.yml runs after converge, and again after the idempotence step (see the test_sequence in
+# config.yml). It checks the observable end result with read-only modules, never by re-running
+# the role.
+#
+# The two checks worth having here are the systemd unit and the journal. The package ships no
+# unit file at all (https://github.com/Graylog2/collector-sidecar/issues/225), so the unit only
+# exists if the role's `graylog-sidecar -service install` task really ran. And since the sidecar
+# has no local status endpoint, the journal is the only place where the running process reveals
+# which server_url it was configured with.
+- name: 'Verify the graylog_sidecar playbook installed a running sidecar'
+  hosts: 'systems_under_test'
+  gather_facts: false
+  tasks:
+
+    # Floor check: the package can only come from the repository that repo_graylog_sidecar
+    # deployed, so this covers both roles.
+    - name: 'Gather the package facts'
+      ansible.builtin.package_facts:  # yamllint disable-line rule:empty-values
+
+    - name: 'Assert that the graylog-sidecar package is installed'
+      ansible.builtin.assert:
+        that: '"graylog-sidecar" in ansible_facts["packages"]'
+
+    # The unit is generated by the sidecar binary, not shipped by the package. Without the role's
+    # `-service install` task there would be nothing to start.
+    - name: 'stat /etc/systemd/system/graylog-sidecar.service'
+      ansible.builtin.stat:
+        path: '/etc/systemd/system/graylog-sidecar.service'
+      register: '__molecule__graylog_sidecar_unit_stat_result'
+
+    - name: 'Assert that the systemd unit was registered'
+      ansible.builtin.assert:
+        that: '__molecule__graylog_sidecar_unit_stat_result["stat"]["exists"]'
+
+    # sidecar.yml holds the Graylog API token, so its mode is a security property of the role,
+    # not a detail of the template module. The package ships the file 0600 and the role must not
+    # widen that.
+    - name: 'stat /etc/graylog/sidecar/sidecar.yml'
+      ansible.builtin.stat:
+        path: '/etc/graylog/sidecar/sidecar.yml'
+      register: '__molecule__graylog_sidecar_config_stat_result'
+
+    - name: 'Assert that the config holding the API token is not readable by other users'
+      ansible.builtin.assert:
+        that: '__molecule__graylog_sidecar_config_stat_result["stat"]["mode"] == "0600"'
+
+    - name: 'Gather the service facts'
+      ansible.builtin.service_facts:  # yamllint disable-line rule:empty-values
+
+    - name: 'Assert that graylog-sidecar.service is enabled and running'
+      ansible.builtin.assert:
+        that:
+          - 'ansible_facts["services"]["graylog-sidecar.service"]["status"] == "enabled"'
+          - 'ansible_facts["services"]["graylog-sidecar.service"]["state"] == "running"'
+
+    # The important check: the running process uses the configured server_url. The sidecar logs
+    # to the journal (/var/log/graylog-sidecar/ stays empty under systemd) and reports every
+    # contact attempt, including the URL it took from the templated sidecar.yml. This proves the
+    # whole chain: group_vars -> template -> service reads the file -> behaviour.
+    - name: 'journalctl --unit graylog-sidecar.service'
+      ansible.builtin.command: 'journalctl --unit=graylog-sidecar.service --no-pager'
+      changed_when: false
+      register: '__molecule__graylog_sidecar_journal_result'
+
+    - name: 'Assert that the sidecar contacts the configured Graylog server'
+      ansible.builtin.assert:
+        that: '"graylog.example.com:9000" in __molecule__graylog_sidecar_journal_result["stdout"]'

playbooks/README.md

@@ -373,6 +373,14 @@ Calls the following roles (in order):
 * [grafana_grizzly](https://github.com/Linuxfabrik/lfops/tree/main/roles/grafana_grizzly)
 
 
+## graylog_sidecar.yml
+
+Calls the following roles (in order):
+
+* [repo_graylog_sidecar](https://github.com/Linuxfabrik/lfops/tree/main/roles/repo_graylog_sidecar): `graylog_sidecar__skip_repo_graylog_sidecar`
+* [graylog_sidecar](https://github.com/Linuxfabrik/lfops/tree/main/roles/graylog_sidecar)
+
+
 ## haveged.yml
 
 Calls the following roles (in order):
@@ -900,6 +908,13 @@ Calls the following roles (in order):
 * [repo_graylog](https://github.com/Linuxfabrik/lfops/tree/main/roles/repo_graylog)
 
 
+## repo_graylog_sidecar.yml
+
+Calls the following roles (in order):
+
+* [repo_graylog_sidecar](https://github.com/Linuxfabrik/lfops/tree/main/roles/repo_graylog_sidecar)
+
+
 ## repo_icinga.yml
 
 Calls the following roles (in order):

playbooks/all.yml

@@ -40,6 +40,7 @@
 - import_playbook: 'glpi_agent.yml'
 - import_playbook: 'grafana.yml'
 - import_playbook: 'grafana_grizzly.yml'
+- import_playbook: 'graylog_sidecar.yml'
 - import_playbook: 'haveged.yml'
 - import_playbook: 'hetzner_vm.yml'
 - import_playbook: 'hostname.yml'
@@ -107,6 +108,7 @@
 - import_playbook: 'repo_gitlab_runner.yml'
 - import_playbook: 'repo_grafana.yml'
 - import_playbook: 'repo_graylog.yml'
+- import_playbook: 'repo_graylog_sidecar.yml'
 - import_playbook: 'repo_icinga.yml'
 - import_playbook: 'repo_influxdb.yml'
 - import_playbook: 'repo_mariadb.yml'

playbooks/graylog_sidecar.yml

@@ -0,0 +1,33 @@
+- name: 'Playbook linuxfabrik.lfops.graylog_sidecar'
+  hosts:
+    - 'lfops_graylog_sidecar'
+
+  pre_tasks:
+    - ansible.builtin.import_role:
+        name: 'shared'
+        tasks_from: 'log-start.yml'
+      tags:
+        - 'always'
+
+    - ansible.builtin.import_role:
+        name: 'shared'
+        tasks_from: 'global-variables.yml'
+      tags:
+        - 'always'
+
+
+  roles:
+
+    - role: 'linuxfabrik.lfops.repo_graylog_sidecar'
+      when:
+        - 'not graylog_sidecar__skip_repo_graylog_sidecar | d(false) | bool'
+
+    - role: 'linuxfabrik.lfops.graylog_sidecar'
+
+
+  post_tasks:
+    - ansible.builtin.import_role:
+        name: 'shared'
+        tasks_from: 'log-end.yml'
+      tags:
+        - 'always'

playbooks/repo_graylog_sidecar.yml

@@ -0,0 +1,29 @@
+- name: 'Playbook linuxfabrik.lfops.repo_graylog_sidecar'
+  hosts:
+    - 'lfops_repo_graylog_sidecar'
+
+  pre_tasks:
+    - ansible.builtin.import_role:
+        name: 'shared'
+        tasks_from: 'log-start.yml'
+      tags:
+        - 'always'
+
+    - ansible.builtin.import_role:
+        name: 'shared'
+        tasks_from: 'global-variables.yml'
+      tags:
+        - 'always'
+
+
+  roles:
+
+    - role: 'linuxfabrik.lfops.repo_graylog_sidecar'
+
+
+  post_tasks:
+    - ansible.builtin.import_role:
+        name: 'shared'
+        tasks_from: 'log-end.yml'
+      tags:
+        - 'always'