Halve the event loop's CPU cost per check

[nook24]

As you know i did some AI shenanigans to reduce the CPU time of Naemons Event Loop. This PR has all the findings in it. We can pick singe commits or merge it all together. How ever, we should do some serious code review :)

---

Naemon's event loop is a single thread, so what it spends per check is a hard ceiling on how large an installation can grow. This series, driven by profiling rather than inspection, takes 48 % off main-thread CPU per check (50.1 → 26.2 µs at 100 000 services, no broker module) and fixes two bugs found along the way. With a broker module loaded — the common case — the gain is about 18 %, because the module then dominates the loop.

Full write-up with methodology and measurements: https://claude.ai/artifact/1RAwMLoi4539y95SE19zjc

Commits

#	Commit	Effect
1	Reject unix socket paths that do not fit in sun_path	crash fix: a query_socket path of 108+ characters overran the stack at startup
2	Do not sift past the end when removing the last event heap node	pre-existing use-after-free in the scheduler, reproduced on master
3	Only flush the descriptor iobroker_write_packet() wrote to	no longer walks every worker socket per dispatched check
4	Keep event heap sort keys in the heap array	−40 % of the heap's cost
5	Cache the current local day for timeperiod lookups	3.4 % → 0.4 % of the loop
6	Skip the NEB result set when nothing is subscribed	two allocations per call saved without a broker module
7	Write status.dat without a fprintf() call per field	writer 2.04 s → 0.83 s per dump (−59 %)
8	Write retention.dat through nm_writebuf as well	~1.0 s → ~0.4 s per dump
9	Carry the checked object in check_result	no lookup by name per result; NEB API 8 → 9
10	Give schedule-only status events their own NEBTYPE	closes #162
11	Add the event loop benchmark tooling	contrib/perfbench/

Commits 1 and 2 stand alone and can be taken without the rest. Every commit builds on its own. The reasoning for each change is in its commit message.

Breaking change: NEB modules need a recompile

check_result gained an object_ptr member, so CURRENT_NEB_API_VERSION goes from 8 to 9. A module built against v8 is refused at load time instead of misreading the struct. A recompile is all that is needed — no module source change is required. Newly available: modules may set cr->object_ptr to the host/service a result belongs to (or leave it NULL for the old lookup), and can recognise NEBTYPE_HOSTSTATUS_SCHEDULE / NEBTYPE_SERVICESTATUS_SCHEDULE.

Behaviour

• status.dat, retention.dat and objects.cache are byte-identical to master's output, verified against a stock build driven to the same state with a fixed set of passive results.
• The status event sent when a check is only rescheduled is now NEBTYPE_*STATUS_SCHEDULE instead of NEBTYPE_*STATUS_UPDATE. Modules that do not look at the type see no difference. The contract, including when a schedule event carries a fresh result, is documented at the type definitions in broker.h.

Testing

• The CI matrix from citest.yml (-O2, -g -O0, AddressSanitizer): make check, make distcheck, flake8, behave — all green, no ASan or LeakSanitizer reports.
• New tests, each confirmed to fail without the change it covers: removing heap entries with equal event times (commit 2), extreme event times in one heap (the saturating key in commit 4), nm_writebuf output against snprintf() up to DBL_MAX (commit 7).
• A stress test for the timeperiod day cache compares it against the uncached computation over 1990–2036 in 16 time zones, including DST transitions at midnight and 30-minute shifts. Cases whose zone file is not installed are skipped rather than silently run as UTC — Ubuntu 26.04 ships no right/ zones.

Not included

• In retention.dat, a host's retry_check_interval is written from check_interval. Fixing it changes what an existing file means, so it is marked FIXME and left for its own patch.
• objects.cache stays on fprintf(): the fcache_* functions are declared in installed headers with a FILE * signature, and the file is written only once at startup.

[sni]

i like the idea of having some docs about certain callbacks. Might be better to have them in the docs in the developer section? What you think?

[sni]

not sure if that's worth the effort. Either doubles are pretty unique, so there will be almost no cache hits. Or it is a round number, then it would be way faster to print is as integer.

[nook24]

What changed

• contrib/perfbench removed. The tooling commit is gone, and no commit message refers to it anymore.

• The NEBTYPE_*STATUS_SCHEDULE explanation moved out of broker.h moved into the Naemon Docs (PR soon)

• The double cache (NM_WB_DBL_CACHE_SIZE) is removed. You were right to be suspicious. I re-measured with realistically spread check intervals (1 min to 1 day, mixed retry intervals, 10 % of services changing state), 100k services, two interleaved rounds each:

  	status.dat dump (median)	cache hit rate
  with cache	180 ms	43 %
  without cache	185 ms	–

  2.5 % of the writer's time is not worth a global cache. nm_wb_dbl() is now a plain snprintf() via nm_wb_printf().

• objects.cache now goes through nm_writebuf as well. The public fcache_*(FILE *) functions keep their signatures and become thin wrappers. The writers proper are nm_fcache_*() in a new, uninstalled objects_fcache.h. The output is byte-identical (including (null) for NULL strings), checked on 7 configs: the 100k one (80 MB), the test configs, and one that covers every object type, every daterange type, escalations, dependencies, coordinates and custom variables. naemon -vp at 100k services: 530 → 425 ms.

• Day cache: timezone switch. As pointed out in the review: zones with different DST rules can share timezone/daylight/tzname, Europe/Amsterdam and Africa/Tunis for one. A use_timezone switch between them left a midnight that was an hour off for the rest of the cached day. read_main_config_file() now resets the cache after tzset(). A new test switches between exactly those two zones through read_main_config_file(); it fails without the fix and passes with it.

Commits

#	Commit		Change since the last push
1	d7b581ea Build test-bufferqueue on 32 bit platforms	unchanged	shared with #548, same SHA
2	ed7e363a Build and pass test-event-heap on 32 bit platforms	unchanged	shared with #548, same SHA
3	38a97f2d Reject unix socket paths that do not fit in sun_path	unchanged
4	c1f041a4 Do not sift past the end when removing the last event heap node	unchanged
5	4be0b058 Only flush the descriptor iobroker_write_packet() wrote to	unchanged
6	9b42ca5f Keep event heap sort keys in the heap array	unchanged
7	5d427cc7 Cache the current local day for timeperiod lookups	changed	cache reset on use_timezone, new test
8	058ddb19 Skip the NEB result set when nothing is subscribed	rebased
9	df7bab34 Write status.dat without a fprintf() call per field	changed	double cache removed, tests and message trimmed
10	830b7908 Write retention.dat through nm_writebuf as well	rebased
11	d351b4ba Write objects.cache through nm_writebuf as well	new
12	a479e0d0 Carry the checked object in check_result	rebased
13	e77e14f1 Give schedule-only status events their own NEBTYPE	changed	comment block in broker.h removed, see docs PR
14	2c24f24d List the event loop changes in NEWS	new	NEWS entry from the old tooling commit, plus objects.cache
–	9e5f84a6 Add the event loop benchmark tooling	removed

"Rebased" means the content is unchanged and only the SHA moved because an earlier commit changed.

Verification

• Every commit builds with -Werror and passes make check on its own.
• CI matrix as in citest.yml (Ubuntu 26.04): -O2, -g -O0 and AddressSanitizer, each with build, make check, make distcheck, flake8 and behave. 15/15 green, no sanitizer reports.
• objects.cache is byte-identical to the previous build, see above.
• On Debian 11/12 i386 the library builds. make check only shows the three failures as I did not pulled Fix the build and make check on 32bit platforms #548.

Den Platzhalter #548 gibt es mehrmals, am einfachsten mit Suchen und Ersetzen.

[inqrphl]

this is its own fix from commit 38a97f2 , only modifies this file.

[inqrphl]

changes on this file are related to nm_writebuffer + fcache usage.

after writing to the buffer it calls nm_writebuf_done to write it into file

[inqrphl]

changes on this file are related to nm_writebuffer + fcache usage.

[inqrphl]

changes on this file are related to nm_writebuffer + fcache usage.

[inqrphl]

changes on this file are related to nm_writebuffer + fcache usage.

[inqrphl]

check_time_against_period uses this function, which this PR replaced with its version using cache

[inqrphl]

changes on this file are related to nm_writebuffer + fcache usage.

[inqrphl]

defining them as macros instead of functions

this is to prevent performance overhead i believe

[inqrphl]

changes on this file are related to nm_writebuffer + fcache usage.

at the end the changes are written into the file using nm_writebuf__done

[inqrphl]

the START_TEST and END_TEST macros come from libcheck

[inqrphl]

this is defined in with_tests option in configure.ac

[inqrphl]

when it says it dereferences two timed_event structs it means the older evheap_compare function:

static inline int evheap_compare(struct timed_event *eva, struct timed_event *evb)
{
	if (eva->event_time.tv_sec < evb->event_time.tv_sec)
		return -1;
	if (eva->event_time.tv_sec > evb->event_time.tv_sec)
		return 1;
	if (eva->event_time.tv_nsec < evb->event_time.tv_nsec)
		return -1;
	if (eva->event_time.tv_nsec > evb->event_time.tv_nsec)
		return 1;
	return 0;
}

evA and evB are the two events in heap, they each point to timed_event.

timed_event has a field event_time of type "struct timespec"

which has separate tv_sec and tv_nsec , so it might do 4 comparsions in total.

the PR combines tv_sec and tv_nsec into one int64_t key , stores it in evheap_entry and uses it as the heap key

[inqrphl]

once this function generates the key to the timestamp i.e event time in nanoseconds, it can be compared in a single comparison

[inqrphl]

the heap used to directly store timed_events

struct timed_event **queue;

array of pointers to timed_event

the array was kept consistent into heap principles, meaning the first one is the smallest.

now instead of storing *timed_event in the array, it stores *evheap_entry

an evheap_entry then points to the same timed_event it was made from

[inqrphl]

new heap is made using evheap_entry , which has the easier to compare nanosecond based key

[nook24]

@inqrphl is there some actual call to action for me in the comments?

[inqrphl]

@inqrphl is there some actual call to action for me in the comments?

No, not at all. I am just looking at it and understanding the changes. Possibly help @sni later on maybe as well.

[sni]

This is not used anymore then. Could it be removed?

[sni]

i don't understand why the tz_name must be compared? It should not change during runtime but only on reload where this cache is cleared anyway.
I mean, it would be nice to have timeperiods or hosts support timezones, but that is a different story.

[nook24]

By default use_timezone is not present in the naemon.cfg (only as comment). If use_timezone is set in the naemon.cfg Naemon set the TZ environment variable on startup. In this case, the timezone name in the cache is not relevant.

	/* adjust timezone values */
	if (use_timezone != NULL)
		set_environment_var("TZ", use_timezone, 1);
	tzset();
	/* zones with different DST rules can look alike to the day cache */
	_reset_day_cache();

But. As mentioned: By default this value is not set. In this case, glibc will use the value of /etc/localtime to determine the timezone. As soon as /etc/localtime change, the next call to mktime() will immediately use the new timezone, no Naemon reload required.

To be able to detect a timezone change, the cache has to hold the timezone fingerprint.

Current Naemon versions are applying timezone changes immediately. If we do not clear the cache on a timezone change, Naemon would calculate wrong values until next midnight.

The current detection is also not 100% bullet proof. A timezone change with the same timezone shortcut but different daylight saving times is not detected. (Europe/Amsterdam and Africa/Tunis for example). For this reason, the reload is resetting the cache anyways.

As id did not find glibc on GitHub, I ask AI to get a little call stack overview, that's the result:

• mktime() calls __tzset() every time (time/mktime.c#l537), as POSIX requires: "Local timezone information shall be set as though mktime() called tzset()." (mktime)
• With TZ unset there is no early return; glibc falls back to TZDEFAULT and calls __tzfile_read() (time/tzset.c#l388)
• which re-reads the file when it changed (time/tzfile.c#l158)

[sni]

shouldn't it be possible to cache midnight, even if there is a daylight change on that day? Midnight does not move, only the day is not exactly 86400 seconds long.

I assume this might raise questions on why naemon is "slower" twice a year :-)

[nook24]

This will be a longer one :)

The current Naemon versions check on every check, if the time is within the check_period. src/naemon/objects_timeperiod.c

static inline time_t get_midnight(time_t when)
{
      struct tm *t, tm_s;

      t = localtime_r((time_t *)&when, &tm_s);   /* timestamp -> date/time, also sets tm_isdst */
      t->tm_sec = 0;
      t->tm_min = 0;
      t->tm_hour = 0;                            /* time to 00:00:00, tm_isdst is kept */
      return mktime(t);                          /* date/time -> timestamp */
}

int check_time_against_period(time_t test_time, const timeperiod *tperiod)
{
      ...
      midnight = get_midnight(test_time);
      for (temp_timerange = _get_matching_timerange(test_time, tperiod); ...) {
              if (timerange_includes_time(temp_timerange, test_time - midnight))
                      return OK;
      }
      return ERROR;
}

_get_matching_timerange() looks up the time-ranges for the day, and calculates the same midnight again. 2*localtime_r() and 2*mktime() called. This is where the 3.4% CPU time was consumed in the event loop.

The cache holds the result for the entire day so a lookup within the same day do not have to do anything.

What midnight means in Naemon
Time-ranges from the configuration are stored as seconds since the start of the day: 09:00-17:00 becomes range_start = 32400, range_end = 61200. They are checked like this:

static inline int timerange_includes_time(struct timerange *range, time_t when)
{
      return (when >= (time_t)range->range_start && when < (time_t)range->range_end);
}
/* called with: test_time - midnight */

So midnight is the origin where Naemon starts counting seconds of the day.
On a normal day that is the real midnight. On a transition day it is not: localtime_r() sets tm_isdst for the tested timestamp, get_midnight() only zeroes hour, minute and second, and mktime() then interprets 00:00 with that inherited tm_isdst. So the origin depends on whether the tested time is before or after the switch.

Europe/Berlin, 2025-03-30:

tested time	"midnight"	09:00 becomes
01:00 CET (before the switch)	03-30 00:00 CET	10:00 CEST
15:00 CEST (after the switch)	03-29 23:00 CET	09:00 CEST
real midnight (tm_isdst=-1)	03-30 00:00 CET	10:00 CEST

Before the switch only times up to 02:00 are tested, and for those both origins agree. After the switch, the shifted origin is what makes 09:00 mean 09:00.

The cache holds one origin per day, but a transition day needs two. Caching one of them would put the time ranges an hour off for part of that day. Avoiding that would mean changing how get_midnight() works, which changes timeperiod behaviour on transition days.

How does the cache detect a transition day? It measures the day length with tm_isdst = -1

      probe = *t;                     /* today, 00:00 */
      probe.tm_isdst = -1;
      probe_midnight = mktime(&probe);

      next = *t;                      /* tomorrow, 00:00 */
      next.tm_mday += 1;
      next.tm_isdst = -1;
      next_midnight = mktime(&next);

      day_cache.tm = *t;              /* the origin as on master, with inherited tm_isdst */
      day_cache.midnight = mktime(&day_cache.tm);

      if (next_midnight - probe_midnight == 86400 &&     /* a normal 24 h day */
          day_cache.midnight == probe_midnight)          /* with a single origin */
              day_cache.valid_until = next_midnight;         /* -> cache it */
      else
              day_cache.valid_until = 0;                     /* -> do not cache */

While going through all of this, I have to admit you where right with the "slower twice a year" part.
On a transition day now 2*localtime_r() and 6*mktime() class are made. (407ns on master, 752ns on this code) I guess we should cache if a day is not cacheable :)
I will address this.

[sni]

looks good, i have some questiond but no blockers. The FIXMEs should be adressed in a follow up PR.