Discrepancies between innxmit and innd accepted sizes

[Julien-Elie]

Jesse Rehmer reports on news.software.nntp that:

I'm observing what feels like a bug in the calculation of "accepted size" in innd's checkpoint log lines with long-running connections. This may also be happening with status log lines, but I haven't focused on those.

In the process of feeding >1 billion articles to a new server. Both servers are running INN and I'm using innxmit on the source machine. After starting innxmit, initially the calculation of the accepted size in the checkpoint log lines seems correct, but over time it seems to miscalculate.

With incominglogfrequency set to 100,000, initially each checkpoint reports accepted size around 220-280MB. Which, by my rough calculations, seems correct. However, over time (8+ hours), I begin to see the accepted size fall dramatically to around 80-110MB for the same amount of articles.

I understand there is going to be fluctuation in the amount of data received per 100,000 articles, but if I stop the innxmit process and start it again, immediately the checkpoints start reporting back in the original range. Then after some time it begins to fall off.

I see this pattern repeat each time I stop/start innxmit (I have removed a lot of output for brevity):

% grep checkpoint log/news.notice | awk '{ print $23 }'
254853120
244031488
240500736
222146560
245587968
221945856
107102208 <--- begins to fall off dramatically
113074176
113967104
98664448
99794944
94420992
89972736
128991232
111230976
80273408
276527360 <--- stopped/started innxmit
254255360
271361536
253618304
271003648
256765952

Best I can tell through watching the CNFS buffer, filesystem, and network metrics, I'm always receiving about the same rate/amount of data throughout the connection lifetime.

Also note the following discrepancies between innxmit and innd:

Jul 18 05:39:12 omnios-inn innxmit[14548]: [ID 738504 news.warning] 192.168.1.75 stats
    offered 27521180 accepted 27520730 refused 0 rejected 418 missing 0 accsize 87812383303 rejsize 696317
Jul 18 05:39:12 omnios-inn innxmit[14548]: [ID 452021 news.warning] 192.168.1.75 times user 476.884
    system 3070.203 elapsed 39294.478

Jul 18 05:39:12 archive innd: [ID 702911 news.notice] 192.168.1.110:29 closed seconds 39295 accepted 27520762
    refused 0 rejected 418 duplicate 32 accepted size 79449063424 duplicate size 99298 rejected size 696317

[jrehmer]

I fed this server over 5TB of articles and then ran news.daily. It's calculation of accept volume is less than half what it ingested.

Incoming volume (innd):
Server                   AcceptVol    DupVol    RejVol  TotalVol %Acc   Vol/Art
192.168.1.110            1983.8 GB   66.3 MB    8.5 GB 1992.3 GB  99%    1.8 KB
192.168.1.35              476.8 MB    0.0 KB   27.0 MB  503.8 MB  94%    3.3 KB
diablo1.usenet.blueworld  384.9 MB    0.0 KB    4.6 MB  389.5 MB  98%    3.3 KB
192.168.1.40               31.0 MB  189.8 MB    1.1 MB  221.9 MB  13%    3.1 KB
diablo2.usenet.blueworld   95.3 KB    0.0 KB    0.6 KB   95.9 KB  99%    4.4 KB
localhost                   0.0 KB    0.0 KB    0.0 KB    0.0 KB   0%         0

TOTAL: 6                 1984.6 GB  256.1 MB    8.6 GB 1993.4 GB  99%    1.8 KB

$ cnfsstat
Class TEXT for groups matching "*"
 Buffer TEXT1, size:   977 GBytes, position:  30.5 MBytes  1.00 cycles
  Newest: 2026-07-08 17:17:49,   33 days, 13:12:33 ago
 Buffer TEXT2, size:   977 GBytes, position:  30.5 MBytes  1.00 cycles
  Newest: 2026-07-11 11:18:50,   30 days, 19:11:32 ago
 Buffer TEXT3, size:   977 GBytes, position:  30.5 MBytes  1.00 cycles
  Newest: 2026-07-15  3:24:00,   27 days,  3:06:22 ago
 Buffer TEXT4, size:   977 GBytes, position:  30.5 MBytes  1.00 cycles
  Newest: 2026-07-18  4:14:34,   24 days,  2:15:48 ago
 Buffer TEXT5, size:   977 GBytes, position:  30.5 MBytes  1.00 cycles
  Newest: 2026-07-21  2:58:21,   21 days,  3:32:01 ago
 Buffer TEXT6, size:   977 GBytes, position:   669 GBytes  0.68 cycles
  Newest: 2026-08-10 18:12:58,    0 days, 12:17:24 ago

[Julien-Elie]

The problem may come from cp->Size (cumulative total of accepted bytes) which is a float in innd whereas STATaccepted is unsigned long in innxmit.
I suspect a float-precision bug, which could explain the "correct at first, degrades over time" symptom. A float can represent integers exactly only up to 2^24 (16 777 216). Beyond that, article sizes are approximate when added to it. For a short-lived connection, cp->Size never gets that big, so the count is correct. But for a long-running connection, every addition of an article size starts silently rounding to the nearest representable float and that number (the "ULP size") keeps growing along with the size.

It may explain the bug. We would need to change the type of our counters (accepted size, duplicated size, etc.) to no longer use floating values, and test that.

[Julien-Elie]

@jrehmer Would it be possible to try the following patch and see whether it fixes the discrepancy with innxmit?
I chose to use unsigned long long (defined in C99) because on 32-bit architectures, unsigned long will not be enough (about 4 GB max).
If the checkpoints now look fine, the sums in innreport will also be correctly computed.

--- a/innd/art.c
+++ b/innd/art.c
@@ -1124,7 +1124,7 @@ ARTreject(Reject_type code, CHANNEL *cp)
     /* Remember why the article was rejected (for the status file) */
 
     cp->Rejected++;
-    cp->RejectSize += (float) (cp->Next - cp->Start);
+    cp->RejectSize += (unsigned long long) (cp->Next - cp->Start);
 
     /* Streaming is used.  Increase the reject counter for TAKETHIS. */
     if (cp->Sendid.size > 3) {
@@ -1134,7 +1134,7 @@ ARTreject(Reject_type code, CHANNEL *cp)
     switch (code) {
     case REJECT_DUPLICATE:
         cp->Duplicate++;
-        cp->DuplicateSize += (float) (cp->Next - cp->Start);
+        cp->DuplicateSize += (unsigned long long) (cp->Next - cp->Start);
         break;
     case REJECT_SITE:
         cp->Unwanted_s++;
@@ -2684,7 +2684,7 @@ ARTpost(CHANNEL *cp)
         cp->ArtMax = i;
     cp->ArtBeg = 0;
 
-    cp->Size += (float) data->BytesValue;
+    cp->Size += (unsigned long long) data->BytesValue;
     if (innconf->logartsize) {
         if (fprintf(Log, " %ld", data->BytesValue) == EOF || ferror(Log)) {
             oerrno = errno;
--- a/innd/chan.c
+++ b/innd/chan.c
@@ -319,22 +319,22 @@ CHANclose_nntp(CHANNEL *cp, const char *name)
     else {
         notice(
             "%s checkpoint seconds %lu accepted %lu refused %lu rejected %lu"
-            " duplicate %lu accepted size %.0f duplicate size %.0f"
-            " rejected size %.0f",
+            " duplicate %lu accepted size %llu duplicate size %llu"
+            " rejected size %llu",
             name, (unsigned long) (Now.tv_sec - cp->Started_checkpoint),
             cp->Received - cp->Received_checkpoint,
             cp->Refused - cp->Refused_checkpoint,
             cp->Rejected - cp->Rejected_checkpoint,
             cp->Duplicate - cp->Duplicate_checkpoint,
-            (double) (cp->Size - cp->Size_checkpoint),
-            (double) (cp->DuplicateSize - cp->DuplicateSize_checkpoint),
-            (double) (cp->RejectSize - cp->RejectSize_checkpoint));
+            cp->Size - cp->Size_checkpoint,
+            cp->DuplicateSize - cp->DuplicateSize_checkpoint,
+            cp->RejectSize - cp->RejectSize_checkpoint);
         notice("%s closed seconds %lu accepted %lu refused %lu rejected %lu"
-               " duplicate %lu accepted size %.0f duplicate size %.0f"
-               " rejected size %.0f",
+               " duplicate %lu accepted size %llu duplicate size %llu"
+               " rejected size %llu",
                name, (unsigned long) (Now.tv_sec - cp->Started), cp->Received,
-               cp->Refused, cp->Rejected, cp->Duplicate, (double) cp->Size,
-               (double) cp->DuplicateSize, (double) cp->RejectSize);
+               cp->Refused, cp->Rejected, cp->Duplicate, cp->Size,
+               cp->DuplicateSize, cp->RejectSize);
     }
     if (cp->Data.Newsgroups.Data != NULL) {
         free(cp->Data.Newsgroups.Data);
--- a/innd/status.c
+++ b/innd/status.c
@@ -33,9 +33,9 @@ typedef struct _STATUS {
     unsigned long Unwanted_g;
     unsigned long Unwanted_s;
     unsigned long Unwanted_f;
-    float Size;
-    float DuplicateSize;
-    float RejectSize;
+    unsigned long long Size;
+    unsigned long long DuplicateSize;
+    unsigned long long RejectSize;
     unsigned long Check;
     unsigned long Check_send;
     unsigned long Check_deferred;
@@ -80,7 +80,7 @@ STATUSinit(void)
 }
 
 static char *
-PrettySize(float size, char *str)
+PrettySize(unsigned long long size, char *str)
 {
     if (size > 1073741824) /* 1024*1024*1024 */
         sprintf(str, "%.1fGb", (double) size / 1073741824.);
@@ -105,9 +105,9 @@ STATUSsummary(void)
     unsigned long accepted = 0;
     unsigned long refused = 0;
     unsigned long rejected = 0;
-    float size = 0;
-    float DuplicateSize = 0;
-    float RejectSize = 0;
+    unsigned long long size = 0;
+    unsigned long long DuplicateSize = 0;
+    unsigned long long RejectSize = 0;
     int peers = 0;
     char TempString[SMBUF];
     char *path;
@@ -300,7 +300,7 @@ STATUSsummary(void)
     if (offered == 0)
         offered = 1; /* to avoid division by zero */
     if (!(size > 0))
-        size = 1.; /* avoid divide by zero here too */
+        size = 1; /* avoid divide by zero here too */
     fprintf(F, "        accepted: %-9lu       %%accepted: %.1f%%\n", accepted,
             (double) accepted / (double) offered * 100);
     fprintf(F, "         refused: %-9lu        %%refused: %.1f%%\n", refused,
@@ -313,18 +313,18 @@ STATUSsummary(void)
             PrettySize(size + DuplicateSize + RejectSize, str));
     fprintf(F, " duplicated size: %-7s  %%duplicated size: %.1f%%\n",
             PrettySize(DuplicateSize, str),
-            (double) (DuplicateSize / size * 100));
+            (double) DuplicateSize / (double) size * 100);
     fprintf(F, "   rejected size: %-7s    %%rejected size: %.1f%%\n",
-            PrettySize(RejectSize, str), (double) (RejectSize / size * 100));
+            PrettySize(RejectSize, str),
+            (double) RejectSize / (double) size * 100);
     fputc('\n', F);
 
     if (innconf->logstatus) {
         notice("%s status seconds %lu accepted %lu "
                "refused %lu rejected %lu duplicate %lu "
-               "accepted size %.0f duplicate size %.0f rejected size %.0f\n",
+               "accepted size %llu duplicate size %llu rejected size %llu\n",
                "ME", (unsigned long) seconds, accepted, refused, rejected,
-               duplicate, (double) size, (double) DuplicateSize,
-               (double) RejectSize);
+               duplicate, size, DuplicateSize, RejectSize);
     }
 
     /* Incoming Feeds */
@@ -375,11 +375,11 @@ STATUSsummary(void)
             notice(
                 "%s status seconds %lu accepted %lu "
                 "refused %lu rejected %lu duplicate %lu "
-                "accepted size %.0f duplicate size %.0f rejected size %.0f\n",
+                "accepted size %llu duplicate size %llu rejected size %llu\n",
                 status->name, (unsigned long) status->seconds,
                 status->accepted, status->refused, status->rejected,
-                status->Duplicate, (double) status->Size,
-                (double) status->DuplicateSize, (double) status->RejectSize);
+                status->Duplicate, status->Size, status->DuplicateSize,
+                status->RejectSize);
         }
 
         tmp = status->next;
--- a/innd/innd.h
+++ b/innd/innd.h
@@ -353,12 +353,12 @@ typedef struct _CHANNEL {
     unsigned long Unwanted_g;
     unsigned long Unwanted_u;
     unsigned long Unwanted_o;
-    float Size;
-    float Size_checkpoint;
-    float DuplicateSize;
-    float DuplicateSize_checkpoint;
-    float RejectSize;
-    float RejectSize_checkpoint;
+    unsigned long long Size;
+    unsigned long long Size_checkpoint;
+    unsigned long long DuplicateSize;
+    unsigned long long DuplicateSize_checkpoint;
+    unsigned long long RejectSize;
+    unsigned long long RejectSize_checkpoint;
     unsigned long Check;
     unsigned long Check_send;
     unsigned long Check_deferred;
--- a/innd/nc.c
+++ b/innd/nc.c
@@ -274,16 +274,16 @@ NCpostit(CHANNEL *cp)
         syslog(L_NOTICE,
                "%s checkpoint seconds %lu accepted %lu refused %lu rejected "
                "%lu duplicate %lu"
-               " accepted size %.0f duplicate size %.0f rejected size %.0f",
+               " accepted size %llu duplicate size %llu rejected size %llu",
                CHANname(cp),
                (unsigned long) (Now.tv_sec - cp->Started_checkpoint),
                cp->Received - cp->Received_checkpoint,
                cp->Refused - cp->Refused_checkpoint,
                cp->Rejected - cp->Rejected_checkpoint,
                cp->Duplicate - cp->Duplicate_checkpoint,
-               (double) (cp->Size - cp->Size_checkpoint),
-               (double) (cp->DuplicateSize - cp->DuplicateSize_checkpoint),
-               (double) (cp->RejectSize - cp->RejectSize_checkpoint));
+               cp->Size - cp->Size_checkpoint,
+               cp->DuplicateSize - cp->DuplicateSize_checkpoint,
+               cp->RejectSize - cp->RejectSize_checkpoint);
         cp->Reported = 0;
         cp->Started_checkpoint = Now.tv_sec;
         cp->Received_checkpoint = cp->Received;
@@ -1420,7 +1420,8 @@ NCproc(CHANNEL *cp)
                 || cp->State == CSeatarticle) {
                 if (cp->Next - cp->Start > innconf->datamovethreshold
                     || (innconf->maxartsize != 0
-                        && cp->Size > (float) innconf->maxartsize)) {
+                        && cp->Size
+                               > (unsigned long long) innconf->maxartsize)) {
                     /* avoid buffer extension for ever */
                     movedata = true;
                 } else {