Speed up members newsletter list by paginating MailDelivery loads

[thibaudgg]

Problem

AppSignal production traces for Members::NewsletterDeliveriesController#index (/newsletters) show a p95 regression (~217ms → ~363ms) with slow samples of 0.8–1.4s across tenants.

On a 1447ms sample (stadsgroenteboer, rev 0760de0…), the 1313.7ms hotspot SQL is:

SELECT "mail_deliveries".* FROM "mail_deliveries"
WHERE "mail_deliveries"."mailable_type" = ?
  AND "mail_deliveries"."state" NOT IN (?, ?)
  AND "mail_deliveries"."member_id" = ?
ORDER BY "mail_deliveries"."created_at" DESC

No LIMIT. This matches Newsletter.deliveries_for, which loaded every processed newsletter delivery for the member (including HTML content) before the controller paginated. That second page query also dropped the in-memory newsletter preload, so _newsletter.html.erb N+1’d Newsletter per row.

Change

1. Paginate first. deliveries_for is now an unloaded relation. The index action loads one page + 1 (LIMIT 21) to decide “show more”, and omits content.
2. Index the list/exists shape. Added idx_mail_deliveries_on_member_mailable_created on (member_id, mailable_type, created_at). A leading state column (as in (member_id, mailable_type, state, created_at)) made SQLite keep using the older (member_id, created_at) index for ORDER BY created_at; EXPLAIN confirms the three-column index is used. state NOT IN (draft, processing) is applied after the seek.
3. Eager-load Newsletter. MailDelivery belongs to source_newsletter via the virtual mailable_id column. deliveries_for uses includes(:source_newsletter), so the collection partial does not query per row.

Behavior/UI is unchanged: same 20-item list, same “show more” offset pagination, same processed-only filter.

Tests

• Index still renders the latest processed deliveries and paginates.
• Query count stays bounded when delivery count grows (25 → 65): the paginated MailDelivery load must be an explicit column list with LIMIT/OFFSET, include subject, and omit content (a SELECT * fails this check).
• EXPLAIN QUERY PLAN uses the new index for deliveries_for.

Review follow-up

Dropped unused Newsletter.preload_sources! (admin still inlines its own preload). Removed the dead Extra 24 pagination assertion. Tightened the content-omit SQL check so SELECT * fails.

Notes

The MailDelivery Exists? gate is unchanged. The large win is avoiding the unbounded SELECT *.

  

[thibaudgg]

The page-size load is the actual fix. Newsletter.preload_sources! has no remaining app caller — includes(:source_newsletter) already does that job. I'd drop the method and the test that only exists for it.

Two assertions in the controller test can't fail: Extra 24 is never inserted (PER_PAGE extras only), and the content-omit check still passes on SELECT *.

[thibaudgg]

No app caller left for this. Admin still inlines its own preload. Safe to delete with the test below.

[thibaudgg]

This insert only creates Extra 0–19, so Extra 24 can never show up. Page 2 is already covered by Subject John Doe appearing.