Indirect calls: function pointers and macro bodies

[rikvanriel]

Indirect calls: function pointers and macro bodies

semcode could not answer the question in issue #9:

semcode> callers tcp_v4_rcv
Info: No functions call 'tcp_v4_rcv'

Nothing calls it by name. The kernel reaches it through net_protocol::handler,
installed once in inet_init and dispatched through in ip_protocol_deliver_rcu.
Every fact needed to say so was in the tree, and none of it was indexed.

The same gap swallowed macros. A #define was stored as text: the calls in its
body were not edges, its parameters were not recorded, and a dispatch or a
registration written inside one existed nowhere.

This series indexes both. A call that goes through a member is recorded as a
dispatch site; a function installed in a member is recorded as a
registration; a query joins them. Macro bodies are parsed with the C grammar
and contribute the same three kinds of fact as any other code.

Three commands are new — implementors, registrations, and indirect results
in callers — and 19 existing MCP tools gain the same answers.

---

Function pointers

1. Who can reach a function nothing calls by name

semcode> callers tcp_v4_rcv

before:

Info: No functions call 'tcp_v4_rcv'

after:

=== Indirect Callers ===
1 call sites can reach it through a function pointer:
  1. ip_protocol_deliver_rcu at net/ipv4/ip_input.c:207 [macro_declared]
     names it at the call site

Note: 25 further call sites go through a member of the same name, but
nothing says their receiver has the type the function was installed in.

The answer comes from INDIRECT_CALL_2(ipprot->handler, tcp_v4_rcv, udp_rcv, skb),
where the macro names its own likely targets. The note is the honest remainder:
25 other calls through a member called handler whose receiver type is unknown.
They are counted, not listed, because they are evidence of a different quality.

2. The VFS shape, resolved through the types table

semcode> callers seq_read

before:

=== Direct Callers ===
1 functions directly call 'seq_read':
  1. pstore_file_read

after:

=== Direct Callers ===
1 functions directly call 'seq_read':
  1. pstore_file_read

=== Indirect Callers ===
3 call sites can reach it through a function pointer:
  1. do_loop_readv_writev at fs/read_write.c:848 [member_arrow]
     installed in file_operations::read at arch/arm64/kvm/ptdump.c:204
     and 343 other places (receiver type matches)
  2. loop_rw_iter at io_uring/rw.c:733 [member_arrow]
     installed in file_operations::read at arch/arm64/kvm/ptdump.c:204
     and 343 other places (receiver type matches)
  3. vfs_read at fs/read_write.c:572 [member_arrow]
     installed in file_operations::read at arch/arm64/kvm/ptdump.c:204
     and 343 other places (receiver type matches)

vfs_read writes file->f_op->read(...). Typing that receiver takes two steps:
the calling file proves file is a struct file, and the types table says
f_op is a struct file_operations *. The first happens while parsing, the
second at query time, because the header declaring struct file is a different
file.

A site is one answer however many places install the target. seq_read sits in
that member in 344 of them, and the answer names one and counts the rest.

3. Two questions that could not be asked at all

semcode> implementors file_operations.read

before:

Error: Unknown command: 'implementors'. Type 'help' for available commands.

after:

=== Implementors ===
1443 installed:
  1. seq_read at .../octeontx2/af/cn20k/debugfs.c:160 in __OCTEONTX2_DEBUGFS_ATTRIBUTE_FOPS [designated_init]
  2. dev_read at drivers/net/ethernet/mellanox/mlx5/core/lib/sd.c:772 [designated_init]
  3. iwl_dbgfs_monitor_data_read at .../iwlwifi/pcie/gen1_2/trans.c:3283 [designated_init]
  ...

and the reverse:

semcode> registrations tcp_v4_rcv

before:

Error: Unknown command: 'registrations'. Type 'help' for available commands.

after:

=== Registrations ===
1 places install it:
  1. net_protocol::handler at net/ipv4/af_inet.c:1934 in inet_init [designated_init]

"What can be called here?" and "where is this hooked up?" are the two questions
review actually asks about a callback, and both are one lookup.

---

Macros

4. A macro body is code

semcode> func wait_event

before:

Declaration:  wait_event(void)

#define wait_event(wq_head, condition)          \
do {                                            \
        might_sleep();                          \
        if (condition)                          \
                break;                          \
        __wait_event(wq_head, condition);       \
} while (0)

Called By: 416

after:

Declaration:  wait_event( wq_head,  condition)

[same definition]

Calls: 2
  → __wait_event
  → might_sleep

Called By: 416

Before, the body was text: no parameters, no callees. 416 callers of
wait_event led nowhere, because the chain stopped at the macro. Now the two
functions it calls are edges, and the parameters it takes are recorded — for a
Linux tree, 133,341 macros gained the 214,769 parameters they declare.

The body is parsed with the C grammar rather than scanned, which is what keeps
while, if and sizeof from being read as callees: 13,137 such edges do not
exist, out of 18,806 wrong ones removed.

5. A macro that dispatches

semcode> func nvkm_memory_target

before:

Declaration:  nvkm_memory_target(void)
#define nvkm_memory_target(p) (p)->func->target(p)

Called By: 22

after:

Declaration:  nvkm_memory_target( p)
#define nvkm_memory_target(p) (p)->func->target(p)

Called By: 22

The definition is unchanged, and so is the absence of a Calls: line — on
purpose. target is a member, not a function, and the accessor is recorded as
a dispatch site through target instead of as a call to something that does
not exist. Whole subsystems put their indirection in a macro like this: 800
dispatch sites in a Linux tree are written inside a macro body.

6. A macro that installs

semcode> func DEFINE_PMC_CORE_ATTR_WRITE

#define DEFINE_PMC_CORE_ATTR_WRITE(__name)                      \
static int __name ## _open(struct inode *inode, struct file *file) \
{ ... }                                                         \
static const struct file_operations __name ## _fops = {         \
        .owner          = THIS_MODULE,                          \
        .open           = __name ## _open,                      \
        .read           = seq_read,                             \
        .write          = __name ## _write,                     \
        .release        = single_release,                       \
}

before: the five installations in that table were not recorded anywhere, so
seq_read did not appear to be installed by this driver at all.

after:

semcode> registrations seq_read
  5. file_operations::read at drivers/platform/x86/intel/pmc/core.h:648
     in DEFINE_PMC_CORE_ATTR_WRITE [designated_init]
114. file_operations::read at drivers/gpu/drm/i915/gt/intel_gt_debugfs.h:17
     in __GT_DEBUGFS_ATTRIBUTE_FOPS [designated_init]
174. file_operations::read at drivers/nvme/target/debugfs.c:24
     in NVMET_DEBUGFS_ATTR [designated_init]

Macros that declare an ops table are how a large part of the kernel installs
callbacks; 1,955 registrations in a Linux tree are written inside one. A body
that states no type of its own — a bare { .read = seq_read } — registers
nothing, since the type would have to be guessed from context.

---

At kernel scale

Whole tree, 1,020,780 functions:

dispatch sites	72,605 — 62,130 through ->, 7,952 through ., 1,378 through a parameter, 530 a local, 437 a dereferenced pointer, 178 named by an indirect-call macro
registrations	575,240 — 330,226 designated initializers, 245,014 assignments
sites with a receiver type	19,269 typed while parsing, 22,042 more resolved through the types table
call edges removed	19,389 that named a keyword, a member, or an assembler directive rather than a function
macro parameters gained	214,769 across 133,341 macros

What this does not do

• A receiver the file does not declare stays untyped, and a chain longer than
  one step (a->b->c->m()) is not resolved. Both are counted in the note, not
  guessed at.
• callchain does not yet expand through an indirect call; resolution returns
  the candidates, and using them there is the next patch.
• There is no policy knob yet for how much fanout to show. Answers are grouped
  by site and capped by the caller.
• Registrations record the target as written, so a member set to a constant is
  stored too. The join against known functions filters those inertly.

Series layout

Four branches, applied in order:

funcpointers                      5   shared prerequisites: candidate-returning
                                      lookups, declarator parsing, loud failure on
                                      an unparsable edge list
funcpointers-indirect-call-sites +5   dispatch sites: member calls, pointer
                                      variables, INDIRECT_CALL_* targets
funcpointers-registrations       +7   registrations, the reverse query, the two new
                                      commands, MCP, end-to-end tests
funcpointers-macro-bodies        +5   macro bodies parsed as C, with the sites and
                                      registrations inside them
funcpointers-receiver-typing     +5   receiver typing, chained receivers resolved
                                      through the types table, one answer per site

27 patches in total, each branch stacked on the one above it.

Every patch builds and passes cargo test, cargo clippy -D warnings and
cargo fmt --check on its own. Each was also run against two real trees, chrony
and Linux, comparing every table's row counts against the previous patch and
re-indexing to confirm the result is idempotent. Three defects in this series
were invisible to the unit tests and caught only by that comparison.

Closes #9
Closes #35

[chucklever]

The separation is right. I tried folding the references into the calls
column on a branch of my own, so that .handler = tcp_v4_rcv makes
inet_init a caller. That answers "who mentions it," and issue #9 asks who
reaches it, which is ip_protocol_deliver_rcu. It also makes calls mean two
things at once, and callchain then walks ops tables as though they were call
edges. Storing the dispatch site and the registration as separate rows and
joining them at query time is what produces the dispatcher. Grading the
evidence is what lets a reader check the answer instead of taking it.

Four things I would want fixed before this goes in.

1. Neither new table gets a scalar index

create_scalar_indices() in src/database/schema.rs:495 has no entry for
dispatch_sites or registrations. find_by_target() and find_by_member()
filter 575,240 and 72,605 rows unindexed, on your own counts.
find_indirect_callers() issues one find_by_member() per distinct member the
target sits in, so a function registered in several slots pays that cost once
per slot.

Adding entries to create_scalar_indices() will not reach a database that
already exists. That function returns early as soon as functions holds more
than 100 rows. The indices have to be created in create_dispatch_sites_table()
and create_registrations_table(), or the early return has to go per-table.
Columns that want one: registrations(target, member, container_type) and
dispatch_sites(member, target, caller_name).

2. type_chained_receivers() reads every version of the type

TypeStore::find_all_by_name() applies no manifest filter, so it returns every
struct file the database holds, including the same struct at older commits.
The rule that every definition must agree on the field then reads one struct at
two commits as a conflict, and the field resolves to None. Index a range that
spans a change to any such field and receiver typing switches off for it. Every
site that depended on it falls out of "receiver type matches" and into the
count at the bottom, and nothing says why. find_indirect_callers() is holding
the manifest already.

3. The upgrade path stamps more than it reads

run_pipeline() replaces the requested range with process_git_tree() at HEAD,
and process_git_tree() calls record_schema_version() when it finishes. A
database built from a commit range keeps version 1 rows for every commit that
is not HEAD and now claims version 2, so those files are never read again. That
is the silence your own commit message argues against, one level up. The path
also drops an explicit --git A..B after printing a line about something else.

Recording the version per file rather than per database collapses the whole
case. Put it in processed_files as a column and include it in the dedup key.
A file indexed by an older extractor then fails the skip test on its own, the
range path works, and the special case in index.rs goes away.

4. The reader never checks the version

schema.rs says a reader that does not understand a version must refuse rather
than guess. index_predates_reader() is called only from the indexer. Point
semcode at a version 1 database and implementors file_operations.read
answers "Nothing is installed," which is exactly what an empty slot looks like.
That is the failure parse_call_list() was changed to stop.

Numbers I am missing

The series reports row counts and no times. Three parse attempts per macro body
across 133,341 macros, and an unindexed scan of the registrations table on
every callers, are both places where this turns from slower into unusable.
What is the indexing wall clock before and after, and what does
callers seq_read cost on a kernel database?

The part that still hurts

I counted member call chains by depth in net/sunrpc and fs/nfsd, since that is
the code I read most:

  69   ops->m()
  95   xprt->xpt_ops->xpo_recvfrom()
   6   rqstp->rq_xprt->xpt_ops->xpo_result_payload()
   2   inode->i_sb->s_export_op->block_ops->commit_blocks()

The second row is the dominant SUNRPC form and the types table resolves it, so
the one-step limit costs 8 sites out of 172 here. Chain depth is not what hurts.

What hurts is the receiver that never gets typed at all. Your table says 41,311
of 72,605 sites end up with a receiver type, so 43% of dispatch sites can only
ever appear as the number in the note. For those, callers prints a count and
the reader goes back to grep. The note is the right thing to print. It is still
most of a large subsystem's indirection.

Typedefs split the join in a way worth calling out. aggregate_type_name()
records a typedef name as a container, so the registration is stored under it.
aggregate_of() returns None for a bare name, so a chained receiver whose
field is declared as that typedef never resolves to the same key. The two
halves cannot meet there. The kernel mostly spells out struct, so this costs
little today. It will not stay cheap in the C++ you say you are aiming at.

Then callchain. callers answers issue #9 and callchain does not. In
SUNRPC every chain worth following crosses a transport dispatch:
callchain svc_recv stops at xprt->xpt_ops->xpo_recvfrom today and still
stops there after this series. The candidates are resolved and sitting right
there by then. I would take that patch ahead of the C++ front end.

Smaller items go inline.

[chucklever]

The smaller items, as promised. None of these blocks.

I see the two commits since. query: refuse to answer from an index older than this build covers point 4 above. I will read both properly rather than comment from the subject lines.

One item has no line to anchor to: docs/schema.md documents 11 tables and this adds dispatch_sites, registrations and schema_meta. CLAUDE.md points at that file as the schema of record. docs/semcode-mcp.md wants the new tools too, since the test now asserts 19.

[chucklever]

caller_name is declared nullable and the comment above it describes sites that sit outside any function. insert_batch() calls append_value(&site.caller_name) unconditionally, and site_from_batch() reads it through text(), which calls .value(row) with no null check.

Nothing writes a null today. If something did, it would read back as an empty string, which is already what the file-scope case stores. Make the column non-null, or read it through the optional path.

[chucklever]

These two keys are written here and never read anywhere.

The doc comment above stored_schema_version() says the marks record when a feature started being populated, and that this is what makes a backfill decidable. Nothing decides a backfill from them. Wire them into the staleness check, or drop them so the comment stops describing a mechanism that is not there.

[chucklever]

dedup() drops adjacent equal elements, and the sort above orders by caller_name, site_file and site_line. Entries that share those three but differ in evidence can sit between two that are fully equal, so the duplicates survive.

It bites when one site yields two identical rows, which the byte_start in the merge key allows: two dispatch sites on the same line become two rows in the table and one IndirectCaller each, since IndirectCaller carries no byte offset. Sorting on the full tuple, or collecting through a HashSet, closes it.

[chucklever]

get_function_callers_git_aware() on the line above generates a manifest, and this generates another. generate_git_manifest() walks the whole tree through walk_tree_at_commit() with no cache, so callers now costs two full tree walks per invocation. On a kernel that is roughly 90k entries, twice.

Threading one manifest down, or memoizing it on the SHA, gets it back to one.

[chucklever]

When confident is empty, this prints with no section header above it, so the reader gets a bare Note: line hanging off the direct callers. further reads oddly there too, since nothing was listed before it.

Print the === Indirect Callers === header whenever either list is non-empty, and word the count for the case where it is the whole answer.

[chucklever]

.long and .short are caught here because they happen to be C keywords. The population you are filtering is assembler directives, and most of them are not: .align, .byte, .word, .quad, .zero, .fill, .org, .set, .type, .size, .rept. .align (n) in a macro body still reads as a dispatch through a member named align.

The list is C89 and C99 only, so _Bool, _Generic, _Alignof and _Static_assert fall through as well. See the note on parse_macro_body() for a filter that covers the class rather than the instances.

[chucklever]

parse_macro_body() returns whether the parse came out clean, and this discards it. Its own doc says a body that parses with errors still yields usable call names, but that its structure is whatever error recovery invented.

sites and registrations further down are nothing but structure, and they are harvested from that tree either way. Gating those two on the clean flag while keeping calls follows the comment, and it covers the assembler case without enumerating directives.

[rikvanriel]

That makes sense, but when Claude tried it, it didn't actually work out. I will fix all the other comments, and update the pull request. Analysis below:

Where I want to push back, and the data for it

One review point, comment (h) on parse_macro_body(), together with the
directive list in comment (g). The diagnosis is right: the keyword filter
catches .long and .short because they happen to be C keywords, and that is
not the class being filtered. Two remedies were proposed. Both are more
expensive than the defect, and one of them removes real answers.

All numbers are from a whole Linux tree, 1,020,780 functions, indexed with the
series applied.

What the defect actually costs

The keyword rule removes 29 dispatch sites — 27 through ., 2 through ->
— and 0 registrations. Every one is an assembler directive read as a member:
_ASM_EXTABLE_HANDLE, __ASM_EXTABLE_RAW on arm64, loongarch and riscv, the
powerpc instruction macros.

Remedy 1: harvest structure only from an error-free parse

Measured, because this is the version I built first and then rejected:

	with the gate
"~/text/semcode-review-pushback.md" 91L, 4090B 1,1 Top

Where I want to push back, and the data for it

One review point, comment (h) on parse_macro_body(), together with the
directive list in comment (g). The diagnosis is right: the keyword filter
catches .long and .short because they happen to be C keywords, and that is
not the class being filtered. Two remedies were proposed. Both are more
expensive than the defect, and one of them removes real answers.

All numbers are from a whole Linux tree, 1,020,780 functions, indexed with the
series applied.

What the defect actually costs

Where I want to push back, and the data for it

One review point, comment (h) on parse_macro_body(), together with the
directive list in comment (g). The diagnosis is right: the keyword filter
catches .long and .short because they happen to be C keywords, and that is
not the class being filtered. Two remedies were proposed. Both are more
expensive than the defect, and one of them removes real answers.

All numbers are from a whole Linux tree, 1,020,780 functions, indexed with the
series applied.

What the defect actually costs

The keyword rule removes 29 dispatch sites — 27 through ., 2 through ->
— and 0 registrations. Every one is an assembler directive read as a member:
_ASM_EXTABLE_HANDLE, __ASM_EXTABLE_RAW on arm64, loongarch and riscv, the
powerpc instruction macros.

Remedy 1: harvest structure only from an error-free parse

Measured, because this is the version I built first and then rejected:

	with the gate

                                                          1,1           Top

Where I want to push back, and the data for it

One review point, comment (h) on parse_macro_body(), together with the
directive list in comment (g). The diagnosis is right: the keyword filter
catches .long and .short because they happen to be C keywords, and that is
not the class being filtered. Two remedies were proposed. Both are more
expensive than the defect, and one of them removes real answers.

All numbers are from a whole Linux tree, 1,020,780 functions, indexed with the
series applied.

What the defect actually costs

The keyword rule removes 29 dispatch sites — 27 through ., 2 through ->
— and 0 registrations. Every one is an assembler directive read as a member:
_ASM_EXTABLE_HANDLE, __ASM_EXTABLE_RAW on arm64, loongarch and riscv, the
powerpc instruction macros.

Remedy 1: harvest structure only from an error-free parse

Measured, because this is the version I built first and then rejected:

	with the gate

                                                          1,1           Top

Where I want to push back, and the data for it

One review point, comment (h) on parse_macro_body(), together with the
directive list in comment (g). The diagnosis is right: the keyword filter
catches .long and .short because they happen to be C keywords, and that is
not the class being filtered. Two remedies were proposed. Both are more
expensive than the defect, and one of them removes real answers.

All numbers are from a whole Linux tree, 1,020,780 functions, indexed with the
series applied.

What the defect actually costs

The keyword rule removes 29 dispatch sites — 27 through ., 2 through ->
— and 0 registrations. Every one is an assembler directive read as a member:
_ASM_EXTABLE_HANDLE, __ASM_EXTABLE_RAW on arm64, loongarch and riscv, the
powerpc instruction macros.

Remedy 1: harvest structure only from an error-free parse

Measured, because this is the version I built first and then rejected:

	with the gate

                                                          1,1           Top

Where I want to push back, and the data for it

One review point, comment (h) on parse_macro_body(), together with the
directive list in comment (g). The diagnosis is right: the keyword filter
catches .long and .short because they happen to be C keywords, and that is
not the class being filtered. Two remedies were proposed. Both are more
expensive than the defect, and one of them removes real answers.

All numbers are from a whole Linux tree, 1,020,780 functions, indexed with the
series applied.

What the defect actually costs

The keyword rule removes 29 dispatch sites — 27 through ., 2 through ->
— and 0 registrations. Every one is an assembler directive read as a member:
_ASM_EXTABLE_HANDLE, __ASM_EXTABLE_RAW on arm64, loongarch and riscv, the
powerpc instruction macros.

Remedy 1: harvest structure only from an error-free parse

Measured, because this is the version I built first and then rejected:

	with the gate

                                                          1,1           Top

[riel@devbig015.atn7 ~/local/semcode (scratch/riel/funcpointers-receiver-typing)]$ curl -o /tmp/feedback #50
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Couldn't connect to server
[riel@devbig015.atn7 ~/local/semcode (scratch/riel/funcpointers-receiver-typing)]$ curl $(fwdproxy-config curl) -o /tmp/feedback #50
Downloading fwdproxy-config-cli...
[ ]Uses thrift-py-deprecated. Migrate to thrift-python. See https://fburl.com/thrift-python and https://fburl.com/wiki/jihy02dr. Future automatic thrift-py-deprecated code generation may stop for non-migrated targets: https://fburl.com/workplace/wer48s4m.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 654k 0 654k 0 0 402k 0 --:--:-- 0:00:01 --:--:-- 403k
[riel@devbig015.atn7 ~/local/semcode (scratch/riel/funcpointers-receiver-typing)]$ less /tmp/feedback
[riel@devbig015.atn7 ~/local/semcode (scratch/riel/funcpointers-receiver-typing)]$
[riel@devbig015.atn7 ~/local/semcode (scratch/riel/funcpointers-receiver-typing)]$ vi ~/text/semcode-review-pushback.md
[riel@devbig015.atn7 ~/local/semcode (scratch/riel/funcpointers-receiver-typing)]$ cat !$
cat ~/text/semcode-review-pushback.md

Where I want to push back, and the data for it

One review point, comment (h) on parse_macro_body(), together with the
directive list in comment (g). The diagnosis is right: the keyword filter
catches .long and .short because they happen to be C keywords, and that is
not the class being filtered. Two remedies were proposed. Both are more
expensive than the defect, and one of them removes real answers.

All numbers are from a whole Linux tree, 1,020,780 functions, indexed with the
series applied.

What the defect actually costs

The keyword rule removes 29 dispatch sites — 27 through ., 2 through ->
— and 0 registrations. Every one is an assembler directive read as a member:
_ASM_EXTABLE_HANDLE, __ASM_EXTABLE_RAW on arm64, loongarch and riscv, the
powerpc instruction macros.

Remedy 1: harvest structure only from an error-free parse

Measured, because this is the version I built first and then rejected:

	with the gate
dispatch sites	−134
registrations	−819

819 is 42% of everything macro-body registrations find. The reason is in the
gate's own premise: a body that fails to parse cleanly can still state plainly
which function it installs. DEFINE_PMC_CORE_ATTR_WRITE and the other
FOPS-defining macros are full of token pasting that the C grammar cannot read,
and their .read = seq_read is unambiguous anyway.

Paying 953 rows to remove 29 wrong ones is the wrong direction, when the 29 can
be removed exactly.

Remedy 2: filter the assembler directives by name

This one removes correct answers. Those names are ordinary struct members in
ordinary C, and nouveau in particular is full of them. Counting sites in the
kernel whose member is one of the proposed names, with the keyword filter
disabled so nothing is hidden:

set   131      zero   5
size   37      type   2
align   9      rept   2
fill    6      word   1

193 sites, and the ones I read are all real:

nvkm_outp_bl_set    outp->ior->func->bl.set(...)
nvkm_fb_vidmem_size fb->func->vidmem.size(...)
shadow_image        mthd->func.size(...)
nvkm_memory_size    (p)->func->size(p)

.set and .size are function-pointer members of nvkm_ior_func_bl and
nvkm_fb_func_vidmem. Filtering the directive names deletes those dispatches
to remove none of the 29, since not one of the 29 is named after a
non-keyword directive.

The predicted population does not appear. .align (n) inside an assembly
macro body was expected to read as a dispatch through align; all 9 align
sites in the tree have real receivers and real callers. Assembly bodies that
are not keyword-shaped do not survive the parse as field expressions at all.

Why the keyword rule is the right shape

It is sound rather than heuristic: C forbids a member named long, so a site
claiming one is a misparse, always, with no tree-specific judgement. The
enumerated-directive rule is unsound in both directions — it misses directives
nobody listed and deletes members that share a name with one.

What I will change

The review is right that the list is incomplete. C11 and C23 keywords are
missing: _Bool, _Generic, _Alignof, _Static_assert, _Atomic,
_Complex, _Imaginary, _Noreturn, _Thread_local, alignas, alignof,
bool, constexpr, false, nullptr, static_assert, thread_local,
true, typeof, typeof_unqual. Those go in, and the rule keeps its
justification: none of them can name a member.

A second sound signal is available and I will take it as well, though it is
small: a member dispatch with an empty receiver expression is not a
dispatch. With the keyword filter disabled, 12 sites have one, all of them
assembly (long 11, if 1). They are a subset of the 29 today, so this adds
nothing immediately — it is there for the next construct that produces
structure with nothing on the left of the dot, which is the shape both
assembler cases took.

Net: the class gets covered by two rules that cannot be wrong, instead of one
list that can be wrong twice, and no correct row is lost.

[chucklever]

You are right, and the measurement settles it. 819 registrations is six times
what the gate would remove, and the reason is the one I missed: a body full of
token pasting never parses cleanly, and its .read = seq_read is unambiguous
regardless.

The directive list is worse than that. I proposed set, size and align
without checking whether they name real members, and outp->ior->func->bl.set()
says they do. Withdrawn.

One thing to carry over. parse_macro_body() still tells the reader that its
structure is whatever error recovery invented, at src/treesitter_analyzer.rs:3370.
The code now leans on that structure on purpose and you have the numbers for
why. Worth saying so there, or the next reader files the comment I just filed.

[rikvanriel]

Four things I would want fixed before this goes in.

Thank you for flagging these issues, Chuck! While addressing the issues you flagged, a whole bunch of other corner cases were discovered, and fixed along the way.

Here is the changelog since your first review:

Every example below is real output. before is the binary at the series
base, after is the tip, both against an index of the same Linux tree.

---

1. Answering the review

Four blockers and eight inline comments, all addressed:

• Scalar indices on both new tables, created with the tables rather than
  in create_scalar_indices(), which returns as soon as the database holds
  rows and so would have reached neither an existing index nor a fresh one.
• Chained receivers resolve at the queried revision. The types table
  holds the same struct at every indexed commit; reading all of them made two
  revisions of one struct look like two definitions disagreeing, which turned
  typing off for that field silently.
• The extractor version lives on each file, in processed_files, inside
  the dedup key — replacing the database-wide mark and, with it, three
  special cases. --git A..B is honoured again rather than being quietly
  replaced by a tree read.
• A stale index is refused, not answered from — by the query command and
  by MCP, which returns it as fields (index_stale, written_by, expected,
  command) rather than prose for a model to interpret.
• Smaller: the answer key includes the byte offset, so two dispatches on one
  line stop merging into one answer with both their counts; the keyword list
  covers C11 and C23; a member call with no receiver is rejected; the indirect
  heading prints whenever there is anything indirect; caller_name is
  declared as what is stored in it; the marks nothing read are gone; the three
  new tables and two new tools are documented.

One review suggestion was declined with a measurement: filtering assembler
directives by name would delete 193 correct sites, because set, size,
align and word are ordinary members — outp->ior->func->bl.set() in
nouveau. The keyword rule covers the same population soundly. Detail in
semcode-review-pushback.md.

2. Defects found by running it

• Re-indexing an existing database failed a batch at a time. A file can
  define one name twice, lance rejects a merge holding two rows for one target
  row, and the whole batch is lost with it. Invisible until now because a fresh
  index has no target row to collide with, and re-indexing skipped every file.
  A fresh index of Linux also holds 1,606 fewer function rows for the fix.
• A macro's dispatch was recorded as a call to its member.
  #define nvkm_memory_target(p) (p)->func->target(p) claimed a call to
  target, which is not a function. The scan that rescues unparsable bodies
  ran whenever no callee was named, and re-read the member as one.
• callers walked the whole tree twice for two manifests, once for direct
  callers and once for indirect.

Both harness blind spots behind these are closed: the regression gate now
re-indexes an existing database as well as building a fresh one.

3. New capability

• Chained receivers of any depth. display->parent->dsb->read() records
  the whole path and resolution walks it. 7,965 of 30,007 chained receivers on
  a Linux tree read more than one field.
• A typedef is a container. Registrations were filed under typedef names
  while resolution refused them, so the two halves of the join could not meet.
• Call chains no longer stop at a dispatch, in the command and in MCP.

4. Performance

callers seq_read      3.06 s -> 2.19 s
callers tcp_v4_rcv    2.62 s -> 1.73 s
indexing Linux        79.3 s -> ~90 s   (+13%, for everything above)

---

Before and after, in the REPL

The question in issue #9

semcode> callers tcp_v4_rcv

before:

Info: No functions call 'tcp_v4_rcv'

after:

=== Indirect Callers ===
1 call sites can reach it through a function pointer:
  1. ip_protocol_deliver_rcu at net/ipv4/ip_input.c:207 [macro_declared]
     names it at the call site

Note: 27 further call sites go through a member of the same name, but
nothing says their receiver has the type the function was installed in.

The VFS shape

semcode> callers seq_read

before:

=== Direct Callers ===
1 functions directly call 'seq_read':
  1. pstore_file_read

after:

=== Direct Callers ===
1 functions directly call 'seq_read':
  1. pstore_file_read

=== Indirect Callers ===
3 call sites can reach it through a function pointer:
  1. do_loop_readv_writev at fs/read_write.c:848 [member_arrow]
     installed in file_operations::read at arch/arm64/kvm/ptdump.c:204
     and 343 other places (receiver type matches)
  2. loop_rw_iter at io_uring/rw.c:733 [member_arrow]
     installed in file_operations::read at arch/arm64/kvm/ptdump.c:204
     and 343 other places (receiver type matches)
  3. vfs_read at fs/read_write.c:572 [member_arrow]
     installed in file_operations::read at arch/arm64/kvm/ptdump.c:204
     and 343 other places (receiver type matches)

Note: 152 further call sites go through a member of the same name...

vfs_read writes file->f_op->read(...). The calling file proves file is a
struct file; the types table says f_op is a struct file_operations *.

A call chain through a transport

semcode> callchain svc_recv

before: the chain listed svc_handle_xprt and its callees, and stopped. Nothing
said the work happens in whatever is installed in xpo_recvfrom.

after, at the end of the chain:

=== Dispatches ===
svc_handle_xprt xprt->xpt_ops->xpo_accept (net/sunrpc/svc_xprt.c:852)
   └─ svc_rdma_accept
   └─ svc_tcp_accept
   └─ svc_udp_accept
svc_handle_xprt xprt->xpt_ops->xpo_recvfrom (net/sunrpc/svc_xprt.c:870)
   └─ svc_rdma_recvfrom
   └─ svc_tcp_recvfrom
   └─ svc_udp_recvfrom

A slot with many implementations reports its count instead of listing them:
file->f_op->read has 934.

Two questions that could not be asked

semcode> implementors file_operations.read

before:

Error: Unknown command: 'implementors'. Type 'help' for available commands.

after:

=== Implementors ===
1443 installed:
  1. seq_read at .../cn20k/debugfs.c:160 in __OCTEONTX2_DEBUGFS_ATTRIBUTE_FOPS
  2. dev_read at drivers/net/ethernet/mellanox/mlx5/core/lib/sd.c:772
  ...

semcode> registrations tcp_v4_rcv

before:

Error: Unknown command: 'registrations'. Type 'help' for available commands.

after:

=== Registrations ===
1 places install it:
  1. net_protocol::handler at net/ipv4/af_inet.c:1934 in inet_init

A macro body is code

semcode> func wait_event

before:

Declaration:  wait_event(void)
[definition]
Called By: 416

after:

Declaration:  wait_event( wq_head,  condition)
[definition]

Calls: 2
  → __wait_event
  → might_sleep

Called By: 416

416 callers led nowhere before, because the chain stopped at the macro. Across
a Linux tree 133,341 macros gained the 214,769 parameters they declare.

A macro that dispatches

semcode> func nvkm_memory_target

#define nvkm_memory_target(p) (p)->func->target(p)

Neither before nor after prints a Calls: line, for different reasons. Before,
macro bodies were text. After, target is recorded as a dispatch through a
member — not as a call to a function that does not exist. 800 dispatch sites
in a Linux tree are written inside a macro body, and 1,955 registrations.

An index older than the binary

New since the first posting, and it has no before: an index built by an older
semcode used to answer, shortened, without saying so.

semcode> callers seq_read
Error: this index was written by an older semcode (version 0; this build
writes 3). It does not hold everything this build extracts, so answers
from it would be incomplete without saying so.

Rebuild it with:

    semcode-index -s /home/riel/linux -d /home/riel/linux

or pass --reindex-if-stale to rebuild it now.

Running semcode-index over such an index reads every file again by itself,
rather than skipping them all as unchanged.

---

Coverage, and what is left

Of 52,702 member dispatch sites in C on a Linux tree, 92.8% carry a receiver
type. The 3,818 that do not are mostly receivers naming a global the file
does not declare; semcode indexes functions, types and macros, not variables.
A receiver that comes back from a call is 65 sites tree-wide, measured rather
than assumed, and is not worth a schema change.

Rust is the larger untyped population — 17,380 member calls, all of them,
because the typing pass reads C declarations. That is the object-oriented work
described in the plan: impls as registrations, method calls as dispatch sites,
the same (container, member) key.

[chucklever]

I am thrilled to see virtual function support moving. This has been open since
December, I put my own attempt aside because tree-sitter would not carry it,
and callers tcp_v4_rcv now answers with ip_protocol_deliver_rcu. That is
the question issue #9 asked. The dispatch line at the end of
callchain svc_recv is the one I have wanted in SUNRPC review for years.

I checked the four blockers against 0eec120 rather than against the changelog.
The indices are created with the tables, chained receivers resolve at the
queried revision, and processed_files carries the extractor version. One path
did not come along with it.

The range path still skips by content alone

process_git_tree() builds its skip set from processed_by_this_extractor(),
at src/git_range.rs:1123. process_git_range() still builds it from
get_all_processed_files(), at src/git_range.rs:1218. Only one of the two was
converted.

That did no harm while index.rs sent every stale index through a tree read.
index: honour --git on an index older than this build narrowed the redirect
to args.git.is_none(), at src/bin/index.rs:1565, which is right on its own
terms. The two together open this:

Take an index written by version 1 and run semcode-index --git A..B. The
redirect does not fire, because a range was named. process_git_range() loads
every processed file whatever extractor read it, so every file whose content
has not changed is skipped, and none of them gains a dispatch site or a
registration. record_index_build() at src/git_range.rs:1360 then stamps
schema_version 2. index_predates_reader() is false from that point on, so
nothing reads those files again, and the query command and MCP answer from them
without saying anything is missing.

That is the failure the per-file version was added to close, surviving in the
path the --git commit opened. Line 1218 wants processed_by_this_extractor().

semcode-lsp does not check

query.rs and semcode-mcp.rs both refuse a stale index. semcode-lsp.rs has no
reference to index_predates_reader(). It answers the same questions from the
same database, and an editor is where a wrong answer is least likely to get
questioned.

[rikvanriel]

Thanks for spotting those last two issues, Chuck!

I'm working addressing them now.

[rikvanriel]

The latest version pushed (commit b8dccd6 at the top) should address your most recent comments.

I discovered some other things to fix in semcode while getting those commits fixed, but that's for another PR...