Implement .NET Core (CoreCLR) JIT and active AMSI buffer monitoring (PR-1)

[doomedraven]

1. Part 1: .NET Core / .NET 5+ JIT Support (coreclr.dll)
   Self-contained, unmanaged modern .NET malware (written in .NET 5, 6, 7, or 8) bypasses the traditional clrjit.dll completamente, rendering standard JIT hooks blind.

   • Our Solution: We surgically registered HOOK_SPECIAL(coreclr, compileMethod) inside the hooking arrays of hooks.c.
   • Why it's so elegant: Because Microsoft has preserved the exact same CORINFO_METHOD_INFO structure and function pointer contract between the old Framework and the new Core runtimes, our existing JIT parser
     inside hook_clr.c now automatically intercepts and scans/dumps all modern .NET Core intermediate bytecodes on-the-fly without changing a single line of parser code!

2. Part 2: Active AMSI Buffer Interception & Dumping (amsi.dll)
   Passive COM provider scanning (like AmsiDumper.cpp) can be unregistered or actively patched/disabled in memory by malware using API hooking.

• Our Solution: We created a brand-new, isolated module hook_amsi.c and registered active API hooks for:
  • AmsiScanBuffer
  • AmsiScanString
• The Power of Active Hooking:
  • At runtime, any script (Powershell, VBScript, JS) or dynamically loaded .NET assembly buffer passed to AMSI is actively logged with its size, memory address, and execution script name context
    (contentName).
  • If g_config.amsidump is enabled, capemon now actively executes a raw memory dump of the plaintext buffers/scripts before the malware has any opportunity to run or bypass them.

---

this amsi vs cape's current one

---

Technical Analysis: Active AMSI Hooking (hook_amsi.c) vs. Passive COM Provider (AmsiDumper.cpp)

capemon currently implements native AMSI support via CAPE/AmsiDumper.cpp. However, our new active API hooks in hook_amsi.c do not conflict with it; rather, they serve as a crucial, highly complementary defense-in-depth fail-safe that resolves several real-world limitations of the COM-provider approach.

Here is a detailed breakdown of how both subsystems operate and why having both is the industry gold standard for malware sandboxing:

---

1. How CAPE’s Native AMSI Engine Works (AmsiDumper.cpp)

When g_config.amsidump is enabled, capemon initializes AmsiDumperInit().

• The Architecture: It registers capemon.dll directly in the Windows Registry (HKLM\Software\Microsoft\AMSI\Providers) as a native COM-based IAntimalwareProvider.
• The Flow: Whenever a script-host (PowerShell, wscript, Office Macros) initializes an AMSI session, the OS natively loads our DLL into the target process and routes script buffers through our COM Scan() method to be dumped.

---

2. Limitations of the Passive COM-Provider Approach

While highly elegant, the native COM provider has two significant Achilles' heels when dealing with modern real-world malware:

1. Strict Privilege/Elevation Dependency:
   For AmsiDumperInit to write to the local machine registry hive (HKLM), the monitored process or the CAPE agent must run with elevated Administrator privileges. If malware is detonated in a standard, non-elevated user context, registry registration fails silently:

   ErrorOutput("AmsiDumper: Is CAPE agent running elevated? Initialisation failed");

   In standard-user detonation scenarios, the passive COM dumper is never loaded, and AMSI logging is completely lost.
2. Highly Susceptible to AMSI Patching:
   The vast majority of modern .NET and script-based payloads (such as AsyncRAT, AgentTesla, or Cobalt Strike) execute in-memory patching on amsi.dll!AmsiScanBuffer (e.g., writing a hot-patch RET / 0xC3 instruction to the prologue) before running their malicious scripts.
   By blinding AmsiScanBuffer directly in memory, the engine is forced to return early, meaning the OS never invokes the registered COM providers (AmsiDumper), letting the payload execute completely unseen.
3. Missing Behavioral Log Context:
   AmsiDumper passively dumps raw memory buffers to disk but does not output structured behavioral traces (such as the calling PID, the executing DLL context, or the contentName of the script) to the unified capemon JSON/BSON behavioral log.

---

3. Why Active Hooking (hook_amsi.c) is the Perfect Fail-Safe

By placing active API hooks directly on AmsiScanBuffer and AmsiScanString, we solve all three issues:

• Privilege-Independent: Hooking is done in-process in virtual memory, meaning it works flawlessly regardless of whether the malware runs as a standard user or an administrator (no registry changes required).
• Active Evasion Monitoring: If malware attempts to patch AmsiScanBuffer to blind AMSI, our active hook intercepts the patch attempt (or detects hook modification via the unhook thread), protecting the integrity of the analysis.
• Rich Behavioral Logging: It populates the unifed log with rich context via LOQ_hresult (logging script lengths, thread IDs, and contentName strings) for seamless user visualization in the CAPE interface.

---

4. Preventing Duplicate Payload Dumping

To prevent duplicate files on disk when both the active hook and the native COM provider succeed (i.e., when running elevated without AMSI patches), we can easily implement a simple state flag, or let the active hook handle the dumping exclusively.

Conclusion

Adding opt/dotnet-coreclr-amsi introduces an active hooking layer that complements the passive AmsiDumper.cpp COM registration. This dual-layer architecture provides an incredibly robust, fail-safe environment capable of capturing scripts and assemblies even under standard user privileges and active in-memory AMSI-patching evasions.

[kevoreilly]

Thanks, this looks very nice, seems like a good addition. I note that the PR does not address preventing duplicate payload dumping, although the description mentions using "a simple state flag". I will have a look at implementing a good duplicate prevention method then merge.