Show a tabular file as a picture, drawn the way the export writes it

CrestApps.Core.slnx

@@ -44,13 +44,9 @@
     <Project Path="src/Primitives/CrestApps.Core.AI.Ollama/CrestApps.Core.AI.Ollama.csproj" />
     <Project Path="src/Primitives/CrestApps.Core.AI.OpenAI.Azure/CrestApps.Core.AI.OpenAI.Azure.csproj" />
     <Project Path="src/Primitives/CrestApps.Core.AI.OpenAI/CrestApps.Core.AI.OpenAI.csproj" />
-    <Project Path="src/Primitives/CrestApps.Core.AI.PostgreSQL/CrestApps.Core.AI.PostgreSQL.csproj">
-      <Build Solution="Debug|*" Project="false" />
-    </Project>
+    <Project Path="src/Primitives/CrestApps.Core.AI.PostgreSQL/CrestApps.Core.AI.PostgreSQL.csproj" />
     <Project Path="src/Primitives/CrestApps.Core.AI.WebRTC/CrestApps.Core.AI.WebRTC.csproj" />
-    <Project Path="src/Primitives/CrestApps.Core.AI.Resilience/CrestApps.Core.AI.Resilience.csproj">
-      <Build Solution="Debug|*" Project="false" />
-    </Project>
+    <Project Path="src/Primitives/CrestApps.Core.AI.Resilience/CrestApps.Core.AI.Resilience.csproj" />
     <Project Path="src/Primitives/CrestApps.Core.AI.WebCrawlers/CrestApps.Core.AI.WebCrawlers.csproj" />
     <Project Path="src/Primitives/CrestApps.Core.AI/CrestApps.Core.AI.csproj" />
     <Project Path="src/Primitives/CrestApps.Core.Azure.AISearch/CrestApps.Core.Azure.AISearch.csproj" />
@@ -59,9 +55,7 @@
     <Project Path="src/Primitives/CrestApps.Core.DataIngestion/CrestApps.Core.DataIngestion.csproj" />
     <Project Path="src/Primitives/CrestApps.Core.Elasticsearch/CrestApps.Core.Elasticsearch.csproj" />
     <Project Path="src/Primitives/CrestApps.Core.Infrastructure/CrestApps.Core.Infrastructure.csproj" />
-    <Project Path="src/Primitives/CrestApps.Core.PostgreSQL/CrestApps.Core.PostgreSQL.csproj">
-      <Build Solution="Debug|*" Project="false" />
-    </Project>
+    <Project Path="src/Primitives/CrestApps.Core.PostgreSQL/CrestApps.Core.PostgreSQL.csproj" />
     <Project Path="src/Primitives/CrestApps.Core.SignalR/CrestApps.Core.SignalR.csproj" />
     <Project Path="src/Primitives/CrestApps.Core.Templates/CrestApps.Core.Templates.csproj" />
     <Project Path="src/Primitives/CrestApps.Core/CrestApps.Core.csproj" />

src/CrestApps.Core.Docs/docs/changelog/2.0.0.md

@@ -1303,6 +1303,50 @@ microphone, desk speakers):
 
 See [Document Processing](../core/document-processing.md#how-large-an-upload-may-be).
 
+## A spreadsheet you can look at
+
+Every tabular tool answered a question *about* the uploaded data. None of them showed it. A user who
+uploaded a workbook and asked what was in it was told about it in prose and had to take that on trust — that
+the right worksheet was read, that the header row was found where they expect it, that a column means what
+the answer says it means.
+
+The Tabular Data Agent now has a `preview_tabular_data` tool that draws the data as a picture of a
+spreadsheet: the lettered column band, the numbered rows, a filled header row, numbers aligned right. It is
+called whenever the user asks to see, view or preview a table, once after the first question about a newly
+uploaded file, and — with a `sql` argument — to show what an export will contain *before* the file is
+written.
+
+- **It is SVG, not a raster image.** Nothing in the process can rasterize one: there is no SkiaSharp, no
+  ImageSharp, no `System.Drawing`, and PDFsharp emits PDF. A vector picture needs no drawing library and no
+  fonts shipped with the host — the browser showing it already has both — so the preview is a few kilobytes
+  of markup that stays sharp at whatever size the chat surface gives it, and the feature ships without a new
+  package or per-RID native assets.
+- **It goes through the paths that already exist.** The picture is stored by `IGeneratedDocumentService` and
+  served by the same authorized document download endpoint as an export, so authorization, cleanup on
+  history clear, and exclusion from the tabular workspace all come for free. It is registered as a `[fig:N]`
+  reference — the mechanism a figure retrieved from a knowledge base already uses — so no chat surface
+  needed a change.
+- **It is bounded, and says so.** A grid large enough to hold a real workbook is scaled down by the chat
+  surface until none of it is readable, so a preview is capped at the first rows and columns and captioned
+  with what it left out: *"Showing 50 of 4,812 rows and 12 of 20 columns"*. `TabularPreviewOptions` moves the
+  caps.
+- **It previews the workspace, not the upload,** so a column added or a value corrected earlier in the
+  conversation is visible in it — consistent with every other tabular tool, and the more useful of the two.
+- **A host that cannot show a picture gets the same window as text.** Where no download endpoint is
+  registered there is no address to draw from, and a marker registered without one reaches the reader as a
+  broken image. The same applies where no writer is registered for the preview format. The tool falls back
+  whole to a Markdown table of the same rows and columns rather than answering with some pictures and some
+  markers. The fallback turns on whether the picture can be *delivered*, not on what the model can read: a
+  vision-capable deployment does not make a host able to draw.
+- **The preview format is resolvable but not requestable.** `AddGeneratedFileWriter` gained a `requestable`
+  overload, and the preview writer is registered with it turned off: the host can write the format, while
+  it stays out of `IGeneratedFileWriterResolver.SupportedExtensions`, which is what `generate_file` and
+  `export_tabular_data` gate on. A preview is markup this host generated and escaped; the same extension
+  reached through a tool whose `content` argument is written verbatim would serve model-supplied markup
+  from the application's own origin.
+
+See [AI Documents](../core/ai-documents.md#seeing-the-data).
+
 ## Change Logs
 
 - **Fixed: several references written in one bracket reached the reader as a raw marker.** A reference is

src/CrestApps.Core.Docs/docs/core/ai-documents.md

@@ -63,6 +63,31 @@ This is the recommended path for tasks such as:
 
 Under the hood, tabular workflows are handled by the built-in **Tabular Data Agent**. It is a code-defined, always-available **system agent** that stays hidden from the AI Profile and Chat Interaction agent pickers, yet still participates in orchestration and is exposed through the A2A host for remote clients.
 
+#### Seeing the data
+
+Everything else the agent does answers a question *about* the data. The hidden `preview_tabular_data` tool shows the data itself: a picture of the sheet, with its header row, lettered columns, numbered rows and numbers aligned right, drawn from the first rows and columns of whatever is currently loaded.
+
+It is what lets a user check for themselves that the right worksheet was read and the header row was found where they expect it, rather than taking the answer's word for it. The agent calls it whenever the user asks to see, view or preview a table, once after the first question about a newly uploaded file, and — given a `sql` argument — to show what an export will contain before the file is created.
+
+The picture is written as SVG and served through the same authorized document download endpoint as any other generated file, so it needs no drawing library, no fonts on the host, and no extra registration. The tool returns one `[fig:N]` marker per table, which the chat surfaces replace with the picture.
+
+Previews are bounded on purpose and say what they left out — *"Showing 50 of 4,812 rows and 12 of 20 columns"* — because a grid large enough to hold everything is scaled down by the chat surface until none of it is readable. The limits are configurable:
+
+```csharp
+builder.Services.Configure<TabularPreviewOptions>(options =>
+{
+    options.MaxRows = 50;
+    options.MaxColumns = 15;
+    options.MaxCellCharacters = 32;
+    options.MaxTables = 4;
+    options.MaxImageWidth = 1100;
+});
+```
+
+A host with no document download endpoint registered cannot serve the picture, and one with no writer registered for the preview format cannot write it; in either case the tool falls back to a Markdown table of the same window of the same data. The user can also ask for the text form directly.
+
+The preview format is registered as a writer the host can resolve but **not** as a format a caller may request, so `generate_file` and `export_tabular_data` will not produce one. A preview is markup this host generated and escaped; the same extension reached through a tool whose `content` argument is written verbatim would serve model-supplied markup from your origin.
+
 #### Spreadsheet formatting
 
 Exported `.xlsx` files are written with real cell types: a numeric column becomes numbers and a date column becomes date serials, so the recipient can sum, sort, filter, and chart the result rather than receiving a sheet of text. A column whose leading zeros matter — a postal code or an account number — is detected and kept as text.

src/Primitives/CrestApps.Core.AI.Chat/Hubs/ChatInteractionHubBase.cs

@@ -1,5 +1,6 @@
 using System.IO.Pipelines;
 using System.Runtime.CompilerServices;
+using System.Text;
 using System.Text.Json;
 using System.Threading.Channels;
 using CrestApps.Core.AI.Capabilities;
@@ -1657,6 +1658,8 @@ protected virtual async Task HandlePromptAsync(
 
             CollectStreamingReferences(services, handlerContext, references, contentItemIds);
 
+            string lastResponseId = null;
+
             await foreach (var chunk in handlerResult.ResponseStream.WithCancellation(cancellationToken))
             {
                 if (string.IsNullOrEmpty(chunk.Text))
@@ -1665,6 +1668,7 @@ protected virtual async Task HandlePromptAsync(
                 }
 
                 builder.Append(chunk.Text);
+                lastResponseId = chunk.ResponseId;
                 CollectStreamingReferences(services, handlerContext, references, contentItemIds);
 
                 var partialMessage = new CompletionPartialMessage
@@ -1682,6 +1686,33 @@ protected virtual async Task HandlePromptAsync(
 
             CollectStreamingReferences(services, handlerContext, references, contentItemIds);
 
+            // A picture the host has already drawn and stored must not depend on the model remembering to
+            // write its marker. This is the rule generated downloads already follow — IsGenerated exists so
+            // an exported file reaches the reader whether or not it was cited — applied to an image, which
+            // is shown where its marker sits rather than listed underneath. Observed repeatedly: the answer
+            // says "the previews are shown above" and writes no marker, so the reader is told about images
+            // that were built, stored and served, and sees none of them.
+            var trailingImageMarkers = BuildUncitedImageMarkers(builder.AsSpan(), references);
+
+            if (!string.IsNullOrEmpty(trailingImageMarkers))
+            {
+                // Streamed as one more chunk rather than only appended to the stored text, so the images
+                // appear in the answer being read now as well as after a reload.
+                builder.Append(trailingImageMarkers);
+
+                await writer.WriteAsync(
+                    new CompletionPartialMessage
+                    {
+                        SessionId = interaction.ItemId,
+                        MessageId = assistantPrompt.ItemId,
+                        ResponseId = lastResponseId,
+                        Content = trailingImageMarkers,
+                        References = references,
+                        Appearance = handlerContext.AssistantAppearance,
+                    },
+                    cancellationToken);
+            }
+
             if (builder.Length > 0)
             {
                 assistantPrompt.Text = builder.ToString();
@@ -2116,6 +2147,67 @@ await Clients.Caller.ReceiveConversationAssistantComplete(
         }
     }
 
+    /// <summary>
+    /// Returns the markers for every servable picture the answer did not name, ready to be appended to it.
+    /// </summary>
+    /// <remarks>
+    /// Only a reference the host can actually turn into a picture is added: one flagged as an image and
+    /// carrying an address this host serves. A marker without a picture behind it would reach the reader as
+    /// the few characters the host typed, which is worse than the omission it was meant to repair.
+    /// <para>
+    /// Ordered by the reference index so the pictures appear in the order the tool produced them rather
+    /// than in whatever order the map happens to enumerate.
+    /// </para>
+    /// </remarks>
+    /// <param name="text">The answer as written.</param>
+    /// <param name="references">The references collected for the answer.</param>
+    /// <returns>The text to append, or <see langword="null"/> when every picture was named.</returns>
+    private static string BuildUncitedImageMarkers(
+        ReadOnlySpan<char> text,
+        Dictionary<string, AICompletionReference> references)
+    {
+        if (references is null || references.Count == 0)
+        {
+            return null;
+        }
+
+        List<KeyValuePair<string, AICompletionReference>> uncited = null;
+
+        foreach (var reference in references)
+        {
+            if (reference.Value?.IsImage != true ||
+                string.IsNullOrWhiteSpace(reference.Value.Link) ||
+                string.IsNullOrEmpty(reference.Key) ||
+                text.Contains(reference.Key, StringComparison.OrdinalIgnoreCase))
+            {
+                continue;
+            }
+
+            (uncited ??= []).Add(reference);
+        }
+
+        if (uncited is null)
+        {
+            return null;
+        }
+
+        uncited.Sort((left, right) => left.Value.Index.CompareTo(right.Value.Index));
+
+        var builder = new StringBuilder(Environment.NewLine + Environment.NewLine);
+
+        for (var index = 0; index < uncited.Count; index++)
+        {
+            if (index > 0)
+            {
+                builder.Append(' ');
+            }
+
+            builder.Append(uncited[index].Key);
+        }
+
+        return builder.ToString();
+    }
+
     private static async Task<Dictionary<string, AICompletionReference>> GetPromptReferencesAsync(
         IServiceProvider services,
         string itemId,

src/Primitives/CrestApps.Core.AI.Documents/Endpoints/DownloadAIDocument.cs

@@ -8,6 +8,7 @@
 using Microsoft.AspNetCore.Http;
 using Microsoft.AspNetCore.Mvc;
 using Microsoft.AspNetCore.Routing;
+using Microsoft.Extensions.Logging;
 
 namespace CrestApps.Core.AI.Documents.Endpoints;
 
@@ -43,17 +44,37 @@ private static async Task<IResult> HandleAsync(
         [FromServices] IAuthorizationService authorizationService,
         [FromServices] ICatalogManager<ChatInteraction> interactionManager,
         [FromServices] IAIChatSessionManager sessionManager,
-        [FromServices] IAIProfileManager profileManager)
+        [FromServices] IAIProfileManager profileManager,
+        [FromServices] ILoggerFactory loggerFactory)
     {
+        var logger = loggerFactory.CreateLogger(typeof(DownloadAIDocument).FullName);
+
         if (string.IsNullOrWhiteSpace(documentId))
         {
             return Results.BadRequest();
         }
 
         var document = await documentStore.FindByIdAsync(documentId);
 
-        if (document is null || string.IsNullOrWhiteSpace(document.StoredFilePath))
+        // Each reason is reported separately. A bare 404 says only that the address did not resolve, which
+        // is the same answer for a document that was never created, one whose row has not been committed
+        // yet, and one whose bytes are missing — three different faults that need three different fixes.
+        if (document is null)
         {
+            logger.LogWarning(
+                "Download of AI document '{DocumentId}' returned 404: no document with that id is readable. If it was created during the request that produced this link, its row may not be committed yet.",
+                documentId);
+
+            return Results.NotFound();
+        }
+
+        if (string.IsNullOrWhiteSpace(document.StoredFilePath))
+        {
+            logger.LogWarning(
+                "Download of AI document '{DocumentId}' ('{FileName}') returned 404: the document record carries no stored file path.",
+                documentId,
+                document.FileName);
+
             return Results.NotFound();
         }
 
@@ -74,9 +95,25 @@ private static async Task<IResult> HandleAsync(
 
         if (stream is null)
         {
+            logger.LogWarning(
+                "Download of AI document '{DocumentId}' ('{FileName}') returned 404: no file exists at '{StoredFilePath}'.",
+                documentId,
+                document.FileName,
+                document.StoredFilePath);
+
             return Results.NotFound();
         }
 
+        if (logger.IsEnabled(LogLevel.Debug))
+        {
+            logger.LogDebug(
+                "Serving AI document '{DocumentId}' ('{FileName}') as '{ContentType}', {FileSize} bytes.",
+                documentId,
+                document.FileName,
+                document.ContentType,
+                document.FileSize);
+        }
+
         return Results.File(
             stream,
             string.IsNullOrWhiteSpace(document.ContentType) ? "application/octet-stream" : document.ContentType,

src/Primitives/CrestApps.Core.AI.Documents/Generation/DefaultGeneratedDocumentService.cs

@@ -1,6 +1,7 @@
 using CrestApps.Core.AI.Ingestion;
 using CrestApps.Core.AI.Models;
 using CrestApps.Core.AI.Orchestration;
+using CrestApps.Core.Services;
 
 namespace CrestApps.Core.AI.Documents.Generation;
 
@@ -25,6 +26,7 @@ public sealed class DefaultGeneratedDocumentService : IGeneratedDocumentService
     private readonly IAIDocumentStore _documentStore;
     private readonly IDocumentFileStore _fileStore;
     private readonly TimeProvider _timeProvider;
+    private readonly IStoreCommitter _committer;
 
     /// <summary>
     /// Initializes a new instance of the <see cref="DefaultGeneratedDocumentService"/> class.
@@ -33,16 +35,19 @@ public sealed class DefaultGeneratedDocumentService : IGeneratedDocumentService
     /// <param name="documentStore">The document metadata store.</param>
     /// <param name="fileStore">The document file store.</param>
     /// <param name="timeProvider">The time provider.</param>
+    /// <param name="committer">The store committer, when the host registered one.</param>
     public DefaultGeneratedDocumentService(
         IGeneratedFileWriterResolver writerResolver,
         IAIDocumentStore documentStore,
         IDocumentFileStore fileStore,
-        TimeProvider timeProvider)
+        TimeProvider timeProvider,
+        IStoreCommitter committer = null)
     {
         _writerResolver = writerResolver;
         _documentStore = documentStore;
         _fileStore = fileStore;
         _timeProvider = timeProvider;
+        _committer = committer;
     }
 
     /// <summary>
@@ -91,7 +96,20 @@ public async Task<GeneratedDocumentResult> CreateAsync(GeneratedDocumentRequest
 
         await _documentStore.CreateAsync(document, cancellationToken);
 
-        var referenceToken = AddDownloadReference(document);
+        // The file is about to be handed out as an address, so the row that address resolves through has to
+        // be readable by the request that follows it. Staged in this request's session it is not: the
+        // conversation commits when the turn ends, and a preview image embedded in the answer is fetched by
+        // the browser the moment the message renders — measured here at 78 ms before that commit, which the
+        // endpoint answered with 404 for every image. A download link survives only because a person takes
+        // seconds to click it.
+        if (_committer is not null)
+        {
+            await _committer.CommitAsync(cancellationToken);
+        }
+
+        var referenceToken = request.RegisterDownloadReference
+            ? AddDownloadReference(document)
+            : null;
 
         return new GeneratedDocumentResult(document, referenceToken);
     }