Ability to index and search over markdown documents

[adityamparikh]

Currently the MCP server can index json, csv and xml formats. We could use a project like https://github.com/docling-project/docling-java or use Apache Tika (in process) to parse documents in various supported formats and export into markdown and then index and be able to search over them.

[shahzadarain]

Hi @adityamparikh I'd like to work on this. I see PR #88 handles client-extracted file text, which is great for Claude Desktop. I'm thinking of a complementary server-side markdown path: a MarkdownDocumentCreator following the existing strategy pattern in indexing/documentcreator/, using CommonMark-Java (lightweight, no Tika-scale dependencies, reflection-free so native-image safe). It would parse YAML frontmatter into metadata fields, extract the H1 as title, index heading hierarchy as a multi-valued field, and the body as content exposed as an index-markdown-documents MCP tool alongside the JSON/CSV/XML ones, with unit + Testcontainers integration tests. Does that direction work for you, or would you prefer it folded into #88?

[adityamparikh]

Hi @shahzadarain,
Thanks for picking this up!
Indexing markdown alongside the existing xml/json/csv — rather than only a flat text blob — is the right call. If you want to create a fresh PR, I am happy to close #88.

The LLM can non-deterministically convert arbitrary formats to markdown and then invoke this tool. We'd need to think about having a stable id to ensure idempotent behavior.

We also need to ensure that the LLM does not try to invoke this tool if xml/json/csv docs are supplied by updating the tool descriptions something like: "Do NOT use for JSON/CSV/XML; use index-json/csv/xml-documents instead, only convert to Markdown when there's no dedicated tool for the source format."