Skip to content

docs: correct the database connector page and document query_timeout - #509

Open
marevol wants to merge 1 commit into
masterfrom
docs/ds-database-accuracy
Open

docs: correct the database connector page and document query_timeout#509
marevol wants to merge 1 commit into
masterfrom
docs/ds-database-accuracy

Conversation

@marevol

@marevol marevol commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

Summary

The 15.9 database connector page described behaviour the connector does not have,
and left out the parts most likely to trip an administrator up. This corrects it
and documents query_timeout, added in the fess-ds-db stack
(codelibs/fess-ds-db#19#28).

All seven languages: ja, en, de, es, fr, ko, zh-cn. 15.9 only.

Corrections

Was Is
Download from Maven Central into WEB-INF/lib maven.codelibs.org, into WEB-INF/plugin - where the admin console installs
Nothing about the JDBC driver beyond "place it in the classpath" It is a separate download, and it has to reach the crawler process classpath: WEB-INF/lib or WEB-INF/env/crawler/lib
"BLOB, CLOB, NCLOB, byte array and binary stream columns are automatically passed through content extraction" Only the binary forms are extracted. CLOB, NCLOB and character streams are read as text, and the MIME hints do not apply to them. Whether a BLOB arrives as java.sql.Blob or as a byte array is the driver's choice
url=url presented as a recipe …with the warning that a missing url column makes the document URL the JDBC connection URL
Scripts see the result columns Scripts see the whole parameter map, credentials included. Column label casing differs per database, and an unresolved label leaves the field unset rather than erroring
"Use environment variables / use the encryption features" Both spelled out: FESS_ENV_ variables expand as ${...} (crawler.data.env.param.key.pattern), and a parameter matching app.encrypt.property.pattern is encrypted at rest. Plus a note that DEBUG logging masks these
Troubleshooting on ClassNotFoundException / Connection refused / SQLException The messages the connector now emits, plus sections for missing parameters, per-row failures and "documents do not appear in search results"
fetch_size: "set to MIN_VALUE for MySQL streaming" …and that other drivers reject the negative value, and what happens then
"Incremental crawling: narrow the SQL by hand" …with what that costs. When a crawl finishes, Fess deletes the documents of that data store configuration that were not part of the crawl that just ran, so a filtered query leaves only the matching rows in the index. The section now says so and points at delete_old_docs=false

The delete_old_docs behaviour is in DataIndexHelper#deleteOldDocs(): it returns
early only when the parameter is literally false, so a partial crawl sweeps the
rest away by default.

New parameter

  • query_timeout, with a note that stopping a job cannot interrupt a query
    blocked inside the driver - the stop is only checked between rows.

Verification

  • Structural parity across all seven files: identical counts of .. note:: (7),
    .. warning:: (4), literal blocks (17) and list-table rows (18); section
    underlines checked with east-asian width taken into account.
  • Every new parameter name, path and error message is present in all seven files,
    and no file still mentions repo1.maven.org, Maven Central,
    mysql-connector-j-8, ClassNotFoundException, No suitable driver or
    Connection refused.
  • Each file parses under docutils with no message other than the expected
    Unknown interpreted text role "doc" from the Sphinx-only :doc: role in the
    "See Also" list.
  • A full Sphinx build was not run: conf/conf.py fails to load under Sphinx 9.1
    with a unicodeescape error at line 228, which is unrelated to this change.

Note on scope

An earlier revision of this branch also documented incremental crawling through
${last_crawl_time}. That connector change (codelibs/fess-ds-db#29) was not
merged, so the parameters and the section describing them have been removed and
the manual-filter section restored - now carrying the delete_old_docs warning
it was missing.

One thing for the reviewer

The Spanish page carries no diacritics (Descripcion, Metodo,
contrasena). That is pre-existing and page-wide; #511, stacked on this branch,
fixes it across the seven affected Spanish pages.

The 15.9 database connector page described behaviour the connector does
not have, and left out the parts most likely to trip an administrator up.
Applied to all seven languages.

Corrections:

- Installation pointed at Maven Central and at WEB-INF/lib. The plugin is
  distributed through maven.codelibs.org, and the admin console installs
  into WEB-INF/plugin.
- The JDBC driver section did not say the driver is a separate download,
  nor that it has to reach the crawler process classpath. Both
  WEB-INF/lib and WEB-INF/env/crawler/lib work; nothing else does.
- "BLOB, CLOB, NCLOB, byte array and binary stream columns are
  automatically passed through content extraction" was wrong. Only the
  binary forms are extracted; CLOB, NCLOB and character streams are read
  as text, and the MIME type hints do not apply to them. Whether a BLOB
  arrives as java.sql.Blob or as a byte array is decided by the driver.
- The url=url recipe was presented without its trap: with no column
  labelled url, the parameter of the same name wins and the document URL
  becomes the JDBC connection URL.
- Scripts see the entire data store parameter map, credentials included,
  not only the result columns. Column label casing differs between
  databases, and a label that does not resolve leaves the field unset
  rather than raising an error.
- "Use environment variables" and "use the encryption features" had no
  instructions. Both exist: FESS_ENV_ variables expand as ${...} inside
  parameters, and a parameter matching app.encrypt.property.pattern is
  encrypted at rest.
- Troubleshooting listed symptoms the connector no longer produces. The
  messages now identify which step failed, and sections for missing
  parameters, per-row failures and missing search results were added.
- fetch_size did not say that MIN_VALUE is a MySQL idiom that other
  drivers reject.
- The incremental crawling section told the reader to narrow the SQL by
  hand without saying what that costs. When a crawl finishes, Fess
  deletes the documents of that data store configuration that were not
  part of the crawl that just ran, so a filtered query leaves only the
  matching rows in the index. The section now says so and points at
  delete_old_docs=false.

New parameter:

- query_timeout, with a note that stopping a job cannot interrupt a query
  blocked inside the driver.

Structure was verified across all seven files: identical note, warning
and literal block counts, list-table rows with a consistent cell count,
and section underlines checked with east-asian width taken into account.
@marevol
marevol force-pushed the docs/ds-database-accuracy branch from 5dfc565 to f9d6394 Compare August 23, 2026 08:08
@marevol marevol changed the title docs: correct the database connector page and document the new parameters docs: correct the database connector page and document query_timeout Aug 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant