Restore comprehensive 2.2 manuals - #650
Conversation
1d9e5e7 to
2a4b8d9
Compare
There was a problem hiding this comment.
Pull request overview
This PR restores substantial bulk_extractor 2.2 LaTeX documentation by reintroducing (and updating) comprehensive user and programmer manuals, preserving historically useful 1.x guidance while clearly labeling 1.x-only workflows (e.g., BEViewer) and recording the restoration in the 2.2 release notes.
Changes:
- Expanded the 2.2 user manual to include operational explanations, investigative workflows, worked examples, troubleshooting, and a BEViewer history appendix.
- Expanded the 2.2 programmer’s manual with detailed architecture, sbuf/recorder guidance, concurrency rules, and scanner examples, plus restored bibliography support.
- Updated manual build metadata (Makefile / Makefile.am dependencies), manual landing page text, and 2.2 release notes to reflect the restored documentation set.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
| doc/RELEASE_NOTES.md | Records the documentation restoration in 2.2 release notes. |
| doc/programmer_manual/references.bib | Adds programmer manual bibliography entries (needs link/macro fixes). |
| doc/programmer_manual/Makefile.am | Adds PDF dependencies for the programmer manual (needs .tex prerequisite). |
| doc/programmer_manual/Makefile | Adds explicit build dependencies for the programmer manual PDF. |
| doc/programmer_manual/BEProgrammersManual.tex | Renames/re-dates manual and restores substantial architecture/API/concurrency content plus bibliography hook. |
| doc/Makefile.am | Ensures programmer manual bibliography is distributed. |
| doc/latex_manuals/references.bib | Adds user manual bibliography entries (needs URL-field handling + link/macro fixes). |
| doc/latex_manuals/Makefile.am | Adds dependency tracking and EXTRA_DIST for the user manual (needs inclusion of main .tex). |
| doc/latex_manuals/index.html | Updates landing page wording for user/programmer manuals (one link label still inconsistent). |
| doc/latex_manuals/BEWorkedExamples.tex | Restores worked investigations content (contains several typos/markup issues). |
| doc/latex_manuals/BECurrentGuide.tex | Rebuilds the user manual content into a full 2.2 user manual with references and appendix material. |
Suppressed comments (1)
doc/latex_manuals/references.bib:118
- With
\bibliographystyle{plain}, theurlfield is typically not rendered, so this entry may lose its link in the PDF. Use ahowpublished(ornote) field that includes\url{...}instead.
url="http://digitalcorpora.org/downloads/bulk_extractor/doc/BEProgrammersManual.pdf",
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.
Suppressed comments (6)
doc/latex_manuals/BEWorkedExamples.tex:290
- Typo/grammar in this paragraph: "The are" should be "They are" (and the surrounding sentence can also capitalize "URLs" for consistency).
The file \texttt{url\_histogram.txt} provides the histogram of the potential urls. In that file, UTF-16 formatted text is converted to UTF-8. Note that not all URLs contained in the histogram file are accurate. The are actually URLs that were typed into a web browser. The following are lines taken from that file:
doc/latex_manuals/BECurrentGuide.tex:668
- The
-poption description is misleading. In the CLI it prints the value of a forensic path expression (<path>[:length][/h][/r]) rather than just printing a path.
\code{-p PATH} & Print a forensic path instead of running a normal scan. \\
doc/latex_manuals/BECurrentGuide.tex:690
- The
--log-levelrow text is unclear. The CLI accepts a severity level string (e.g., trace/debug/info/warning/error/critical/off).
\code{--log-level LEVEL} & Set trace through off diagnostic severity. \\
doc/latex_manuals/BEWorkedExamples.tex:301
- Typo: "UT-16" should be "UTF-16".
Because the histogram file converts the UT-16 formatted text to UTF-8, the histogram file is more human readable than the \texttt{url.txt} file alone. The files \texttt{url\_facebook.txt}, \texttt{url\_microsoft-live}, \texttt{url\_services} and \texttt{url\_searches} all extract specific types of information from URLs. The most useful for digital media triage is likely the file \texttt{url\_searches.txt} because it shows histogram of searches from the disk image. Searches frequently convey intent. The following is an excerpt from that file:
doc/latex_manuals/references.bib:55
- BibTeX uses the
numberfield for issue numbers;issuesis non-standard and will be ignored by common styles (includingplain), so the issue info may be dropped in the rendered bibliography.
@article{
dfxmlpaper,
Author={Simson Garfinkel},
title = {Digital forensics XML and the DFXML toolset},
journal ={Digital Investigation},
volume={8},
issues={3-4},
month={February},
year={2012},
doc/programmer_manual/references.bib:44
- BibTeX uses the
numberfield for issue numbers;issuesis non-standard and will be ignored by common styles (includingplain), so the issue info may be dropped in the rendered bibliography.
@article{
dfxmlpaper,
Author={Garfinkel, Simson},
title = {Digital forensics XML and the DFXML toolset},
journal ={Digital Investigation},
volume={8},
issues={3-4},
month={February},
year={2012},
|
Addressed all six additional findings from Copilot's suppressed current-head review in commit
Validation completed:
The pull request remains a draft while the current-head Copilot re-review and GitHub Actions checks run. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.
Suppressed comments (3)
doc/latex_manuals/BEWorkedExamples.tex:206
- Typo/grammar: “users activity” should be possessive (“user's activity”).
Many of these domains are part of the operating system, such as openoffice.org, but some are not, such as www.uspto.gov. The histogram file provides insight into the users activity on the machine and which sites they were most frequently visiting. \\
doc/latex_manuals/BEWorkedExamples.tex:208
- Typo: “information form HTTP requests” should be “information from HTTP requests”.
The file \texttt{rfc822.txt} primarily provides email headers and HTTP headers both of which are in a format specified by RFC822, the Internet Message Standard. It can be useful to see the subject of emails that have been sent and information form HTTP requests. The following is an excerpt from the text file:
doc/latex_manuals/BEWorkedExamples.tex:220
- Grammar: “This following numbers” should be “The following numbers”.
Telephone numbers found on the disk image are stored in \texttt{telephone.txt}. This following numbers found in the file are clearly for technical support (found within installed software):
|
Addressed all three additional grammar findings from Copilot's suppressed review in commit
Validation completed:
The pull request remains a draft while the current-head Copilot re-review and GitHub Actions checks run. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.
Suppressed comments (2)
doc/programmer_manual/references.bib:12
\bulkis already defined as\textit{bulk\_extractor}in this manual, so wrapping it in\textit{...}again will cancel italics (nested\textit{\textit{...}}renders upright). Use\bulkdirectly in the BibTeX title.
title={Readme File for \textit{\bulk}},
doc/latex_manuals/references.bib:12
\bulkis already defined as\textit{bulk\_extractor}in the user manual, so\textit{\bulk}results in nested italics that typically render upright. Use\bulkdirectly in the BibTeX title.
title={Readme File for \textit{\bulk}},
|
Addressed both additional bibliography-formatting findings from Copilot's suppressed review in commit
Validation completed:
The pull request remains a draft while the current-head Copilot re-review and GitHub Actions checks run. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.
Suppressed comments (2)
doc/latex_manuals/BEWorkedExamples.tex:566
- Typo in the example narrative: the email address is shown as "pat@m57.biz" in the listing, but the text says "pat@m57biz" (missing dot), which can confuse readers searching for the address.
The email address "pat@m57biz" was found in the file \texttt{Documents and Settings/Charlie/}\newline\texttt{Application Data/Thunderbird/Profiles/4zy34x9h.default/Mail/Local Folders/Inbox} and investigators can refer to that location on the disk image to view the full text.\\
doc/latex_manuals/BEWorkedExamples.tex:574
- Grammar: "features differences" should be singular ("feature differences").
The output shows the features differences on the disk image. The following is an excerpt of that output:
|
Addressed both additional worked-example findings from Copilot's suppressed review in commit
Validation completed:
The pull request remains a draft while the current-head Copilot re-review and GitHub Actions checks run. |
|
Added the requested docs-only CI isolation in commit
Validation completed:
The pull request remains a draft while the current-head Copilot re-review and GitHub Actions checks run. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.
Suppressed comments (1)
doc/latex_manuals/Makefile.am:7
$(wildcard ...)is a GNU make extension; Automake-generated Makefiles are expected to be portable to non-GNUmake(notably the default BSDmakeon macOS). The repo’s top-levelMakefile.ameven calls out avoiding$(wildcard)for this reason (Makefile.am:18-20). As written,make -C doc/latex_manualscan fail on non-GNU make, andEXTRA_DIST/deps expansion becomes build-dir dependent.
USER_MANUAL_DEPS = BECurrentGuide.tex BEWorkedExamples.tex references.bib \
$(wildcard archPics/*.pdf installPics/* otherPics/* viewerPics/*) \
carvedJPEG.jpg charlieRunOutput.png
|
Addressed Copilot's portability finding in commit
Validation completed:
The pull request remains a draft while the current-head Copilot re-review and GitHub Actions checks run. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.
Suppressed comments (3)
doc/latex_manuals/BEWorkedExamples.tex:30
- This sentence is missing the possessive apostrophe: "each computers memory" should be "each computer's memory".
The 2009-M57-Patents scenario tracks the first four weeks of corporate history of the (fictional) M57 Patents company. The company started operation on Friday, November 13th, 2009, and ceased operation on Saturday, December 12, 2009. This specific scenario was built to be used as a teaching tool both as a disk forensics exercise and as a network forensics exercise. The scenario data is also useful for computer forensics research because the hard drive of each computer and each computers memory were imaged every day. In this example, we are not particularly interested in the exercises related to illegal activity, exfiltration and eavesdropping; they do however provide interesting components for us to examine in the example data\cite{m57scenario}.
doc/latex_manuals/BEWorkedExamples.tex:727
- The Prefetch paragraph has a missing verb and awkward phrasing ("carved files Windows Prefetch" / "This extremely useful"). Rewording improves readability and fixes the grammar.
The file \texttt{winprefetch.txt} contains the information from carved files Windows Prefetch that were discovered anywhere on the drive. \bulk will carve the Prefetch files from unallocated space. This extremely useful because Prefetch files are frequently deleted. A single line in the prefetch output file is also very long. The following is only the beginning of one line from the file:
doc/latex_manuals/BEWorkedExamples.tex:24
- In the encoding section, the phrase "features files" is grammatically incorrect and should be "feature files".
This issue also appears in the following locations of the same file:
- line 30
- line 727
(\charpicture{otherPics/U+0428}) or Arabic (\charpicture{otherPics/U+062D}) may show up in features files as all
|
Addressed all three additional prose findings from Copilot's suppressed review in commit
Validation completed:
The pull request remains a draft while the current-head Copilot re-review and GitHub Actions checks run. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.
Suppressed comments (1)
doc/RELEASE_NOTES.md:241
- This release-note bullet implies the code/coverage/Windows workflows ignore docs-only changes generally, but the workflows’
paths-ignoreonly applies to thepull_requesttrigger; they will still run on pushes tomain(e.g., after merging a docs-only PR). Clarify the wording so readers don’t assume docs-only merges avoid those workflows.
before a pull request is marked ready for review. The code, coverage, and
Windows workflows ignore changes limited to documentation and this LaTeX
workflow. The user-manual build uses an explicit, portable asset manifest
rather than a GNU `make`-specific wildcard.
|
Addressed Copilot's release-note scope finding in commit
Validation completed:
The pull request remains a draft while the current-head Copilot re-review and GitHub Actions checks run. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.
Suppressed comments (4)
doc/latex_manuals/BEWorkedExamples.tex:143
- Spelling: "identify information" should be "identity information" in this context.
To answer many of these questions, we look at the identify information on the drive including email addresses, credit card information, search terms, Facebook IDs, domain names and vCard data. The output files created by \bulk contain all of this type of information that was found on the disk image. \\
doc/latex_manuals/BEWorkedExamples.tex:145
- Grammar: "the user of the this drive" has an extra "the".
The scenario setup leads us to believe that Charlie is the user of the this drive (based on the name of the disk image). First, we look at \texttt{email.txt} to find information about the email addresses contained on the disk. The first two lines of the email features found are the following (each block of text represents one long line of offset, feature and context):
doc/latex_manuals/BEWorkedExamples.tex:737
- Spelling: "useable" should be "usable".
Printing the line out here would cover almost two pages. It includes a lot of information about the Prefetch file including the name of the executable, the name of the DLLs, the directory of DLLs, the atime, the number of runs, the serial number, and the ctime. The Prefetch file is searchable and useable by investigators searching for EXEs or DLLs related to a malware investigation.\\
doc/latex_manuals/BEWorkedExamples.tex:139
- Grammar: this list item contains a duplicated "have" ("have they have been").
This issue also appears in the following locations of the same file:
- line 143
- line 145
- line 737
\item What kinds of websites have they have been visiting most often?
|
Addressed all four additional historical-prose findings from Copilot's suppressed review in commit
Validation completed:
The pull request remains a draft while the current-head Copilot re-review and GitHub Actions checks run. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.
Suppressed comments (1)
doc/latex_manuals/BECurrentGuide.tex:257
- The Windows raw-device examples use
\code{\\\\.\\PhysicalDriveN}/\code{\\\\.\\X:}. In LaTeX,\\is a line-break command, so this will not reliably typeset literal backslashes (and can produce incorrect output). Use\verb|...|(or\textbackslash{}) for Windows paths that include backslashes.
\code{\\\\.\\PhysicalDriveN}, \code{\\\\.\\X:}, and a named volume path
|
Addressed Copilot's Windows-path rendering finding in commit
Validation completed:
The pull request remains a draft while the current-head Copilot re-review and GitHub Actions checks run. |
Summary
The prior rewrite had substantially reduced both manuals and removed useful material originally developed for the 1.x documentation. This change restores that material wherever it remains applicable and explicitly labels historical-only behavior.
Validation
make -C doc pdfsgit diff --cached --checkThe generated PDFs are build artifacts; this PR commits their LaTeX sources and build metadata.