Cross-Language Sort Performance Benchmark

Course: CSCI 334, Communication for Computing Sciences, Fall 2025

Assignment: Analysis-Experiment

Assignment Intent

The assignment was a "Communication for Computing Sciences" research-writing exercise: run a genuine numerical experiment, then write it up as a formal research paper and present it as a slide deck, following a detailed academic-writing checklist covering document structure (a system-parameters table, numbered/highlighted code, a math environment for equations, explicit float references, a fully-cited bibliography, active voice, double spacing, page numbers) and presentation structure (self-contained slides with explicit URLs, terse bullet points, visuals prioritized over paragraphs, and content that is a subset of the paper's).

The underlying numerical experiment is what's implemented in this repository: three independent sort-performance drivers (sort.cpp, Sort.java, sort.py) timing each language's built-in general-purpose sort across five dataset sizes (50K-800K lines, 500 iterations each), logged to a shared CSV and analyzed in analysis.ipynb with warm-up/outlier removal (MAD), one-way ANOVA and effect size, and cross-language stability comparisons -- see Overview below for the full breakdown. That part is verified directly against the code.

docs/Report.pdf (added to this repository) is the actual submitted paper/slide artifact. It has been scanned and confirmed clean of identifiable information (no student ID or personal file paths). Checking its content against the formatting checklist above (fonts, floats, bibliography, etc.) has not been done yet; that verification is pending.

Status

This is student coursework, not an actively maintained project. It was written while learning basic performance benchmarking and statistics, and development has stopped. Nothing here is "finished" in a production sense — see Known Issues below before reusing any of it.

Overview

It benchmarks each language's built-in general-purpose sort (std::sort in C++, Collections.sort in Java, list.sort() in Python) against the same line-of-text datasets (50K / 100K / 200K / 400K / 800K lines), running 500 timed iterations per dataset per language, logging every run to a shared CSV file, and then analyzing the results in a Jupyter notebook.

Verified against source: sort.cpp, Sort.java, and sort.py each define ITERATIONS = 500 and iterate over the same five input filenames, so each language produces 5 * 500 = 2,500 log rows per run (7,500 total across all three languages). src/dataset/times.csv currently contains exactly 7,500 rows, consistent with one completed run of all three drivers. The CSV has no header row; columns are Language, Type, Iteration, DataSize, Time (time in milliseconds), which is what analysis.ipynb's load_data() assumes when it reads the file with header=None, names=[...].

The notebook (notebooks/analysis.ipynb) is the actual analytical work. Verified by reading the notebook's cells and their saved outputs:

• Removes the first N "warm-up" iterations per language/dataset-size group (remove_warmup_runs, called with n_warmup=25 in the saved run)
• Removes statistical outliers using Median Absolute Deviation, MAD (mad_outliers_by_group, threshold 3.5 in the final pipeline; the notebook also sweeps thresholds 2.5-4.5 to show sensitivity)
• Runs one-way ANOVA and effect size, eta squared, per dataset size (perform_statistical_tests, using scipy.stats.f_oneway)
• Compares stability across languages using coefficient of variation, CV (analyze_stability)
• Normalizes timing by n * log(n) to compare scaling behavior independent of raw speed (analyze_dataset_performance, plot_time_per_element_combined_and_individual)

One inconsistency found while reading the notebook, not present in the original README, worth flagging for anyone extending the analysis: the RQ1 statistics/ANOVA cells run against cleaned_data (post warm-up-removal and post-MAD-removal), while the RQ2 stability and RQ3 dataset-size cells run against the raw data DataFrame. This is a real inconsistency in the analysis, not a security or dead-code issue, so it isn't broken out in Known Issues below, but a future contributor should decide whether RQ2/RQ3 are supposed to use cleaned data too.

Research questions the notebook answers (see its markdown cells):

1. Which language performs best overall, and per dataset size?
2. Which language is most stable (lowest run-to-run variance)?
3. Which dataset size is each language most/least efficient at?

Repository Structure

src/
  sort.cpp            C++ benchmark driver
  Sort.java            Java benchmark driver (class Sort)
  sort.py              Python benchmark driver
  inputs/
    50K.txt ... 800K.txt   Line-based text datasets sorted by each benchmark
                            (verified line counts: 50000, 100000, 200000, 400000, 800000)
  outputs/              Sorted output written by each language (gitignored; not present
                         in this checkout)
  dataset/
    times.csv           Combined timing log all three benchmarks append to (7,500 rows,
                         no header row)
notebooks/
  analysis.ipynb        Statistical analysis notebook (the real deliverable), reading
                         ../src/dataset/times.csv

Each of sort.cpp / Sort.java / sort.py is a separate, independent program. They are not called from each other; each one is run by hand and each one appends its own rows to dataset/times.csv (relative to src/, i.e. src/dataset/times.csv).

Dependencies

C++ (sort.cpp)

• A C++17 compiler (uses <filesystem>)
• GCC example: g++ -std=c++17 -O2 sort.cpp -o sort (older GCC/libstdc++ may need -lstdc++fs appended)
• No third-party libraries — standard library only (<iostream>, <vector>, <string>, <algorithm>, <fstream>, <chrono>, <iomanip>, <filesystem>)

Java (Sort.java)

• JDK 8 or newer (uses java.nio.file, no version-specific language features)
• Compile: javac Sort.java (produces Sort.class)
• Run: java Sort
• No third-party libraries
• (Fixed) This file was previously named sort.java (lowercase) while declaring public class Sort (capitalized). Java requires a public top-level class to live in a file named exactly <ClassName>.java — a compiler-enforced, case-sensitive rule independent of the filesystem's own case sensitivity — so javac sort.java did not actually compile as documented. The file has been renamed to Sort.java to match the class name; the command above now works as written.

Python (sort.py and the analysis notebook)

• sort.py itself has zero third-party dependencies. Verified imports: os, time, csv (all standard library).
• The notebook requires the packages listed in requirements.txt: pandas, numpy, scipy, matplotlib, seaborn, jupyter. Verified against the notebook's actual import statements — all five are used (pandas, matplotlib.pyplot, numpy, scipy.stats, seaborn).
• No versions are pinned in requirements.txt and no lock file exists. If this project is picked back up, pin exact versions once a working environment is confirmed.

Environment Setup

python -m venv venv
venv/bin/pip install -r requirements.txt      (or venv\Scripts\pip on Windows)

Running a Benchmark

All three drivers expect to be run from inside src/, since inputs/, outputs/, and dataset/ are all referenced as relative paths in the source:

cd src
g++ -std=c++17 -O2 sort.cpp -o sort_cpp && ./sort_cpp
javac Sort.java && java Sort
python sort.py

Each run appends 500 * 5 = 2,500 rows per language to dataset/times.csv, so re-running a benchmark does not overwrite prior results — it accumulates. Delete or archive times.csv first if a clean run is needed.

Known Issues

This section separates "intentionally incomplete because the assignment ended" from "actually broken." Nothing below has been fixed — these are notes for a future contributor if this project is ever resumed. Each item below was checked directly against the files in this checkout; a couple of claims from an earlier draft of this README did not hold up and have been corrected or removed (see notes inline).

Dead Code and Repository Hygiene

1. Compiled binary committed to the repo (Sorts/sort) -- FIXED

Confirmed: Sorts/sort was a compiled ELF 64-bit binary (file reported "ELF 64-bit LSB pie executable, x86-64 ... dynamically linked ... for GNU/Linux 3.2.0"), almost certainly produced by running g++ on sort.cpp and committed by accident. Binaries should not be checked into version control.

• Severity: Low (hygiene, not a functional or security bug)
• Resolution: Sorts/sort has been removed from the working tree and from git's index (git rm --cached). .gitignore now has an explicit src/sort rule (matching the exact filename that was previously committed, updated to the Sorts/ -> src/ rename described in Repository Structure above) plus generic *.o and *.out patterns to catch other compiled artifacts, so the binary cannot be re-added accidentally. Regenerate it locally with: cd src && g++ -std=c++17 -O2 sort.cpp -o sort (see the C++ Dependencies section above for compiler requirements). Note the binary's removal is only from the current working tree/index; it still exists in this repository's prior git history unless that history is separately rewritten.

2. .gitignore references a much larger, abandoned project scope

Confirmed by reading .gitignore directly: it contains rules for Comparison/C/fish_tank, Comparison/C/simulation_output/, Comparison/Rust/simulation_output/, src/sort_go, src/sort_rust (renamed from Sorts/sort_go / Sorts/sort_rust along with the Sorts/ -> src/ rename described in Repository Structure above), a .NET project at Sorts/SortProgram/, and a root-level Multithreading-Analysis.sln solution file. None of these exist in this repository. This is further corroborated by the git history (git log), whose earliest commits are literally titled "Fish tank in C" and "main.c captures metrics" (a fluid-simulation project), followed later by commits about creating sort programs — confirming the repo's actual origin was a larger, multi-language, multi-experiment comparison that was scaled back to just this three-language sort benchmark. The .gitignore also still allowlists Sorts/input_50000.txt through Sorts/input_500000.txt, an older input-naming scheme that no longer matches the current inputs/50K.txt-style files.

• Severity: Low (hygiene; does not affect current functionality)
• Fix-it plan: if resumed, either (a) restore the additional language implementations and simulation experiment referenced by the ignore rules, or (b) if the scope reduction to "just sorting, just these 3 languages" is permanent, delete the dead ignore rules so the file reflects reality.

3. Identical logic triplicated across three languages

Confirmed by direct comparison: readData/writeData/writeLog/ performSortingBenchmark (and their Python/Java equivalents) are near-verbatim copies across all three files, including the same hardcoded constants (ITERATIONS = 500, PROGRESS_INTERVAL = 10, the same five input filenames, the same directory names). This is normal for a first pass at a cross-language comparison — you cannot actually share implementation logic across languages — but the configuration values didn't need to be hardcoded three separate times.

• Severity: Low (maintainability only)
• Fix-it plan: if this benchmark is extended (new dataset sizes, different iteration counts, etc.), move the shared configuration into a single file (e.g. small JSON/YAML read by all three programs) instead of editing three source files in lockstep.

4. No one-command way to regenerate results, and no output/figure artifacts saved -- PARTIALLY ADDRESSED

Confirmed: the notebook's plots (warm-up behavior, normalized time, CV by dataset size) are only ever displayed inline via plt.show(); grepping the notebook found no calls to savefig or similar. There is no Makefile/CMakeLists.txt despite .gitignore anticipating CMake artifacts (CMakeFiles/, CMakeCache.txt, compile_commands.json) that were never actually produced.

A CI workflow (.github/workflows/ci.yml, badge at the top of this README) has since been added, which does automatically compile all three drivers and smoke-test each against the smallest dataset (50K.txt, full 500 iterations) on every push. That covers "does this still build and run at all," but it is intentionally not a full regeneration of results: it does not run all five dataset sizes, does not re-execute the analysis notebook, and does not save any figures. There is still no one-command way to reproduce the full times.csv and notebook outputs from scratch.

• Severity: Low
• Fix-it plan: if this project continues, add a small run script (shell or make) that builds all three benchmarks, runs them against all five dataset sizes, and re-executes the notebook (e.g. via jupyter nbconvert --execute), and have the notebook save figures to a figures/ directory instead of only calling plt.show().

5. Analysis inconsistency: mixing raw and cleaned data across research questions

New finding, not in the prior README. calculate_statistics and perform_statistical_tests (RQ1: best performer, ANOVA) are called on cleaned_data — the DataFrame that has had warm-up iterations and MAD outliers removed. analyze_stability (RQ2: CV/stability) and analyze_dataset_performance (RQ3: dataset-size efficiency) are instead called directly on the raw data DataFrame, with no warm-up or outlier removal applied. This means RQ1's conclusions and RQ2/RQ3's conclusions are not computed on the same underlying dataset, which undermines direct comparison between them.

• Severity: Low-Medium (affects correctness/consistency of the analysis, not security)
• Fix-it plan: decide whether RQ2 and RQ3 should also run on cleaned_data (most likely intent, given the whole point of the warm-up/MAD cleaning step earlier in the notebook) and update those two calls accordingly, then re-run and compare whether the stability/efficiency rankings change.

Security Findings

A direct read-through of sort.py, sort.cpp, Sort.java, and analysis.ipynb found:

• No hardcoded credentials, API keys, tokens, or secrets anywhere in the repository (checked all source files, notebook, and config files for common secret patterns).
• No network calls in any of the three benchmark drivers or the notebook (no requests, urllib, socket, or HTTP client usage).
• No shell/command execution (no subprocess, os.system, eval, exec, Runtime.getRuntime, or ProcessBuilder usage in any file).
• No unsafe deserialization (no pickle, no dynamic class loading).
• File I/O in all three drivers reads/writes only from a small, hardcoded list of filenames (inputs/50K.txt etc.) under fixed relative directories (inputs/, outputs/, dataset/). None of these paths are built from external or user-supplied input, so there is no path-traversal or injection risk in normal use.

No security findings requiring a fix. This is a low-risk, offline, file-based benchmarking tool with no attack surface beyond running it locally.

Status

Coursework, not maintained. Development stopped once the assignment concluded. All five research-question findings and the outlier/warm-up analysis described above were reproduced from the notebook's own saved cell outputs, not re-run, so they reflect the last time the author executed the notebook.