Harden 0.4.0: fix the [bracket] DDL crash, make the CLI scriptable, add CI

.github/workflows/ci.yml

@@ -0,0 +1,36 @@
+name: CI
+
+on:
+  push:
+    branches: [main, dev]
+  pull_request:
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v6
+        with:
+          enable-cache: true
+      - run: uv sync --locked
+      - run: uv run ruff check .
+      - run: uv run ruff format --check .
+
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
+    env:
+      UV_PYTHON: ${{ matrix.python-version }}
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v6
+        with:
+          enable-cache: true
+          python-version: ${{ matrix.python-version }}
+      # --locked fails the build if uv.lock drifted from pyproject.toml
+      - run: uv sync --locked
+      - run: uv run pytest

.github/workflows/publish.yml

@@ -0,0 +1,17 @@
+name: Publish
+
+on:
+  push:
+    tags: ["v*"]
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    permissions:
+      # Required for PyPI trusted publishing: no API token to store as a secret.
+      id-token: write
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v6
+      - run: uv build
+      - run: uv publish --trusted-publishing always

Makefile

@@ -1,12 +1,16 @@
 
-run:
-	python -m sqlite2duckdb 
-
-build:
-	rm -Rf dist/ ; python -m build
+dev:
+	uv sync
+
+lint:
+	uv run ruff check .
+	uv run ruff format --check .
 
 test:
-	python -m pytest
+	uv run pytest
+
+build:
+	rm -rf dist/ && uv build
 
 publish:
-	python -m twine upload  dist/*
+	uv publish

README.md

@@ -1,32 +1,46 @@
 # sqlite2duckdb
 
+![CI](https://github.com/dridk/sqlite2duckdb/actions/workflows/ci.yml/badge.svg)
 ![PyPI - Python Version](https://img.shields.io/pypi/pyversions/sqlite2duckdb)
 ![PyPI - Downloads](https://img.shields.io/pypi/dm/sqlite2duckdb)
 
 A tool for converting a [sqlite](https://www.sqlite.org/) database into a [duckdb](https://duckdb.org/) database
 
-
-## Description 
+## Description
 
 Sqlite is an embedded online database designed for transactional reading and writing.
 Duckdb is also an embedded database, but column-oriented, designed for analytical process with a very high reading efficiency.
 
 For more details [https://towardsdatascience.com/forget-about-sqlite-use-duckdb-instead-and-thank-me-later-df76ee9bb777](https://towardsdatascience.com/forget-about-sqlite-use-duckdb-instead-and-thank-me-later-df76ee9bb777)
 
+Requires Python >= 3.9 and duckdb >= 1.1.0 (indexes are only copied from that version on).
+
+## Installation
+
+With [uv](https://docs.astral.sh/uv/), no installation is required. `uvx` downloads and runs the tool in one go:
+
+```bash
+uvx sqlite2duckdb source.db target.db
+```
 
-## Installation 
+To keep it around:
 
+```bash
+uv tool install sqlite2duckdb
 ```
+
+Or with pip:
+
+```bash
 pip install sqlite2duckdb
 ```
 
-## Usage 
+## Usage
 
-### As a command line 
+### As a command line
 
 ```
-
-usage: sqlite2duckdb <sqlite_path> <duckdb_path>
+usage: sqlite2duckdb [-f] <sqlite_path> <duckdb_path>
 
 Convert Sqlite database to Duckdb database
 
@@ -36,35 +50,74 @@ positional arguments:
 
 options:
   -h, --help     show this help message and exit
+  -f, --force    overwrite the duckdb file if it already exists
+  -q, --quiet    only report errors
+  --verbose      report every step
   -v, --version  show program's version number and exit
-
-
 ```
 
-### Examples 
+The tool never overwrites an existing target silently. On a terminal it asks for
+confirmation; anywhere else (a script, a CI job, a pipe) it exits with code 1 and tells you
+to pass `--force`. Progress is written to stderr, so stdout stays free for pipelines.
+
+### Examples
 
 ```bash
-sqlite2duckdb source.db target.db
+uvx sqlite2duckdb source.db target.db
+uvx sqlite2duckdb --force source.db target.db   # overwrite target.db without asking
 ```
 
-### From python 
+### From python
 
 ```python
+from sqlite2duckdb import sqlite_to_duckdb
 
-from sqlite2duckdb  import sqlite_to_duckdb
-sqlite_to_duckdb("source.sqlite", "target.duckdb")
-
+result = sqlite_to_duckdb("source.sqlite", "target.duckdb")
+print(result.tables, result.elapsed)
 ```
 
-## Todo 
+`sqlite_to_duckdb(sqlite_db, duck_db, *, overwrite=False)` accepts `str` or `pathlib.Path`
+and returns a `ConversionResult` (`target`, `tables`, `elapsed`). It raises
+`FileNotFoundError` if the source is missing and `FileExistsError` if the target already
+exists and `overwrite` is False. If the conversion fails halfway, the partially written
+target file is removed rather than left behind. Progress is reported through the standard
+`logging` module (logger `sqlite2duckdb.sqlite_to_duckdb`), never printed.
 
-- [ ] Custom mapping
-- [ ] Relation and constraint 
+## What is converted
 
+| | |
+|---|---|
+| Tables and data | ✅ |
+| Primary keys, NOT NULL constraints, indexes | ✅ |
+| UNIQUE, FOREIGN KEY and CHECK constraints | ❌ |
+| Views | ❌ (silently dropped) |
 
-### See also 
+Duckdb's sqlite extension does not expose the last two on the attached database, so they
+cannot be copied. Reading them back from `sqlite_master` would be needed.
 
-- [Harlequin](https://github.com/tconbeer/harlequin): A nice duckdb IDE for your terminal
+Tables are recreated from the DDL duckdb derives for the attached database, then filled
+from it, and the indexes are read back from `sqlite_master`. This is what makes sqlite
+files that quote their DDL with `[brackets]` (chinook.db, MS Access exports) convert
+correctly: duckdb's own parser rejects that syntax, so the quoting is translated first.
+
+## Todo
+
+- [ ] Custom type mapping
+- [x] Primary keys, NOT NULL constraints and indexes
+- [ ] Views, and UNIQUE / FOREIGN KEY / CHECK constraints
 
+## Contributing
 
+The project uses [uv](https://docs.astral.sh/uv/) for everything:
 
+```bash
+make dev     # uv sync — installs duckdb plus the test deps (pytest, faker, ruff)
+make test    # uv run pytest
+make lint    # uv run ruff check . && uv run ruff format --check .
+make build   # uv build
+make publish # uv publish (PyPI trusted publishing, also run on tags by CI)
+```
+
+### See also
+
+- [Harlequin](https://github.com/tconbeer/harlequin): A nice duckdb IDE for your terminal

pyproject.toml

@@ -1,23 +1,29 @@
 [build-system]
-requires = ["hatchling", "build", "twine"]
+requires = ["hatchling>=1.27"]
 build-backend = "hatchling.build"
 
 
 [project]
 name = "sqlite2duckdb"
-version = "0.3.0"
+version = "0.4.0"
 authors = [{name="Sacha Schutz", email="sacha.schutz@pm.me"}]
 description = "A tool to convert sqlite database to duckdb database"
 readme = "README.md"
-requres-python = ">=3.8"
+requires-python = ">=3.9"
+license = "MIT"
+license-files = ["LICENSE"]
 keywords = ["sqlite", "duckdb", "database", "olap", "oltp"]
 classifiers = [
     "Programming Language :: Python :: 3",
-    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
     "Operating System :: OS Independent",
 ]
 dependencies = [
-    'duckdb >= 0.10.0'
+    'duckdb >= 1.1.0'
 ]
 
 
@@ -27,3 +33,28 @@ Issues = "https://github.com/dridk/sqlite2duckdb/issues"
 
 [project.scripts]
 sqlite2duckdb = "sqlite2duckdb.__main__:main_cli"
+
+[dependency-groups]
+dev = [
+    "pytest >= 7.0",
+    "faker",
+    "ruff",
+]
+
+[tool.hatch.build.targets.wheel]
+packages = ["sqlite2duckdb"]
+
+# Allow list rather than a deny list, so that anything new landing in the repo
+# stays out of the distribution unless it is explicitly wanted.
+[tool.hatch.build.targets.sdist]
+include = ["sqlite2duckdb", "tests", "README.md"]
+
+[tool.pytest.ini_options]
+pythonpath = ["."]
+testpaths = ["tests"]
+
+[tool.ruff]
+target-version = "py39"
+
+[tool.ruff.lint]
+extend-select = ["I", "UP"]

sqlite2duckdb/__init__.py

@@ -1,4 +1,14 @@
 import importlib.metadata
-from sqlite2duckdb.sqlite_to_duckdb import sqlite_to_duckdb
 
-__VERSION__ = importlib.metadata.version("sqlite2duckdb")
+from sqlite2duckdb.sqlite_to_duckdb import ConversionResult, sqlite_to_duckdb
+
+try:
+    __version__ = importlib.metadata.version("sqlite2duckdb")
+except importlib.metadata.PackageNotFoundError:
+    # Running from a source checkout that was never installed.
+    __version__ = "0.0.0.dev0"
+
+# Deprecated alias, kept so that existing imports keep working.
+__VERSION__ = __version__
+
+__all__ = ["ConversionResult", "__version__", "sqlite_to_duckdb"]

sqlite2duckdb/__main__.py

@@ -1,38 +1,76 @@
-import duckdb
+from __future__ import annotations
+
 import argparse
+import logging
 import os
-from sqlite2duckdb import sqlite_to_duckdb, __VERSION__
+import sys
+
+from sqlite2duckdb import __version__, sqlite_to_duckdb
 
 
-def main_cli():
+def main_cli() -> int:
     parser = argparse.ArgumentParser(
+        prog="sqlite2duckdb",
         description="Convert Sqlite database to Duckdb database",
-        usage="sqlite2duckdb <sqlite_path> <duckdb_path>",
+        usage="sqlite2duckdb [-f] <sqlite_path> <duckdb_path>",
     )
 
     parser.add_argument("sqlite_path", type=str, help="sqlite file path")
     parser.add_argument("duckdb_path", type=str, help="duckdb file path")
     parser.add_argument(
-        "-v", "--version", action="version", version=f"sqlite2duckdb {__VERSION__}"
+        "-f",
+        "--force",
+        action="store_true",
+        help="overwrite the duckdb file if it already exists",
+    )
+    parser.add_argument("-q", "--quiet", action="store_true", help="only report errors")
+    parser.add_argument("--verbose", action="store_true", help="report every step")
+    parser.add_argument(
+        "-v", "--version", action="version", version=f"sqlite2duckdb {__version__}"
     )
 
-    # Analyser les arguments
     args = parser.parse_args()
 
-    if os.path.exists(args.duckdb_path):
-        delete_input = (
-            input(
-                f"{args.duckdb_path} already exists. do you want to delete this file ? (yes/no): "
+    if args.quiet:
+        level = logging.WARNING
+    elif args.verbose:
+        level = logging.DEBUG
+    else:
+        level = logging.INFO
+    # Progress goes to stderr so that stdout stays free for pipelines.
+    logging.basicConfig(level=level, format="%(message)s", stream=sys.stderr)
+
+    overwrite = args.force
+    if not overwrite and os.path.exists(args.duckdb_path):
+        if not sys.stdin.isatty():
+            print(
+                f"{args.duckdb_path} already exists. Use --force to overwrite it.",
+                file=sys.stderr,
+            )
+            return 1
+        try:
+            answer = (
+                input(
+                    f"{args.duckdb_path} already exists. do you want to delete this file ? (yes/no): "
+                )
+                .strip()
+                .lower()
             )
-            .strip()
-            .lower()
-        )
-        if delete_input in ("yes", "y"):
-            os.remove(args.duckdb_path)
-        else:
-            exit(1)
-    sqlite_to_duckdb(args.sqlite_path, args.duckdb_path)
+        except EOFError:
+            print(f"{args.duckdb_path} already exists.", file=sys.stderr)
+            return 1
+        if answer not in ("yes", "y"):
+            return 1
+        overwrite = True
+
+    try:
+        sqlite_to_duckdb(args.sqlite_path, args.duckdb_path, overwrite=overwrite)
+    except (FileNotFoundError, FileExistsError) as error:
+        print(f"error: {error}", file=sys.stderr)
+        return 1
+
+    return 0
 
 
 if __name__ == "__main__":
-    main_cli()
+    sys.exit(main_cli())