Migrate from TFS (TFVC) to GitLab - #1
Open
soumyaparida-vtb wants to merge 11 commits into
Open
Conversation
First half of TFS support: read a TFVC history over the REST API and convert it into the same kind of mirror the Subversion side produces, so the export, push, verify and sync stages can be reused rather than duplicated. REST is used deliberately instead of tf.exe or the .NET client libraries, which are Windows-only and would reintroduce exactly the third-party dependency this project just removed from the SVN path. Every endpoint shape here was checked against the published API reference rather than written from memory, which turned up three defaults that silently corrupt a migration: * `maxCommentLength` defaults to 80, so every substantial commit message comes back truncated with only a quiet `commentTruncated` flag to say so; * `changeType` is a flags enum serialised as "rename, edit" - handling it as a single value leaves a renamed-and-edited file holding its previous content; * `hashValue` is a base64 MD5, which is useful for deduplicating blobs and catching truncated downloads, but only when decoded as such. Branch creation is resolved with fast-import's `ls`, as SVN directory copies are: TFVC emits a `branch` change per item, and replaying those literally would re-download an entire branch's content that Git already holds. The fake TFS server the tests run against is deliberately awkward in the same ways the real API is. That immediately earned its keep: the paging test caught a real bug where a short page was treated as the last page. Since the server caps page size independently of `$top`, that would have migrated only the first page of changesets and reported success. Not yet wired: config, CLI, export/push/verify/sync integration. Nothing here has run against a real TFS instance - the first customer run must be a rehearsal.
Completes TFS support. `source.kind: tfvc` now runs the same pipeline Subversion does — preflight, analyze, convert, export, push, verify, sync — with only the history-reading stages differing. The seam that makes this work is `MirrorFormat`: both sources produce a mirror of the same shape (refs under `refs/remotes/<prefix>` plus a trailer naming the source version per commit), so export, LFS, the GitLab push and incremental sync are shared code rather than a second implementation. Added: the TFVC mirror with incremental resume, a TFVC analyze stage producing the same report the SVN side does, and TFVC verification that fetches each file at the migrated changeset and compares Git blob hashes — checking the item's MD5 first, so a truncated download is caught rather than reported as a content difference. Proven end to end against the fake servers: a TFVC project migrates to GitLab, a fresh clone returns the expected bytes including binaries, and verification passes. A companion test tampers with the content to prove verification can actually fail. TFVC differences are reported rather than glossed: no tags, merges become ordinary commits with correct content but simplified topology, no ignore translation, and the cutover lock stays Subversion-only. Still not run against a real TFS instance. The first customer run must be `analyze` followed by a dry run.
The Windows job failed comparing a cloned file to the expected content. The migration was correct: Git converts line endings on checkout when core.autocrlf is on, which it is by default on Windows, so the working tree legitimately differs from what is stored. Compare the blob instead. The binary is still checked through the working tree, since it must survive checkout untouched whatever the eol setting. The equivalent SVN assertion is accidentally immune because `read_text` opens in text mode and normalises newlines on read; mine used `read_bytes`, which does not. Also adds a guard that repositories this tool creates pin core.autocrlf=false. Everything the project claims about reproducibility assumes the bytes written are the bytes stored - if Git were free to translate them, a Windows conversion and a Linux conversion would produce different blobs, and the golden master, the byte-for-byte verification and the fast-forward push would all quietly stop meaning what they say.
Three things needed for a client to install and test TFS support from the installer rather than from source. `doctor` now reports readiness per source kind. It previously demanded git, svn and svnadmin for everything, so a machine set up purely to migrate TFS - which reads history over HTTPS and needs nothing but git - would have been told it was broken, and the installer's post-install check would have repeated the same wrong advice. `--source svn|tfvc` gates on one specifically; without it the command succeeds if either kind can run, and the table says which. The installer gains a source-type choice, and offers to download and install Git for Windows when it is missing. Subversion is deliberately not auto-installed: there is no official Windows build with a stable download URL, so guessing at a third-party repackaging would be less reliable than telling the operator what to install. TFS migrations do not need it at all. Git is downloaded rather than bundled because redistributing it inside this installer carries GPLv2 obligations, including a matching source offer, that cannot be discharged from here - and fetching on demand means a current, patched build rather than whatever was current when the package was cut. The pinned asset URL is verified during the release build, so a stale link fails there rather than at a customer site. The first URL I wrote pointed at a release that does not exist, which is precisely why that check is in. Tags carrying a suffix now publish as prereleases, so a build cut for testing cannot be mistaken for a general release. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…Subversion Two fixes that both block a client migration, shipped together as 1.1.0-rc2. **Windows Server 2016.** The installer refused to run on it: MinVersion was 10.0.17763, which is Server 2019, while Server 2016 is 10.0.14393. That gate was inherited from when 2019 was the only stated target and never revisited - nothing in the tool needs anything newer. Python 3.12, which the frozen build uses, supports Server 2012 R2 and later, so 2016 was always inside support. Lowering the setup gate is only half of it. `MinVersion` controls whether *setup* runs; the frozen executable carries its own floor in its PE header, and the Windows loader refuses an image stamped newer than the running OS. A build produced on a current runner can therefore install happily on Server 2016 and then fail to start - the worst place to find out being a client's server mid-migration. So the Windows build is pinned to windows-2022 (windows-2019 has been retired, so that is the oldest available), and `installer/check_os_floor.py` reads the PE subsystem version of the built binary and fails the build if it demands anything newer than 10.0. It runs in CI on every push as well as in the release, and its parser is tested against synthetic images including one that must be rejected - a check that cannot fail would be decoration. **TFVC sync and cutover.** Both asked the Subversion client for the head revision and author list regardless of source kind, so every TFVC sync died with `svn: E170013: Unable to connect`, and cutover with it, since cutover runs a final sync. Both are now source-aware, and cutover states plainly that a TFVC project is frozen by denying "Check in" in TFS rather than by a hook this tool can install. Found by exercising sync and cutover directly rather than assuming the migrate path covered them. Regression tests added for both, including a new committer appearing after the migration - the case that previously aborted every subsequent sync.
…to the installer Found against a real TFS server during a client migration. The first two are blockers; the third is why the installer failed at that site. **recursionLevel is case-sensitive.** We sent `full`; TFS answers 400 and helpfully enumerates the valid values (None, OneLevel, OneLevelPlusNestedEmptyFolders, Full). In analyze this silently reported 0 files and 0 bytes at HEAD. Worse, verification reads the same tree listing, so every branch would have failed to list, been marked skipped, and the migration would have failed at its final stage. It fails loudly rather than passing green, but no verified migration was possible. **An arbitrary branch became the mainline.** When no branch is named Main, Trunk or Master, the first alphabetically was chosen and nothing said so. On a real project that made `CR-Merger-1` - a transient change-request branch - the default branch everyone would clone. An explicitly configured `branch_roots` list is now treated as an instruction, with its first entry as the mainline, and a guessed mainline raises a warning naming the consequence. **Git is bundled rather than downloaded.** Download-on-demand failed at the client, and always would have: migration hosts sit behind VPNs and proxies that block public DNS. Git for Windows (MinGit) is fetched when the release is built and shipped inside the package, so installation needs no network. The build proves it by stripping every Git directory from PATH and confirming the tool still resolves one from inside its own package. GPLv2 is discharged by shipping Git's licence and a source offer naming the exact upstream release and its SHA-256. Also: query parameters now encode spaces as %20 rather than +, since TFVC paths routinely contain spaces and TFS reads the plus literally; TFS's own error text is no longer discarded in favour of a worse guess; and analyze checks the project root exists up front, listing the projects that do. The fake TFS returned only files for a one-level listing where the real server returns folders, which made the new project-root guard silently untestable. Fixed, and the guard is now confirmed to fire.
…h stages agree on
Two defects, found by verifying a real 994-changeset migration and confirmed against
the customer's TFS. Either one alone would have published a wrong repository.
Branch creation copied the wrong tree. `_commit` treated a folder-level `branch`
change as a whole-tree copy from the *converted* source branch's head, and skipped
the per-file `branch` records TFVC emits alongside it. But TFVC branches from any
version, not only the source's tip, so that head is frequently the wrong tree - and a
branch cut from an already-stale branch inherits the error and adds its own. On the
customer repository this left ~4,816 files stale across ten branches: untouched on
`main` and `v16`, identical on three sibling copies, and worst on the branches cut
from other branches. The per-file records carry each file's own item version and are
now the authority. It stays cheap because `_blob_for` dedupes on the content hash
*before* fetching, so a branch copy costs metadata rather than bandwidth - the
existing "does not redownload the tree" budget still holds unchanged.
The conversion and the analysis disagreed about the mainline. `analyze` passed
`explicitly_configured` to `detect_layout` and honoured the operator's ordering; the
pipeline did not, and fell back to picking alphabetically. With the customer's
thirteen branch roots the report named `Main-UI-Dev` as trunk while the conversion
published `CR-Merger-1` as `main`, and `mainline-guessed` stayed silent because
analyze had made the right choice.
Verification cannot catch that second one: it compares `main` against whatever the
conversion decided `main` meant, so a side branch published as the default branch
verifies perfectly. Neutering the new consistency check makes the run report
`succeeded` with the wrong default branch, which is why the check exists and why it
is asserted rather than assumed. The layouts must now agree or the run stops.
Reporting fixes, all of which made these harder to find:
* `difference_count` counted the list *after* `max_reported_diffs` truncated it, so
a branch with 1,245 differences reported 200. The count is never capped now; only
the listing is, and `differences_omitted` says how much was withheld.
* `note` and `skip_reason` were written to an attribute `BranchVerification` does
not declare, so the truncation warning and every skip reason were silently
discarded. Both are real fields now.
* The manifest dropped all verification detail when verification failed - the one
case it matters - because the stage raises before returning a result. It falls
back to the in-memory report.
* Extensionless files bucketed under `""`, which Windows PowerShell's
ConvertFrom-Json refuses to load, making the manifest unreadable on the platform
the tool is built for. They bucket under `(no extension)`.
Tests: the fake only ever branched from the tip, so it could not express any of this
and 41 TFVC tests passed straight through the defect. It now branches from an older
version, and `tests/test_tfvc_pipeline.py` covers the pipeline path - which had no
coverage at all, and is where both defects lived. Each new test was run against the
unfixed code and fails there.
Ships the TFVC branch-content and mainline-selection fixes from 4b08b0a. rc3 must not be used for a TFVC migration: it converts branch content from the source branch's converted head rather than TFVC's per-file records, and publishes the alphabetically-first branch root as `main` regardless of configured ordering.
The installer version was a hand-maintained second copy of the project version and it drifted: v1.1.0-rc4 published an installer named svn2gitlab-setup-1.1.0-rc3.exe, built from the right source but labelled with the previous release. The Add/Remove Programs entry was wrong too. ISCC now takes the version from the tag, and the build fails if the file it produced is not named after that tag. A mislabelled installer is worse than a missing one: it gets installed, and then bugs are reported against a build nobody ran.
`touches_content` counted `merge` as a content-bearing flag. It is not. In TFVC `merge` records a relationship and the action travels beside it - `merge, edit`, `merge, branch`, `merge, delete`. A bare `merge` means a merge happened that did not change the item, and when what was merged is a deletion, TFS reports exactly that: a bare `merge` on a path that is already deleted and stays deleted. Confirmed against the customer's TFS. `$/…/Optimization/src/app/Allertbox/ Allertbox.module.ts` is deleted at changeset 2274 and 404s at every version after, yet changeset 2355 carries a `merge` record for it. The converter wrote the file back. Across two branches that restored 730 deleted files. Nothing caught it. `_blob_for` found the content hash in its cache and served the bytes without a request, so the 404 that would have exposed it was never issued - and verification's per-branch listing cap hid the count behind content errors until that cap was fixed earlier today. The fake TFS made the same assumption: `_deleted_at` treated any non-delete change as a resurrection and `tree_at` re-added the path, so a fake-backed test would have agreed with the bug. Both now model the server's actual behaviour, and the new test fails against the unfixed converter. Bumps to 1.1.0rc5.
Three more defects in the same family as the merge fix: change flags whose handling was never exercised because the fake only ever emitted the easy shape. A rename with no edit deleted the file. `rename` carries no content flag, so `_blob_for` produced nothing and `_apply` removed the old path and wrote nothing at the new one. Only `rename, edit` had coverage, where the edit supplied the bytes and hid it. A rename now moves the object already in the parent tree, which is both exact and free - a rename does not change content, so there is nothing to download. Folder deletes removed nothing and folder renames moved nothing. Every folder change was skipped on the grounds that folders carry no content, but TFVC can delete or rename a folder with no per-file records at all, and both operations move the whole subtree. A deleted folder stayed in the migrated branch forever. Folder changes that delete or rename are now replayed; the mode comes from the parent tree, so a renamed folder moves as a tree rather than being written as a file. None of these fired on the customer repository - its verification reported no missing-in-git on any of thirteen branches - but a sync or a different history would hit them, and each was found by asking what the untested flag combinations do rather than by waiting for a migration to fail.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Team Foundation Version Control as a second source, so the tool migrates both SVN and TFS to GitLab.
How it fits
source.kind: tfvcreads a TFVC history over the REST API and converts changesets into Git commits using the samegit fast-importwriter the Subversion engine uses. AMirrorFormatseam means both sources produce a mirror of the same shape, so export, Git LFS, the GitLab push, verification and incremental sync are shared code, not a second implementation.REST rather than
tf.exeor the .NET client libraries — deliberately, so the migrator stays cross-platform and free of the Windows-only dependency this project removed from the SVN path.Grounded in the API reference, not memory
Every endpoint was checked against Microsoft's docs. Three documented defaults silently corrupt a migration and are handled explicitly:
maxCommentLengthdefaults to 80, truncating every substantial commit message with only a quietcommentTruncatedflagchangeTypeis a flags enum —"rename, edit"means both; handling it as one value leaves a renamed file holding stale contenthashValueis a base64 MD5, used to deduplicate blobs and catch truncated downloadsBranch creation is resolved with fast-import's
lsrather than replaying the per-filebranchchanges TFVC emits, so branching a large tree costs one operation instead of re-downloading it.Verified
End to end against the fake TFS and fake GitLab servers: a TFVC project migrates, a fresh clone returns the expected bytes including binaries, and verification passes. A companion test tampers with content to prove verification can fail.
The fake TFS is deliberately awkward in the same ways the real API is. That earned its keep immediately — it caught a real bug where a short page was treated as the last page, which would have migrated only the first 100 changesets and reported success.
327 tests, all passing. The SVN path is unchanged.
Reported, not glossed
No tags (TFVC has none), merges become ordinary commits with correct content but simplified topology, no ignore translation, and the cutover lock stays Subversion-only — for TFVC, deny "Check in" in TFS.
The gap
Nothing here has run against a real TFS instance. It proves we handle the API as documented; TFS 2015 vs 2018 will differ in ways docs do not capture. First customer run must be
analyzefollowed by a dry run.