Merge pull request git#391 from newren/master

rn-54: add an introduction to git-filter-repo

Author: chriscool

rev_news/drafts/edition-54.md

@@ -78,6 +78,253 @@ This edition covers what happened during the month of July 2019.
 ### Support
 -->
 
+## An Introduction to git-filter-repo
+
+There is a new tool available for surgery on git repositories:
+[git-filter-repo](https://github.com/newren/git-filter-repo). It
+claims to have [many new unique
+features](https://github.com/newren/git-filter-repo#design-rationale-behind-filter-repo-why-create-a-new-tool),
+[good
+performance](https://public-inbox.org/git/CABPp-BGOz8nks0+Tdw5GyGqxeYR-3FF6FT5JcgVqZDYVRQ6qog@mail.gmail.com/),
+and an ability to scale -- from making simple history rewrites
+trivial, to facilitating the creation of entirely new tools which
+leverage existing capabilities to handle more complex cases.
+
+You can read more about [common usecases and base capabilities of
+filter-repo](https://github.com/newren/git-filter-repo/blob/ae43a0ef6d2c7af8f38c5bba38ca0b22942463cf/Documentation/git-filter-repo.txt#L17-L55),
+but in this article, I'd like to focus on two things: providing a simple
+example to give a very brief flavor for git-filter-repo usage, and answer a
+few likely questions about its purpose and rationale (including a short
+comparison to other tools). I will provide several links along the way for
+curious folks to learn more.
+
+### A simple example
+
+Let's start with a simple example that has come up a lot for me:
+extracting a piece of an existing repository and preparing it to be
+merged into some larger monorepository. So, we want to:
+
+* extract the history of a single directory, src/. This means that only
+ paths under src/ remain in the repo, and any commits that only touched
+ paths outside this directory will be removed.
+* rename all files to have a new leading directory, my-module/ (e.g. so that
+ src/foo.c becomes my-module/src/foo.c)
+* rename any tags in the extracted repository to have a 'my-module-'
+ prefix (to avoid any conflicts when we later merge this repo into
+ something else)
+
+Doing this with filter-repo is as simple as the following command:
+```shell
+ git filter-repo --path src/ --to-subdirectory-filter my-module --tag-rename '':'my-module-'
+```
+(the single quotes are unnecessary, but make it clearer to a human that we
+are replacing the empty string as a prefix with `my-module-`)
+
+By contrast, filter-branch comes with a pile of caveats even once you
+figure out the necessary (os-dependent) invocation(s):
+
+```shell
+ git filter-branch --index-filter 'git ls-files | grep -v ^src/ | xargs git rm -q --cached; git ls-files -s | sed "s-$(printf \\t)-&my-module/-" | git update-index --index-info; git ls-files | grep -v ^my-module/ | xargs git rm -q --cached' --tag-name-filter 'echo "my-module-$(cat)"' --prune-empty -- --all
+ git clone file://$(pwd) newcopy
+cd newcopy
+ git for-each-ref --format="delete %(refname)" refs/tags/ | grep -v refs/tags/my-module- | git update-ref --stdin
+ git gc --prune=now
+```
+
+BFG is not capable of this type of rewrite, and this type of rewrite is
+difficult to perform safely using fast-export and fast-import directly.
+
+You can find a lot more examples in [filter-repo's
+manpage](https://github.com/newren/git-filter-repo/blob/ae43a0ef6d2c7af8f38c5bba38ca0b22942463cf/Documentation/git-filter-repo.txt#L434).
+(If you are curious about the "pile of caveats" mentioned above or the
+reasons for the extra steps for filter-branch, you can [read more
+details about this
+example](https://github.com/newren/git-filter-repo#example-usage-comparing-to-filter-branch)).
+
+### Why a new tool instead of contributing to other tools?
+
+There are two well known tools in the repository rewriting space:
+
+*[git-filter-branch](https://git-scm.com/docs/git-filter-branch)
+*[BFG Repo Cleaner](https://rtyley.github.io/bfg-repo-cleaner/)
+
+and two lesser-known tools:
+
+*[reposurgeon](http://www.catb.org/~esr/reposurgeon/reposurgeon.html)
+*[git-fast-export](https://git-scm.com/docs/git-fast-export) and
+[git-fast-import](https://git-scm.com/docs/git-fast-import)
+
+(While fast-export and fast-import themselves are well known, they are
+usually thought of as export-to-another-VCS or import-from-another-VCS
+tools, though they also work for git->git transitions.)
+
+I will briefly discuss each.
+
+#### filter-branch and BFG
+
+It's natural to ask why, if these well-known tools lacked features I
+wanted, they could not have been extended instead of creating a new tool.
+In short, they were philosophically the wrong starting point for extension
+and they also had the wrong architecture or design to support such an
+effort.
+
+From the philosophical angle:
+
+* BFG: easy to use flags for some common cases, but not extensible
+* filter-branch: relatively versatile capability via user-specified
+ shell commands, but rapidly becomes very difficult to use beyond
+ trivial cases especially as usability defaults increasingly
+ conflict and cause problems.
+
+I wanted something that made the easy cases simple like BFG, but which
+would scale up to more difficult cases and have versatility beyond that
+which filter-branch provides.
+
+From the technical architecture/design angle:
+
+* BFG: works on packfiles and packed-refs, directly rewriting tree and
+ blob objects; Roberto proved you can get a lot done with this design
+ with his work on the BFG (as many people who have used his tool can
+ attest), but this design does not permit things like differentiating
+ paths in different directories with the same basename nor could it be
+ used to allow renaming of paths (except within the same directory).
+ Further, this design even sadly runs into a
+[lot](https://github.com/newren/git-filter-repo/blob/ae43a0ef6d2c7af8f38c5bba38ca0b22942463cf/contrib/filter-repo-demos/bfg-ish#L32-L39)
+[of](https://github.com/newren/git-filter-repo/blob/ae43a0ef6d2c7af8f38c5bba38ca0b22942463cf/contrib/filter-repo-demos/bfg-ish#L29-L31)
+[roadblocks](https://github.com/newren/git-filter-repo/blob/ae43a0ef6d2c7af8f38c5bba38ca0b22942463cf/contrib/filter-repo-demos/bfg-ish#L23-L26)
+[and](https://github.com/newren/git-filter-repo/blob/ae43a0ef6d2c7af8f38c5bba38ca0b22942463cf/contrib/filter-repo-demos/bfg-ish#L64-L66)
+[limitations](https://github.com/newren/git-filter-repo/blob/ae43a0ef6d2c7af8f38c5bba38ca0b22942463cf/contrib/filter-repo-demos/bfg-ish#L27-L28)
+ even within its intended usecase of removing big or sensitive content.
+
+* filter-branch: performance really shouldn't matter for a one shot
+ usage tool, but filter-branch can turn a few hour rewrite
+ (allowing an overnight downtime) into an intractable three month
+ wait. Further, its design architecture leaks through every level
+ of the interface, making it nearly impossible to change anything
+ about the slow design without having backward compatibility
+ issues. These issues are well known, but what is less well known
+ is that even ignoring performance, [the usability choices in
+ filter-branch rapidly become increasingly conflicting and
+ problematic](https://github.com/newren/git-filter-repo/blob/ae43a0ef6d2c7af8f38c5bba38ca0b22942463cf/contrib/filter-repo-demos/filter-lamely#L9-L61)
+ for users with larger repos and more involved rewrites,
+ difficulties that again cannot be ameliorated without breaking
+ backward compatibility.
+
+#### reposurgeon
+
+Some brief impressions about reposurgeon:
+
+* Appears to be
+[almost](http://www.catb.org/~esr/reposurgeon/features.html)
+[exclusively](http://www.catb.org/~esr/reposurgeon/dvcs-migration-guide.html)
+ focused on transitioning between different version control systems
+ (cvs, svn, hg, bzr, git, etc.), and in particular handling the myriad
+ edge and corner cases that arise in transitioning from CVS or SVN to a
+ DVCS.
+* Provides very thorough reference-style documentation; if you read all
+ reposurgeon documentation, you will likely feel as though you can take
+ an existing example and modify it in many ways.
+*[Absolutely no full-fledged
+ examples](https://public-inbox.org/git/CAA01Csq0eX2L5cKpjjySs+4e0Sm+vp=10C_SAkE6CLpCHBWZ8g@mail.gmail.com/)
+[or user-guide style
+ documentation](https://public-inbox.org/git/CAA01Csp+RpCXO4noewBOMY6qJiBy=Gshv3rUh83ZY2RJ5Th3Ww@mail.gmail.com/)
+ are provided for getting started.
+* Appears to not have any facilities for quick (in terms of time spent by
+ the user) conversions similar to filter-branch, BFG, or filter-repo.
+ Users who want such capabilities are likely to be frustrated by
+ reposurgeon and give up.
+* Strikes me as "GDB for history rewriting" it has lots of facilities
+ for manually inspecting and editing, but is not intended for the
+ first-time or casual history spelunker. Only those who view history
+ spelunking as a frequent hobby or job are likely to dive in. And it's
+ not quite clear whether it is only useful to those transitioning from
+ CVS/SVN or whether the facilities would also be useful to others.
+* Built on top of fast-export and fast-import, which I contend is the
+ right architecture for a history filtering tool (see below).
+
+I have read the reposurgeon documentation multiple times over the years,
+and am almost at a point where I feel like I know how to get started with
+it. I haven't had a need to convert a CVS or SVN repo in over a decade; if
+I had such a need, perhaps I'd persevere and learn more about it. I
+suspect it has some good ideas I could apply to filter-repo. But I haven't
+managed to get started with reposurgeon, so clearly my impressions of it
+should be taken with a grain of salt.
+
+#### fast-export and fast-import
+
+Finally, fast-export and fast-import can be used with a little editing of
+the fast-export output to handle a number of history rewriting cases. I
+have done this many times, but it has some
+[drawbacks](https://public-inbox.org/git/CABPp-BGL-3_nhZSpt0Bz0EVY-6-mcbgZMmx4YcXEfA_ZrTqFUw@mail.gmail.com/):
+
+* Dangerous for programmatic edits: It's tempting to use sed or perl
+ one-liners to e.g. try to modify filenames, but you risk accidentally
+ also munging unrelated data such as commit messages, file contents, and
+ branch and tag names.
+* Easy to miss corner cases: for example, fast-export only quotes
+ filenames when necessary; as such, your attempt to rename a directory
+ might leave files with spaces or UTF-8 characters in their original
+ location.
+* Difficult to directly provide higher level facilities: for example,
+ rewriting (possibly abbreviated) commit hashes in commit messages to
+ refer to the new commit hashes, or stripping of non-merge commits which
+ become empty or merge commits which become degenerate and empty.
+* Misses a lot of pieces needed to round things out into a usable
+ tool
+
+However, fast-export and fast-import are the right architecture for
+building a repository filtering tool on top of; they are fast, provide
+access to almost all aspects of a repository in a very machine-parseable
+format, and will continue to gain features and capabilities over time
+(e.g. when replace refs were added, fast-export and fast-import immediately
+gained support). To create a full repository surgery tool, you "just" need
+to [combine fast-export and fast-import together with a whole lot of
+parsing and
+glue](https://github.com/newren/git-filter-repo#how-filter-repo-works),
+which, in a nutshell, is what filter-repo is.
+
+#### Upstream improvements
+
+But to circle back to the question of improving existing tools, during the
+development of filter-repo and its predecessor, lots of [improvements to
+both fast-export and
+fast-import](https://github.com/newren/git-filter-repo/tree/develop#upstream-improvements)
+were submitted and included in git.git.
+
+(Also, [filter-repo started in early 2009 as
+git_fast_filter.py](https://public-inbox.org/git/[email protected]/)
+and therefore technically predates both BFG and reposurgeon.)
+
+### Why not a builtin command?
+
+One could ask why this new command is not written in C like most of git.
+While that would have several advantages, it doesn't meet the necessary
+design requirements. See the ["VERSATILITY" section of the
+manpage](https://github.com/newren/git-filter-repo/blob/ae43a0ef6d2c7af8f38c5bba38ca0b22942463cf/Documentation/git-filter-repo.txt#L306-L326)
+or see the "Versatility" section under the [Design Rationale of the
+README](https://github.com/newren/git-filter-repo#design-rationale-behind-filter-repo-why-create-a-new-tool).
+
+Technically, we could perhaps provide a mechanism for people to write
+and compile plugins that a builtin command could load, but having users
+write filtering functions in C sounds suboptimal, and requiring gcc for
+filter-repo sounds more onerous than using python.
+
+### Where to from here?
+
+This was just a quick intro to filter-repo, and I've provided a lot of
+links above if you want to learn more. Just a few more that might be of
+interest:
+
+*[Ramifications of repository
+ rewrites](https://github.com/newren/git-filter-repo/blob/ae43a0ef6d2c7af8f38c5bba38ca0b22942463cf/Documentation/git-filter-repo.txt#L340-L350);
+ including
+[some](https://github.com/newren/git-filter-repo/blob/ae43a0ef6d2c7af8f38c5bba38ca0b22942463cf/Documentation/git-filter-repo.txt#L376-L410)
+[tips](https://github.com/newren/git-filter-repo/blob/ae43a0ef6d2c7af8f38c5bba38ca0b22942463cf/Documentation/git-filter-repo.txt#L426-L431)
+ (not specific to filter-repo)
+*[Finding big objects/directories/extensions (and renames) in your
+ repo](https://github.com/newren/git-filter-repo/blob/ae43a0ef6d2c7af8f38c5bba38ca0b22942463cf/Documentation/git-filter-repo.txt#L356-L361)
+ (can be used together with tools other than filter-repo too)
+*[Creating new history rewriting tools](https://github.com/newren/git-filter-repo/tree/master/contrib/filter-repo-demos)
 
 ## Developer Spotlight: Jean-Noël Avila