Handle large files with GIT

Question:

Scenario

Some time ago, I tried to use GIT to version some backups, mostly small files, git behaved very well versioning them when the changes from one commit to another weren't big, but on a specific server, there were big files binaries, which GIT couldn't handle, didn't even manage to do the initial commit.


Problem

If git didn't behave well with these files (errors were related to memory problems), the real limitations of handling binaries with GIT remain open, it is clear that handling binaries is not the purpose of GIT, but it was not clear enough information I got at that time.


Question

  1. What is the relationship between the limit of a binary file to be viewed in GIT with the processing capacity and machine memory?
  2. Is it safe to keep binaries in GIT, even small versioned in many commits?
  3. What method can we use to optimize GIT so that it behaves better when versioning binaries cannot be avoided?

You can cite solutions like Git Annex or Git Bup, but just as an aid to the answer, it refers to the behavior of pure GIT, without plugins or Forks

Answer:

The primary reason git does not support very large files is that it passes files through xdelta , which usually means that it tries to load the entire contents of the file into memory at once .

If it didn't, you would have to store the entire contents of every revision of every file, even when you only changed a few bytes of that file. This would be terribly inefficient in terms of disk usage, and git is known for its extremely efficient repository format.

You can try tinkering with these server parameters:

[core]
  packedGitLimit = 128m
  packedGitWindowSize = 128m

[pack]
  deltaCacheSize = 128m
  packSizeLimit = 128m
  windowMemory = 128m

I think that git-annex and these types of solutions really are the best because of the way git is built. There's a way around these issues, but you'll have an extremely customized git server and it wouldn't work "right away" in other environments if you need to migrate the server.

Scroll to Top