Uploading many files to GitHub repository

Uploading many files to GitHub repository

As GitHub introduced unlimited private repositories, I uploaded gigabytes of data to GitHub repository. Here is some technical notes for uploading many files to GitHub repository.
Storage and bandwidth limit of large files
Git Large File Storage (LFS) is required to push a file larger than 100 MB to GitHub repository. There is a storage and bandwidth limit for LFS, and if you exceed the limit you need to buy extra storage and bandwidth.
You may want to use LFS to handle the large files, or you may just want to ignore the large files. Both methods are written in this article.
Create a repository
Sign in to GitHub and create a repository with New reposiroty button (Don’t check Initialize this repository with a README). If there is no REAME.md, create a tentative file with
echo “# test” >> README.md

and run the following commands to initialize the repository and add README.md (Rewrite USER and REP).
git init
git add README.md
git commit -m “First commit”
git remote add origin git@github.com:USER/REP.git
git push -u origin master

Remove spaces from the file names
This command replace spaces " " into underscores “" in the file names under the current directory.
for A in $(find.|grep""|sed-es/""/x3Exe/g);domv"$(echo KaTeX parse error: Double superscript at position 22: …d -e s/x3Exe/' '̲/g)" "(echo $A | sed -e s/x3Exe/’’/g)”; done

If there are spaces in the name of the directory, error may arise. Just repeat this command until no error is shown.
Manage file with LFS
Skip this step when not using LFS. For using LFS, install LFS first. Then setup LFS to Git by
git lfs install

Files larger than 100 MB can be listed with
find . -size +100M | xargs du -sh

Designate the file to manage with LFS from this list. For example, to track files of .psd extention:
git lfs track “*.psd”

Ignore large files
When not using LFS, large files can be ignored with .gitignore file.
To add all the files larger than 100 MB to .gitignore:
find . -size +100M | sed -e ‘s/^.///’ >> .gitignore

Increase the HTTP post buffer size
When pushing large files, error may arise
packet_write_wait: Connection to 192.30.252.123: Broken pipe
fatal: The remote end hung up unexpectedly
error: failed to push some refs to ‘git@github.com:USER/REP.git’
To avoid this, increasing the HTTP post buffer size is recommended. To increase the buffer size to 50 MB,
git config http.postBuffer 52428800

Adding files to repository
Standard way to adding all the files to repository is git add -A; git commit; git push, but it does not succeed when trying to add gigabytes of files; fatal: The remote end hung up unexpectedly error arises even when the HTTP buffer size is increased. Therefore I made the following shell script, gitadd, to add all the files in the current directory step by step.
When you get error by git add -A; git commit; git push, you can reset the commit and index by git reset HEAD~ and run gitadd after that.

将许多文件上传到GitHub存储库
2016年6月6日作者Katsutoshi Seki分类
： 英语 git github
当GitHub引入无限的私有存储库时，我将千兆字节的数据上传到GitHub存储库。这里是一些技术说明，用于将许多文件上传到GitHub存储库。
大文件的存储和带宽限制
需要Git大文件存储（LFS）才能将大于100 MB的文件推送到GitHub存储库。LFS有存储和带宽限制，如果超出限制，则需要购买额外的存储和带宽。
您可能想使用LFS处理大文件，或者只想忽略大文件。两种方法都写在本文中。
创建一个仓库
登录GitHub并使用New reposiroty按钮创建一个存储库（不要检查Initialize this repository with a README）。如果没有REAME.md，请使用以下命令创建一个暂定文件
echo “# test” >> README.md

并运行以下命令来初始化存储库并添加README.md（Rewrite USER and REP）。
git init
git add README.md
git commit -m “First commit”
git remote add origin git@github.com:USER/REP.git
git push -u origin master

从文件名中删除空格
此命令将空格“”替换为当前目录下文件名中的下划线“ ”。
for A in $(find.|grep""|sed-es/""/x3Exe/g);domv"$(echo KaTeX parse error: Double superscript at position 22: …d -e s/x3Exe/' '̲/g)" "(echo $A | sed -e s/x3Exe/’’/g)"; done

如果目录名称中有空格，则可能会出现错误。只需重复此命令，直到没有错误显示。
使用LFS管理文件
不使用LFS时跳过此步骤。要使用LFS，请先安装LFS。然后将LFS设置为Git
git lfs install

大于100 MB的文件可以列出
find . -size +100M | xargs du -sh

从此列表中指定要使用LFS管理的文件。例如，要跟踪.psd扩展名的文件：
git lfs track “*.psd”

忽略大文件
当不使用LFS时，大文件可以被.gitignorefile 忽略。
要将所有大于100 MB的文件添加到.gitignore：
find . -size +100M | sed -e ‘s/^.///’ >> .gitignore

增加HTTP发布缓冲区的大小
推入大文件时，可能会出现错误
packet_write_wait：连接到192.30.252.123：损坏的管道
致命：远程端意外挂起
错误：无法将某些引用推送到’git@github.com：USER / REP.git’
为避免这种情况，建议增加HTTP后缓冲区的大小。要将缓冲区大小增加到50 MB，
git config http.postBuffer 52428800

将文件添加到存储库
将所有文件添加到存储库的标准方法是git add -A; git commit; git push，但尝试添加千兆字节的文件时不会成功；fatal: The remote end hung up unexpectedly即使增加HTTP缓冲区的大小也会出现错误。因此，我制作了以下shell脚本，gitadd以逐步添加当前目录中的所有文件。
当您收到的错误时git add -A; git commit; git push，可以重置提交和索引，git reset HEAD~然后再运行gitadd。