合并上游主分支
git remote add upstream https://github.com/Intel-bigdata/OAP.git
git fetch upstream
git checkout master(branch-0.6-spark-2.4.x)
git merge upstream/master(branch-0.6-spark-2.4.x)
或强制覆盖
git reset --hard upstream/master
报错了
检查状态
# git status
# On branch Cache-Orc
# All conflicts fixed but you are still merging.
# (use "git commit" to conclude merge)
#
# Changes to be committed:
#
# modified: src/main/java/org/apache/parquet/hadoop/MrOapRecordReader.java
# modified: src/main/java/org/apache/parquet/hadoop/ParquetFiberDataReader.java
# modified: src/main/scala/org/apache/spark/sql/execution/datasources/oap/DataSourceMeta.scala
# modified: src/main/scala/org/apache/spark/sql/execution/datasources/oap/OptimizedOrcFileFormat.scala
# modified: src/main/scala/org/apache/spark/sql/execution/datasources/oap/filecache/FiberSensor.scala
# modified: src/main/scala/org/apache/spark/sql/execution/datasources/oap/index/BitmapReader.scala
# modified: src/main/scala/org/apache/spark/sql/execution/datasources/oap/index/IndexScanner.scala
# modified: src/main/scala/org/apache/spark/sql/execution/datasources/oap/index/indexPlans.scala
# modified: src/main/scala/org/apache/spark/sql/execution/datasources/oap/io/ParquetDataFiberReaderWriter.scala
# modified: src/main/scala/org/apache/spark/sql/internal/oap/OapConf.scala
# modified: src/main/scala/org/apache/spark/sql/oap/OapRuntime.scala
# modified: src/main/scala/org/apache/spark/sql/oap/adapter/SqlConfAdapter.scala
# modified: src/test/scala/org/apache/spark/sql/execution/datasources/oap/DataSourceMetaSuite.scala
# modified: src/test/scala/org/apache/spark/sql/execution/datasources/oap/OptimizedParquetFilterSuite.scala
# modified: src/test/scala/org/apache/spark/sql/execution/datasources/oap/filecache/FiberSensorSuite.scala
# modified: src/test/scala/org/apache/spark/sql/execution/datasources/oap/index/BitMapIndexSuite.scala
先提交
# git commit
[Cache-Orc 083d542] Merge remote-tracking branch 'upstream/master' into Cache-Orc
再检查
# git merge upstream/master
Already up-to-date.
merge origin to upstream
先切换到 upstream 上的分支
git checkout upstream/<upstream-branch>
git merge origin/<origin-branch>
merge upstream and resolve
$ git merge upstream/master
Auto-merging native-sql-engine/cpp/src/benchmarks/CMakeLists.txt
CONFLICT (content): Merge conflict in native-sql-engine/cpp/src/benchmarks/CMakeLists.txt
Auto-merging native-sql-engine/cpp/src/CMakeLists.txt
CONFLICT (content): Merge conflict in native-sql-engine/cpp/src/CMakeLists.txt
Automatic merge failed; fix conflicts and then commit the result.
解决这两个CMakeLists.txt,然后
git add 这两个CMakeLists.txt
git commit
这样即可完成conflict解决
checkout 上游分支
git remote add upstream https://github.com/Intel-bigdata/OAP.git
git fetch upstream
git checkout -b independent-offheapCache upstream/branch-1.0
Force merge
git A分支合并B分支,并强制使用B分支代码
git checkout A
git merge --strategy-option=theirs B
git checkout A
git reset --hard B
git A分支合并B分支,并强制使用A分支代码
git checkout A
git merge --strategy-option=ours B
查看 git 日志
查看指定时间段的日志
# git log --after="2019-1-7"
commit bd67a34aff0efd6e7b83866a7473cba4cbf03513
Author: zhixingheyi-tian <41657774+zhixingheyi-tian@users.noreply.github.com>
Date: Wed Nov 20 13:54:31 2019 +0800
Optimize ORC File reading For ORC columnVector cache (#1103)
commit 49714e5c6c4f130d1efb0ef8f78c6f2ab2a989bd
Author: JiaKe <ke.a.jia@intel.com>
Date: Wed Nov 13 11:37:56 2019 +0800
change the notify threshold from 0 to the dcpmm wait threshold (#1097)
commit 3236f0959d0a8e8dea491fa99e70e2316cd35120
Author: JiaKe <ke.a.jia@intel.com>
Date: Wed Nov 13 09:38:55 2019 +0800
sync fiber cache when the reference count is to 0 and if failed, still keep the origin release code path (#1098)
commit 9a8f415f3350e10124eac3e636a94da64c053a06
Author: zhixingheyi-tian <41657774+zhixingheyi-tian@users.noreply.github.com>
Date: Fri Oct 25 18:14:36 2019 +0800
Update Spark-OAP-Integration-Guide.md
git log 寻找某段时间的 Author
# git log --after="2019-1-7" | grep "Author"
Author: zhixingheyi-tian <41657774+zhixingheyi-tian@users.noreply.github.com>
Author: JiaKe <ke.a.jia@intel.com>
Author: JiaKe <ke.a.jia@intel.com>
Author: zhixingheyi-tian <41657774+zhixingheyi-tian@users.noreply.github.com>
Author: zhixingheyi-tian <41657774+zhixingheyi-tian@users.noreply.github.com>
Author: zhixingheyi-tian <41657774+zhixingheyi-tian@users.noreply.github.com>
Author: zhixingheyi-tian <1564481943@qq.com>
Author: shaowenzh <zshaowen@126.com>
Author: zhixingheyi-tian <41657774+zhixingheyi-tian@users.noreply.github.com>
Author: shaowenzh <zshaowen@126.com>
Author: shaowenzh <zshaowen@126.com>
Author: Greg Lee <lihaowhu@gmail.com>
Author: YangJie <yangjie01@baidu.com>
Author: shaowenzh <zshaowen@126.com>
Author: JiaKe <ke.a.jia@intel.com>
Author: shaowenzh <zshaowen@126.com>
Author: shaowenzh <shaowen.zhang@intel.com>
Author: zhixingheyi-tian <41657774+zhixingheyi-tian@users.noreply.github.com>
Author: Greg Lee <lihaowhu@gmail.com>
Author: YangJie <yangjie01@baidu.com>
Author: YangJie <yangjie01@baidu.com>
Author: YangJie <yangjie01@baidu.com>
Author: Greg Lee <lihaowhu@gmail.com>
Author: Greg Lee <lihaowhu@gmail.com>
Author: WangGuangxin <wangguangxin.cn@gmail.com>
Author: YangJie <yangjie01@baidu.com>
Author: DuRipeng <453243496@qq.com>
Author: zhixingheyi-tian <41657774+zhixingheyi-tian@users.noreply.github.com>
Author: YangJie <yangjie01@baidu.com>
Author: YangJie <yangjie01@baidu.com>
Author: zhixingheyi-tian <41657774+zhixingheyi-tian@users.noreply.github.com>
Author: YangJie <yangjie01@baidu.com>
Author: YangJie <yangjie01@baidu.com>
Author: YangJie <yangjie01@baidu.com>
git log 安装author 进行过滤
git log --author="xxx"
git log --grep
注意,需要先切换到 包含 该message 的 分支上
$ git checkout master
Switched to branch 'master'
Your branch is up to date with 'origin/master'.
$ git log --grep=1319
commit 1c7c40bf8963e8be9b400560f26a5d4b8a320fee
Author: RickyMa <rickyma@tencent.com>
Date: Wed Dec 6 21:12:07 2023 +0800
[#1319] fix(server): Add shaded com.google.guava:failureaccess dependency to prevent NoClassDefFoundError (#1352)
### What changes were proposed in this pull request?
Add shaded com.google.guava:failureaccess dependency in server pom.xml to prevent NoClassDefFoundError
### Why are the changes needed?
For [#1319](https://github.com/apache/incubator-uniffle/issues/1319)
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
By running server successfully.
其他时间操作命令
git log --since=2.weeks
安装 Author 分类查找
git shortlog 把输出按作者名字排序,
传入 -n 选项来按每个作者提交数量排序。
# git shortlog -n --after="2019-1-7"
YangJie (10):
Refactor SharedOapContextBase Inheritance hierarchy (#994)
[OapExtensions] Inject OapFileSourceStrategy to SparkSessionExtensions (#995)
[OapExtensions] Inject OapStrategies to SparkSessionExtensions #999 (#1001)
Use WriteJobStatsTracker to pass IndexBuildResults (#1004)
[FOLLOWUP]Move OapSparkSqlParser to oap package #1005
Avoid possible overflow (#1028)
Fix query failed when Some index file not exists(#1031) (#1032)
Remove redundant configuration from OapConf (#1042)
Add a constant timestamps config for create oindex (#1039)
Use a sparksession to init conf of OapSparkSqlParser (#1065)
zhixingheyi-tian (9):
Inject OapSparkSqlParser by SparkSessionExtension (#1005)
Fix errors reported by Coverity (#1020)
ORC cache Support (#1017)
Add a simple feature of caching hot tables for OAP 0.6 (#1081)
add Integration-Guide
Update Spark-OAP-Integration-Guide.md
Update Spark-OAP-Integration-Guide.md
Update Spark-OAP-Integration-Guide.md
Optimize ORC File reading For ORC columnVector cache (#1103)
git patch
git diff
# 新的 commitid 在后
git diff commit-id-1 commit-id-2 > diff.patch
- 注意: 新的 commitid 在后
git format-patch
两个点之间不能有空格
git format-patch a71b0bdbe1eea4ed4dec7d260afd1d4c7044dd8f..f488a0b7a35d5433434f4644dffb990d057a5076
git apply patch
正常情况打patch 的步骤
# git patch
--stat instead of applying the patch, output diffstat for the input
--check instead of applying the patch, see if the patch is applicable
git apply --stat ../hashjoin-revert.patch
git apply --check ../2.3.2.numa.patch
git apply ../2.3.2.numa.patch
修复空格
git apply --whitespace=fix your.patch
git apply --reject
能合进的patch 自然能合进,不能合进的patch 会生成 *.rej 文件。
先检查
# git apply --check 0004-introduce-ParquetFileReader-from-parquet-1.8.3-1083.patch
error: patch failed: pom.xml:645
error: pom.xml: patch does not apply
然后合并
# git apply --reject 0004-introduce-ParquetFileReader-from-parquet-1.8.3-1083.patch
0004-introduce-ParquetFileReader-from-parquet-1.8.3-1083.patch:31: trailing whitespace.
/*
0004-introduce-ParquetFileReader-from-parquet-1.8.3-1083.patch:39: trailing whitespace.
*
0004-introduce-ParquetFileReader-from-parquet-1.8.3-1083.patch:41: trailing whitespace.
*
Checking patch pom.xml...
error: while searching for:
<configuration>
<sources>
<source>src/main/spark2.3.2/java</source>
<source>src/main/spark2.3.2/scala</source>
</sources>
</configuration>
error: patch failed: pom.xml:645
Checking patch src/main/parquet-1.8.3/java/org/apache/parquet/hadoop/ParquetFileReader.java...
Applying patch pom.xml with 1 reject...
Rejected hunk #1.
Applied patch src/main/parquet-1.8.3/java/org/apache/parquet/hadoop/ParquetFileReader.java cleanly.
./pom.xml.rej
每个不能完全 merge 的文件,都会生成一个 带有 rej 后缀的文件。 后续可以观测,进行手动 merge
git 切换主分支
在当前分支进行重命名
然后push
git branch -m branch-1.0-1212 master-20191212
git push -f origin master-20191212
删除branch
# 删除本地branch
git branch -d branch-0.6-spark-3.0.0-preview
# 删除远程branch
git push origin --delete branch-0.7
删除tag
查看所有tag
git tag
删除某个tag
git tag -d v0.6.0-rc1
git push origin :refs/tags/v0.6.0-rc1
git 撤销
git add 之前
git checkout -- <file>
//或者使用下面的指令来放弃所有的更改
git checkout .
注意:新建的文件此时对git而言是未知的,直接手动删除即可。
git add 之后
git status 先看一下add 中的文件
git reset HEAD 如果后面什么都没有, 就是上一次add 里面的全部撤销了
git reset HEAD XXX.java 就是对某个java文件进行撤销了
git reset HEAD file 即对file文件夹进行撤销
撤销commit
commit id 为需要回退到的那个版本 (想变成的那个版本)
git reset --hard 034f08adbc488b8a2720c99435e3c858d78b8b38
git push origin orc-bianry-cache --force
# Changes to be committed:
#
# modified: cpp/src/arrow/dataset/file_orc_test.cc
可以使用: git reset HEAD cpp/src/arrow/dataset/file_orc_test.cc
进行撤销
# Unmerged paths:
# (use "git add <file>..." to mark resolution)
#
# both modified: cpp/src/arrow/adapters/orc/adapter.cc
# both modified: cpp/src/arrow/adapters/orc/adapter.h
# both modified: python/pyarrow/_orc.pxd
# both modified: python/pyarrow/_orc.pyx
# both modified: python/pyarrow/orc.py
可执行以下命令撤销```
git reset cpp/src/arrow/adapters/orc/adapter.cc
git checkout cpp/src/arrow/adapters/orc/adapter.cc
Undo cherry-pick
# git cherry-pick 7c819571baa8bea7baa79bb47303d2f7d23b7dc2
error: could not apply 7c81957... ARROW-13797: [C++][Python] Column projection pushdown for ORC dataset reading + use liborc for column selection
hint: after resolving the conflicts, mark the corrected paths
hint: with 'git add <paths>' or 'git rm <paths>'
hint: and commit the result with 'git commit'
[root@vsr532 arrow-test-cheery]# git status
# On branch orcbackport
# You are currently cherry-picking.
# (fix conflicts and run "git commit")
#
# Changes to be committed:
#
# modified: cpp/src/arrow/dataset/file_orc.cc
# modified: cpp/src/arrow/dataset/file_orc_test.cc
# modified: python/pyarrow/tests/test_dataset.py
# modified: python/pyarrow/tests/test_orc.py
#
# Unmerged paths:
# (use "git add <file>..." to mark resolution)
#
# both modified: cpp/src/arrow/adapters/orc/adapter.cc
# both modified: cpp/src/arrow/adapters/orc/adapter.h
# both modified: python/pyarrow/_orc.pxd
# both modified: python/pyarrow/_orc.pyx
# both modified: python/pyarrow/orc.py
git cherry-pick --abort
一键撤销
git cherry-pick
合并某个commit 到指定的branch
#6bbf6b4为非当前branch上的commitId
git cherry-pick 6bbf6b4
git强制覆盖本地文件(与git远程仓库保持一致)
git fetch --all
git reset --hard origin/master
git pull
git fetch是将远程主机的最新内容拉到本地,用户在检查了以后决定是否合并到工作本机分支中。
而git pull 则是将远程主机的最新内容拉下来后直接合并,即:git pull = git fetch + git merge,
HEAD
HEAD本质上仅仅是个指向 commit 对象的可变指针。git 是如何知道你当前在哪个分支上工作的呢?
其实它保存着一个名为 HEAD 的特别指针。在 git 中,它是一个指向你正在工作中的本地分支的指针,可以将 HEAD 想象为当前分支的别名。
清理 Untracked files
# git status
# On branch master
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
#
# src/main/java/org/apache/spark/unsafe/NativeLoader.java
# src/main/java/org/apache/spark/unsafe/VMEMCacheJNI.java
# src/main/native/vmemcache/
nothing added to commit but untracked files present (use "git add" to track)
git clean
usage: git clean [-d] [-f] [-n] [-q] [-e <pattern>] [-x | -X] [--] <paths>...
-q, --quiet do not print names of files removed
-n, --dry-run dry run
-f, --force force
-d remove whole directories
-e, --exclude <pattern>
add <pattern> to ignore rules
-x remove ignored files, too
-X remove only ignored files
# git clean -fd
Removing src/main/java/org/apache/spark/unsafe/NativeLoader.java
Removing src/main/java/org/apache/spark/unsafe/VMEMCacheJNI.java
Removing src/main/native/vmemcache/
# git status
# On branch master
nothing to commit, working directory clean
直接抓取某个 PR
直接抓取进当前branch
git clone https://github.com/Intel-bigdata/OAP.git
cd OAP
git pull origin pull/1073/head
抓取进新建的branch
cd OAP
git fetch origin pull/1075/head:pull_1075
git checkout pull_1075
rebase upstream
git remote add upstream https://github.com/apache/arrow.git
git fetch upstream
git checkout master
git rebase upstream/master
clone specific tag
clone some tag
git clone -b v2.2.0 https://github.com/IntelAI/models.git
Cases
git fetch upstream
# 先备份
git checkout -b orcJNI-1105 orcJNI
git checkout orcJNI
# 回到一个不容易有分歧的commit id
# 是为了下一步的准确merge服务
git reset --hard a2c70e8b2d923fde4310b2ce4e253ab3b75a95e3
# 同步主分支
git merge upstream/arrow-4.0.0-oap
# 从备份分支,获取需要 submit 的 commit id
git cherry-pick d2a4b1c4050a89233c3187536e94ad413ad4fa76
# 强制 push
git push origin orcJNI --force
其他问题也可以参照 rebase 方式解决
refer to https://github.com/edx/edx-platform/wiki/How-to-Rebase-a-Pull-Request
git statsh
# git merge upstream/master
Updating 10ce3ca..190dff4
error: Your local changes to the following files would be overwritten by merge:
native-sql-engine/core/src/main/scala/com/intel/oap/extension/ColumnarOverrides.scala
Please, commit your changes or stash them before you can merge.
Aborting
# git stash
# git merge upstream/master
Updating 10ce3ca..190dff4
Fast-forward
docs/Installation.md | 8 +-
docs/Prerequisite.md | 2 +-
docs/SparkInstallation.md | 10 +-
docs/User-Guide.md | 3 +-
docs/limitations.md | 2 +-
docs/memory.md | 2 +-
docs/performance.md | 2 +-
native-sql-engine/core/src/main/java/com/intel/oap/vectorized/JniUtils.java | 14 +-
.../core/src/main/scala/com/intel/oap/execution/ColumnarBatchScanExec.scala | 78 +++
.../core/src/main/scala/com/intel/oap/extension/ColumnarOverrides.scala | 23 +-
.../core/src/main/scala/com/intel/oap/extension/columnar/ColumnarGuardRule.scala | 22 +-
.../main/scala/org/apache/spark/sql/execution/ColumnarCollapseCodegenStages.scala | 24 +-
.../scala/org/apache/spark/sql/execution/python/ColumnarArrowPythonRunner.scala | 1 -
.../scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala | 5 +-
shims/spark311/src/main/scala/com/intel/oap/execution/ColumnarBatchScanExec.scala | 52 --
.../src/main/scala/com/intel/oap/execution/ColumnarBatchScanExecBase.scala | 30 +
shims/spark311/src/main/scala/com/intel/oap/sql/shims/spark311/Spark311Shims.scala | 1 -
shims/spark321/src/main/scala/com/intel/oap/execution/ColumnarBatchScanExec.scala | 53 --
.../src/main/scala/com/intel/oap/execution/ColumnarBatchScanExecBase.scala | 31 +
shims/spark321/src/main/scala/com/intel/oap/sql/shims/spark321/Spark321Shims.scala | 1 -
tools/README.md | 9 +
tools/gazelle_analysis.ipynb | 1308 +++++++++-----------------------------
tools/sparklog.ipynb | 415 +++++++++---
23 files changed, 816 insertions(+), 1280 deletions(-)
create mode 100644 native-sql-engine/core/src/main/scala/com/intel/oap/execution/ColumnarBatchScanExec.scala
delete mode 100644 shims/spark311/src/main/scala/com/intel/oap/execution/ColumnarBatchScanExec.scala
create mode 100644 shims/spark311/src/main/scala/com/intel/oap/execution/ColumnarBatchScanExecBase.scala
delete mode 100644 shims/spark321/src/main/scala/com/intel/oap/execution/ColumnarBatchScanExec.scala
create mode 100644 shims/spark321/src/main/scala/com/intel/oap/execution/ColumnarBatchScanExecBase.scala
# git stash pop
Auto-merging native-sql-engine/core/src/main/scala/com/intel/oap/extension/ColumnarOverrides.scala
# On branch nativecolumnarbatch
# Changes not staged for commit:
# (use "git add <file>..." to update what will be committed)
# (use "git checkout -- <file>..." to discard changes in working directory)
#
# modified: native-sql-engine/core/src/main/scala/com/intel/oap/GazellePluginConfig.scala
# modified: native-sql-engine/core/src/main/scala/com/intel/oap/extension/ColumnarOverrides.scala
# modified: native-sql-engine/cpp/src/jni/jni_wrapper.cc
#
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
#
# native-sql-engine/core/src/main/java/com/intel/oap/vectorized/ArrowCoalesceBatchesJniWrapper.java
# native-sql-engine/core/src/main/scala/com/intel/oap/execution/ArrowCoalesceBatchesExec.scala
# native-sql-engine/core/src/test/scala/org/apache/spark/shuffle/ArrowCoalesceBatchesSuite.scala
# native-sql-engine/core/src/test/scala/org/apache/spark/shuffle/HashjoinSuite.scala
# git stash drop
No stash found.
其他操作可参考 https://www.cnblogs.com/yw0219/p/6020878.html
git submodule
example:
https://github.com/apache/arrow/blob/release-4.0.0/docs/source/developers/cpp/development.rst
The unit tests depend on an environment variable PARQUET_TEST_DATA that depends on a git submodule to the repository https://github.com/apache/parquet-testing:
git submodule update --init
export PARQUET_TEST_DATA=$ARROW_ROOT/cpp/submodules/parquet-testing/data
# cat .gitmodules
[submodule "cpp/submodules/parquet-testing"]
path = cpp/submodules/parquet-testing
url = https://github.com/apache/parquet-testing.git
[submodule "testing"]
path = testing
url = https://github.com/apache/arrow-testing
Git 相关配置
这个是使用nano进行编辑提交的页面,退出方法为:
Ctrl + X然后输入y再然后回车,就可以退出了
或者将默认编辑器换成VIM:
git config --global core.editor “vim”
然后.gitconfig 里自动填写配置
# cat /root/.gitconfig
[core]
editor = vim
使用其他branch 的 file
git checkout <other-branch-name> -- path/to/your/folder
git checkout upstream/master -- ./cpp/src/parquet/arrow/reader_internal.cc
# Changes to be committed:
# (use "git reset HEAD <file>..." to unstage)
#
# modified: ../src/parquet/arrow/reader_internal.cc
#
push 到 github 项目
git remote add origin
示例如下:
git remote add zhixing https://github.com/xxx.git
git checkout branch-1.5
git push -u zhixing branch-1.5
这种方式需要先 git checkout 到本地分支,否则会报以下错误
error: src refspec branch-1.5 does not match any
git push zhixing --all
–all 命令,可以将本地所有分支一起push 上去。
修改默认分支的的页面如下:
mirror
参考链接:
- https://blog.csdn.net/azxcv12345err/article/details/137148858
git clone --mirror https://github.com/xxx.git
cd xxx.git
git remote add zhixing https://github.com/xxx.git
git push zhixing --all
git push zhixing --tags
git clone --bare https://github.com/xxx.git
cd xxx.git
git push --mirror https://github.com/newxxx.git
Two-factor methods
验证路径
使用手机端 Authenticator app ,直接进行扫描绑定
How to Revert a Single File
All that is left is to revert the file. Once you’ve opened a terminal and changed to the working directory, you use git checkout to revert the file. The format of the git command will look like this:
git checkout [commit ID] -- path/to/file
迁移仓库
参考链接:
- https://docs.github.com/zh/repositories/creating-and-managing-repositories/transferring-a-repository
自动重定向
“Repository name”(仓库名称)旁边的 “Rename”(重命名)按钮,重命名仓库后,GitHub 将自动为你设置 URL 重定向。旧的仓库 URL 将自动重定向到新的仓库 URL。
对于仓库迁移,当你将仓库从一个账户或组织迁移到另一个账户或组织时,GitHub 会自动处理重定向。这意味着旧的仓库 URL 将自动重定向到新的仓库 URL,以便用户可以无缝地访问和引用仓库。
FAQ
出现以下错误,可以将 Repo 设置成 private,即可解决。
This repository does not have Secret Scanning enabled, but is eligible. Enable Secret Scanning to view and manage detected secrets.
remote: Visit the repository settings page,