fedora之rpkg，dist-git介绍个使用

https://clime.github.io/2018/09/18/starting-your-own-distro.html

https://pagure.io/rpkg-util

http://copr.fedorainfracloud.org/

https://github.com/release-engineering/dist-git

 

https://fedoraproject.org/wiki/Category:Copr 介绍copy和koji的区别

第一部安装DistGit

First, install the dist-git package:

DistGit（分发Git）是具有附加数据存储的Git。它旨在保留源rpm的内容，并包含以下三个主要组件：

1. Git存储库
2. 后备缓存以存储源tarball 一般是http 
3. 同时管理这两个脚本

在这里阅读有关最新版本的信息：https : //github.com/release-engineering/dist-git/wiki

它是如何工作的

RPM源软件包通常包含一个规范文件和源（上游tarball +其他补丁）。源代码包是二进制文件，并且可能很大，因此不太适合放在Git存储库中。每次更新时，Git都会产生巨大的，毫无意义的差异。这就是为什么引入DistGit的原因，因为它采用了可以存储tarball的高效后备缓存。然后，可以让Git存储库本身做其最擅长的事情：跟踪规格文件，下游补丁和名为的额外文本文件的更改，该文件sources包含指向后备缓存中源tarball的链接。

# dnf install dnf-plugins-core
# dnf copr enable clime/dist-git
# dnf install dist-git

on CentOS7 or RHEL7 with EPEL7 (yum install epel-release) enabled:

# yum install yum-plugin-copr
# yum copr enable clime/dist-git
# yum install dist-git

 其实就是安装一个仓库

[root@localhost yum.repos.d]# cat _copr_clime-dist-git.repo 
[copr:copr.fedorainfracloud.org:clime:dist-git]
name=Copr repo for dist-git owned by clime
baseurl=https://download.copr.fedorainfracloud.org/results/clime/dist-git/epel-7-$basearch/
type=rpm-md
skip_if_unavailable=True
gpgcheck=1
gpgkey=https://download.copr.fedorainfracloud.org/results/clime/dist-git/pubkey.gpg
repo_gpgcheck=0
enabled=1
enabled_metadata=1

配置

There is /etc/httpd/conf.d/dist-git/lookaside-upload.conf.example provided by the dist-git package itself for ssl uploading with authentication by client certificates but we will use something much more simple. Put the following lines into /etc/httpd/conf.d/dist-git/lookaside-upload.conf:

cp lookaside-upload.conf.example  lookaside-upload.conf

[root@koji dist-git]# cat lookaside-upload.conf
<VirtualHost _default_:80>
    # This alias must come before the /repo/ one to avoid being overridden.
    ScriptAlias /repo/pkgs/upload.cgi /var/lib/dist-git/web/upload.cgi

    Alias /repo/ /var/lib/dist-git/cache/lookaside/

    LogLevel trace8

    # provide username manually to upload.cgi
    SetEnv SSL_CLIENT_S_DN_CN joe

    <Location /repo/pkgs/upload.cgi>
        Options +ExecCGI
        Require all granted
    </Location>
</VirtualHost>

 systemctl enable httpd --now

If you hit problems with localhost ssl certs missing on httpd start, move /etc/httpd/conf.d/ssl.conf to /etc/httpd/conf.d/ssl.conf.off and try again.

We will now create two users to carry out all the unprivileged tutorial actions.

 

User joe will be responsible for all client operations (i.e. cloning/pushing) and user admin will be responsible for all server operations (i.e. setting up a new package repo/chilling out). Both joe and admin need to belong to packager group that got created on installation of the dist-git package.

# useradd admin -G packager
# useradd joe -G packager

There is very few things missing to set up the server part at this point. First, start dist-git.socket service so that git:// protocol works for anonymous read-only access (we shall use it later):

# systemctl start dist-git.socket
# systemctl status dist-git.socket  # state should be "active (listening)"

Now we will make sure sshd is up and running which we will be used for authorized Git read/write access.

To install ssh server on Fedora or centos, run:

# yum install openssh-server
# systemctl start sshd
# systemctl status sshd  # state should be "active (running)"

Also, let’s configure public key access to localhost for user joe:

# su joe
joe@localhost / $ cd
joe@localhost ~ $ ssh-keygen  # press enter on everything
joe@localhost ~ $ cat .ssh/id_rsa.pub >> .ssh/authorized_keys
joe@localhost ~ $ chmod 600 .ssh/authorized_keys
joe@localhost ~ $ ssh localhost  # on Fedora, you might need to do rm /run/nologin as root

第二部安装rpkg

Now for the client part, install the rpkg package. We will use the latest package version from copr.fedoraproject.org/clime/rpkg-util project.

On Fedora, you can invoke:

# dnf copr enable clime/rpkg-util
# dnf install rpkg

On EPEL, you can do:

# yum copr enable clime/rpkg-util
# yum install rpkg
# yum install git  # to upgrade git from the enabled copr repo

同理其实也是添加一个库

[root@localhost yum.repos.d]# cat _copr_clime-rpkg-util.repo 
[copr:copr.fedorainfracloud.org:clime:rpkg-util]
name=Copr repo for rpkg-util owned by clime
baseurl=https://download.copr.fedorainfracloud.org/results/clime/rpkg-util/epel-7-$basearch/
type=rpm-md
skip_if_unavailable=True
gpgcheck=1
gpgkey=https://download.copr.fedorainfracloud.org/results/clime/rpkg-util/pubkey.gpg
repo_gpgcheck=0
enabled=1
enabled_metadata=1

Put the following configuration into /etc/rpkg.conf (by replacing the default content): 可以在另一台主机上面操作

[root@koji dist-git]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.10.3.109 koji.superred.com koji
10.10.3.136 kojibuilder01.superred.com kojibuilder01




root@koji:/etc # cat rpkg.conf
[rpkg]

preprocess_spec = True
# auto-packing is deprecated: 自动解包
auto_pack = False
base_output_path = /tmp/rpkg
# module_name = @group/project
#[git]
#lookaside = https://src.fedoraproject.org/repo/pkgs/%(ns1)s/%(name)s/%(filename)s/%(hashtype)s/%(hash)s/%(filename)s
#lookaside_cgi = https://src.fedoraproject.org/repo/pkgs/upload.cgi
#gitbaseurl = ssh://%(user)s@pkgs.fedoraproject.org/%(module)s
#anongiturl = https://src.fedoraproject.org/%(module)s
[git]
lookaside = http://koji.superred.com/repo/pkgs/%(ns1)s/%(name)s/%(filename)s/%(hashtype)s/%(hash)s/%(filename)s
lookaside_cgi = http://koji.superred.com/repo/pkgs/upload.cgi
gitbaseurl = ssh://%(user)s@koji.superred.com/var/lib/dist-git/git/%(module)s #用的本地目录，然后通过cgit去查看
#gitbaseurl = ssh://git@10.10.3.104:2222/%(module)s #用的是gitlab地址
anongiturl = git://koji.superred.com/%(module)s

可以查看配置信息

crudini --format=sh --get /etc/rpkg.conf rpkg | git

 

That’s it! Let’s create our first DistGit repository.

# su admin
admin@localhost / $ /usr/share/dist-git/setup_git_package package  # creates Git repo on the server
Generating initial grok manifest...
Done.
admin@localhost / $ ls /var/lib/dist-git/git/rpms
package.git

Now you can already interact with the created DistGit repository by using rpkg:

# su joe
joe@localhost / $ cd
joe@localhost ~ $ rpkg clone package  # clones the package.git repo
joe@localhost ~ $ cd package
joe@localhost package $ ls
sources



如果上面#gitbaseurl = ssh://git@10.10.3.104:2222/%(module)s #用的是gitlab地址的话
root@koji:~/rpkg # rpkg clone superredsos/centos-indexhtml    #superredsos为gitla的组，centos-indexhtml为组里面的项目，地址头在ssh://git@10.10.3.104:2222/%(module)s

流程如下
root@koji:~/rpkg # rpkg clone superredsos/centos-indexhtml
root@koji:~/rpkg/centos-indexhtml(c7⚡) # git co c7
root@koji:~/rpkg/centos-indexhtml(c7⚡) # into_srpm.sh
root@koji:~/rpkg/centos-indexhtml(c7⚡) # rpkg import SRPMS/centos-indexhtml-7-9.el7.src.rpm



root@koji:/var/lib/dist-git # find ./                                                                                                                                                                                             1 ↵
./
./cache
./cache/lookaside
./cache/lookaside/pkgs
./cache/lookaside/pkgs/superredsos
./cache/lookaside/pkgs/superredsos/centos-indexhtml
./cache/lookaside/pkgs/superredsos/centos-indexhtml/centos-indexhtml-7.tar.gz
./cache/lookaside/pkgs/superredsos/centos-indexhtml/centos-indexhtml-7.tar.gz/sha512
./cache/lookaside/pkgs/superredsos/centos-indexhtml/centos-indexhtml-7.tar.gz/sha512/7c90b8b7b5c3028e5f864ae9b6d5e5357f2c551970c5ee09687a586ee9774ba45aa597f8c0834c5860a797f3a84fd496c180c6900b7368642874addd07653923
./cache/lookaside/pkgs/superredsos/centos-indexhtml/centos-indexhtml-7.tar.gz/sha512/7c90b8b7b5c3028e5f864ae9b6d5e5357f2c551970c5ee09687a586ee9774ba45aa597f8c0834c5860a797f3a84fd496c180c6900b7368642874addd07653923/centos-indexhtml-7.tar.gz
./git
./web
./web/upload.cgi

Now we are in our local cloned Git repository. Let’s initialize it with some public source rpm:

joe@localhost package $ curl https://mirrors.tuna.tsinghua.edu.cn/centos-vault/7.6.1810/os/Source/SPackages/centos-indexhtml-7-9.el7.centos.src.rpm -o /tmp/centos-indexhtml-7-9.el7.centos.src.rpm
joe@localhost package $ rpkg import /tmp/centos-indexhtml-7-9.el7.centos.src.rpm  # unpack src.rpm, upload tarball into dist-git's lookaside and modify local repo accordingly
joe@localhost package $ git status  # display what has been changed in the local git repo
On branch master
Your branch is up-to-date with 'origin/master'.

Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

        modified:   .gitignore
        new file:   prunerepo.spec
        modified:   sources

joe@localhost package $ cat sources  # let's display the pointer to the lookaside cache for the uploaded tarball
SHA512 (prunerepo-1.13.tar.gz) = 25c3f6e42f390e4e2215f0f24fea4a0482ee910ce7fa129c8d91c33bf350d31c564796721437a053ad34bdddb67c36cbb8130b5e54c5bf6af9d68bed0e983244

joe@localhost package $ git config --global user.email "joe@localhost"  # set user git commit info
joe@localhost package $ git config --global user.name "joe"

joe@localhost package $ git commit -m "DistGit test update" -a  # commit changes to the local Git repo
joe@localhost package $ git rev-parse master  # show commit hash, output will differ for you
d8d68e0d8e47455ca686516b45a65e37d752fbbd
joe@localhost package $ rpkg srpm  # build srpm just to test things out
joe@localhost package $ rpkg push  # push local git changes to DistGit
Wrote: /tmp/centos-indexhtm-qzygm0hc/centos-indexhtm.spec
Wrote: /tmp/centos-indexhtm-1-qzygm0hc/centos-indexhtm-1.13-1.fc28.src.rpm
joe@localhost package $ rpkg build  # build package in Copr BuildSystem, this needs copr-cli tool to be installed
...


yum install copr-cli
还要配置http://copr.fedorainfracloud.org/服务

And you can, of course, now clone the repo and start doing something from scratch. Let’s do it with -a switch, which uses the git:// scheme and the read-only git-smart-http Git backend.

joe@localhost package $ cd
joe@localhost ~ $ rpkg clone -a package package-copy
joe@localhost package-copy $ cd package-copy
joe@localhost package-copy $ ls
prunerepo.spec  sources
joe@localhost package-copy $ rpkg sources  # fetch the tarball
Downloading prunerepo-1.13.tar.gz from lookaside cache at localhost
######################################################################## 100.0%
joe@localhost package-copy $ ls
prunerepo-1.13.tar.gz  prunerepo.spec  sources
joe@localhost package-copy $ rpkg srpm  # build srpm again just to see that it works
Wrote: /tmp/rpkg/prunerepo-2-_w0wu16l/prunerepo.spec
Wrote: /tmp/rpkg/prunerepo-2-_w0wu16l/prunerepo-1.13-1.fc27.src.rpm

您基本上可以从这个非常基本的初始设置开始自己的linux发行版。

不用说，到目前为止，我们仅使用传统打包资源（规范+补丁+ tarball）展示了案例，这就是Fedora，CentOS，RHEL，Mageia和其他发行版的工作方式。

此设置（DistGit + rpkg）的有趣之处在于，您也可以在DistGit中解压缩源存储库（spec +原始源文件），这要归功于rpkg实用程序中对此功能的支持。

 

完整流程

Let’s take https://pagure.io/hello_rpkg project, which is raw sources with spec, and import it to our setup here.

# Here we are no longer mentioning commands for switching between users and dirs.

admin@localhost / $ /usr/share/dist-git/setup_git_package hello_rpkg

joe@localhost ~ $ rpkg clone hello_rpkg
joe@localhost ~ $ cd hello_rpkg
joe@localhost hello_rpkg $ git pull https://pagure.io/hello_rpkg --allow-unrelated-histories  # on EPEL, omit --allow-unrelated-histories switch
joe@localhost hello_rpkg $ ls
Makefile  README.md  hello_rpkg.spec.rpkg  main.c  sources

So we have imported code and history from https://pagure.io/hello_rpkg project. There is sources file in addition to the content at https://pagure.io/hello_rpkg, which was created by /usr/share/dist-git/setup_git_package script. We may remove it as we won’t probably be using lookaside cache for this particular project.

joe@localhost hello_rpkg $ rm sources
joe@localhost hello_rpkg $ git commit -a -m 'remove unneeded sources file'

Note: In the current upstream version of dist-git at https://github.com/release-engineering/dist-git, the empty ‘sources’ file is no longer being pregenerated.

And let’s push:

joe@localhost hello_rpkg $ rpkg push

to get the code and history import finished.

Now it is time a play around with the code a little bit. So let’s again try to generate an srpm, this time from unpacked sources:

joe@localhost hello_rpkg $ rpkg srpm
git_dir_pack: packing path /home/joe/hello_rpkg
git_dir_pack: Wrote: /tmp/rpkg/hello_rpkg-1-2oz9vvwk/hello_rpkg-0.0.git.8.1a2615b.tar.gz
Wrote: /tmp/rpkg/hello_rpkg-1-2oz9vvwk/hello_rpkg.spec
Wrote: /tmp/rpkg/hello_rpkg-1-2oz9vvwk/hello_rpkg-0.0.git.8.1a2615b-1.fc28.src.rpm

You can see that it works as well as it worked for the packed case (spec+patches+tarballs). How is that even possible? You will find out when you closer examine the hello_rpkg.spec.rpkg file, which is an rpkg spec file template. Particularly, let’s examine the line defining an rpm source ('Source:'), which is usually a name of tarball stored in the lookaside cache:

joe@localhost hello_rpkg $ grep 'Source:' hello_rpkg.spec.rpkg
Source:     {{{ git_dir_pack }}}

{{{ git_dir_pack }}} is a special rpkg macro, which tells rpkg that the tarball should be dynamically generated from the Git checked-out content. That generated tarball will be then used to build the final srpm. This is different from the standard procedure where the tarball is statically present next to the spec file (even though just as a link to the lookaside until you download it) and can be just used to build an srpm.

This feature of rpkg utility enables you to work with the sources in their unpacked form and only pack them when you need to build them. With this feature, instead of adding patch files and Patch: directives into the spec file, you could just commit the changes without needing to generate patch files at all.

That’s it. This tutorial should give you the basic gist of how it works under the hood in Fedora and similarly, in other rpm distros. All those distros still use just the traditional spec+patches+tarballs approach. So if you use the setup presented here, you are going to be ahead of them as far as Git package maintenance goes.

Anything else?

The DistGit upstream is hosted at https://github.com/release-engineering/dist-git.

The rpkg-util upstream is hosted at https://pagure.io/rpkg-util.

Please, send us patches and requests there.

 

=====================================================================================

 

用户指南 dist-git的web客户端

1.生成并安装软件包：

该项目准备作为RPM软件包构建。您可以使用称为rpkg的工具在启用了EPEL7的Fedora或CentOS上轻松构建它。要构建当前版本，请在repo目录中使用以下命令：

$ rpkg srpm && rpkg local

安装生成的RPM软件包：

# dnf| yum install ./noarch/*

清理：

$ rpkg clean -x

2.配置：

通过使用和修改示例httpd配置来启用后备缓存：

# cd /etc/httpd/conf.d/dist-git/
# cp lookaside-upload.conf.example lookaside-upload.conf
# vim lookaside-upload.conf

后备缓存使用https通信，并且使用ssl客户端证书对客户端进行身份验证。Dist Git服务提供商需要为每个用户颁发客户端证书。

3.用户和组：

所有DistGit用户都需要：

1. 可以通过私钥身份验证访问ssh服务器
2. 在服务器上的打包程序组中
3. 向其提供ssl客户端证书以向后备缓存进行身份验证

4.安装DistGit Web界面：

安装Cgit，

Git的Web界面

 rpkg是cli   是连个工具

# dnf | yum install cgit

并将其指向DistGit存储库：

echo "scan-path=/var/lib/dist-git/git/" >> /etc/cgitrc

注释掉/etc/cgitrc 中的 cache-size条目（或将其设置为零）以在每次页面刷新时始终获取最新的存储库状态很有用。

网络界面将在上提供，例如http://your-server/cgit。

5.系统服务：

# systemctl start sshd
# systemctl start httpd
# systemctl start dist-git.socket

6. DistGit客户端工具：

要与DistGit服务器进行交互，可以使用rpkg或fedpkg命令行工具。

7.部署

您可以在Fedora Infastructure和Copr中看到部署示例。

开发人员指南

单元测试

nosetests -vs .

整合测试

请参阅beaker-tests/README.md。

执照

整个项目使用MIT许可证。文件upload.cgi使用GPLv1。