Creating Local Mirrors for Updates or Installs

If you have a lot of systems you may want to consider setting up a local mirror for updates or network installs. Lets assume we want to set up the share for CentOS 5.8 as /share/CentOS/5/... The $releasever variable in yum repo definitions always maps to the current major release number, in this case "5", that is typically a symbolic link pointing to the current minor release directory tree, now "5.8".

On the server machine become root and create the share:


mkdir -p /share/CentOS/5.8
cd /share/CentOS
ln -s 5.8 5

Adjust the path to a filesystem that has enough space, or mount a new disk/partition under /share or your preferred mountpoint. On my current system with a full mirror including ISO p_w_picpaths and some local packages (see HowTos/CreateLocalRepos), this requires ~28GB.

Find a spacer.gifCurrent Mirror near you that supports rsync, set up a directory structure already populated with content from the 5.8 DVDs or CDs. Can add ISO p_w_picpaths and any updates you already have downloaded.

The structure should look like this (or you may choose to only use a subset, such as "os" just for installs):


addons:
i386  x86_64
centosplus:
i386  x86_64
contrib:
i386  x86_64
cr:
i386  x86_64
extras:
i386  x86_64
fasttrack:
i386  x86_64
isos:
i386  x86_64
os:
i386  x86_64
updates:
i386  x86_64

(The above is current for 5.8.)

Populate according to your preferred architecture, or do both i386 and x86_64:


mkdir /tmp/mnt
mount -ro loop /share/CentOS/5.8/isos/x86_64/CentOS-5.8-x86_64-bin-DVD-1of2.iso /tmp/mnt
rsync -avHPS /tmp/mnt/ /share/CentOS/5.8/os/x86_64/
umount /tmp/mnt
mount -ro loop /share/CentOS/5.8/isos/x86_64/CentOS-5.8-x86_64-bin-DVD-2of2.iso /tmp/mnt
rsync -avHPS /tmp/mnt/ /share/CentOS/5.8/os/x86_64/
umount /tmp/mnt

If using CD ISOs then copy the contents of all CDs to the same directories for each architecture, overwriting files of the same names.

Then for CentOS 5.8 use a script like:


#!/bin/bash
if [ -d /share/CentOS/5.8 ] ; then
    rsync  -avSHP --delete --exclude "local*" --exclude "isos" nearby.rsync.centos.net::CentOS/5.8/ /share/CentOS/5.8/
else
    echo "Target directory /share/CentOS/5.8 not present."
fi

This can be run in cron to keep current. If you only want i386 or x86_64, or want to exclude things like [extras] or [centosplus] adapt as required. If running in cron one should probably be a bit more robust and use a lock file to assure that a job is not already running.


#!/bin/bash
if [ -f /var/lock/subsys/rsync_updates ]; then
    echo "Updates via rsync already running."
    exit 0
fi
if [ -d /share/CentOS/5.8 ] ; then
    touch /var/lock/subsys/rsync_updates
    rsync  -avSHP --delete --exclude "local*" --exclude "isos" nearby.rsync.centos.net::CentOS/5.8/ /share/CentOS/5.8/
    /bin/rm -f /var/lock/subsys/rsync_updates
else
    echo "Target directory /share/CentOS/5.8 not present."
fi

Create a link pointing to the current release, if it has changed.


cd /share/CentOS/
ln -fs 5.8 5

On the server you can use NFS to export the directory. NFS has the advantage over HTTP or FTP that updates will be used "in place" rather than copies to the cache being made by yum.


cat >> /etc/exports
/share  192.168.1.0/24(rw,mountpoint)

Substitute your local subnet or an explicit list of IP addresses.

If NFS is not already enabled:


exportfs -a
chkconfig nfs on
service nfs start

otherwise just


exportfs -r

Then on clients mount the share. Can use an entry in /etc/fstab like:


myserver.my.net:/share   /share   nfs   rw  0 0

Or if using autofs:


cd /
ln -s net/myserver/share

Set up /etc/yum.repos.d/CentOS-Base.repo like:


[base]
name=CentOS-$releasever - Base
baseurl=file:/share/CentOS/$releasever/os/$basearch/
gpgcheck=1
gpgkey=file:/etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5
protect=1
priority=1
enabled=1
[updates]
name=CentOS-$releasever - Updates
baseurl=file:/share/CentOS/$releasever/updates/$basearch/
gpgcheck=1
gpgkey=file:/etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5
protect=1
priority=1
enabled=1
[extras]
name=CentOS-$releasever - Extras
baseurl=file:/share/CentOS/$releasever/extras/$basearch/
gpgcheck=1
gpgkey=file:/etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5
protect=1
priority=1
enabled=1
[centosplus]
name=CentOS-$releasever - Plus
baseurl=file:/share/CentOS/$releasever/centosplus/$basearch/
exclude=kernel*
gpgcheck=1
enabled=1
gpgkey=file:/etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5
protect=0
priority=1
[contrib]
name=CentOS-$releasever - Contrib
baseurl=file:/share/CentOS/$releasever/contrib/$basearch/
gpgcheck=1
enabled=0
gpgkey=file:/etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5
protect=0
priority=3

Be sure to set which repos are enabled and setup protectbase or priorities as you require for your needs.

Alternatively - use HTTP or FTP rather than, or in addition to, NFS to share the updates - left as an exercise for the reader, but some help for HTTP can be found here.


Variations on the Theme

If you prefer to maintain your own subset of locally tested updates, rather than just mirroring the released updates, you will need to generate the metadata for yum. Populate the updates directories as desired then run


createrepo /share/CentOS/5/updates/i386
createrepo /share/CentOS/5/updates/x86_64

If createrepo is not found (it is not installed by default)


yum install createrepo



  • If you have the bandwidth available, please consider becoming a public mirror.



  • Alternative approaches would be to use Dag Wieers' mrepo, lftp, or reposync from the yum-utils package that's hosted in the [os] repository.

Quoting Karanbir Singh from a centos-docs post:

  • reposync works for centos-3/4/5 and since it uses the internal yum setup, you can make it do quite nice things like share a cache for X local install setups and also use it to keep repo's in real sync without needing to get every package that is hosted in the repo. Also, works well for include / exclude segments so you can easily (and in a more yum friendly manner) manage the package set. With a bit more creativity, its possible to use that and squid to setup an avahi based lan area zeroconf yum mirror with failback to remote urls if required.

And Akemi Yagi on lftp:

  • You might want to add 'lftp' as an alternative method to rsync. I have been using lftp because rsync at work is capped at a miserably low speed. Also, there are more http / ftp sites available than rsync sites. It is also a one-liner. For example:


lftp -e 'open http://<some.site>/centos/ && mirror -c --delete  5.8 && exit'
for example:
lftp  -e "mirror --delete --only-newer --verbose 4.7.0/ bak/" http://archive.cloudera.com/cdh4/redhat/6/x86_64/cdh/
as such: This will mirror site's 4.7.0 to /bak,local direcotry structure is bak/4.7.0/

  • will mirror the whole 5.8 under the remote centos/ directory. The major shortcoming of lftp is that it is not as robust about preserving existing files and will often re-download perfectly good files if things are not exactly right with the time stamp, but ignoring time may prevent downloads of things like repodata that have the same names but new content and time stamps.


Migrating to a new Point Release

A new point release typically consists or a new [base] repo containing the latest files from the previous release with released updates rolled in plus a large set of new updates. To prepare for 5.9


cd /share/CentOS
cp -al 5.8 5.9
mv 5.9/updates/i386/RPMS/* 5.9/os/i386/CentOS
mv 5.9/updates/x86_64/RPMS/* 5.9/os/x86_64/CentOS

When 5.9 is released and the mirrors are updated, then change the mirror scripts to reflect the 5.8 to 5.9 transition and sync with the mirrors. Then


cd /share/CentOS
/bin/rm -f 5
/bin/ln -s 5.9 5



  • This page created and maintained by PhilSchaffner. Other Wiki contributors are invited to make corrections, additions, or modifications.