9

Creating an official Debian mirror with apt-mirror, debmirror or ftpsync

 3 years ago
source link: http://www.linux-admins.net/2013/11/creating-official-debian-mirror-with.html
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
neoserver,ios ssh client

Creating an official Debian mirror with apt-mirror, debmirror or ftpsync

In this article I'll demonstrate 3 ways of creating a local Debian mirror using the apt-mirror, debmirror and the ftpsync tools. I'll be mirroring Debian Wheezy, which will download about 79GB of packages. If your environment consists of many server builds each day, having a local Debian mirror will save you time and bandwidth.

I'll be using a Debian Wheezy server for this purpose.

First install the apt-mirror package and apache:

[root@mirror:~]# apt-get install apt-mirror apache2

The apt-mirror config file is /etc/apt/mirror.list and should contain the following:

[root@mirror:~]# cat /etc/apt/mirror.list deb http://ftp.us.debian.org/debian wheezy main contrib non-free deb-src http://ftp.us.debian.org/debian wheezy main contrib non-free clean http://ftp.us.debian.org/debian

You can put whatever mirror is closer to you and  add or remove different repository sections. In this example I am interested in mirroring the main, contrib and the non-free sections.

Now that the config is there just run:

[root@mirror:~]# apt-mirror /etc/apt/mirror.list Downloading 22 index files using 20 threads... Begin time: Wed Nov  6 15:53:51 2013 [20]... [19]... [18]... [17]... [16]... [15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]... End time: Wed Nov  6 15:53:53 2013

Proceed indexes: [SP] 79.7 GiB will be downloaded into archive. Downloading 89109 archive files using 20 threads... Begin time: Wed Nov  6 15:54:04 2013 [20]...[19]... [18]... [17]... [16]... [15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]... End time: Wed Nov  6 16:38:47 2013

0.0 bytes in 0 files and 0 directories can be freed. Run /var/spool/apt-mirror/var/clean.sh for this purpose.

Running the Post Mirror script ... (/var/spool/apt-mirror/var/postmirror.sh)

Post Mirror script has completed. See above output for any possible errors.

This will have the packages downloaded to

/var/spool/apt-mirror/mirror/ftp.us.debian.org/debian

Let's expose the repo over HTTP:

[root@mirror:~]# ln -s /var/spool/apt-mirror/mirror/ftp.us.debian.org/debian /var/www/debian

And use it:

[root@mirror:~]# cat /etc/apt/sources.list deb http://YOUR_MIRROR_IP/debian/ wheezy main contrib non-free deb-src http://YOUR_MIRROR_IP/debian/ wheezy main contrib non-free

[root@mirror:~]# apt-get update Get:1 http://YOUR_MIRROR_IP wheezy Release.gpg [1,672 B] Get:2 http://YOUR_MIRROR_IP wheezy Release [168 kB] Get:3 http://YOUR_MIRROR_IP wheezy/main Sources [5,959 kB] Get:4 http://YOUR_MIRROR_IP wheezy/contrib Sources [47.8 kB] Get:5 http://YOUR_MIRROR_IP wheezy/non-free Sources [93.4 kB] Get:6 http://YOUR_MIRROR_IP wheezy/main amd64 Packages [5,848 kB] Get:7 http://YOUR_MIRROR_IP wheezy/contrib amd64 Packages [42.0 kB] Get:8 http://YOUR_MIRROR_IP wheezy/non-free amd64 Packages [80.8 kB] Ign http://YOUR_MIRROR_IP wheezy/contrib Translation-en Ign http://YOUR_MIRROR_IP wheezy/main Translation-en Ign http://YOUR_MIRROR_IP wheezy/non-free Translation-en Fetched 12.2 MB in 5s (2,344 kB/s) Reading package lists... Done

[root@mirror:~]# apt-get install iptraf Reading package lists... Done Building dependency tree Reading state information... Done The following NEW packages will be installed: iptraf 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded. Need to get 175 kB of archives. After this operation, 609 kB of additional disk space will be used. Get:1 http://YOUR_MIRROR_IP/debian/ wheezy/main iptraf amd64 3.0.0-8.1 [175 kB] Fetched 175 kB in 0s (5,723 kB/s) Selecting previously unselected package iptraf. (Reading database ... 27685 files and directories currently installed.) Unpacking iptraf (from .../iptraf_3.0.0-8.1_amd64.deb) ... Processing triggers for man-db ... Setting up iptraf (3.0.0-8.1) ...

In some cases you might need to have a stable section in your repo. In this case you can just create a symlink to Wheezy:

[root@mirror:~]# cd /var/spool/apt-mirror/mirror/ftp.us.debian.org/debian/dists [root@mirror:~]# ln -s wheezy/ stable

To keep the repository in sync with the upstream Debian you can create a cronjob that pulls new packages at midnight:

[root@mirror:~]# crontab -l 0 0 * * * /usr/bin/apt-mirror  /etc/apt/mirror.list > /var/spool/apt-mirror/var/cron.log

Alternatively you can mirror any repo e.g. Percona using debmirror with this one liner:

[root@mirror:~]# debmirror -a amd64 -v --ignore-release-gpg --no-source -s main -h repo.percona.com -d wheezy -r /apt --diff=none --progress -e rsync /var/www/mirror/percona

The recommended way of mirroring Debian repository is by using the ftpsync mirroring tool [1].

Let's install it and configure it to mirror wheezy and squeesze:

[root@mirror:~]# adduser --disabled-password archvsync [root@mirror:~]# mkdir -p /srv/mirrors/debian [root@mirror:~]# chown -R archvsync:archvsync /srv/mirrors [root@mirror:~]# su - archvsync [archvsync@mirror:~]# wget http://ftp-master.debian.org/ftpsync.tar.gz [archvsync@mirror:~]# tar zxfv ftpsync.tar.gz [archvsync@mirror:~]# mv distrib/bin/ . [archvsync@mirror:~]# mv distrib/etc/ . [archvsync@mirror:~]# mkdir log [archvsync@mirror:~]# cp etc/ftpsync.conf.sample etc/ftpsync.conf

The config file should at a minimum look like this (if you only want to mirror i386 and amd64 architectures):

[archvsync@mirror:~]# cat etc/ftpsync.conf | grep -vi "^#" | sed '/^$/d'

MIRRORNAME=`hostname -f` TO="/srv/mirrors/debian" RSYNC_PATH="debian" RSYNC_HOST=ftp.us.debian.org LOGDIR="${BASEDIR}/log" EXCLUDE="--exclude=jessie* --exclude=oldstable* --exclude=sid* --exclude=testing* --exclude=unstable* --exclude=experimental* --exclude=Debian* --exclude=stable*" ARCH_EXCLUDE="alpha arm armel armhf hppa hurd-i386 ia64 kfreebsd-amd64 kfreebsd-i386 m68k mipsel mips powerpc s390 s390x sh sparc source"

With all that in place time to rsync the files:

[archvsync@mirror:~]# bin/ftpsync sync:all

Or if you rather run the two rsync stages one at a time, execute:

[archvsync@mirror:~]# bin/ftpsync sync:stage1 [archvsync@mirror:~]# bin/ftpsync sync:stage2

This really just runs these two rsync commands one after the other:

rsync --bwlimit=0 -prltvHSB8192 --timeout 3600 --stats --filter=protect_Archive-Update-in-Progress-ftpsync --filter=protect_project/trace/ftpsync --filter=protect_Archive-Update-Required-ftpsync --exclude=Packages* --exclude=Sources* --exclude=Release* --exclude=InRelease --exclude=i18n/* --exclude=ls-lR* --exclude=jessie* --exclude=oldstable* --exclude=sid* --exclude=testing* --exclude=unstable* --exclude=experimental* --exclude=Debian* --exclude=stable* --exclude=.~tmp~/ ftp.us.debian.org::debian /srv/mirrors/debian rsync --bwlimit=0 -prltvHSB8192 --timeout 3600 --stats --filter=protect_Archive-Update-in-Progress-heroku-test --filter=protect_project/trace/heroku-test --filter=protect_Archive-Update-Required-heroku-test --max-delete=40000 --delay-updates --delete --delete-excluded --delete-delay --exclude=.~tmp~/ ftp.us.debian.org::debian/dists/wheezy /srv/mirrors/debian

To mirror the OpenVZ repo at download.openvz.org first let's find what rsync shares are supported:

[archvsync@mirror:~]# rsync download.openvz.org:: openvz-download OpenVZ download site

After some trial and error the openvz-download is the share that works. The config file should look like so:

[archvsync@mirror:~]# cat etc/ftpsync-OPENVZ.conf | grep -vi "^#" | sed '/^$/d' MIRRORNAME=`hostname -f` TO="/srv/mirrors/openvz" RSYNC_PATH="openvz-download/debian" RSYNC_HOST=download.openvz.org LOGDIR="${BASEDIR}/log" LOG="${LOGDIR}/${NAME}.log"

To rsync the repo run:

[archvsync@mirror:~]# bin/ftpsync sync:archive:OPENVZ

Notice how the config file has the OPENVZ name in it and how we specify which config to run by passing the sync:archive:OPENVZ parameters to the ftpsync script.

Resources:

[1]. http://ftp-master.debian.org/ftpsync.tar.gz

report

Recommend

About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK