generated from smedev/Template-for-SMEServer-Core-upstream
tidy
This commit is contained in:
parent
3ded4d19c9
commit
d0022253cc
@ -1,55 +0,0 @@
|
||||
%define name ${REPO_NAME}
|
||||
%define version 1.0
|
||||
%define release 1
|
||||
Summary: This is what ${REPO_NAME} does.
|
||||
Name: %{name}
|
||||
Version: %{version}
|
||||
Release: %{release}%{?dist}
|
||||
Source: %{name}-%{version}.tar.gz
|
||||
License: GNU GPL version 2
|
||||
Group: SMEserver/addon
|
||||
BuildRoot: %{_tmppath}/%{name}-buildroot
|
||||
Prefix: %{_prefix}
|
||||
BuildArchitectures: noarch
|
||||
BuildRequires: e-smith-devtools
|
||||
Requires: e-smith-release >= 10.0
|
||||
AutoReqProv: no
|
||||
|
||||
%description
|
||||
${REPO_DESCRIPTION}
|
||||
|
||||
%changelog
|
||||
* Day MMMM DD YYYY <brianr@koozali.org> 1.0-1.sme
|
||||
- Initial code - create RPM [SME:99999]
|
||||
|
||||
%prep
|
||||
|
||||
%setup -q
|
||||
|
||||
%build
|
||||
perl createlinks
|
||||
|
||||
%install
|
||||
rm -rf $$RPM_BUILD_ROOT
|
||||
(cd root ; find . -depth -print | cpio -dump $$RPM_BUILD_ROOT)
|
||||
rm -f %{name}-%{version}-filelist
|
||||
/sbin/e-smith/genfilelist $$RPM_BUILD_ROOT \
|
||||
> %{name}-%{version}-filelist
|
||||
#echo "%doc COPYING" >> %{name}-%{version}-filelist
|
||||
#--dir <dir> 'attr(755,user,grp)' \
|
||||
#--file <file> 'attr(755,root,root)' \
|
||||
|
||||
%clean
|
||||
cd ..
|
||||
rm -rf %{name}-%{version}
|
||||
|
||||
%pre
|
||||
|
||||
%preun
|
||||
|
||||
%post
|
||||
|
||||
%postun
|
||||
#uninstall
|
||||
%files -f %{name}-%{version}-filelist
|
||||
%defattr(-,root,root)
|
||||
266
README.md
266
README.md
@ -5,3 +5,269 @@ SME build of the debian ftpsync scrip found at https://salsa.debian.org/mirror-t
|
||||
Two main scripts are :
|
||||
- ftpsync : Used to sync an archive using rsync
|
||||
- runmirrors : Used to notify leaf nodes of available updates
|
||||
|
||||
# Archvsync
|
||||
|
||||
This is the central repository for the Debian mirror scripts. The scripts
|
||||
in this repository are written for the purposes of maintaining a Debian
|
||||
archive mirror (and shortly, a Debian bug mirror), but they should be
|
||||
easily generalizable.
|
||||
|
||||
|
||||
Currently the following scripts are available:
|
||||
|
||||
* ftpsync - Used to sync an archive using rsync
|
||||
* runmirrors - Used to notify leaf nodes of available updates
|
||||
|
||||
## Usage
|
||||
|
||||
For impatient people, short usage instruction:
|
||||
|
||||
* Create a dedicated user for the whole mirror.
|
||||
* Create a seperate directory for the mirror, writeable by the new user.
|
||||
* Place the ftpsync script in the mirror user's $HOME/bin (or just $HOME)
|
||||
* Place the ftpsync.conf.sample into $HOME/etc as ftpsync.conf and edit
|
||||
it to suit your system. You should at the very least change the TO=
|
||||
and RSYNC_HOST lines.
|
||||
* Create $HOME/log (or wherever you point $LOGDIR to)
|
||||
* If only you receive an update trigger,
|
||||
Setup the .ssh/authorized_keys for the mirror user and place the public key of
|
||||
your upstream mirror into it. Preface it with
|
||||
`no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,command="~/bin/ftpsync",from="IPADDRESS"`
|
||||
and replace $IPADDRESS with that of your upstream mirror.
|
||||
* You are finished
|
||||
|
||||
In order to receive different pushes or syncs from different archives,
|
||||
name the config file ftpsync-$ARCHIVE.conf and call the ftpsync script
|
||||
with the commandline "sync:archive:$ARCHIVE". Replace $ARCHIVE with a
|
||||
sensible value. If your upstream mirror pushes you using runmirrors
|
||||
bundled together with this sync script, you do not need to add the
|
||||
"sync:archive" parameter to the commandline, the scripts deal with it
|
||||
automatically.
|
||||
|
||||
Common names used in the Debian mirror network are:
|
||||
* standard debian archive: default empty $ARCHIVE (ftpsync.conf)
|
||||
* debian-ports archive: ARCHIVE=ports (ftpsync-ports.conf)
|
||||
* debian-security archive: ARCHIVE=security (ftpsync-security.conf)
|
||||
|
||||
|
||||
## Debian mirror script minimum requirements
|
||||
|
||||
As always, you may use whatever scripts you want for your Debian mirror,
|
||||
but we *STRONGLY* recommend you to not invent your own. However, if you
|
||||
want to be listed as a mirror it *MUST* support the following minimal
|
||||
functionality:
|
||||
|
||||
* Must perform a 2-stage sync
|
||||
The archive mirroring must be done in 2 stages. The first rsync run
|
||||
must ignore the index files. The correct exclude options for the
|
||||
first rsync run are:
|
||||
`--exclude Packages* --exclude Sources* --exclude Release* --exclude=InRelease --include=i18n/by-hash/** --exclude=i18n/* --exclude ls-lR*`
|
||||
|
||||
The first stage must not delete any files.
|
||||
|
||||
The second stage should then transfer the above excluded files and
|
||||
delete files that no longer belong on the mirror.
|
||||
|
||||
Rationale: If archive mirroring is done in a single stage, there will be
|
||||
periods of time during which the index files will reference files not
|
||||
yet mirrored.
|
||||
|
||||
* Must not ignore pushes whil(e|st) running.
|
||||
If a push is received during a run of the mirror sync, it MUST NOT
|
||||
be ignored. The whole synchronization process must be rerun.
|
||||
|
||||
Rationale: Most implementations of Debian mirror scripts will leave the
|
||||
mirror in an inconsistent state in the event of a second push being
|
||||
received while the first sync is still running. It is likely that in
|
||||
the near future, the frequency of pushes will increase.
|
||||
|
||||
* Should understand multi-stage pushes.
|
||||
The script should parse the arguments it gets via ssh, and if they
|
||||
contain a hint to only sync stage1 or stage2, then ONLY those steps
|
||||
SHOULD be performed.
|
||||
|
||||
Rationale: This enables us to coordinate the timing of the first
|
||||
and second stage pushes and minimize the time during which the
|
||||
archive is desynchronized. This is especially important for mirrors
|
||||
that are involved in a round robin or GeoDNS setup.
|
||||
|
||||
The minimum arguments the script has to understand are:
|
||||
* `sync:stage1`: Only sync stage1
|
||||
* `sync:stage2`: Only sync stage2
|
||||
* `sync:all`: Do everything. Default if none of stage1/2 are present.
|
||||
|
||||
There are more possible arguments, for a complete list see the
|
||||
ftpsync script in our git repository.
|
||||
|
||||
## ftpsync
|
||||
|
||||
This script is based on the old anonftpsync script. It has been rewritten
|
||||
to add flexibilty and fix a number of outstanding issues.
|
||||
|
||||
Some of the advantages of the new version are:
|
||||
* Nearly every aspect is configurable
|
||||
* Correct support for multiple pushes
|
||||
* Support for multi-stage archive synchronisations
|
||||
* Support for hook scripts at various points
|
||||
* Support for multiple archives, even if they are pushed using one ssh key
|
||||
* Support for multi-hop, multi-stage archive synchronisations
|
||||
|
||||
### Correct support for multiple pushes
|
||||
When the script receives a second push while it is running and syncing
|
||||
the archive it won't ignore it. Instead it will rerun the
|
||||
synchronisation step to ensure the archive is correctly synchronised.
|
||||
|
||||
Scripts that fail to do that risk ending up with an inconsistent archive.
|
||||
|
||||
### Can do multi-stage archive synchronisations
|
||||
The script can be told to only perform the first or second stage of the
|
||||
archive synchronisation.
|
||||
|
||||
This enables us to send all the binary packages and sources to a
|
||||
number of mirrors, and then tell all of them to sync the
|
||||
Packages/Release files at once. This will keep the timeframe in which
|
||||
the mirrors are out of sync very small and will greatly help things like
|
||||
DNS RR entries or even the planned GeoDNS setup.
|
||||
|
||||
### Multi-hop, multi-stage archive synchronisations
|
||||
The script can be told to perform a multi-hop multi-stage archive
|
||||
synchronisation.
|
||||
|
||||
This is basically the same as the multi-stage synchronisation
|
||||
explained above, but enables the downstream mirror to push his own
|
||||
staged/multi-hop downstreams before returning. This has the same
|
||||
advantage than the multi-stage synchronisation but allows us to do
|
||||
this over multiple level of mirrors. (Imagine one push going from
|
||||
Europe to Australia, where then locally 3 others get updated before
|
||||
stage2 is sent out. Instead of 4times transferring data from Europe to
|
||||
Australia, just to have them all updated near instantly).
|
||||
|
||||
|
||||
### Can run hook scripts
|
||||
ftpsync currently allows 5 hook scripts to run at various points of the
|
||||
mirror sync run.
|
||||
|
||||
* Hook1: After lock is acquired, before first rsync
|
||||
* Hook2: After first rsync, if successful
|
||||
* Hook3: After second rsync, if successful
|
||||
* Hook4: Right before leaf mirror triggering
|
||||
* Hook5: After leaf mirror trigger (only if we have slave mirrors; HUB=true)
|
||||
|
||||
Note that Hook3 and Hook4 are likely to be called directly after each other.
|
||||
The difference is that Hook3 is called *every* time the second rsync
|
||||
succeeds even if the mirroring needs to re-run due to a second push.
|
||||
Hook4 is only executed if mirroring is completed.
|
||||
|
||||
### Support for multiple archives, even if they are pushed using one ssh key
|
||||
If you get multiple archives from your upstream mirror (say Debian,
|
||||
Debian-Backports and Volatile), previously you had to use 3 different ssh
|
||||
keys to be able to automagically synchronize them. This script can do it
|
||||
all with just one key, if your upstream mirror tells you which archive.
|
||||
See "Commandline/SSH options" below for further details.
|
||||
|
||||
For details of all available options, please see the extensive documentation
|
||||
in the sample configuration file.
|
||||
|
||||
### Commandline/SSH options
|
||||
|
||||
Script options may be set either on the local command line, or passed by
|
||||
specifying an ssh "command". Local commandline options always have
|
||||
precedence over the SSH_ORIGINAL_COMMAND ones.
|
||||
|
||||
Currently this script understands the options listed below. To make them
|
||||
take effect they MUST be prepended by "sync:".
|
||||
|
||||
| Option | Behaviour |
|
||||
|--------------|-----------|
|
||||
| stage1 | Only do stage1 sync |
|
||||
| stage2 | Only do stage2 sync |
|
||||
| all | Do a complete sync (default) |
|
||||
| mhop | Do a multi-hop sync |
|
||||
| archive:foo | Sync archive foo (if the file $HOME/etc/ftpsync-foo.conf exists and is configured) |
|
||||
| callback | Call back when done (needs proper ssh setup for this to work). It will always use the "command" callback:$HOSTNAME where $HOSTNAME is the one defined in config and will happen before slave mirrors are triggered. |
|
||||
|
||||
So, to get the script to sync all of the archive behind bpo and call back when
|
||||
it is complete, use an upstream trigger of
|
||||
ssh $USER@$HOST sync:all sync:archive:bpo sync:callback
|
||||
|
||||
|
||||
Mirror trace files
|
||||
==================
|
||||
Every mirror needs to have a 'trace' file under project/trace.
|
||||
The filename has to be the full hostname (eg. hostname -f), or in the
|
||||
case of a mirror participating in RR DNS (where users will never use
|
||||
the hostname) the name of the DNS RR entry, eg. security.debian.org
|
||||
for the security rotation.). (Note: ftp.$COUNTRY.debian.org is always
|
||||
wrong, don't use that).
|
||||
|
||||
The contents are defined as:
|
||||
|
||||
First line is always the output of "date -u", ideally run with LANG
|
||||
and LC_ALL set to POSIX.
|
||||
|
||||
Lines two to the end of file follow a RFC822 style format, though the
|
||||
field names can have spaces. Currently the following fields are
|
||||
defined, listed in the order as output by ftpsync, though the order is
|
||||
not mandantory:
|
||||
|
||||
| Field | Content | Example |
|
||||
|-------|---------|---------|
|
||||
| `Date` | Date in RFC822 Format when the tracefile got generated, ie. end of mirror run. | `Sun, 28 Feb 2016 09:40:29 +0000` |
|
||||
| `Date-Started` | As Date, but the time the mirror run started. | `Sun, 28 Feb 2016 09:33:55 +0000` |
|
||||
| `Archive Serial` | Archive serial for mirror run, taken from main archives tracefile. | `2016022802` |
|
||||
| `Creator` | Name and version of used software | `ftpsync 20170204`
|
||||
| `Running on host` | FQDN of mirror host. This may not match the actual mirror name, its the real hostname. | `klecker.debian.org` |
|
||||
| `Maintainer` | Groups and people responsible for this mirror. | `Admins <admins@example.com>` |
|
||||
| `Sponsor` | Organizations sponsoring this mirror. | `Example <https://example.com>` |
|
||||
| `Country` | The ISO 3361-1 code of the country hosting this mirror. | `DE` |
|
||||
| `Location` | The location this mirror is hosted in. | `Example` |
|
||||
| `Throughput` | Available throughput for this mirror per second. | `10Gb` |
|
||||
| `Trigger` | Trigger used for this run.` | `cron`, `ssh (comment)` |
|
||||
| `Architectures` | List of architectures included in the mirror. | `all amd64 i386 source` |
|
||||
| `Architectures-Configuration` | Architecture list as specified in config. | `ALL`, `INCLUDE amd64 i386 source`, `EXCLUDE armel` |
|
||||
| `Upstream-Mirror` | From where does the mirror get its data. | `ftp-master.debian.org` |
|
||||
| `Rsync-Transport` | Transport to connect to upstream used by rsync | `plain`
|
||||
| `Total bytes received in rsync` | rsync --stats output, bytes received | `1109846675` |
|
||||
| `Total time spent in stage1 rsync` | Seconds of runtime for stage1 | `347` |
|
||||
| `Total time spent in stage2 rsync` | Seconds of runtime for stage2 | `47` |
|
||||
| `Total time spent in rsync` | How long in total for rsync | `394` |
|
||||
| `Average rate` | How fast did the sync go | `2816869 B/s` |
|
||||
|
||||
The third line in the legacy format and the hostname for the "Running
|
||||
on Host" line for the new format MUST NOT be the DNS RR name, even if
|
||||
the mirror is part of it. It MUST BE the hosts own name. This is in
|
||||
contrast to the filename, which SHOULD be the DNS RR name.
|
||||
|
||||
|
||||
## runmirrors
|
||||
|
||||
This script is used to tell leaf mirrors that it is time to synchronize
|
||||
their copy of the archive. This is done by parsing a mirror list and
|
||||
using ssh to "push" the leaf nodes. You can read much more about the
|
||||
principle behind the
|
||||
[push](http://blog.ganneff.de/blog/2007/12/29/ssh-triggers.html),
|
||||
essentially it tells the receiving
|
||||
end to run a pre-defined script. As the whole setup is extremely limited
|
||||
and the ssh key is not usable for anything else than the pre-defined
|
||||
script this is the most secure method for such an action.
|
||||
|
||||
This script supports two types of pushes: The normal single stage push,
|
||||
as well as the newer multi-stage push.
|
||||
|
||||
The normal push, as described above, will simply push the leaf node and
|
||||
then go on with the other nodes.
|
||||
|
||||
The multi-staged push first pushes a mirror and tells it to only do a
|
||||
stage1 sync run. Then it waits for the mirror (and all others being pushed
|
||||
in the same run) to finish that run, before it tells all of the staged
|
||||
mirrors to do the stage2 sync.
|
||||
|
||||
This way you can do a nearly-simultaneous update of multiple hosts.
|
||||
This is useful in situations where periods of desynchronization should
|
||||
be kept as small as possible. Examples of scenarios where this might be
|
||||
useful include multiple hosts in a DNS Round Robin entry.
|
||||
|
||||
For details on the mirror list please see the documented
|
||||
runmirrors.mirror.sample file.
|
||||
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user