Automate mirror usage and checksums for ISO downloads

[twanj]

Sabayon could use metalink for ISO downloads. It's an XML file format listing mirrors and checksums for higher availability and integrity. It corrects errors during ftp/http downloads. A few older ISO are listed at http://www.metalinker.org/samples.html

There are automated tools for creating metalinks. More info at http://en.wikipedia.org/wiki/Metalink

[twanj]

Updated metalinks for 3.3 ISOs are at http://www.metalinker.org/samples.html#sabayon

These list all mirrors & checksums (chunk checksums are omitted here to save space, but are in the metalinks to repair downloads. About 20 programs support metalink, the best is probably aria2 but it's command line. openSUSE, OpenOffice.org, cURL and many other distributions use metalink.

Using metalinks with checksums means you don't have errors, and you don't have to instruct users to do things like this: http://www.sabayonlinux.org/wiki/index. ... our_MD5SUM

Here's what the metalink files look like:

Code: Select all

<?xml version="1.0" encoding="utf-8"?>
<metalink version="3.0" generator="Metalink Editor version 1.0.0" xmlns="http://www.metalinker.org/">
  <publisher>
    <name>SabayonLinux</name>
    <url>http://www.sabayonlinux.org/</url>
  </publisher>
  <identity>SabayonLinux</identity>
  <version>3.3</version>
  <description>SabayonLinux is a Gentoo based Linux distro.</description>
  <files>
    <file name="SabayonLinux-x86-3.3.iso">
      <size>3349598208</size>
      <os>Linux-x86</os>
      <verification>
        <hash type="md5">e8f2a1100f2292edec55f3296c1d1f5b</hash>
        <hash type="sha1">911d745d96517e59cc5ce0cb52088d1b4bbab3c1</hash>
      </verification>
      <resources>
        <url type="http" location="au" preference="100">http://mirror.internode.on.net/pub/sabayonlinux/SabayonLinux-x86-3.3.iso</url>
        <url type="ftp" location="au" preference="100">ftp://mirror.internode.on.net/pub/sabayonlinux/SabayonLinux-x86-3.3.iso</url>
        <url type="http" location="it" preference="100">http://bo.mirror.garr.it/mirrors/sabayonlinux/SabayonLinux-x86-3.3.iso</url>
        <url type="ftp" location="it" preference="100">ftp://bo.mirror.garr.it/mirrors/sabayonlinux/SabayonLinux-x86-3.3.iso</url>
        <url type="ftp" location="us" preference="100">ftp://dotdot.mine.nu/sabayon/SabayonLinux-x86-3.3.iso</url>
        <url type="https" location="dk" preference="100">https://binstar.org/sabayon/SabayonLinux-x86-3.3.iso</url>
        <url type="http" location="de" preference="100">http://sabayon.domaindorf.de/SabayonLinux-x86-3.3.iso</url>
        <url type="http" location="us" preference="100">http://mirror.cs.vt.edu/pub/SabayonLinux/SabayonLinux-x86-3.3.iso</url>
        <url type="ftp" location="us" preference="100">ftp://mirror.cs.vt.edu/pub/SabayonLinux/SabayonLinux-x86-3.3.iso</url>
        <url type="ftp" location="pt" preference="100">ftp://glua.ua.pt/sabayon/SabayonLinux-x86-3.3.iso</url>
      </resources>
    </file>
  </files>
</metalink>

[voxiac]

How do you include bittorrent mirrors?
Is it like:

Code: Select all

<url type="bittorrent" preference="100">http://linuxtracker.org/download.php?id=3744&name=SabayonLinux-x86-3.3.iso.torrent</url>

Will this work?

Also here will come the problem of instructing users about right download managers...
Don't get me wrong though - I like the idea.

[twanj]

How do you include bittorrent mirrors?
Is it like:

Code: Select all

<url type="bittorrent" preference="100">http://linuxtracker.org/download.php?id=3744&name=SabayonLinux-x86-3.3.iso.torrent</url>

Will this work?

Also here will come the problem of instructing users about right download managers...
Don't get me wrong though - I like the idea.

Yes, that will work.

Instruction is a problem, since it is less familiar than bittorrent. But a lot of people are familiar with neither. Metalink also has the possibility of being supported by a lot of programs that don't support torrents...Right now, only about 20 programs support it, but there are at least some on all platforms.

Here's how other distros instruct their users, see above for how they also explain bittorrent.

http://en.opensuse.org/Released_Version#Metalink

[twanj]

metalinks for 3.4loop2 are at http://www.metalinker.org/samples.html#sabayon now.

[teedog]

Here's a metalink for SabayonLinux-x86-3.4f.iso.
http://www.metalinker.org/samples.html#sabayon

Code: Select all

<?xml version="1.0" encoding="utf-8"?>
<metalink version="3.0" generator="Metalink Generator v1.00.0037" xmlns="http://www.metalinker.org/">
  <publisher>
    <name>Sabayon Linux</name>
    <url>http://www.sabayonlinux.org/</url>
  </publisher>
  <files>
    <file name="SabayonLinux-x86-3.4f.iso">
      <os>Linux-x86</os>
      <verification>
        <hash type="md5">9fabc6ce4a033f55b537ed19cf86be8e</hash>
      </verification>
      <resources>
        <url type="ftp" preference="100">ftp://glua.ua.pt/sabayon/SabayonLinux-x86-3.4f.iso</url>
        <url type="ftp" preference="100">ftp://mirror.aarnet.edu.au/pub/SabayonLinux/SabayonLinux-x86-3.4f.iso</url>
        <url type="http" preference="100">http://bo.mirror.garr.it/mirrors/sabayonlinux/SabayonLinux-x86-3.4f.iso</url>
        <url type="http" preference="100">http://cesium.di.uminho.pt/pub/sabayon/SabayonLinux-x86-3.4f.iso</url>
        <url type="http" preference="100">http://cross-lfs.sabayonlinux.org/SabayonLinux-x86-3.4f.iso</url>
        <url type="http" preference="100">http://mirror.internode.on.net/pub/sabayonlinux/SabayonLinux-x86-3.4f.iso</url>
        <url type="bittorrent" preference="100">http://www.linuxtracker.org/download.php?id=4564&name=SabayonLinux-x86-3.4f.iso.torrent</url>
      </resources>
    </file>
  </files>
</metalink>

[Nicolas]

I have metalinks of the newer 3.4 miniEditions at http://impfarm.imp.org/~nico/sabayon/ (the "http-only" metalink was to test a download manager bug).

I'll see if I can add chunk checksums later.

[WarraWarra]

If anyone wants to or have the time to make new metalinks or update the current ones for mostly new versions of SL or all versions just shout as the ibiblio.org server now has a metalink folder so we can post them on there , redistribute them from there etc etc . Just buzz me / send me the links and I will download / post them there or something like this.

Just did not have time myself to update the metlinks with all server links yet.

PS> When making / updating the metalinks.
Just ignore the ibiblio.org http link for downloads as it still has internal linking issues or something and working on it, note a few missing iso files in http side.
Should actually for the time being temporarily remove the http ibiblio.org from the SL downloads page until fixed.

http://distro.ibiblio.org/pub/linux/dis ... Metalinks/
ftp://ftp.ibiblio.org/pub/linux/distrib ... /Metalinks
rsync://ibiblio.org/pub/linux/distribut ... /Metalinks

[Nicolas]

If anyone wants to or have the time to make new metalinks or update the current ones for mostly new versions of SL or all versions just shout as the ibiblio.org server now has a metalink folder so we can post them on there , redistribute them from there etc etc . Just buzz me / send me the links and I will download / post them there or something like this.

They're already here: http://www.metalinker.org/samples/sabayon/

[twanj]

If anyone wants to or have the time to make new metalinks or update the current ones for mostly new versions of SL or all versions just shout as the ibiblio.org server now has a metalink folder so we can post them on there , redistribute them from there etc etc . Just buzz me / send me the links and I will download / post them there or something like this.

They're already here: http://www.metalinker.org/samples/sabayon/

I think it's great to have them on ibiblio.org but still link to metalinker.org on the download page, like things are now.