binary repository

[random guy]

those are some good points whilo, i would be completely fine with a bittorent host, not sure why to have or not have a tracker though but whatever. only thing about that is that it should be like linuxtracker and not just let any random guy mess up ebuilds. there should be someone to make sure it works before hundreds of people track or a way to authenticate them.

just a few other stuff

- CHOST, CFLAGS, USE differences are unchecked and you may decide on your own, if you install a binary package. In the case of some dependencies this can be deadly and you most likely do not know what you do, if you install an incompatible version. Portage should check that first and maybe ask you then to ignore the problems or it simply rejects the binary package and uses sources instead.

doesnt portage already do that? if it really doesnt than that is something that needs to be ironed out.

- Configuration issues. I don't know exactly if installing binary packages respects the configuration protection, but for distributions like Sabayon, there should be a possibilitiy to add a script to tweak your configuration in the 'right' way. Debian and Ubuntu do this in some ways, e.g. in the case of X11. It could be really helpful. You could even integrate a Python/Ruby/Kommander script to give a graphical wizard for configuration of the binary package. Of course this only makes sense on binary sub-distributions like Sabayon not on Gentoo itself ( see below ).

i think the config would probably work out too, otherwise a binary package would never work for anyone other than the one who made it. cant say for sure because i havent tested it out though.

ok so even if gentoo doesnt make a binhost we will still need them to fix some stuff in portage?

[Goatee]

No chance of being able to use the apt binhost is there?

[random guy]

No chance of being able to use the apt binhost is there?

i shudder at the thought.

anyway there are cross package manager programs out there, like alien is an app that converts rpm to deb so you can use rpm files in debian. although you sometimes will have problems like whilo had outlined above.

thing is someone probably could develop such an app to convert rpms for gentoo but the thing is our binaries are already a lot better than rpm stuff so why sacrifice a lot of good stuff we got to use rpm's?

[whilo]

only thing about that is that it should be like linuxtracker and not just let any random guy mess up ebuilds. there should be someone to make sure it works before hundreds of people track or a way to authenticate them.

BitTorrent handles all that stuff. Trackerless is relatively new and means it uses DHT (distributed hash tables) for an independent network. You don't need a tracker to share the file, the leecher only needs to download the *.ebuild file once. This takes load from a ebuild-binary-packagename.torrent server. BitTorrent also has a checksum to verify the package. Therefore we don't have to do much about, BitTorrent is made for it .
As long as you download the right *.ebuild file you will get the right package in the end.

The problem I fear here is connectivity. If you download 100 packages for that, you may have 100 torrent clients in the background sharing and handling 100 * 20 (e.g.) = 2000 connections. This can kill your connection and you can get a lot of overhead. You even have to share the files longer than you actually download to get the most out of BitTorrent. This can be really tricky.

Either you maximally download, let's say 10 files at once and have a maximal background seeding of 20 files, but then we have to be good at randomly choosing different packages for upload in the background to spread the files perfectly over the net. We could use a central server like a master seeder, seeding all ebuild-binary-packagename files constantly, so you won't get dead files and we can even dynamically add 'mirrors'. If we do that, we can also use it to track (central tracker) the files, but this can be really nasty again.

I'm neither a big network pro nor a BitTorrent pro, but these problems come to my mind when using BitTorrent. Still BitTorrent seems to be the best solution in the long term and for other projects, too. It even gives us the possibility to start 3rd party BitTorrent servers for different CHOST-march settings, e.g. pentium 4 for Sabayon.

doesnt portage already do that? if it really doesnt than that is something that needs to be ironed out.

No. Binary packages are simple *.tar.bz2 files. They are installed without any protection and are therefore a lot like *.rpms. The file format actually need not to change, since all dependency information is effectively handled by portage, but the validation of the content must be added. This is new to package management, since binary distributions don't need these checks. They have only one consistant state of their repository on their hosts. I have to check how Arch handles that.

And no!, mixing package managers is a no-no to me.

Cheers,
w

Besides: Binary packages make a lot of sense now. Sabayon cannot have all Gentoo packages in it and you have a lot of packages you may want to add after installation.

[random guy]

I'm neither a big network pro nor a BitTorrent pro, but these problems come to my mind when using BitTorrent.

apparently a lot more than me

well i cant imagine people really downloading 100 packages all at same time but ya i can see how even with 10 that can be a problem. if we could get such a thing started we wouldnt really be killing for seeds because we would have all the mainstream gentoo users seeding aswell and especially nice would be gentoo servers who are updating packages i could see this reaching the potencial that bittorent was originally intended for: to be used by a large number of people who seed what they download. in theory bittorent should be faster than standard http or ftp but in practice i have never seen that actually happen but maybe with this there could be a shot.

about the md5 check: so the users are not going to make the binary files themselves. because if a user is making the files and sharing thier own custom creations we will not only have 50 firefox torrents going but people will make mistakes and mess it up. so i htink we would need somene to donate binaries that then get shared with all. sabayon and kororaa make a lot but what about nonincluded stuff?

by the way this would also be good to relieve the stress on gentoo servers, i mean sync-ing with the repo more than once every 2 weeks is considered a lot, heck in ubuntu i do it every day soemtimes multiple times.

EDIT: can we fix the thing were messeges get sent to hell?

[cvill64]

that was my fear also, sense only SL would release our binaries, if through BT, they could be easily altered

[random guy]

man it is lokoing like i dont know that much about bittorent, i think it is possible to make it such that only authorized people are able to upload programs to the tracker.

in the card i was thinking about this and i could see myself using it. we could even have our own bittorent client modified from the source of another or built up from scratch. it could have an integrated search and maybe the ability to connect to multiple such trackers (maybe of kororaa or other derivatives, so each one is seperate but still all can benieft from eachother)...

just my 2 cents.

[cvill64]

if SL were to do a binhost, it would be a binhost of all of our creations and settings

[whilo]

O.k. I try to make myself clearer. I don't think we need a hacked BitTorrent version and I don't want that. Keep it as simple as possible, once we have reasonable model.

My idea is:

Setup some configuration, either in make.conf or if we don't want to integrate with Portage (unnecessary options in make.conf are not evil...) we can do it elsewhere, e.g. /etc/conf.d/binarypackages .
The environment could look like:
TORHOST="http://sabayonlinux.org/torrents"
TORDIR="/var/tmp/portage_torrents"

What portage should do afterwards is:
1. fetch the file using wget to download the *.torrent file from TORHOST
2. start a bittorrent client in the background to fetch the torrent file to PKGDIR
3. verify the PKGDIR contents to the current system (CHOST,CFLAGS,USE,ldd check,...)*
resume installation of the binary package as usual...

* is not strictly necessary as long as you stay on the same platform, but that means you may not use any Gentoo source capabilities anymore and this once again is what I don't want. I would like to set it on top of Gentoo. Give the simple average user/setup binary packages back...

As long as you download the right .torrent file from the right TORHOST you are all setup and nobody can fake anything up. BitTorrent uses digest verification.

A simple solution could be a FETCHCOMMAND bash script. Start wget, fetch the file, and load the bittorrent client. The command can wait until bittorrent finishes, move the file into PKGDIR and done.

Advantages:
- very simple bash setup
- clean integration into portage (we can move the env vars somewhere else) due to FETCHCOMMAND
- compatible with FEATURE="parallel-fetch" (?), fetching in a background queue (It is a queue and not truely parallel, isn't it?)

Disadvantages:
- the clients only share during download

The much more complex solution is using a daemon in the background which is talking to the shell script and handles uploads (random selection) of packages constantly. This could be a longterm solution.

Advantages:
- more stable
- faster downloads
- more distributed

Disadvantages:
- an additional daemon
- many configuration options of the daemon: uploads, max connections, bandwith...
- constant upload, should be possible to turn off

We should offer the whole portage tree compiled with Sabayon Linux settings. Otherwise binary packages don't make a lot of sense.

O.k. I have to test if Portage uses the FETCHCOMMAND also for binary packages and I have to test some first setup. I'll tell you later.

Cheers,
w

P.S.: I've started a thread about my idealogical problems here and would like to get to some conclusion there first before I start to work on Sabayon seriously.

[whilo]

O.k. in my opinion you release VERY often. As long as you keep that, a binary repository for updates wouldn't make too much sense. Upgrading via a new dvd image is much easier and helps spreading the whole update as an iso file via bittorrent. The only advantage of a binary host left would be adding additional Gentoo packages not on the dvd easily without building them.

Still a binary packages solution for Gentoo would enrich the Gentoo platform a lot, but are there any other advantages for Sabayon in particular?

Cheers,
whilo