How to recompile Sabayon without leaving the tree?

[Odysseus]

Now at least we're closer to understanding each other, and you finally hit the nail on the head. What I was describing in my original post was something along these lines: someone downloads and installs Sabayon (it doesn't matter if it's the Core or full versions) then stay with the pure binaries and have a fully functioning system or recompile if they want (of course you couldn't do both) while maintaining their functioning system.

The way I suggest would actually reduce your server load because they wouldn't have to be serving as many binary files. Instead most of the load would be hosting and serving the git files needed to make up the Distribution, but the actual source tarballs would be downloaded from Gentoo.

You folks only release a new version a few times a year, but back port security updates for previous versions like you do now. Let's say someone is running Sabayon 4.2, when they emerge world, instead of getting every freaking update from Gentoo ~x86 they would instead only get the security updates. If they need software that isn't in the 4.2 release only then would they have to get it directly from the Gentoo ~x86 branch. Then when ver. 5 gets released you would use entropy to "hop" to 5. You could stay with entropy and stick with the binaries or recompile. Yes, there would be a ton of recompiling whenever there was a new release, but you'd only have to do it a few times a years instead of constantly, like if you run ~x86 now. Developers and testers could "hop" to the latest version before general release to help stabilization.

I would run "pure" Sabayon binaries if they didn't force you to install all of the language files (have them be available for those who want them as a choice like other distros); and all of the unneeded driver files (much of which hasn't been sold for over ten years). For example: why should one have to have installed on their system all of the X video card drivers for hardware made in the 80's and 90's; especially old isa/pci/vlb cards that were already out of favor when the Pentium-Pro (the first 686 CPU) was released?

And quite frankly the default Sabayon kernel configuration settings baffle me. They're all adjusted (as I mentioned previously) for large scale enterprise solutions. The reason these settings don't seem logical is that there isn't an admin in the world who is going to run Sabayon or Gentoo ~x86 on a production machine. Anyone whose job and reputation is on the line is going to be running a more mature stable OS. They can't afford down-time. I know I would be running the most stable system I could, if my livelihood were on the line. No one is going to be running a server farm or Beowolf cluster on Sabayon so why use kernel settings optimized for this purpose as a default?

Instead the default kernel should be geared towards "real-time" desktop usage, because this is who your users are. For example, one of the low latency preemptable schedulers and timers vs. the current default. There are many other optimizations I noticed... but you get the point.

These are some ideas... I have others but I believe the proper place would be in the general forum not here. Anyhow thanks for the feedback.

Ciao

[Fitzcarraldo]

And quite frankly the default Sabayon kernel configuration settings baffle me. They're all adjusted (as I mentioned previously) for large scale enterprise solutions. The reason these settings don't seem logical is that there isn't an admin in the world who is going to run Sabayon or Gentoo ~x86 on a production machine. Anyone whose job and reputation is on the line is going to be running a more mature stable OS. They can't afford down-time. I know I would be running the most stable system I could, if my livelihood were on the line. No one is going to be running a server farm or Beowolf cluster on Sabayon so why use kernel settings optimized for this purpose as a default?

Instead the default kernel should be geared towards "real-time" desktop usage, because this is who your users are. For example, one of the low latency preemptable schedulers and timers vs. the current default. There are many other optimizations I noticed... but you get the point.

Could you be more specific, please? Could you list the specific kernel parameters in the SL 2.6.31-sabayon kernel config file that are set for servers as opposed to desktops? I would be interested to know which these are. I do know that the SL kernel uses the default kernel CPU scheduler, the CFS (Completely Fair Scheduler), however that is not the recommended CPU scheduler for servers (many kernel tuning guides for enterprise servers recommend the Deadline Scheduler, for example), so I would be interested to know which specific parameters in the SL kernel are server-oriented in your opinion. If you could also recommend the specific changes to the SL kernel config file to improve its performance for a modern desktop environment and a legacy desktop environment then that would also be helpful. Thanks.

[Odysseus]

Could you be more specific, please? Could you list the specific kernel parameters in the SL 2.6.31-sabayon kernel config file that are set for servers as opposed to desktops? I would be interested to know which these are. I do know that the SL kernel uses the default kernel CPU scheduler, the CFS (Completely Fair Scheduler), however that is not the recommended CPU scheduler for servers (many kernel tuning guides for enterprise servers recommend the Deadline Scheduler, for example), so I would be interested to know which specific parameters in the SL kernel are server-oriented in your opinion. If you could also recommend the specific changes to the SL kernel config file to improve its performance for a modern desktop environment and a legacy desktop environment then that would also be helpful. Thanks.

I'm running Windows at the moment so I'm going to make these suggestions off the top of my head.
First thing since you're going through the trouble of compiling you own kernel is to select the processor type closest to what's in your rig. In my case it's pentium-m since I have a single core processor I unset anything related to SMP, you however would leave as is. Next is to unset generic x86 since you're building a kernel specific to your machine. I set the tickless timer option because I'm running on a fairly recent laptop with all power saving features enabled and don't want interrupts while asleep.

The CFQ (CFS) is a good i/o scheduler it's lightweight and simple and I use it, however I have several musician friends who swear by the Anticipatory Scheduler (which is more complex and gives priority to already running processes). I've tried both but don't notice much of a difference in my usage. Noop should only be selected by those who have hardware scsi arrays.

Preemption: You now have 3 choices None, voluntary (which is a compromise between no preemption and preemption, and the older preemptive model. I select preemptive. When preemptive is selected, you also get an additional choice of RCU schedulers which labeled as preemptive, I select this as well.

For config_hz the default is usually 100 or 250, I change this to 300 because as the help guide states it's divisible by 50 or 60 which are the two world standards for video. Since I do some AV editing, and DVD cloning and shrinking. I choose this setting. It gives me more responsiveness than the lower values. I used to choose 1000 until reading about how much increase in overhead this caused (see this link http://kerneltrap.org/node/5411). But from what I've read those using skype or other VOIP devices should select 1000.

I also choose the SLUB allocator over the SLAB allocator. SLUB is simpler and has less overhead than SLAB many Distros now default to SLUB over SLAB.

I unset anything having to do with profiling, and debugging these are just aren't needed, I'm not a kernel developer so it just adds to kernel size and overhead. Same goes for group scheduling. These are intended for multi-user machines which need to allocate CPU time-slices to various users /processes. They add nothing of value to a single user realtime box.

As I said these were off the top of my head. There are others. I hope this helps.

Ciao

[Fitzcarraldo]

Thanks for the reply. My memory failed me in my previous post: CFS is a CPU scheduler, whereas Deadline is an I/O scheduler (as is CFQ). They (CPU schedulers and I/O schedulers) are of course two different things. Anyway, I changed from the CFQ I/O scheduler to the Anticipatory I/O scheduler (adding "elevator=as" in grub.conf is easier than rebuilding the kernel) a year ago, and there was a noticeable improvement on my laptop. The Deadline I/O scheduler made matters worse, which is not surprising as it is recommended for database servers.

There was also an improvement when I added the "nopat" kernel boot option in grub.conf. AS and nopat together got rid of the stuttering (specifically) in videos in my case (see SL Forum thread Poor performance (unresponsive, pausing, stuttering etc.) for further details).

In the above-mentioned thread I also mentioned problems with the CFS CPU scheduler which I could resolve using the Group scheduler extension to the CFS (see post viewtopic.php?f=57&t=14972&start=40#p86029 in the above-mentioned thread), although using the Group scheduler extension is a real pain to set up.

I read with some interest about the new BFS CPU scheduler written by the well known Con Kolivas (see the Phoronix Web site for benchmarks, and also a recent thread about the BFS in the SL Forums) but it does not appear to be conclusively superior to the CFS, with reports of degraded performance from some users, so I'm not going to bother trying it.

I'll study with interest your other suggestions.

[Odysseus]

Interesting threads. As I said in my other posts I'm not an expert or professional, just a hobbyist who's been playing with Linux for a number of years. I'm completely self-taught with no formal computer training at all. I have no experience at all with multiprocessor or 64bit rigs. My experience with Linux generally is to KISS (keep it simple stupid). If I don't need it I don't activate it, And I only activate a new feature if I find I need to. I try to make my kernel as "monolithic" as possible, compiling in drivers I know I actually have and use, and deselecting those I don't. For example I compile into the kernel my specific alsa driver and deselect anything else. The two big exceptions being network and video drivers, I've found over the years that they work best when compiled as modules. But again I only compile what I need and deselect the rest. It takes me quite awhile to configure my kernel when I first set it up. But later kernel compilations are done in no time because I'm not compiling modules for gadgets I don't have, need, nor want.

I never have needed to set or adjust the group schedulers. My understanding is that one only needs these features to prioritize/ allocate CPU time-slices, which is much more important in a multiuser setting than real-time desktop use. Instead when I've had problems in the past (and this was before the acceptance of preemption into the main kernel tree) I just changed the "niceness" (if you reduce the niceness you make it less nice) giving priority to the process I wanted that way. Or if a process was hogging the CPU I made it more nice. I'll admit that it's a kludge, but as in just about anything in Linux there's more than one way to "skin a cat". Here is an informative link regarding nice and renice http://www.linux.com/archive/articles/58638 .

I hope this clarified what I said earlier.

Ciao

[Fitzcarraldo]

Interestingly I have found that the CPU scheduler can have a big influence on a desktop system, as explained in the thread I referenced earlier. nice and renice won't necessarily solve a CPU scheduling problem, as explained in the following post: Re: Poor performance (unresponsive, pausing, stuttering etc.) in that thread. If CONFIG_USER_SCHED=y in the kernel configuration then tasks are grouped by the CFS according to their user id, and niceness only applies within a group. The above-mentioned post explains it in more detail.

The CPU scheduler was the big 'battle' between Con Kolivas and the official Linux kernel developers, leading to his two-year self-imposed exile from kernel development. He was (and now is again) a major champion of CPU scheduler development for desktops as opposed to servers. Read his official release notes for the BFS and you'll see his zeal. There appears to be some, shall we say, rivalry between Ingo Molnar (the official kernel team's CPU scheduler developer) and Con Kolivas. One thing is for sure, something is not right with scheduling in the Linux kernel (see the last post in the above-mentioned thread). Actually, as far as I understand it, Ingo Molnar's CFS was inspired by Con Kolivas' CPU scheduler work.

Wikipedia entry on Con Kolivas: http://en.wikipedia.org/wiki/Con_Kolivas

[Odysseus]

It's funny all of the politics going on in the kernel development area. It's been going on for years if not the last decade. I remember reading on slashdot in the late 90's all of the conflicts between Linus and Allen Cox, it was like a soap opera then when there was only a few people and not much financial backing. Now with all of the big guns like IBM, Novell, Red Hat, Sun (now Oracle) ploughing money into development it must be like trying to referee a death-cage tag-team match in pro wrestling. I guarantee that most of the corporate dollars wants an emphasis on new enterprise development, where as many of the developers and most end-users want more spent on the desktop. With Linus in the middle trying to keep the money flowing arguing with both. Anyhow....

Another interesting link. Again I reiterate that I've been hacking my kernel for a long time, I'm an older fellow and because of that a bit less risk adverse. I often don't active new features until I read that I should use it, and usually don't do that until the "experimental" tag gets removed. The way I usually update my kernel is just to copy my old .config over to the new source directory and that's it. Of course occasionally I run into problems like when they changed the order of things a while back. But I deal with it then. If it ain't broke, don't fix it! I've ruined too many otherwise awesome setups over the years just to get the latest greatest.

That's how I ended up here in the first place. I was running a perfectly well performing Gentoo stable setup that had been performing perfectly since I installed it on my laptop four years ago. Then one weekend about a month and a half ago I decided I wanted to try out the new KDE. I figured what the hell I might as well finally upgrade X to take advantage of HAL and Evdev. Big mistake... It took me all weekend to get X to the desktop and when I did I found my KDE4 installation wasn't working correctly. I posted at Gentoo forums but couldn't recover. So I came here, like I said in the original post, hoping to get a more up-to-date, but still relatively stable setup going again while still having a usable box.

Anyhow.... Thanks for the info

Ciao