Poor performance (unresponsive, pausing, stuttering etc.)

[Fitzcarraldo]

I downloaded the ISO for Fedora 9 x86_64 LiveCD and had a look at /proc/sys/kernel -- see below. The maintainer of the kernel CPU scheduler works for Red Hat, so if any distro is going to have the CFS Scheduler in it, it's going to be Fedora.

Code: Select all

[fedora@localhost ~]$ cd /usr/src
[fedora@localhost src]$ ls
[fedora@localhost src]$ ls -la
total 8
drwxr-xr-x  2 root root 4096 2008-04-07 17:44 .
drwxr-xr-x 14 root root 4096 2008-05-07 22:20 ..
[fedora@localhost src]$ su
[root@localhost src]# zcat /proc/config.gz > /usr/src/config
gzip: /proc/config.gz: No such file or directory
[root@localhost src]# cd /proc
[root@localhost proc]# ls
1     2     3144  3525  4033  acpi           locks
10    2217  3145  3526  422   asound         mdstat
1017  2220  3147  3558  424   buddyinfo      meminfo
1018  224   3162  3561  4294  bus            misc
1070  225   3178  3588  4359  cgroups        modules
1072  226   3205  3589  4360  cmdline        mounts
1093  2437  3214  3591  442   cpuinfo        mtrr
11    2438  3216  3597  4451  crypto         net
1104  2439  3230  3626  4453  devices        pagetypeinfo
1113  2623  3239  3628  4782  diskstats      partitions
1118  2624  3247  3649  4784  dma            sched_debug
1121  273   3261  3699  4785  dri            schedstat
1136  274   3314  3795  4786  driver         scsi
1161  2803  3323  3807  4788  execdomains    self
1163  2804  3337  3831  4793  fb             slabinfo
1170  2895  3346  3845  4819  filesystems    stat
1171  2897  3347  3846  4829  fs             swaps
1246  2910  3357  3847  4866  interrupts     sys
1250  2918  3378  3850  5     iomem          sysrq-trigger
1252  2937  3381  3852  6     ioports        sysvipc
1254  2956  3382  3856  62    irq            timer_list
1276  2984  3383  3861  63    kallsyms       timer_stats
1281  2985  3384  3863  65    kcore          tty
1283  2992  3386  3869  66    keys           uptime
1339  3     3391  3871  7     key-users      version
160   3005  3418  3878  773   kmsg           vmcore
162   3046  3428  3880  779   kpagecount     vmstat
167   3054  3437  3882  781   kpageflags     zoneinfo
170   3057  3473  3898  8     latency_stats
1908  3058  3492  4     9     loadavg
[root@localhost proc]# cd sys
[root@localhost sys]# cd kernel
[root@localhost kernel]# ls
acct                      pid_max
acpi_video_flags          poweroff_cmd
bootloader_type           print-fatal-signals
cad_pid                   printk
compat-log                printk_ratelimit
core_pattern              printk_ratelimit_burst
core_uses_pid             pty
ctrl-alt-del              random
domainname                randomize_va_space
exec-shield               real-root-dev
hostname                  sched_batch_wakeup_granularity_ns
hotplug                   sched_child_runs_first
hung_task_check_count     sched_compat_yield
hung_task_timeout_secs    sched_domain
hung_task_warnings        sched_features
io_delay_type             sched_latency_ns
kstack_depth_to_print     sched_migration_cost
latencytop                sched_min_granularity_ns
maps_protect              sched_nr_migrate
max_lock_depth            sched_rt_period_us
modprobe                  sched_rt_runtime_us
msgmax                    sched_wakeup_granularity_ns
msgmnb                    sem
msgmni                    shmall
ngroups_max               shmmax
nmi_watchdog              shmmni
osrelease                 softlockup_thresh
ostype                    sysrq
overflowgid               tainted
overflowuid               threads-max
panic                     unknown_nmi_panic
panic_on_oops             version
panic_on_unrecovered_nmi  vsyscall64
[root@localhost kernel]# sysctl kernel
bash: sysctl: command not found
[root@localhost kernel]# cat sched_batch_wakeup_granularity_ns
20000000
[root@localhost kernel]# cat sched_child_runs_first
1
[root@localhost kernel]# cat sched_compat_yield
0
[root@localhost kernel]# cat sched_domain
cat: sched_domain: Is a directory
[root@localhost kernel]# cat sched_features
15
[root@localhost kernel]# cat sched_latency_ns
40000000
[root@localhost kernel]# cat sched_migration_cost
500000
[root@localhost kernel]# cat sched_min_granularity_ns
8000000
[root@localhost kernel]# cat sched_nr_migrate
32
[root@localhost kernel]# cat sched_rt_period_us
1000000
[root@localhost kernel]# cat sched_rt_runtime_us
950000
[root@localhost kernel]# cat sched_wakeup_granularity_ns
10000000
[root@localhost kernel]#

Unfortunately the kernel config file is not on the LiveCD, so I can't tell what the Fedora 9 x86_64 config is, and neither is /proc/config.gz so I can't zcat it either. Nevertheless, the CFS CPU scheduler tuning parameters appear to be present, albeit with slightly different name. But then so are sched_rt_period_us and sched_rt_runtime_us. So I'm none the wiser.

[Fitzcarraldo]

Some more progress to report.

At last I've managed to enable the CFS Scheduler tuning parameters:

Code: Select all

localhost / # cd /proc/sys/kernel
localhost kernel # ls
acct                    msgmnb                    sched_compat_yield
acpi_video_flags        msgmni                    sched_domain
bootloader_type         ngroups_max               sched_features
cad_pid                 nmi_watchdog              sched_latency_ns
compat-log              osrelease                 sched_migration_cost
core_pattern            ostype                    sched_min_granularity_ns
core_uses_pid           overflowgid               sched_nr_migrate
ctrl-alt-del            overflowuid               sched_rt_period_us
domainname              panic                     sched_rt_runtime_us
hostname                panic_on_oops             sched_wakeup_granularity_ns
hotplug                 panic_on_unrecovered_nmi  sem
hung_task_check_count   pid_max                   sg-big-buff
hung_task_timeout_secs  poweroff_cmd              shmall
hung_task_warnings      print-fatal-signals       shmmax
io_delay_type           printk                    shmmni
keys                    printk_ratelimit          softlockup_thresh
kstack_depth_to_print   printk_ratelimit_burst    tainted
latencytop              pty                       threads-max
maps_protect            random                    unknown_nmi_panic
max_lock_depth          randomize_va_space        version
modprobe                real-root-dev             vsyscall64
msgmax                  sched_child_runs_first
localhost kernel #

Even though I had included CONFIG_SCHED_DEBUG in the kernel config file, it was not being picked up by menuconfig (genkernel). Apparently it is not a good idea to edit the config file yourself. I decided to look for the parameter in menuconfig and set it there:

Code: Select all

-> Kernel hacking
  -> Kernel debugging (DEBUG_KERNEL [=y])

So now I can finally try to tune the CFS Scheduler.

By the way, I have now found out that, although the CFS Scheduler was included by default in kernel 2.6.23, the CFS Scheduler developer (Ingo Molnar) published patches for the 2.6.21 and 2.6.22 kernels (http://people.redhat.com/mingo/cfs-scheduler/). I have also just noticed that Fabio included the patch for the CFS Scheduler into the 2.6.22 kernel in SL 3.4 Stable Release (see the Press Release Sabayon Linux x86/x86-64 3.4: Stable Release). Presumably he included it in the other 3.4* releases too? And anything using kernel 2.6.23 and onwards would have had it in the kernel by default.

[Fitzcarraldo]

EDIT: Just found this:

There is only one central tunable (you have to switch on CONFIG_SCHED_DEBUG):

/proc/sys/kernel/sched_granularity_ns

which can be used to tune the scheduler from 'desktop' (low
latencies) to 'server' (good batching) workloads. It defaults to a
setting suitable for desktop workloads. SCHED_BATCH is handled by the
CFS scheduler module too.

from Linux Kernel Documentation::scheduler:sched-design-CFS.txt

That documentation is wrong: there is more than one tunable -- see the following two pages:

About documentation...

Multiprocessing with the Completely Fair Scheduler

[mhtt]

so Fitzcarraldo, Did you manage to get your hands on that granularity thing? I'm growing impatiet as it seems that it might be an easy solution to at least half of our problems

[Fitzcarraldo]

I'm still working on it. To make matters worse, I have just found out that the documentation for the 2.6.26 kernel regarding the CFS Scheduler is out of date, i.e. plain wrong, and as a result I have lost a couple of days building kernels and wondering where the Hell the CONFIG_FAIR_CGROUP_SCHED parameter was. Now I find out that, despite it being mentioned all over the Web, it doesn't exist any more: it's now called something else! Anyway, the latest documentation I've found for the CFS Scheduler is the following document patch on the Web:

http://retis.sssup.it/~scordino/project ... -CFS.patch

As I understand it from the above document patch (and the MythTV thread that I referenced a few posts back), the kernel config parameters I need to focus on are named:

CONFIG_GROUP_SCHED
CONFIG_FAIR_GROUP_SCHED
CONFIG_CGROUPS
CONFIG_CGROUP_SCHED

What I'm going to try to do is put media players in a task group that has more CPU share than browsers, in order to stop Firefox (a greedy CPU and memory hog) pausing videos that are playing on KMPlayer. I will try to do this by creating a task group that I will name "multimedia" and a task group that I will name "browsers" and put a value in /dev/cpuctl/multimedia/cpu.shares that is higher than /dev/cpuctl/browsers/cpu.shares, and tune these if necessary, using the tunables mentioned in my earlier posts.

In principle, one could create one or more task groups and allocate all those tasks to it that one does not want to give a high CPU share to. To start with I'm going to focus on Firefox. If I can stop that interrupting my multimedia tasks then I'll expand and create one or more further task groups to give lower CPU share to.

That's the theory as far as I currently understand it, but I've still got to get the kernel to build to support what I want to do, and that is proving difficult for me over the last couple of days (mainly because of the poor documentation), but I'm building my test kernel no. 13 as I type this, so hopefully I'll know within a day or so whether separating my tasks into groups and allocating them different CPU shares improves or not the performance.

[WarraWarra]

Is it not easier to run something similar to "renice" as we used in OSX and then fine tune some of the configs for this to get a bit more speed.

http://www.oreillynet.com/linux/cmd/cmd ... h=r/renice

More crazy man rantings:
Something else that could be useful is to get grub to boot faster using it as what is the word , kernel boot image instead of going from grub to stage 1 , stage 2 , stage3 and eventually pass on to GUI / KDE , ie: combine several of this boot stages into one.
Sry for the lack of words / terms but you get the idea.

It is actually such a waist to pass something from one clown to the next to the next if one can do it or 2 in parallel.
Something similar to EFI that apple tried but just not in the bios , after the bios onwards.

Imagine if grub can load and then during the boot selection window it continues to load the modules and other junk you might need that would in any case load + adding say extra options in parallel from the boot selection window and continuing as this could cut down a few more seconds on boot.

Example:
Say XEN'd grub that can then take 5 parallel independant SL's and close once finished then load gui on top of 1 or leave the rest for other functions all independant of each other and each handling useful functions with load sharing of resources ??

Currently booting , it is like a 3 year old child that gets to carry his older brothers and sister + then the father + mother and then the grandparents + extended family climbes onto his back as well = does not make sense.

Sry got side tracked there.

http://www.ibm.com/developerworks/linux ... linuxboot/
http://www.bootchart.org/
http://www.google.co.uk/search?q=fastes ... =firefox-a

Might as well let several user specific stuff load at gui login / on demand after say kde loaded with 5 seconds delay then start if needed ?

Booting linux in 5 seconds:
http://lwn.net/Articles/299483/

[wolfden]

2.6.27 gentoo-sources out - may want to spin that

[Fitzcarraldo]

^Thanks, I will have a look at it.

[Fitzcarraldo]

Ok, some interesting things to report. I've finally managed to build a kernel with the configuration the way I want it:

Code: Select all

localhost / # grep GROUP_SCHED /usr/src/config-2.6.26.zcat14
CONFIG_GROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
CONFIG_RT_GROUP_SCHED=y
CONFIG_CGROUP_SCHED=y
localhost / # grep USER_SCHED /usr/src/config-2.6.26.zcat14
# CONFIG_USER_SCHED is not set
localhost / # grep CONFIG_SCHED_DEBUG /usr/src/config-2.6.26.zcat14
CONFIG_SCHED_DEBUG=y
localhost / #

(I don't know what real-time tasks have been grouped in SL, but CONFIG_RT_GROUP_SCHED is in the stock 2.6.26 SL kernel so I left it in.)

I then carried out the following experiment:

1. Started a video playing in KMPlayer.
2. Launched Firefox.
3. Dragged the Firefox window slider up and down, which still caused pauses in the video.
4. Opened a terminal window and launched htop as root.
5. Observed in the htop window the pid of Firefox (12958) and mplayer (12933).
6. Opened another terminal window and typed the following commands:

Code: Select all

$ su
Password:
localhost fitzcarraldo # mkdir /dev/cpuctl
localhost fitzcarraldo # mount -t cgroup -ocpu none /dev/cpuctl
localhost fitzcarraldo # cd /dev/cpuctl
localhost cpuctl # mkdir multimedia
localhost cpuctl # mkdir browser
localhost cpuctl # cat multimedia/cpu.shares
1024
localhost cpuctl # cat browser/cpu.shares
1024
localhost cpuctl # echo 4096 > multimedia/cpu.shares
localhost cpuctl # echo 12958 > browser/tasks
localhost cpuctl # echo 12933 > multimedia/tasks
localhost cpuctl #

7. Dragged the Firefox window slider up and down, which still caused pauses in the video.
8. Reduced the CPU share of the browser group further:

Code: Select all

localhost cpuctl # echo 512 > browser/cpu.shares
localhost cpuctl #

9. Dragged the Firefox window slider up and down, which still caused pauses in the video.
10. Reduced the CPU share of the browser group further still:

Code: Select all

localhost cpuctl # echo 2 > browser/cpu.shares
localhost cpuctl #

11. Dragged the Firefox window slider up and down. No more pauses in the video!

So the Group scheduler extension to the CFS scheduler does work. The problem, as pointed out in the Ubuntu Bug Report No. 188226 and MythTV thread I referred to in an earlier post, is that the /dev/cpuctl pseudo filesystem will be recreated every time I boot, so cgroup configuration will have to be performed in an init script (daemon). I need to work out how to write a script that will automate what I did manually, i.e. give multimedia tasks a higher CPU share than certain other tasks such as browsers, editors, OpenOffice etc. so that my videos don't have pauses in them. Not a simple thing to do, especially as the pid of the tasks is likely to be different every time.

What the above has not achieved, though, is to stop the pause in the playing video when I maximise a window (any window, not just Firefox). I need to investigate that one too, because that is also annoying and does not happen to me in Windows XP or Vista.

[wolfden]

nice nice nice



I can almost see the howto in the wiki for group scheduler extension