Poor performance (unresponsive, pausing, stuttering etc.)

[Fitzcarraldo]

My new line of thought is that the CPU scheduler could be the cause of the pauses and stutter. The following blog post explains in easy-to-understand terms the issues with the kernel CPU scheduler: Linux kernel: The battle of the CPU schedulers. I had read about Con Kolivas' 'resignation' last year in Interview with Con Kolivas part 2: his effort to improve Linux performance on the desktop but did not understand the issue at the time. Surfing the Web, it seems that Linux scheduler performance is still an issue for many. Coincidentally, I came across a thread in the Amarok Forum mentioning the good (=responsive) desktop performance of PCLinuxOS, apparently due to the use of the CK kernel patches. CK = Con Kolivas.

As it happens, I've just had to hit the Power Off button to get out of a situation that happens on my laptop occasionally: the HDD starts to read/write continuously and the desktop response becomes very, very sluggish: the mouse cursor takes ages to move, it takes ages for windows to maximise/minimise, windows grey-out for a while then the colour returns, then they grey-out again, the time on the panel clock is frozen, etc. This phenomenon is not due to a crontab run of ubdatedb (which also makes my desktop unresponsive, by the way). The phenomenon was also reported in the second Gentoo thread I listed in my first post. The difference this time is that I happened to have htop running and, although htop was only able to update its display sporadically, Core 1 of the CPU was running at typically 17% and Core 2 was running at 0%. Memory was virtually full (1979 MB out of a total 2001 MB) and Swap was empty. I also noticed that 66.5% of the memory was being used by /lib/ld-linux.so.2 (ld.so, ld-linux.so* - dynamic linker/loader).

EDIT: From my surfing I understand that the I/O scheduler and the CPU scheduler are two different things. CFS is a CPU scheduler, whilst CFQ is an I/O scheduler.

[Fitzcarraldo]

dunsurfin, could you please do me another favour? As you're using the CFQ I/O scheduler (the default in the SL kernel), then you should find the following files in the directory /sys/block/<drive>/queue/iosched/:

quantum (the number of requests to be placed in the dispatch queue in each cycle)
queued (the number of requests allowed in each of the 64 internal queues)

Where <drive>=sda or whatever your drive is. Would you please cat those two files and tell me the contents? If you find other files in /sys/block/sda/queue/iosched then please do the same.

Also, as I think you're using the default CFS CPU scheduler, could you please have a look in directory /proc/sys/kernel/ and cat any file which starts with "sched_" and tell me the contents.

Many thanks.


By the way, apparently the CFS CPU scheduler appeared for the first time in kernel 2.6.23 (http://www.linuxinsight.com/cfs-scheduler-to-appear-in-linux-kernel-2.6.23.html). Interesting, as the performance I experienced with the SL 2.6.22 kernel was fine, as far as I remember.

[dunsurfin]

Couldn't find "queued" but here are the rest:

localhost robin # cat /sys/block/sda/queue/iosched/quantum
4
localhost robin # cat /sys/block/sda/queue/iosched/back_seek_max
16384
localhost robin # cat /sys/block/sda/queue/iosched/back_seek_penalty
2
localhost robin # cat /sys/block/sda/queue/iosched/fifo_expire_async
250
localhost robin # cat /sys/block/sda/queue/iosched/fifo_expire_sync
125
localhost robin # cat /sys/block/sda/queue/iosched/slice_async
40
localhost robin # cat /sys/block/sda/queue/iosched/slice_async_rq
2
localhost robin # cat /sys/block/sda/queue/iosched/slice_idle
8
localhost robin # cat /sys/block/sda/queue/iosched/slice_sync
100
localhost robin # cat /proc/sys/kernel/sched_compat_yield
0
localhost robin # cat /proc/sys/kernel/sched_rt_period_us
1000000
localhost robin # cat /proc/sys/kernel/sched_rt_runtime_us
950000

Is this all a conspiracy to make me learn new things?

[Fitzcarraldo]

Well, I don't know about you, but I'm certainly learning a lot! I'm now reading e-mails between kernel developers and actually understanding some of what they're discussing.

Thanks for checking those scheduler tuning parameters. I switched temporarily back to the CFQ I/O scheduler and here are the parameters for my installation:

Code: Select all

localhost kernel # cd /proc/sys/kernel/
localhost kernel # ls
acct                   msgmnb                    random
acpi_video_flags       msgmni                    randomize_va_space
bootloader_type        ngroups_max               real-root-dev
cad_pid                nmi_watchdog              sched_compat_yield
compat-log             osrelease                 sched_rt_period_us
core_pattern           ostype                    sched_rt_runtime_us
core_uses_pid          overflowgid               sem
ctrl-alt-del           overflowuid               sg-big-buff
domainname             panic                     shmall
hostname               panic_on_oops             shmmax
hotplug                panic_on_unrecovered_nmi  shmmni
io_delay_type          pid_max                   tainted
keys                   poweroff_cmd              threads-max
kstack_depth_to_print  print-fatal-signals       unknown_nmi_panic
maps_protect           printk                    version
max_lock_depth         printk_ratelimit          vsyscall64
modprobe               printk_ratelimit_burst
msgmax                 pty
localhost kernel # cd /sys/block/sda/queue/iosched/
localhost kernel # ls
back_seek_max      fifo_expire_async  quantum      slice_async_rq  slice_sync
back_seek_penalty  fifo_expire_sync   slice_async  slice_idle
localhost kernel # cat /sys/block/sda/queue/iosched/quantum
4
localhost kernel # cat /sys/block/sda/queue/iosched/back_seek_max
16384
localhost kernel # cat /sys/block/sda/queue/iosched/back_seek_penalty
2
localhost kernel # cat /sys/block/sda/queue/iosched/fifo_expire_async
250
localhost kernel # cat /sys/block/sda/queue/iosched/fifo_expire_sync
125
localhost kernel # cat /sys/block/sda/queue/iosched/slice_async
40
localhost kernel # cat /sys/block/sda/queue/iosched/slice_async_rq
2
localhost kernel # cat /sys/block/sda/queue/iosched/slice_idle
8
localhost kernel # cat /sys/block/sda/queue/iosched/slice_sync
100
localhost kernel # cat /proc/sys/kernel/sched_compat_yield
0
localhost kernel # cat /proc/sys/kernel/sched_rt_period_us
1000000
localhost kernel # cat /proc/sys/kernel/sched_rt_runtime_us
950000

As you can see, the same as yours. So another theory of mine looks to be out of the window. Nil desperandum.

[Fitzcarraldo]

What I don't understand is why, if the CFS CPU scheduler is enabled in my 2.6.26 kernel (I assume that's what CONFIG_FAIR_GROUP_SCHED=y does), the tuning files /proc/sys/kernel/min-timeslice and /proc/sys/kernel/max-timeslice do not exist on my laptop. Do you have them?

Code: Select all

localhost kernel # grep SCHED /usr/src/linux/.config
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_GROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
CONFIG_RT_GROUP_SCHED=y
CONFIG_USER_SCHED=y
# CONFIG_CGROUP_SCHED is not set
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_DEFAULT_IOSCHED="cfq"
CONFIG_SCHED_SMT=y
CONFIG_SCHED_MC=y
CONFIG_SCHED_HRTICK=y
CONFIG_NET_SCHED=y
CONFIG_USB_EHCI_TT_NEWSCHED=y
localhost kernel #

[rhomp2002]

I have a bunch of distros installed so I went around and checked them out:

Sabayon, Bluewhite64 (2 versions), OpenSuse are all 64-bit. Bluewhite64 (essentially Slackware) is +300 FPS over Sabayon and Open Suse is +200 FPS over sabayon.

Mandriva, Ubuntu, Kubuntu, Pardus are all 32-bit. Mandriva is +300 over Sabayon and the others are +100 over Sabayon when it comes to FPS.

No idea what that means but Sabayon is slower than the others.

[dunsurfin]

What I don't understand is why, if the CFS CPU scheduler is enabled in my 2.6.26 kernel (I assume that's what CONFIG_FAIR_GROUP_SCHED=y does), the tuning files /proc/sys/kernel/min-timeslice and /proc/sys/kernel/max-timeslice do not exist on my laptop. Do you have them?

Not here.

[Fitzcarraldo]

rhomp2002, thanks for the benchmarks. This is interesting, because I've read before now in a few places that SL performance has been observed as slower than some other distros.

dunsurfin, that's also interesting. This appears to indicate that the CFS CPU scheduler is in fact not being used by SL. In which case, what is, I wonder? I need to dig further. I have also come across several mentions of incorrect or suboptimal setup of CPU scheduling in distros, so I'm going to look into that. It's a tedious process, because every change requires rebuilding the kernel.


EDIT: The following Ubuntu bug report no. 188226 looks very interesting. Notice that it discusses CPU scheduling in detail and some changes that the devs had to make to stop interruptions to e.g. sound and video on desktop machines:

https://bugs.launchpad.net/ubuntu/+sour ... bug/188226

I also came across the following MythTV thread (which referred me to the above-mentioned Ubuntu thread) which discusses changes to the kernel:

http://www.gossamer-threads.com/lists/m ... ers/349300

[Fitzcarraldo]

What I don't understand is why, if the CFS CPU scheduler is enabled in my 2.6.26 kernel (I assume that's what CONFIG_FAIR_GROUP_SCHED=y does), the tuning files /proc/sys/kernel/min-timeslice and /proc/sys/kernel/max-timeslice do not exist on my laptop. Do you have them?

Not here.

I now think that these two tuning parameters are not in /proc/sys/kernel/ because CONFIG_SCHED_DEBUG=y is not in the SL 2.6.26 kernel config file. I'll add it and report back. So it could be that SL is using the CFS CPU scheduler.


EDIT: Just found this:

There is only one central tunable (you have to switch on CONFIG_SCHED_DEBUG):

/proc/sys/kernel/sched_granularity_ns

which can be used to tune the scheduler from 'desktop' (low
latencies) to 'server' (good batching) workloads. It defaults to a
setting suitable for desktop workloads. SCHED_BATCH is handled by the
CFS scheduler module too.

from Linux KernelDocumentation::scheduler:sched-design-CFS.txt

[Fitzcarraldo]

Well, I added CONFIG_SCHED_DEBUG=y but still no sched_granularity_ns in /proc/sys/kernel/. What on Earth is going on?

Code: Select all

localhost kernel # pwd
/proc/sys/kernel
localhost kernel # ls
acct                   msgmnb                    random
acpi_video_flags       msgmni                    randomize_va_space
bootloader_type        ngroups_max               real-root-dev
cad_pid                nmi_watchdog              sched_compat_yield
compat-log             osrelease                 sched_rt_period_us
core_pattern           ostype                    sched_rt_runtime_us
core_uses_pid          overflowgid               sem
ctrl-alt-del           overflowuid               sg-big-buff
domainname             panic                     shmall
hostname               panic_on_oops             shmmax
hotplug                panic_on_unrecovered_nmi  shmmni
io_delay_type          pid_max                   tainted
keys                   poweroff_cmd              threads-max
kstack_depth_to_print  print-fatal-signals       unknown_nmi_panic
maps_protect           printk                    version
max_lock_depth         printk_ratelimit          vsyscall64
modprobe               printk_ratelimit_burst
msgmax                 pty
localhost kernel # sysctl kernel
kernel.sched_rt_period_us = 1000000
kernel.sched_rt_runtime_us = 950000
kernel.sched_compat_yield = 0
kernel.panic = 0
kernel.core_uses_pid = 0
kernel.core_pattern = core
kernel.tainted = 1
kernel.real-root-dev = 0
kernel.print-fatal-signals = 0
kernel.ctrl-alt-del = 0
kernel.modprobe = /sbin/modprobe
kernel.hotplug =
kernel.sg-big-buff = 32768
kernel.acct = 4 2       30
kernel.cad_pid = 1
kernel.threads-max = 32744
kernel.random.poolsize = 4096
kernel.random.entropy_avail = 3585
kernel.random.read_wakeup_threshold = 64
kernel.random.write_wakeup_threshold = 128
kernel.random.boot_id = 4daff42b-5edf-4b9e-b7f0-a393bb151d4c
kernel.random.uuid = c3bef9fe-f72e-4c24-9199-3577a223c0e6
kernel.overflowuid = 65534
kernel.overflowgid = 65534
kernel.pid_max = 32768
kernel.panic_on_oops = 0
kernel.printk = 0       4       1       7
kernel.printk_ratelimit = 5
kernel.printk_ratelimit_burst = 10
kernel.ngroups_max = 65536
kernel.unknown_nmi_panic = 0
kernel.nmi_watchdog = 0
kernel.panic_on_unrecovered_nmi = 0
kernel.bootloader_type = 113
kernel.kstack_depth_to_print = 12
kernel.io_delay_type = 0
kernel.randomize_va_space = 1
kernel.acpi_video_flags = 0
kernel.compat-log = 1
kernel.max_lock_depth = 1024
kernel.maps_protect = 0
kernel.poweroff_cmd = /sbin/poweroff
kernel.keys.maxkeys = 200
kernel.keys.maxbytes = 20000
kernel.keys.root_maxkeys = 200
kernel.keys.root_maxbytes = 20000
kernel.vsyscall64 = 1
kernel.ostype = Linux
kernel.osrelease = 2.6.26-sabayon
kernel.version = #1 SMP Fri Oct 10 23:48:37 BST 2008
kernel.hostname = localhost
kernel.domainname = (none)
kernel.shmmax = 33554432
kernel.shmall = 2097152
kernel.shmmni = 4096
kernel.msgmax = 8192
kernel.msgmni = 4002
kernel.msgmnb = 16384
kernel.sem = 250        32000   32      128
kernel.pty.max = 4096
kernel.pty.nr = 2
localhost kernel #

However, I noticed that there were still the parameters /proc/sys/kernel/sched_rt_period_us and /proc/sys/kernel/sched_rt_runtime_us. Why were those there? Aren't they something to do with a Real Time kernel? If the CFS scheduler is in use, should they be there? So I then changed:

Code: Select all

CONFIG_RT_GROUP_SCHED=y

to:

Code: Select all

# CONFIG_RT_GROUP_SCHED is not set

and rebuilt the kernel yet again. But still there was no change to /proc/sys/kernel/. Seems I'm flailing around in the dark here. I need the help of a kernel expert.