Poor performance (unresponsive, pausing, stuttering etc.)

[Fitzcarraldo]

Just some more background information on the performance of my laptop:

Code: Select all

$ # First with Compiz-Fusion enabled:
$ glxgears
26957 frames in 5.0 seconds = 5391.328 FPS
27139 frames in 5.0 seconds = 5427.670 FPS
27118 frames in 5.0 seconds = 5423.543 FPS
27134 frames in 5.0 seconds = 5426.786 FPS
27137 frames in 5.0 seconds = 5427.318 FPS
27143 frames in 5.0 seconds = 5428.509 FPS
27136 frames in 5.0 seconds = 5427.050 FPS
27148 frames in 5.0 seconds = 5429.476 FPS
27100 frames in 5.0 seconds = 5419.985 FPS
27142 frames in 5.0 seconds = 5428.338 FPS
27054 frames in 5.0 seconds = 5410.652 FPS
^C
$ # Now with Metacity instead of Compiz-Fusion:
$ glxgears
26647 frames in 5.0 seconds = 5329.240 FPS
26598 frames in 5.0 seconds = 5319.457 FPS
26597 frames in 5.0 seconds = 5319.302 FPS
26602 frames in 5.0 seconds = 5320.207 FPS
26602 frames in 5.0 seconds = 5320.397 FPS
26602 frames in 5.0 seconds = 5320.359 FPS
26617 frames in 5.0 seconds = 5323.233 FPS
26604 frames in 5.0 seconds = 5320.695 FPS
26605 frames in 5.0 seconds = 5320.832 FPS
26620 frames in 5.0 seconds = 5323.810 FPS
26458 frames in 5.0 seconds = 5291.586 FPS
^C
$

Back in the days of SL 3.26/3.3 I used to get 9000 FPS with XGL and Beryl. When AIGLX replaced XGL and Compiz-Fusion replaced Beryl it dropped to around 5000 and has been that way every since, as the AMD Catalyst driver and Compiz-Fusion packages have 'advanced'.

[Fitzcarraldo]

I have now done some experiments with the I/O Scheduler. The default (CFQ) is by far the worst in my case, giving regular pauses and stuttering when playing a video. I then tried DEADLINE, NOOP and AS. The memory and CPU usage with all three is noticeably less than with CFQ. They are all better than CFQ with the best being AS, so I'm switching to AS for the time being. There is still always a pause in the video whenever I open a window, though. Memory use with the AS scheduler is significantly better than with CFQ on my laptop. So, a postive development.

Code: Select all

title=Sabayon Linux x86_64 (2.6.26-sabayon)
   root (hd0,2)
   kernel /kernel-genkernel-x86_64-2.6.26-sabayon  root=/dev/ram0 ramdisk=8192 real_root=UUID=6c2e8b37-21cb-4d0f-9ef9-e47598fe1763  quiet  init=/linuxrc splash=silent,theme:sabayon vga=791 CONSOLE=/dev/tty1 pci=nomsi nmi_watchdog=0 elevator=as
   initrd /initramfs-genkernel-x86_64-2.6.26-sabayon

The I/O Scheduler can be defined in grub.conf or in the kernel. For the moment I'm doing it in grub.conf as that's easier for me to experiment with. Notice also that I have switched off the Non-Maskable Interrupt Watchdog by adding nmi_watchdog=0 in the boot line in grub.conf. That didn't seem to make a noticeable difference, but it shouldn't do any harm.

Onward with the experiments!

EDIT: It's CFQ, not CFX. My mistake.

[wolfden]

x86_64 3.5
07:00.0 VGA compatible controller: nVidia Corporation G70 [GeForce 7600 GT]
x11-drivers/nvidia-drivers-173.14.12-r10

C-F

wolf64 wolfden # glxgears
37930 frames in 5.0 seconds = 7585.507 FPS
38226 frames in 5.0 seconds = 7645.174 FPS
37961 frames in 5.0 seconds = 7592.031 FPS
38164 frames in 5.0 seconds = 7632.682 FPS
38254 frames in 5.0 seconds = 7650.683 FPS
37800 frames in 5.0 seconds = 7559.959 FPS

Metacity

wolf64 wolfden # glxgears
48396 frames in 5.0 seconds = 9679.119 FPS
48122 frames in 5.0 seconds = 9624.312 FPS
48314 frames in 5.0 seconds = 9662.673 FPS
48322 frames in 5.0 seconds = 9664.206 FPS
48199 frames in 5.0 seconds = 9639.769 FPS
48142 frames in 5.0 seconds = 9628.372 FPS
48186 frames in 5.0 seconds = 9637.073 FPS
^C

[dunsurfin]

By the way, /usr/src/linux is a symlink, so you should still have it on your PC. When you do an "eselect kernel list" it will show what the symlink is currently pointing to (2.6.25-r1 in your case, 2.6.26 in my case). When you do an "eselect kernel set <x>" it changes the symlink to point to the kernel number <x> you select from the list:

Yes, it is there but the .config for it (see attachment) is for 2.6.26 so I assume it's only used if I boot into the 2.6.26 and I don't do that yet as I know I'll have some problems if I do.

What does the FPS rate signify in your calculations? I seem to have better overall performance but my glxgears is far, far lower:

9057 frames in 5.0 seconds = 1811.233 FPS
9075 frames in 5.0 seconds = 1814.820 FPS
8773 frames in 5.0 seconds = 1754.492 FPS
9081 frames in 5.0 seconds = 1816.044 FPS
9087 frames in 5.0 seconds = 1817.267 FPS
9077 frames in 5.0 seconds = 1815.294 FPS
9083 frames in 5.0 seconds = 1816.540 FPS
9079 frames in 5.0 seconds = 1815.770 FPS

[Fitzcarraldo]

Well, glxgears is notoriously unreliable as a benchmark, but basically it looks like my graphics performance is better than yours. If you were able to see the rotating desktop cube with my skydome, you'd be surprised that I have trouble with pausing/stuttering videos and my mouse/keyboard pausing: my laptop's graphics performance under SL with Compiz-Fusion is smooth and fast. This is why I suspect my problem is disk I/O-related.

That said, the AMD ATI fglrx driver is notorious (and I suspect the AIGLX and X.Org code could be improved too). As I understand it, AMD and X.Org are currently working on updates which would enable the fglrx and X.Org server to work together with OpenGL (currently one has to use the X11 or XHSM output driver with fglrx, as the OpenGL driver does not work properly with fglrx under Compiz-Fusion). So, as I mentioned in an earlier post (and so did KelloggsFrosties), the fglrx driver might possibly have something to do with the pauses and stuttering I'm seeing.

However, I still think the problem is due to the kernel. Having looked into it over the last few days, the pausing/stuttering problem (which includes pauses of keyboard entry and mouse cursor movement, temporary greying out of windows, and so on) occurs in various distros, and it seems more for x86_64 than x86. Here's just one other example, this time under Arch Linux: x86_64 Disk IO. The more I read and experience, the more I feel that the Linux kernel is not as good as it should be. For example, there's an article in the August issue of Linux Magazine about the kernel's locking system, and the change of the BKL (big kernel lock) from a spinlock to a semaphore, with benefits for some but performance problems for others, so it was reverted. Reading all the posts about I/O scheduling, and the many kernel hacks proposed, it seems to me that the kernel could well be the cause of these poor performance problems with some people's hardware.

[Fitzcarraldo]

Another experiment. I rebuilt the 2.6.26 kernel using the config changes suggested in Arch Linux bug report "FS#10512 - x86_64 Disk I/O Issues in Kernel26" (http://bugs.archlinux.org/task/10512):

Description: The default CFS I/O scheduler options in the kernel essentially cause the system to halt for disk I/O operations due to the way CFS handles root processes. CFS assigns the majority of the I/O slices to root and forgets about the user, causing the system to become unresponsive until disk I/O has ceased. While same disk transfers leave the system sluggish with intermittent pauses.

This problem was not readily apparent in the i686 kernel and was only noticed when changing to the x86_64 Arch. Similar behavior has been noted in both Gentoo and Ubuntu see links below.

The problem is solved by enabling CONFIG_CGROUP_SCHED see diff attached.

Additional info:

Reproduced on...
- kernel26-2.6.25.4 (Stock Arch)
- zen-kernel26-2.6.25-zen2 (Using Arch config)
- vanilla-kernel-2.6.25.4 (Using Arch config)

Solved by...
- kernel26-2.6.25-4 (Stock arch config + attached diff)
- zen-kernel26-2.6.25-zen2 (Using BFQ I/O scheduler + Stock arch config)

http://forums.gentoo.org/viewtopic-t-48 ... amd64.html
https://bugs.launchpad.net/ubuntu/+sour ... bug/188226

Steps to reproduce:
Use stock Arch x86_64 kernel and transfer large amounts of data from disk -> disk, or same disk large file transfers.

Below is precisely what I changed in the kernel:

Code: Select all

localhost src # diff config-2.6.26.test1 config-2.6.26.zcat0
83c83
< # CONFIG_CGROUP_DEBUG is not set
---
> CONFIG_CGROUP_DEBUG=y
86c86
< # CONFIG_CPUSETS is not set
---
> CONFIG_CPUSETS=y
89,94c89,94
< # CONFIG_FAIR_GROUP_SCHED is not set
< # CONFIG_RT_GROUP_SCHED is not set
< # CONFIG_USER_SCHED is not set
< CONFIG_CGROUP_SCHED=y
< # CONFIG_CGROUP_CPUACCT is not set
< # CONFIG_RESOURCE_COUNTERS is not set
---
> CONFIG_FAIR_GROUP_SCHED=y
> CONFIG_RT_GROUP_SCHED=y
> CONFIG_USER_SCHED=y
> # CONFIG_CGROUP_SCHED is not set
> CONFIG_CGROUP_CPUACCT=y
> CONFIG_RESOURCE_COUNTERS=y
localhost src #   

Performance seemed a bit better subjectively, but still did not stop the pausing and stuttering completely. I hammered the HDD with:

Code: Select all

dd if=/dev/zero of=./chunk bs=1024000 count=20000

and browsed the Web while playing the video, and it coped quite well but still paused or stuttered from time to time. Then I rebuilt the kernel to revert to the previous linux-sabayon-2.6.26 version (with my hwclock fix: see another thread) and recompiled the various modules and all the Compiz stuff. However the compiz building failed part way through with an out of memory message. Whaaaat?! htop was running in a window and showed me that the memory was all used up and so was my swap (517 Mb), which is a first. Perhaps a coincidence, perhaps not, but anyway I'm back with the stock linux-sabayon-2.6.26 kernel (with my hwclock fix). So still no silver bullet.

To recap, I've still got in place my two grub.conf boot line additions (see a previous post) and my fstab additions (also see a previous post).

[Fitzcarraldo]

I've now had a chance to compare dunsurfin's config files with mine.

First, his config.txt (which I had assumed is the stock linux-sabayon-2.6.25-r1, but appears to be modified?) compared with my config file for the linux-sabayon-2.6.25-r1 kernel still installed on my laptop:

Code: Select all

localhost Desktop # diff config.txt /usr/src/linux-2.6.25-sabayon-r1/.config
4c4
< # Fri Jul 11 17:47:13 2008
---
> # Tue Jul  1 18:18:23 2008
1695,1696c1695
< CONFIG_IWL4965_QOS=y
< CONFIG_IWL4965_HT=y
---
> # CONFIG_IWL4965_QOS is not set
1701c1700
< CONFIG_IWL3945_QOS=y
---
> # CONFIG_IWL3945_QOS is not set
2795,2797c2794,2796
< CONFIG_SND=m
< CONFIG_SND_TIMER=m
< CONFIG_SND_PCM=m
---
> CONFIG_SND=y
> CONFIG_SND_TIMER=y
> CONFIG_SND_PCM=y
2803,2804c2802,2803
< CONFIG_SND_MIXER_OSS=m
< CONFIG_SND_PCM_OSS=m
---
> CONFIG_SND_MIXER_OSS=y
> CONFIG_SND_PCM_OSS=y
3426,3432d3424
< CONFIG_AUFS=y
< CONFIG_AUFS_BRANCH_MAX_127=y
< # CONFIG_AUFS_BRANCH_MAX_511 is not set
< # CONFIG_AUFS_BRANCH_MAX_1023 is not set
< # CONFIG_AUFS_BRANCH_MAX_32767 is not set
< CONFIG_AUFS_STAT=y
< # CONFIG_AUFS_DEBUG is not set
3443a3436,3453
> CONFIG_AUFS=y
>
> #
> # These options are for
> #
> CONFIG_AUFS_FAKE_DM=y
> CONFIG_AUFS_BRANCH_MAX_127=y
> # CONFIG_AUFS_BRANCH_MAX_511 is not set
> # CONFIG_AUFS_BRANCH_MAX_1023 is not set
> # CONFIG_AUFS_BRANCH_MAX_32767 is not set
> CONFIG_AUFS_ROBR=y
> # CONFIG_AUFS_DLGT is not set
> # CONFIG_AUFS_SHWH is not set
> CONFIG_AUFS_RR_SQUASHFS=y
> CONFIG_AUFS_BR_XFS=y
> CONFIG_AUFS_WORKAROUND_FUSE=y
> # CONFIG_AUFS_DEBUG is not set
> # CONFIG_AUFS_COMPAT is not set
3562,3598c3572,3608
< CONFIG_NLS_CODEPAGE_437=y
< CONFIG_NLS_CODEPAGE_737=y
< CONFIG_NLS_CODEPAGE_775=y
< CONFIG_NLS_CODEPAGE_850=y
< CONFIG_NLS_CODEPAGE_852=y
< CONFIG_NLS_CODEPAGE_855=y
< CONFIG_NLS_CODEPAGE_857=y
< CONFIG_NLS_CODEPAGE_860=y
< CONFIG_NLS_CODEPAGE_861=y
< CONFIG_NLS_CODEPAGE_862=y
< CONFIG_NLS_CODEPAGE_863=y
< CONFIG_NLS_CODEPAGE_864=y
< CONFIG_NLS_CODEPAGE_865=y
< CONFIG_NLS_CODEPAGE_866=y
< CONFIG_NLS_CODEPAGE_869=y
< CONFIG_NLS_CODEPAGE_936=y
< CONFIG_NLS_CODEPAGE_950=y
< CONFIG_NLS_CODEPAGE_932=y
< CONFIG_NLS_CODEPAGE_949=y
< CONFIG_NLS_CODEPAGE_874=y
< CONFIG_NLS_ISO8859_8=y
< CONFIG_NLS_CODEPAGE_1250=y
< CONFIG_NLS_CODEPAGE_1251=y
< CONFIG_NLS_ASCII=y
< CONFIG_NLS_ISO8859_1=y
< CONFIG_NLS_ISO8859_2=y
< CONFIG_NLS_ISO8859_3=y
< CONFIG_NLS_ISO8859_4=y
< CONFIG_NLS_ISO8859_5=y
< CONFIG_NLS_ISO8859_6=y
< CONFIG_NLS_ISO8859_7=y
< CONFIG_NLS_ISO8859_9=y
< CONFIG_NLS_ISO8859_13=y
< CONFIG_NLS_ISO8859_14=y
< CONFIG_NLS_ISO8859_15=y
< CONFIG_NLS_KOI8_R=y
< CONFIG_NLS_KOI8_U=y
---
> CONFIG_NLS_CODEPAGE_437=m
> CONFIG_NLS_CODEPAGE_737=m
> CONFIG_NLS_CODEPAGE_775=m
> CONFIG_NLS_CODEPAGE_850=m
> CONFIG_NLS_CODEPAGE_852=m
> CONFIG_NLS_CODEPAGE_855=m
> CONFIG_NLS_CODEPAGE_857=m
> CONFIG_NLS_CODEPAGE_860=m
> CONFIG_NLS_CODEPAGE_861=m
> CONFIG_NLS_CODEPAGE_862=m
> CONFIG_NLS_CODEPAGE_863=m
> CONFIG_NLS_CODEPAGE_864=m
> CONFIG_NLS_CODEPAGE_865=m
> CONFIG_NLS_CODEPAGE_866=m
> CONFIG_NLS_CODEPAGE_869=m
> CONFIG_NLS_CODEPAGE_936=m
> CONFIG_NLS_CODEPAGE_950=m
> CONFIG_NLS_CODEPAGE_932=m
> CONFIG_NLS_CODEPAGE_949=m
> CONFIG_NLS_CODEPAGE_874=m
> CONFIG_NLS_ISO8859_8=m
> CONFIG_NLS_CODEPAGE_1250=m
> CONFIG_NLS_CODEPAGE_1251=m
> CONFIG_NLS_ASCII=m
> CONFIG_NLS_ISO8859_1=m
> CONFIG_NLS_ISO8859_2=m
> CONFIG_NLS_ISO8859_3=m
> CONFIG_NLS_ISO8859_4=m
> CONFIG_NLS_ISO8859_5=m
> CONFIG_NLS_ISO8859_6=m
> CONFIG_NLS_ISO8859_7=m
> CONFIG_NLS_ISO8859_9=m
> CONFIG_NLS_ISO8859_13=m
> CONFIG_NLS_ISO8859_14=m
> CONFIG_NLS_ISO8859_15=m
> CONFIG_NLS_KOI8_R=m
> CONFIG_NLS_KOI8_U=m
3647c3657
< CONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE=0
---
> CONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE=1
localhost Desktop #

^The July 1st config is mine. I can't see anything in the differences that relate to I/O scheduling (correct me if I'm wrong, anyone), although dunsurfin appears to be using AUFS (AnotherUnionFS) -- I wonder if that is improving performance on his PC when watching videos? But I don't think his kernel config would give me any better performance.

Now, let's compare my 2.6.26 kernel with the one dunsurfin built:

Code: Select all

localhost Desktop # diff config2.txt /usr/src/linux-2.6.26-sabayon/.config
4c4
< # Sat Aug  9 11:19:24 2008
---
> # Tue Oct  7 22:46:11 2008
252c252
< CONFIG_I8K=m
---
> # CONFIG_I8K is not set
3340c3340
< CONFIG_RTC_DRV_CMOS=m
---
> CONFIG_RTC_DRV_CMOS=y
localhost Desktop #   

^The October 7 config is mine (I've been rebuilding the 2.6.26 kernel rather a lot over the last few days!). The only differences are that dunsurfin has included the module for a Dell laptop:

Code: Select all

CONFIG_I8K
   Say Y if you intend to run this kernel on a Dell laptop.
   Say N otherwise.

and he has not included the fix for the SL 2.6.26 kernel hwclock bug (see 2.6.26-sabayon kernel cannot access hardware clock [Solved]). Other than that, our 2.6.26 kernels are identical and his kernel config would not make any difference to my laptop's performance.

[dunsurfin]

I've now had a chance to compare dunsurfin's config files with mine.

First, his config.txt (which I had assumed is the stock linux-sabayon-2.6.25-r1, but appears to be modified?)

snip

Now, let's compare my 2.6.26 kernel with the one dunsurfin built:

snip

The only differences are that dunsurfin has included the module for a Dell laptop:

and he has not included the fix for the SL 2.6.26 kernel hwclock bug (see 2.6.26-sabayon kernel cannot access hardware clock [Solved]). Other than that, our 2.6.26 kernels are identical and his kernel config would not make any difference to my laptop's performance.

This won't make any difference but I'd just like to point out that I haven't specifically modified my config files and any modifications would probably have been made when I've been installing using equo and spritz or the very few occasions I've used portage. Since I don't use the 2.6.26 kernel I haven't made any changes there either. (The 2.6.26 was installed via spritz).

[Fitzcarraldo]

Thanks for the clarification, dunsurfin. It was still a useful exercise to compare kernel configs because I wanted to see if there was anything in yours that could explain why you are getting better performance.

Anyway, I have some news: In addition to the entries in fstab and grub.conf mentioned previously, I have now added "nopat" to grub.conf, and my laptop's performance is noticeably better. Putting nopat in grub.conf switches off x86 PAT (Page Attribute Tables) -- see Linux 2 6 26 and Documentation/x86/pat.txt. Since I added nopat, memory use and CPU core loads have remained low and, more importantly, pauses have been reduced significantly under Compiz-Fusion, even under heavy load (KMPlayer playing a video; browsing the Web with Firefox; emerge --sync running in a terminal window; htop running in a terminal window; various windows open; rotating the Compiz-Fusion desktop cube; copying large files).

As I had seen memory and swap both fill up completely during my experiments -- and they had not changed after I closed all applications and windows -- I decided to increase the size of my swap partition from 517 MB to 1 GiB. That will certainly do no harm (although no swap has been used since I did it, and memory has not been more than half full). I also increased the cache in KMPlayer from 128 kB to 512 kB (although I have not bothered to do it for other media players such as xine, VLC etc. and they work fine).

However I can still cause a pause by maximising a minimised window or by scrolling a Firefox window up and down very fast. Such pauses don't happen under Windows XP on the same laptop, or under Vista on my tower PC, so I'm going to experiment some more, as I think I can improve things further. But the current situation is now fine for normal use and I will no longer feel embarrassed about showing off SL's capabilities to colleagues and friends.

EDIT: Fixed a typo.

[dunsurfin]

As I had seen memory and swap both fill up completely during my experiments -- and they had not changed after I closed all applications and windows -- I decided to increase the size of my swap partition from 517 GiB to 1 GiB. That will certainly do no harm (although no swap has been used since I did it, and memory has not been more than half full).

I'm pleased you've found a partial cure.
Just as an aside, I ran 3.4e quite happily without a swap (despite Wolfden's warning) and am tempted to turn it off here just to see what will happen. I know the process is different in XP, but I always ran that without a swap file and found better performance.