Full persistence, i.e. fully updateable liveUSB

Discuss all artwork and development - Suggestions needed

Moderator: Moderators

Full persistence, i.e. fully updateable liveUSB

Postby olafsen » Sun Sep 06, 2009 15:10

Hi all,

I'm playing a bit with Sabayon 4.2.
I made a persistent LiveDVD install on USB stick.

The problem is: "/home" persistence (xsistence) is not enough for me. :-)

I need :
- LiveDVD style installation, because i will be using my flash drive on multiple computers, and don't want to modify grub.lst each time i boot on a new PC... And... i do like the LiveDVD desktop ;-)
- Custom software collection on key (IDEs, developper docs, disk tools, and so on...), meaning i am able to merge/unmerge/sync using the sabayon repositories, and get the same environment next time i boot using my flash drive.

This is easily done with LiveDVDs such as Knoppix and Sidux (persistent file is fully merged with '/', not '/home', and i may do everything i want using APT).

Well, guys I am missing this functionality a lot, and i really want to give gentoo a try :-(

Could this be a new feature in next release ? a new "xtension" boot parameter ?? ;-)

Well, that's just an idea, but it could make many users happy.


Olivier
olafsen
Baby Hen
 
Posts: 1
Joined: Sun Sep 06, 2009 14:29
Location: France

Re: Full persistence, i.e. fully updateable liveUSB

Postby totedati » Mon Sep 07, 2009 2:25

indeed ... a more userfriendly liveusb mods can be a good thing ...

you can raise your sugestion here:

[Input Needed] Sabayon 5 Brainstorm

to be considered for next sabayon major release ...

in theory sabayon liveusb is easy and a no brainer quick setup. You slap the iso to usbstick, slap the grub machinery on usbstick MBR then reboot and pray .... all things is suposed to be just like an livedvd session ... easy, not so?

custom installed progs can be done using custom iso with applications aplliance, like is slax modules ... no mistery here, a slax module is simply a squasfs iso image who, when loaded, is mounted on top of main sistem ... this is a thing that can be done in sabayon ... with little work ... a binary equo package is almost like an slax module ... binary ... compressed ... but not in squasfs format ...

then, for persistent user setings, /etc, /var, /tmp and /home, i think i picked all of it, is just a need for a separate partition with a little aufs magic ... here sabayon setup must be a little tricky because we don't want too much write cycles to our usbstick not so?

so, here is how can be done:

1. load all usb persistent settings, /etc, /var, /tmp and /home, from usbstick as readonly branches
2. create a tmpfs in ram memory with all this persistent settings created again as readwrite branches
3. mount it on top of readonly usbstick filesystem

then use it .... and when you need to shutdown this live session, or after some time of continuos running, when that tmpfs is begining to be a little heavy and memory hungry, you do THIS:

aufs - brsync
Brsync -- synchronize files between two aufs branches
J. R. Okajima

Let's assume a system such like this,
- aufs with only two branches
- one lower readonly branch on SSD
- one upper read/write branch on tmpfs
- using the system for a while, usage of the tmpfs grows
- you may want to reduce usage of tmpfs and regain the system main
memory
- usage of SSD never change since it is readonly, even if you remove
some larger files in aufs
- you also may want to reduce usage of SSD
- yes, it is ASUS EeePC. :-)

In this case, I'd recommend you to try aubrsync script in aufs2-util.git
tree. It executes rsync(1) between the two branches.

SYNTAX
----------------------------------------------------------------------
aubrsync Options move | move_with_wh | copy \
mntpnt src_branch dst_branch [ options for rsync ]

generic form:
aubrsync [ -w | --wh ] [ -i | --inotify ] Options \
mntpnt cmd [ parameters for cmd ]

Options:
[ -n | --dry_run ]
[ -q | --quiet ]
----------------------------------------------------------------------

SIMPLE EXAMPLES
----------------------------------------------------------------------
1.
# mount -t aufs -o br:/rw:/ro none /u
# aubrsync copy /u /rw /ro

The script executes rsync(1) and,
- remove the whiteout-ed files in /ro
- COPY the non-whiteouted files in /rw to /ro

2.
# mount -t aufs -o br:/rw:/ro none /u
# aubrsync move /u /rw /ro

This is similar to above except COPY.
The operation 'move' removes the non-whiteouted files in /rw by rsync(1).
After rsync(1), the script finds all whiteouts in /rw and removes them
too.
After this aubrsync, /rw will be almost empty.


now, i hope you understand ... as you can see is ... real magic :alien: :alien:

when you want to make you changes persistent you simply brsync that readonly /etc, /var, /tmp and /home settings from you readwrite memory tmpfs branch to you readonly usbstick branch... heh ... so sweet not so? you move a file from a readwrite partition to an readonly partition ... aufs magic ...

next time, when you boot, all settings should be already here, on you usbstick ...

to resume a little, you have
1. a readonly main sabayon squashfs iso image
2. on top of it an bunch of readonly custom little squashfs equo modules ala slax modules, for you custom installed applications
3. on top of it an persistent /etc, /var, /tmp and /home and so on readonly usbstick image
4. and on top of it a readwrite tmpfs ram based partition

in everyday operation, sistem files is loaded on demand only from squashfs modules to save as much as can be from you precious ram memory
all new and growing files, mainly from /etc, /var, /tmp and /home will be stored in tmpfs

from time to time to free you ram memory, you do a brsync, and save you memory ... also, when shutdown that sabayon liveusb you do also a final brsync ....

then ... reboot ... and work again, on another computer ... country ... planet ... galaxy ... always, you lovely sabayon linux liveusb will be here, with you, with all settings and files already in place on that little usbstick ....

can be done? i think so! i think is not so hard ... comparing this setup with a livedvd setup we have only two extrasteps:

1. a readonly main sabayon squashfs iso image <= is already done, and working well in livedvd setup
2. on top of it an bunch of readonly custom little squashfs equo modules ala slax modules, for you custom installed applications <=a new feature for liveusb setup
3. on top of it an persistent /etc, /var, /tmp and /home and so on readonly usbstick image <=a new feature for liveusb setup
4. and on top of it a readwrite tmpfs ram based partition <=is already done, and working well in livedvd setup

on step 3 can be reused xsistence feature but extended to all liveusb setup, begining from /, not only /home ... here, for my taste, i think is too much to push all user setings in an separate xsistence iso ... if will be simply a separate partition on usbstick, that user files can be accesed more easy and more quickly from another linux distro .... just like an simple usbstick ... but can be imagined a lot of variations here, for example:

- a normal /boot partition on usbstick, no compresion, no iso, no squasfs ..
- the main sabayon.squasfs split in a bunch of main appliances: one core.squasfs, hey! we have already this module! look like is Sabayon Linux CoreCD 4.2 not so!?, :shock: :shock:, one kde4.squasfs, one kde3.squasfs, one kernel.squasfs, one gnome.squasfs, x11.squasfs, apache.squasfs and so on .... portage and equo sets feature, portage - User-defined Package Sets, is the key for a good and logical segmentation here ....

and so on ...

can be sabayon linux the first live distro to do all this things, ALL REQUIRED STEPS, not only a part of it?

will see ... will see ....

Gentoo Linux LiveUSB HOWTO and
wiki.sabayonlinux.org - What_is_XsistenCe
is the reference documentation, of course ...

pendrivelinux.com - USB Gentoo 2007.0 install
can be considered reference documentation also ...

and then, is a lot of sabayon forum threads discussing this problem. i list all forum threads i found in order of importance, for me:
forum.sabayonlinux.org - liveCD to liveUSB
forum.sabayonlinux.org - Live USB image

forum.sabayonlinux.org - Can't find the live USB tutorial anymore
forum.sabayonlinux.org - installing sabayon on a pendrive [SOLVED][Solved]
forum.sabayonlinux.org - Post LiveUSB problems here.
forum.sabayonlinux.org - Help-How to install Sabayon on usb HDD?
Last edited by totedati on Mon Sep 07, 2009 4:24, edited 3 times in total.
linux is free, the expertise to harness it is not!
you don't make so much money selling open source software!
You make MORE money USING it, just like google!
linux registered user #352479
User avatar
totedati
Technological Hen
 
Posts: 400
Joined: Thu Jan 11, 2007 0:24
Location: Sibiu, Romania

Re: Full persistence, i.e. fully updateable liveUSB

Postby wolfden » Mon Sep 07, 2009 2:58

this isn't going to happen any time soon that is for sure

I would like to see people that are interested in doing it work/start up the project. Got to remember, man power is thin and if people are willing to step up and take on the project, that would be super.

You can probably get help and info from dev ML
http://mailman.sabayonlinux.org/cgi-bin ... info/devel
User avatar
wolfden
Sharecropper
 
Posts: 8824
Joined: Sat Jan 14, 2006 0:55
Location: Midwest USA

Re: Full persistence, i.e. fully updateable liveUSB

Postby totedati » Mon Sep 07, 2009 3:39

wolfden wrote:I would like to see people that are interested in doing it work/start up the project. Got to remember, man power is thin and if people are willing to step up and take on the project, that would be super.


true ... lack of time is the main issue here ... we need a bunch of real monkey who do already required first steps and type and test first all this ideeas and who will like to share with all of us what they find ... the most advanced liveusb distro look like is Slax - your pocket operating system ... for me at least ...

i tested a bit this distro and this is what i find:

1. the first setup is fine ... you got a cute liveusb distro to play ...
2. then ... of course, you need this thing ... and this thing ... not in the main setup ... you search and find a user provided solution for you needs ... this application bring to you sistem libproblem.x.y.so version ....
3. installed ... then you need another application ... you find another user provided squasfs module, but this module use libproblem.x.y+1.so library version .... then another apps ... another libproblem.x-2.y+1.so library file ... but libproblem.so.interface symlink, or hardlink, is pointing only to one of this three libproblem versions ... not necesary the newest ... not necesary the best ... but the last installed ....

and so on ... in time you end up with a lot of sistem library files where abstract interface point only to newest installed library ... the main problem to all liveusb setup is that from a readonly branch you can hide, or virtual owerwrite to speach so, an old file only when his name is identical with the upper, in readwrite branch, and newer file ... if is not identical, the full path + full file name, is not deleted, or replaced, is still here ... and as you install and reinstall a lot of custom application you end with a mess of old and new library files ...

in time application begin to behave very strange ... to crash randonly ... and all you liveusb experience begin to be very ugly ....

for a good liveusb distro this problem, old library files coexisting with new library files should be in a very tight and controled relation .... if not, you end up with an horror story ...

for my humble linux lib mechanics experience, learned until now, this scheme libproblem.x.y.so <-> libproblem.so.interface is a deep skin vitamin to all important linux applications ... library versioning help here but only with careful generated binary packages ....

this means all userprovided aplication aplliances should pass a very tight set of requirements ... is not too much room for variations from a reference sabayon linux liveusb mainline trunk of binaries .... but here, is already true that is dangerous to play and mix too much entropy installed binaries with portage instaled applications ... is already a dangerous game ....

;-D
linux is free, the expertise to harness it is not!
you don't make so much money selling open source software!
You make MORE money USING it, just like google!
linux registered user #352479
User avatar
totedati
Technological Hen
 
Posts: 400
Joined: Thu Jan 11, 2007 0:24
Location: Sibiu, Romania

Re: Full persistence, i.e. fully updateable liveUSB

Postby wolfden » Mon Sep 07, 2009 4:31

I don't know much about it, but hear that pendrive linux seems to be really good across multiple computers even. I myself have never needed such a thing, but I do see where it could be a very handy thing.
User avatar
wolfden
Sharecropper
 
Posts: 8824
Joined: Sat Jan 14, 2006 0:55
Location: Midwest USA

Re: Full persistence, i.e. fully updateable liveUSB

Postby amoebios » Fri Sep 25, 2009 3:05

i've been running around with Puppy on my stick. Compared to Puppy, Sabayon always feels a bit slow. :/
User avatar
amoebios
Growing Hen
 
Posts: 168
Joined: Wed Dec 03, 2008 22:25


Return to Artwork and Development Suggestions

Who is online

Users browsing this forum: No registered users and 0 guests