Problem setting up Community Repo [Solved]

Anything that pertains to Entropy, Equo or Sulfur

Moderator: Moderators

Re: Problem setting up Community Repo

Postby sjieke » Fri Jul 04, 2008 16:21

I will try to execute that command and see what the results are when I get home after work.

I had some spare time now (not much to do at work...) so I used it to browse a little bit in the entropy code (http://svn.sabayonlinux.org/projects/entropy/trunk/)

If found the routine that logs the 'Repository is being updated' messages in the 'libraries/entropy.py' file
Code: Select all
    def handle_repository_lock(self, repo):
        # get database lock
        unlocked = self.is_repository_unlocked(repo)
        if not unlocked:
            mytxt = "%s: %s. %s." % (
                bold(_("Attention")),
                red(_("Repository is being updated")),
                red(_("Try again in a few minutes")),
            )
            self.Entropy.updateProgress(
                mytxt,
                importance = 1,
                type = "warning",
                header = "\t"
            )
            return True
        return False


Going further down the line, following the routines I found the following (also in 'libraries/entropy.py)
Code: Select all
def is_repository_unlocked(self, repo):

        self.__validate_repository_id(repo)

        rc = self.download_item("lock", repo)
        if rc: # cannot download database
            self.syncErrors = True
            return False
        return True

and
Code: Select all
def download_item(self, item, repo, cmethod = None, lock_status_func = None):

        self.__validate_repository_id(repo)
        url, filepath = self.__construct_paths(item, repo, cmethod)

        #some code here, left out by me (poster of this messages...)
       
        rc = fetchConn.download()
        del fetchConn
        if rc in ("-1","-2","-3"):
            return False
        self.Entropy.setup_default_file_perms(filepath)
        return True


From what I could understand further down the line (to much code to paste :shock: ) I could conclude that it tries to download the file 'packages.db.download.lock' from the database folder (in my case http://users.telenet.be/brulee/standard/brulee/database/x86/)
If it failes then it concludes that it is unlocked, if it succeeds it concludes that it is locked.

This is all logical to me, but for some reason it succeeds to download a file that isn't event there?

P.S. I must say I was pleasantly suprised when I saw how much code there was for entropy. It was a little hard to browse through because of the large files, but then I don't have a python ide to ease the browsing between classes (I used notepad 8) ). Never the less, I found my way in less than 1h. So I can only conclude that it has a clear, structured and object oriented design, knowing that I have no experience with Python. So my deepest respect for the creator, since this is a one man project if I'm not mistaken :?:
sjieke
Technological Hen
 
Posts: 321
Joined: Thu Mar 01, 2007 10:46
Location: Maldegem, Belgium

Re: Problem setting up Community Repo

Postby lxnay » Fri Jul 04, 2008 17:02

Yes, you are right.
Thanks :)
Image
Join us on IRC (chat.freenode.net #sabayon or WebChat)
Submit bugs to our Bug Tracker
Follow me on Twitter
Add me on Facebook
Add me on Google+
lxnay
Land Owner
 
Posts: 3421
Joined: Thu Oct 13, 2005 23:16
Location: Italy

Re: Problem setting up Community Repo

Postby sjieke » Fri Jul 04, 2008 17:52

You're welcome,

I did some more testing, but still have problems.
If I lock the database with 'equo community mirrors db-download-lock', the file 'packages.db.download.lock' gets created.
If then I try to update I get a 'database does not exist online' exception
Code: Select all
Crashbox joachim # equo community mirrors db-download-lock
/usr/lib/portage/pym/portage_const.py:39: DeprecationWarning: DEPRECATION NOTICE: The portage_const module was replaced by portage.const
>>  @@ Entropy Server Mirrors Interface loaded:
>>    # mirror: ftp://t836565:xxxxxxxx@users.telenet.be/
>>  @@ Entropy Server Interface Instance on repository: brulee (type: community repository)
>>  @@ Currently configured repositories:
>>    # brulee
>>    # __system__
>>  * Starting to lock download mirrors databases ...
>>  * [ftp:users.telenet.be] connecting with user: t836565
>>  * [ftp:users.telenet.be] switching to: /
>>  * [ftp:users.telenet.be] switching to: /
>>  * [ftp:users.telenet.be] switching to: standard/brulee/database/x86/
>>  * [ftp:users.telenet.be] switching to: /
>>  * [ftp:users.telenet.be] switching to: standard/brulee/database/x86/
>>  @@ [repo:brulee|users.telenet.be] mirror successfully locked for download
>>  * Download mirrors lock complete
Crashbox joachim # equo update
>> Repository brulee hasn't been downloaded yet.
>> Repository brulee is not available. Cannot validate
>>  @@ Repositories synchronization ...
>>   #  (1/2) XFCE oriented repository
>>   # Database URL: http://users.telenet.be/brulee/standard/brulee/database/x86
>>   # Database local path: /var/lib/entropy/client/database/x86/brulee/standard/x86
>>   # Database EAPI: 2
>>      Downloading repository database ...
>>      Attention: database does not exist online.
>>      [svn.sabayonlinux.org:1026] Successfully connected to host
>>   #  (2/2) Sabayon Linux Official Repository
>>   # Database URL: http://svn.sabayonlinux.org/entropy/standard/sabayonlinux.org/database/x86
>>   # Database local path: /var/lib/entropy/client/database/x86/sabayonlinux.org/standard/x86
>>   # Database EAPI: 3
>>      Attention: database is already up to date.
>> Repository brulee is not available. Cannot validate


If I unlock it again and update it gives me the 'Repository is being updated' exception.
Code: Select all
Crashbox joachim # equo community mirrors db-download-unlock
/usr/lib/portage/pym/portage_const.py:39: DeprecationWarning: DEPRECATION NOTICE: The portage_const module was replaced by portage.const
>>  @@ Entropy Server Mirrors Interface loaded:
>>    # mirror: ftp://t836565:xxxxxxxx@users.telenet.be/
>>  @@ Entropy Server Interface Instance on repository: brulee (type: community repository)
>>  @@ Currently configured repositories:
>>    # brulee
>>    # __system__
>>  * Starting to unlock download mirrors databases ...
>>  * [ftp:users.telenet.be] connecting with user: t836565
>>  * [ftp:users.telenet.be] switching to: /
>>  * [ftp:users.telenet.be] switching to: /
>>  * [ftp:users.telenet.be] switching to: standard/brulee/database/x86/
>>  * [ftp:users.telenet.be] switching to: /
>>  * [ftp:users.telenet.be] switching to: standard/brulee/database/x86/
>>  * [repo:brulee|users.telenet.be] mirror successfully unlocked
>>  * Download mirrors unlock complete
Crashbox joachim # equo update
>> Repository brulee hasn't been downloaded yet.
>> Repository brulee is not available. Cannot validate
>>  @@ Repositories synchronization ...
>>   #  (1/2) XFCE oriented repository
>>   # Database URL: http://users.telenet.be/brulee/standard/brulee/database/x86
>>   # Database local path: /var/lib/entropy/client/database/x86/brulee/standard/x86
>>   # Database EAPI: 2
>>      Attention: Repository is being updated. Try again in a few minutes.
>>      [svn.sabayonlinux.org:1026] Successfully connected to host
>>   #  (2/2) Sabayon Linux Official Repository
>>   # Database URL: http://svn.sabayonlinux.org/entropy/standard/sabayonlinux.org/database/x86
>>   # Database local path: /var/lib/entropy/client/database/x86/sabayonlinux.org/standard/x86
>>   # Database EAPI: 3
>>      Attention: database is already up to date.
>> Repository brulee is not available. Cannot validate
>>  @@ Something bad happened. Please have a look.


If I have time, I will try to investigate it further, but I don't know how to debug python... Maybe I put some extra output in it...
Or maybe somebody here recognizes something and can tell me what configuration mistake I made... :roll:
sjieke
Technological Hen
 
Posts: 321
Joined: Thu Mar 01, 2007 10:46
Location: Maldegem, Belgium

Re: Problem setting up Community Repo

Postby sjieke » Fri Jul 04, 2008 23:17

Hello,

I figured out some things.
First of all, the reason why it still thinks the repo is being updated is because if you request a file that doesn't exist my provider replaces it with a not found page.
I added some extra 'DEBUG' lines to find out
Code: Select all
>>  @@ Repositories synchronization ...
>>   #  (1/2) XFCE oriented repository
>>   # Database URL: http://users.telenet.be/brulee/standard/brulee/database/x86
>>   # Database local path: /var/lib/entropy/client/database/x86/brulee/standard/x86
>>   # Database EAPI: 2
>>      DEBUG:: url -> http://users.telenet.be/brulee/standard/brulee/database/x86/packages.db.download.lock; filepath -> /dev/null
>>      DEBUG: Before fetchConn.download().
>>      DEBUG:: After fetchConn.download() -> rc: d41d8cd98f00b204e9800998ecf8427e.
>>      Attention: Repository is being updated. Try again in a few minutes.

In the same way it thinks probably that the 'md5sum.php' file is present, resulting in the problems I had with the validation.

Second thing I found was, when locking the database, so the file is present I got the following (note the lines with 'DEBUG' in it are added by me)
Code: Select all
>>  @@ Repositories synchronization ...
>>   #  (1/2) XFCE oriented repository
>>   # Database URL: http://users.telenet.be/brulee/standard/brulee/database/x86
>>   # Database local path: /var/lib/entropy/client/database/x86/brulee/standard/x86
>>   # Database EAPI: 2
>>      DEBUG:: url -> http://users.telenet.be/brulee/standard/brulee/database/x86/packages.db.download.lock; filepath -> /dev/null
>>      DEBUG: Before fetchConn.download().
DEBUG: Error during urllib2.urlopen(req)
DEBUG: ex ->  <class 'urllib2.HTTPError'>
DEBUG: ex ->  HTTP Error 405: Method Not Allowed
>>      DEBUG:: After fetchConn.download() -> rc: -3.
>>      Downloading repository database ...
>>      DEBUG:: url -> http://users.telenet.be/brulee/standard/brulee/database/x86/packages.db.download.lock; filepath -> /dev/null
>>      DEBUG: Before fetchConn.download().
>>      DEBUG:: url -> http://users.telenet.be/brulee/standard/brulee/database/x86/packages.db.dump.bz2; filepath -> /var/lib/entropy/client/database/x86/brulee/standard/x86/packages.db.dump.bz2
>>      DEBUG: Before fetchConn.download().
DEBUG: Error during urllib2.urlopen(req)
DEBUG: ex ->  <class 'urllib2.HTTPError'>
DEBUG: ex ->  HTTP Error 405: Method Not Allowed
>>      DEBUG:: After fetchConn.download() -> rc: -3.
DEBUG: Error during urllib2.urlopen(req)
DEBUG: ex ->  <class 'urllib2.HTTPError'>
DEBUG: ex ->  HTTP Error 405: Method Not Allowed
>>      DEBUG:: After fetchConn.download() -> rc: -3.
>>      DEBUG:: url -> http://users.telenet.be/brulee/standard/brulee/database/x86/packages.db.bz2; filepath -> /var/lib/entropy/client/database/x86/brulee/standard/x86/packages.db.bz2
>>      DEBUG: Before fetchConn.download().
>>      DEBUG:: url -> http://users.telenet.be/brulee/standard/brulee/database/x86/packages.db.download.lock; filepath -> /dev/null
>>      DEBUG: Before fetchConn.download().
DEBUG: Error during urllib2.urlopen(req)
DEBUG: ex ->  <class 'urllib2.HTTPError'>
DEBUG: ex ->  HTTP Error 405: Method Not Allowed
>>      DEBUG:: After fetchConn.download() -> rc: -3.
DEBUG: Error during urllib2.urlopen(req)
DEBUG: ex ->  <class 'urllib2.HTTPError'>
DEBUG: ex ->  HTTP Error 405: Method Not Allowed
>>      DEBUG:: After fetchConn.download() -> rc: -3.
>>      Attention: database does not exist online.
>>      [svn.sabayonlinux.org:1026] Successfully connected to host
>>   #  (2/2) Sabayon Linux Official Repository
>>   # Database URL: http://svn.sabayonlinux.org/entropy/standard/sabayonlinux.org/database/x86
>>   # Database local path: /var/lib/entropy/client/database/x86/sabayonlinux.org/standard/x86
>>   # Database EAPI: 3
>>      Attention: database is already up to date.
>> Repository brulee is not available. Cannot validate

So when the file is present a 'HTTP Error 405: Method Not Allowed' occurres.

Conclussion: the problem is located and the cause is partly discovered. No it is time to start looking for a solution :?:

Do you have enough information, if not ask and I will provide it. I don't know python very well so I don't know what 'urllib2.urlopen(req)' is trying to do, and why it returns a 'HTTP Error 405: Method Not Allowed'...

Hoping someone can help me with this
sjieke
Technological Hen
 
Posts: 321
Joined: Thu Mar 01, 2007 10:46
Location: Maldegem, Belgium

Re: Problem setting up Community Repo

Postby sjieke » Sat Jul 05, 2008 12:43

I got my repo updated :alien: , but had to alter the code in 'libraries/entropy.py' a bit :oops:

What I changed is 2 things
First, in the download method of the urlFetcher class there is a block of code with comment 'get file size if available'. As you can see in the code snippet I added an extra check after the retrieval of the remote file to see if the requested url is the same as the retrieved url. This handles the auto-redirect of my provider.
Code: Select all
# get file size if available
try:
    self.remotefile = urllib2.urlopen(req)
    if self.url <> self.remotefile.geturl():
        self.close()
        self.status = "-3"
        return self.status         
except KeyboardInterrupt:
    self.close()
    raise
except:
    self.close()
    self.status = "-3"         
    return self.status


The second thing I altered was to solve the 'HTTP Error 405: Method Not Allowed' exception
I changed
Code: Select all
if self.url.startswith("http://"):
    headers = { 'User-Agent' : self.user_agent }
    req = urllib2.Request(self.url, self.extra_header_data, headers)
else:
    req = self.url

To
Code: Select all
req = self.url


For some reason my provider doesn't like the user agent being set, but I also don't know the meaning of it (need to read more to understand or someone explainging it to me)
The question now is, what did I break by making these changes...

Looking forward to your response, I hope you don't mind I'm thinkering with it this much. Just trying to make it better (and solve my problem)...
sjieke
Technological Hen
 
Posts: 321
Joined: Thu Mar 01, 2007 10:46
Location: Maldegem, Belgium

Re: Problem setting up Community Repo

Postby sjieke » Sat Jul 05, 2008 12:43

Sorry, this was a double post of the previous one...

Still looking forward to your response :D
sjieke
Technological Hen
 
Posts: 321
Joined: Thu Mar 01, 2007 10:46
Location: Maldegem, Belgium

Re: Problem setting up Community Repo

Postby lxnay » Sat Jul 05, 2008 13:59

I am uploading Entropy 0.15.23.8 which should sort out all the issues described in this thread.
Image
Join us on IRC (chat.freenode.net #sabayon or WebChat)
Submit bugs to our Bug Tracker
Follow me on Twitter
Add me on Facebook
Add me on Google+
lxnay
Land Owner
 
Posts: 3421
Joined: Thu Oct 13, 2005 23:16
Location: Italy

Re: Problem setting up Community Repo [SOLVED]

Postby sjieke » Sat Jul 05, 2008 19:59

Updated to the new version, played and tested with it and everything works perfectly know :mrgreen:

Adding solved to the topic...
sjieke
Technological Hen
 
Posts: 321
Joined: Thu Mar 01, 2007 10:46
Location: Maldegem, Belgium

Re: Problem setting up Community Repo [SOLVED]

Postby lxnay » Sat Jul 05, 2008 20:11

wow! thanks lxnay for having made such a miracle! :geek:
Image
Join us on IRC (chat.freenode.net #sabayon or WebChat)
Submit bugs to our Bug Tracker
Follow me on Twitter
Add me on Facebook
Add me on Google+
lxnay
Land Owner
 
Posts: 3421
Joined: Thu Oct 13, 2005 23:16
Location: Italy

Previous

Return to Entropy|Equo|Rigo Package Managers

Who is online

Users browsing this forum: No registered users and 1 guest