linkchecker debug

DEBUG 2014-01-09 16:59:37,070 MainThread Python 2.7.5+ (default, Sep 19 2013, 13:48:49) 
[GCC 4.8.1] on linux2
DEBUG 2014-01-09 16:59:37,070 MainThread reading configuration from ['/root/.linkchecker/linkcheckerrc']
WARNING 2014-01-09 16:59:37,072 MainThread Running as root user; dropping privileges by changing user to nobody.
DEBUG 2014-01-09 16:59:37,079 MainThread configuration: [('anchors', False),
 ('authentication', []),
 ('blacklist', {'filename': '~/.linkchecker/blacklist'}),
 ('checkcss', False),
 ('checkhtml', True),
 ('clamavconf', '/etc/clamav/clamd.conf'),
 ('complete', False),
 ('connectionlimits', {}),
 ('cookiefile', None),
 ('csv',
  {'filename': 'linkchecker-out.csv', 'quotechar': '"', 'separator': ';'}),
 ('debugmemory', False),
 ('dot', {'encoding': 'ascii', 'filename': 'linkchecker-out.dot'}),
 ('externlinks', []),
 ('fileoutput', []),
 ('gml', {'filename': 'linkchecker-out.gml'}),
 ('gxml', {'filename': 'linkchecker-out.gxml'}),
 ('html',
  {'colorbackground': '#fff7e5',
   'colorborder': '#000000',
   'colorerror': '#db4930',
   'colorlink': '#191c83',
   'colorok': '#3ba557',
   'colorurl': '#dcd5cf',
   'colorwarning': '#e0954e',
   'filename': 'linkchecker-out.html'}),
 ('ignorewarnings', []),
 ('internlinks', []),
 ('localwebroot', None),
 ('logger', 'TextLogger'),
 ('loginextrafields', {}),
 ('loginpasswordfield', 'password'),
 ('loginurl', None),
 ('loginuserfield', 'login'),
 ('maxnumurls', None),
 ('maxrunseconds', None),
 ('nntpserver', None),
 ('none', {}),
 ('output', 'text'),
 ('proxy', {}),
 ('quiet', False),
 ('recursionlevel', -1),
 ('scanvirus', False),
 ('sendcookies', False),
 ('sitemap', {'encoding': 'utf-8', 'filename': 'linkchecker-out.sitemap.xml'}),
 ('sql',
  {'dbname': 'linksdb', 'filename': 'linkchecker-out.sql', 'separator': ';'}),
 ('status', True),
 ('status_wait_seconds', 5),
 ('storecookies', False),
 ('text',
  {'colorbase': 'purple',
   'colordlsize': 'default',
   'colordltime': 'default',
   'colorinfo': 'default',
   'colorinvalid': 'bold;red',
   'colorname': 'default',
   'colorparent': 'default',
   'colorreal': 'cyan',
   'colorreset': 'default',
   'colorurl': 'default',
   'colorvalid': 'bold;green',
   'colorwarning': 'bold;yellow',
   'filename': 'linkchecker-out.txt'}),
 ('threads', 100),
 ('timeout', 60),
 ('trace', False),
 ('useragent',
  u'Mozilla/5.0 (compatible; LinkChecker/8.4; +http://wummel.github.com/linkchecker/)'),
 ('verbose', False),
 ('wait', 0),
 ('warningregex', None),
 ('warningregex_max', 5),
 ('warnings', True),
 ('warnsizebytes', None),
 ('warnsslcertdaysvalid', 14),
 ('xml', {'filename': 'linkchecker-out.xml'})]
DEBUG 2014-01-09 16:59:37,079 MainThread HttpUrl handles url http://guide.dexy.it
DEBUG 2014-01-09 16:59:37,079 MainThread checking syntax
DEBUG 2014-01-09 16:59:37,080 MainThread Content cache key u'http://guide.dexy.it'
DEBUG 2014-01-09 16:59:37,080 MainThread URL cache key u'http://guide.dexy.it'
DEBUG 2014-01-09 16:59:37,080 MainThread Add intern pattern u'https?://(www\\.|)guide\\.dexy\\.it'
DEBUG 2014-01-09 16:59:37,080 MainThread Link pattern u'https?://(www\\.|)guide\\.dexy\\.it' strict=False
DEBUG 2014-01-09 16:59:37,081 MainThread queueing http link
base_url=u'http://guide.dexy.it'
parent_url=None
base_ref=None
recursion_level=0
url_connection=None
line=0
column=0
name=u''
anchor=u''
cache_key=u'http://guide.dexy.it'
LinkChecker 8.4              Copyright (C) 2000-2013 Bastian Kleineidam
LinkChecker comes with ABSOLUTELY NO WARRANTY!
This is free software, and you are welcome to redistribute it
under certain conditions. Look at the file `LICENSE' within this
distribution.
Get the newest version at http://wummel.github.com/linkchecker/
Write comments and bugs to https://github.com/wummel/linkchecker/issues
Support this project at http://wummel.github.com/linkchecker/donations.html

Start checking at 2014-01-09 16:59:37+000
DEBUG 2014-01-09 16:59:37,082 CheckThread-http://guide.dexy.it Checking http link
base_url=u'http://guide.dexy.it'
parent_url=None
base_ref=None
recursion_level=0
url_connection=None
line=0
column=0
name=u''
anchor=u''
cache_key=u'http://guide.dexy.it'
DEBUG 2014-01-09 16:59:37,083 CheckThread-http://guide.dexy.it checking connection
send: u'GET /robots.txt HTTP/1.1\r\nHost: guide.dexy.it\r\nUser-Agent: Mozilla/5.0 (compatible; LinkChecker/8.4; +http://wummel.github.com/linkchecker/)\r\nConnection: close\r\nAccept-Encoding: x-gzip,gzip,deflate\r\n\r\n'
reply: 'HTTP/1.1 404 Not Found\r\n'
header: Server: nginx/1.4.4
header: Date: Thu, 09 Jan 2014 16:59:37 GMT
header: Content-Type: text/html
header: Transfer-Encoding: chunked
header: Connection: close
header: Content-Encoding: gzip
DEBUG 2014-01-09 16:59:37,107 CheckThread-http://guide.dexy.it page info 404 Server: nginx/1.4.4
Date: Thu, 09 Jan 2014 16:59:37 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: close
Content-Encoding: gzip

DEBUG 2014-01-09 16:59:37,109 CheckThread-http://guide.dexy.it u'http://guide.dexy.it/robots.txt' allow all (HTTP error)
DEBUG 2014-01-09 16:59:37,110 CheckThread-http://guide.dexy.it u'http://guide.dexy.it/robots.txt' check allowance for:
  user agent: 'Mozilla/5.0 (compatible; LinkChecker/8.4; +http://wummel.github.com/linkchecker/)'
  url: u'http://guide.dexy.it' ...
DEBUG 2014-01-09 16:59:37,111 CheckThread-http://guide.dexy.it  ... allow all.
DEBUG 2014-01-09 16:59:37,111 CheckThread-http://guide.dexy.it Connecting to u'guide.dexy.it'
DEBUG 2014-01-09 16:59:37,112 CheckThread-http://guide.dexy.it Closing None
DEBUG 2014-01-09 16:59:37,112 CheckThread-http://guide.dexy.it Acquired lock for http://guide.dexy.it:80
send: 'HEAD / HTTP/1.1\r\nHost: guide.dexy.it\r\nUser-Agent: Mozilla/5.0 (compatible; LinkChecker/8.4; +http://wummel.github.com/linkchecker/)\r\nAccept-Encoding: x-gzip,gzip,deflate\r\nAccept-Charset: utf-8,ISO-8859-1;q=0.7,*;q=0.3\r\nDNT: 1\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: nginx/1.4.4
header: Date: Thu, 09 Jan 2014 16:59:37 GMT
header: Content-Type: text/html
header: Last-Modified: Thu, 09 Jan 2014 16:01:58 GMT
header: Connection: keep-alive
header: Content-Encoding: gzip
DEBUG 2014-01-09 16:59:37,119 CheckThread-http://guide.dexy.it Response: 200 OK
DEBUG 2014-01-09 16:59:37,120 CheckThread-http://guide.dexy.it Headers: Server: nginx/1.4.4
Date: Thu, 09 Jan 2014 16:59:37 GMT
Content-Type: text/html
Last-Modified: Thu, 09 Jan 2014 16:01:58 GMT
Connection: keep-alive
Content-Encoding: gzip

DEBUG 2014-01-09 16:59:37,121 CheckThread-http://guide.dexy.it follow all redirections
DEBUG 2014-01-09 16:59:37,122 CheckThread-http://guide.dexy.it HTML title parser
DEBUG 2014-01-09 16:59:37,122 CheckThread-http://guide.dexy.it Get content of u'http://guide.dexy.it'
DEBUG 2014-01-09 16:59:37,123 CheckThread-http://guide.dexy.it Connecting to u'guide.dexy.it'
DEBUG 2014-01-09 16:59:37,123 CheckThread-http://guide.dexy.it Closing <HttpConnection state=Idle
 method=HEAD
 sock=<socket._socketobject object at 0x21c9d00>
 buffer=[]
 response=<HTTPResponse status=200 reason=OK chunked=0 length=0>>
DEBUG 2014-01-09 16:59:37,124 CheckThread-http://guide.dexy.it Release lock for http://guide.dexy.it:80 and expiration None
DEBUG 2014-01-09 16:59:37,126 CheckThread-http://guide.dexy.it Acquired lock for http://guide.dexy.it:80
send: 'GET / HTTP/1.1\r\nHost: guide.dexy.it\r\nUser-Agent: Mozilla/5.0 (compatible; LinkChecker/8.4; +http://wummel.github.com/linkchecker/)\r\nAccept-Encoding: x-gzip,gzip,deflate\r\nAccept-Charset: utf-8,ISO-8859-1;q=0.7,*;q=0.3\r\nDNT: 1\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: nginx/1.4.4
header: Date: Thu, 09 Jan 2014 16:59:37 GMT
header: Content-Type: text/html
header: Last-Modified: Thu, 09 Jan 2014 16:01:58 GMT
header: Transfer-Encoding: chunked
header: Connection: keep-alive
header: Content-Encoding: gzip
DEBUG 2014-01-09 16:59:37,135 CheckThread-http://guide.dexy.it Response: 200 OK
DEBUG 2014-01-09 16:59:37,135 CheckThread-http://guide.dexy.it follow all redirections
DEBUG 2014-01-09 16:59:37,142 CheckThread-http://guide.dexy.it Stopped parsing: found <title> tag
 1 URL active,     0 URLs queued,    0 URLs checked, runtime 1 seconds
DEBUG 2014-01-09 16:59:38,350 CheckThread-http://guide.dexy.it Closing <HttpConnection state=Idle
 method=GET
 sock=<socket._socketobject object at 0x21c9d00>
 buffer=[]
 response=<HTTPResponse status=200 reason=OK chunked=1 length=None>>
DEBUG 2014-01-09 16:59:38,350 CheckThread-http://guide.dexy.it Release lock for http://guide.dexy.it:80 and expiration 1389287078.35
DEBUG 2014-01-09 16:59:38,350 CheckThread-http://guide.dexy.it task_done http link
base_url=u'http://guide.dexy.it'
parent_url=None
base_ref=None
recursion_level=0
url_connection=None
line=0
column=0
name=u''
anchor=u''
cache_key=u'http://guide.dexy.it'


********** Oops, I did it again. *************

You have found an internal error in LinkChecker. Please write a bug report
at https://github.com/wummel/linkchecker/issues
and include the following information:
- the URL or file you are testing
- the system information below

When using the commandline client:
- your commandline arguments and any custom configuration files.
- the output of a debug run with option "-Dall"

Not disclosing some of the information above due to privacy reasons is ok.
I will try to help you nonetheless, but you have to give me something
I can work with ;) .

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/linkcheck/director/task.py", line 29, in run
    line: self.run_checked()
    locals:
      self = <local> <Checker(CheckThread-http://guide.dexy.it, started 139866167351040)>
      self.run_checked = <local> <bound method Checker.run_checked of <Checker(CheckThread-http://guide.dexy.it, started 139866167351040)>>
  File "/usr/lib/python2.7/dist-packages/linkcheck/director/checker.py", line 48, in run_checked
    line: self.check_url()
    locals:
      self = <local> <Checker(CheckThread-http://guide.dexy.it, started 139866167351040)>
      self.check_url = <local> <bound method Checker.check_url of <Checker(CheckThread-http://guide.dexy.it, started 139866167351040)>>
  File "/usr/lib/python2.7/dist-packages/linkcheck/director/checker.py", line 56, in check_url
    line: self.check_url_data(url_data)
    locals:
      self = <local> <Checker(CheckThread-http://guide.dexy.it, started 139866167351040)>
      self.check_url_data = <local> <bound method Checker.check_url_data of <Checker(CheckThread-http://guide.dexy.it, started 139866167351040)>>
      url_data = <local> <http link, base_url=u'http://guide.dexy.it', parent_url=None, base_ref=None, recursion_level=0, url_connection=None, line=0, column=0, name=u'', anchor=u'', cache_key=u'http://guide.dexy.it'>
  File "/usr/lib/python2.7/dist-packages/linkcheck/director/checker.py", line 71, in check_url_data
    line: url_data.check()
    locals:
      url_data = <local> <http link, base_url=u'http://guide.dexy.it', parent_url=None, base_ref=None, recursion_level=0, url_connection=None, line=0, column=0, name=u'', anchor=u'', cache_key=u'http://guide.dexy.it'>
      url_data.check = <local> <bound method HttpUrl.check of <http link, base_url=u'http://guide.dexy.it', parent_url=None, base_ref=None, recursion_level=0, url_connection=None, line=0, column=0, name=u'', anchor=u'', cache_key=u'http://guide.dexy.it'>>
  File "/usr/lib/python2.7/dist-packages/linkcheck/checker/urlbase.py", line 490, in check
    line: self.local_check()
    locals:
      self = <local> <http link, base_url=u'http://guide.dexy.it', parent_url=None, base_ref=None, recursion_level=0, url_connection=None, line=0, column=0, name=u'', anchor=u'', cache_key=u'http://guide.dexy.it'>
      self.local_check = <local> <bound method HttpUrl.local_check of <http link, base_url=u'http://guide.dexy.it', parent_url=None, base_ref=None, recursion_level=0, url_connection=None, line=0, column=0, name=u'', anchor=u'', cache_key=u'http://guide.dexy.it'>>
  File "/usr/lib/python2.7/dist-packages/linkcheck/checker/urlbase.py", line 545, in local_check
    line: self.check_content()
    locals:
      self = <local> <http link, base_url=u'http://guide.dexy.it', parent_url=None, base_ref=None, recursion_level=0, url_connection=None, line=0, column=0, name=u'', anchor=u'', cache_key=u'http://guide.dexy.it'>
      self.check_content = <local> <bound method HttpUrl.check_content of <http link, base_url=u'http://guide.dexy.it', parent_url=None, base_ref=None, recursion_level=0, url_connection=None, line=0, column=0, name=u'', anchor=u'', cache_key=u'http://guide.dexy.it'>>
  File "/usr/lib/python2.7/dist-packages/linkcheck/checker/urlbase.py", line 798, in check_content
    line: self.check_html()
    locals:
      self = <local> <http link, base_url=u'http://guide.dexy.it', parent_url=None, base_ref=None, recursion_level=0, url_connection=None, line=0, column=0, name=u'', anchor=u'', cache_key=u'http://guide.dexy.it'>
      self.check_html = <local> <bound method HttpUrl.check_html of <http link, base_url=u'http://guide.dexy.it', parent_url=None, base_ref=None, recursion_level=0, url_connection=None, line=0, column=0, name=u'', anchor=u'', cache_key=u'http://guide.dexy.it'>>
  File "/usr/lib/python2.7/dist-packages/linkcheck/checker/urlbase.py", line 859, in check_html
    line: return self.check_html_w3()
    locals:
      self = <local> <http link, base_url=u'http://guide.dexy.it', parent_url=None, base_ref=None, recursion_level=0, url_connection=None, line=0, column=0, name=u'', anchor=u'', cache_key=u'http://guide.dexy.it'>
      self.check_html_w3 = <local> <bound method HttpUrl.check_html_w3 of <http link, base_url=u'http://guide.dexy.it', parent_url=None, base_ref=None, recursion_level=0, url_connection=None, line=0, column=0, name=u'', anchor=u'', cache_key=u'http://guide.dexy.it'>>
  File "/usr/lib/python2.7/dist-packages/linkcheck/checker/urlbase.py", line 937, in check_html_w3
    line: self.check_w3_errors(u.read(), "W3C HTML")
    locals:
      self = <local> <http link, base_url=u'http://guide.dexy.it', parent_url=None, base_ref=None, recursion_level=0, url_connection=None, line=0, column=0, name=u'', anchor=u'', cache_key=u'http://guide.dexy.it'>
      self.check_w3_errors = <local> <bound method HttpUrl.check_w3_errors of <http link, base_url=u'http://guide.dexy.it', parent_url=None, base_ref=None, recursion_level=0, url_connection=None, line=0, column=0, name=u'', anchor=u'', cache_key=u'http://guide.dexy.it'>>
      u = <local> <addinfourl at 36538992 whose fp = <socket._fileobject object at 0x2216c50>>
      u.read = <local> <bound method _fileobject.read of <socket._fileobject object at 0x2216c50>>
  File "/usr/lib/python2.7/dist-packages/linkcheck/checker/urlbase.py", line 911, in check_w3_errors
    line: dom = parseString(xml)
    locals:
      dom = <not found>
      parseString = <local> <function parseString at 0x7f351805eed8>
      xml = <local> 'Status: 403 Forbidden\nVary: Referer\nContent-type: text/html\n\n<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"\n    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">\n<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">\n  <head>\n    <meta http-equiv="Content-Type" ..., len = 2931
  File "/usr/lib/python2.7/xml/dom/minidom.py", line 1931, in parseString
    line: return expatbuilder.parseString(string)
    locals:
      expatbuilder = <local> <module 'xml.dom.expatbuilder' from '/usr/lib/python2.7/xml/dom/expatbuilder.pyc'>
      expatbuilder.parseString = <local> <function parseString at 0x7f3518071938>
      string = <local> 'Status: 403 Forbidden\nVary: Referer\nContent-type: text/html\n\n<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"\n    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">\n<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">\n  <head>\n    <meta http-equiv="Content-Type" ..., len = 2931
  File "/usr/lib/python2.7/xml/dom/expatbuilder.py", line 940, in parseString
    line: return builder.parseString(string)
    locals:
      builder = <local> <xml.dom.expatbuilder.ExpatBuilderNS instance at 0x7f3518066128>
      builder.parseString = <local> <bound method ExpatBuilderNS.parseString of <xml.dom.expatbuilder.ExpatBuilderNS instance at 0x7f3518066128>>
      string = <local> 'Status: 403 Forbidden\nVary: Referer\nContent-type: text/html\n\n<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"\n    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">\n<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">\n  <head>\n    <meta http-equiv="Content-Type" ..., len = 2931
  File "/usr/lib/python2.7/xml/dom/expatbuilder.py", line 223, in parseString
    line: parser.Parse(string, True)
    locals:
      parser = <local> <pyexpat.xmlparser object at 0x22e9b40>
      parser.Parse = <local> <built-in method Parse of pyexpat.xmlparser object at 0x22e9b40>
      string = <local> 'Status: 403 Forbidden\nVary: Referer\nContent-type: text/html\n\n<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"\n    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">\n<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">\n  <head>\n    <meta http-equiv="Content-Type" ..., len = 2931
      True = <builtin> True
ExpatError: syntax error: line 1, column 0
System info:
LinkChecker 8.4
Python 2.7.5+ (default, Sep 19 2013, 13:48:49) 
[GCC 4.8.1] on linux2

Statistics:
Downloaded: 56.22KB
Robots.txt cache: 0 hits, 1 miss
No statistics available since no URLs were checked.

That's it. 0 links checked. 0 warnings found. 0 errors found.
Stopped checking at 2014-01-09 16:59:38+000 (1 seconds)
Modules: CSSutils, Sqlite
Local time: 2014-01-09 16:59:38+000
sys.argv: ['/usr/bin/linkchecker', '--check-html', '-Dall', 'http://guide.dexy.it']
LANG = 'en_US.UTF-8'
Default locale: ('en', 'UTF-8')

 ******** LinkChecker internal error, over and out ********
WARNING 2014-01-09 16:59:38,904 CheckThread-http://guide.dexy.it internal error occurred