Skip to content

Instantly share code, notes, and snippets.

@zuriby
Created March 6, 2011 18:49
Show Gist options
  • Save zuriby/857532 to your computer and use it in GitHub Desktop.
Save zuriby/857532 to your computer and use it in GitHub Desktop.
tail -n 250 /opt/itrust/crawlers/gsg-api/debug.log
2011-03-06 14:09:54,913-DEBUG-Got []as candidates to add, but all of them already existed
2011-03-06 14:09:54,914-INFO-creating entities: [('http://zeze0556.appspot.com/iA2', 0L), ('http://zeze0556.appspot.com/iA2/followers', 0L), ('http://zeze0556.appspot.com/iA2/following', 0L), ('http://zeze0556.appspot.com/iA2/lists/memberships', 0L)]
2011-03-06 14:09:55,120-INFO-got a new entity id: 23612
2011-03-06 14:09:55,120-DEBUG-adding a new entity: id:23612, provider: 0, name: http://zeze0556.appspot.com/iA2
2011-03-06 14:09:55,121-DEBUG-adding alias for existing entity: id:23612, provider: 0, new alias: http://zeze0556.appspot.com/iA2/followers
2011-03-06 14:09:55,122-DEBUG-adding alias for existing entity: id:23612, provider: 0, new alias: http://zeze0556.appspot.com/iA2/following
2011-03-06 14:09:55,123-DEBUG-adding alias for existing entity: id:23612, provider: 0, new alias: http://zeze0556.appspot.com/iA2/lists/memberships
2011-03-06 14:09:55,123-DEBUG-created entities: names:[('http://zeze0556.appspot.com/iA2', 0L), ('http://zeze0556.appspot.com/iA2/followers', 0L), ('http://zeze0556.appspot.com/iA2/following', 0L), ('http://zeze0556.appspot.com/iA2/lists/memberships', 0L)], created_uids:{0L: 27859L}
2011-03-06 14:09:55,124-INFO-creating entities: [('http://twitter2foaf.appspot.com/data/l_lysakowski', 0L)]
2011-03-06 14:09:55,171-INFO-got a new entity id: 23613
2011-03-06 14:09:55,172-DEBUG-adding a new entity: id:23613, provider: 0, name: http://twitter2foaf.appspot.com/data/l_lysakowski
2011-03-06 14:09:55,186-DEBUG-created entities: names:[('http://twitter2foaf.appspot.com/data/l_lysakowski', 0L)], created_uids:{0L: 27860L}
2011-03-06 14:09:55,186-INFO-creating entities: [('http://vps.zzzcn.info/iA', 0L), ('http://vps.zzzcn.info/iA/followers', 0L), ('http://vps.zzzcn.info/iA/following', 0L), ('http://vps.zzzcn.info/iA/lists/memberships', 0L)]
2011-03-06 14:09:55,382-INFO-got a new entity id: 23614
2011-03-06 14:09:55,383-DEBUG-adding a new entity: id:23614, provider: 0, name: http://vps.zzzcn.info/iA
2011-03-06 14:09:55,384-DEBUG-adding alias for existing entity: id:23614, provider: 0, new alias: http://vps.zzzcn.info/iA/followers
2011-03-06 14:09:55,385-DEBUG-adding alias for existing entity: id:23614, provider: 0, new alias: http://vps.zzzcn.info/iA/following
2011-03-06 14:09:55,385-DEBUG-adding alias for existing entity: id:23614, provider: 0, new alias: http://vps.zzzcn.info/iA/lists/memberships
2011-03-06 14:09:55,386-DEBUG-created entities: names:[('http://vps.zzzcn.info/iA', 0L), ('http://vps.zzzcn.info/iA/followers', 0L), ('http://vps.zzzcn.info/iA/following', 0L), ('http://vps.zzzcn.info/iA/lists/memberships', 0L)], created_uids:{0L: 27861L}
2011-03-06 14:09:55,386-INFO-creating entities: [('https://seiyaggg.appspot.com/iA2', 0L), ('https://seiyaggg.appspot.com/iA2/followers', 0L), ('https://seiyaggg.appspot.com/iA2/following', 0L), ('https://seiyaggg.appspot.com/iA2/lists/memberships', 0L)]
2011-03-06 14:09:55,607-INFO-got a new entity id: 23615
2011-03-06 14:09:55,608-DEBUG-adding a new entity: id:23615, provider: 0, name: https://seiyaggg.appspot.com/iA2
2011-03-06 14:09:55,609-DEBUG-adding alias for existing entity: id:23615, provider: 0, new alias: https://seiyaggg.appspot.com/iA2/followers
2011-03-06 14:09:55,609-DEBUG-adding alias for existing entity: id:23615, provider: 0, new alias: https://seiyaggg.appspot.com/iA2/following
2011-03-06 14:09:55,610-DEBUG-adding alias for existing entity: id:23615, provider: 0, new alias: https://seiyaggg.appspot.com/iA2/lists/memberships
2011-03-06 14:09:55,611-DEBUG-created entities: names:[('https://seiyaggg.appspot.com/iA2', 0L), ('https://seiyaggg.appspot.com/iA2/followers', 0L), ('https://seiyaggg.appspot.com/iA2/following', 0L), ('https://seiyaggg.appspot.com/iA2/lists/memberships', 0L)], created_uids:{0L: 27862L}
2011-03-06 14:09:55,611-INFO-creating entities: [('http://ameosly.appspot.com/iA/followers', 0L), ('http://ameosly.appspot.com/iA/following', 0L), ('http://ameosly.appspot.com/iA/lists/memberships', 0L), ('http://informationarchitects.jp/', 0L)]
2011-03-06 14:09:55,813-DEBUG-Finding entity by name http://informationarchitects.jp/, resulted in 26919
2011-03-06 14:09:55,814-WARNING-Notice! found a problematic provider: http://informationarchitects.jp/
2011-03-06 14:09:55,815-DEBUG- since there is an unknown provider with the id, creating a new entity id: 23616 (next msg is misleading)
2011-03-06 14:09:55,815-DEBUG- using an already existing entity_id: 23616
2011-03-06 14:09:55,815-DEBUG-adding a new entity: id:23616, provider: 0, name: http://ameosly.appspot.com/iA/followers
2011-03-06 14:09:55,816-DEBUG-adding alias for existing entity: id:23616, provider: 0, new alias: http://ameosly.appspot.com/iA/following
2011-03-06 14:09:55,817-DEBUG-adding alias for existing entity: id:23616, provider: 0, new alias: http://ameosly.appspot.com/iA/lists/memberships
2011-03-06 14:09:55,817-DEBUG-adding alias for existing entity: id:23616, provider: 0, new alias: http://informationarchitects.jp/
2011-03-06 14:09:55,818-DEBUG-created entities: names:[('http://ameosly.appspot.com/iA/followers', 0L), ('http://ameosly.appspot.com/iA/following', 0L), ('http://ameosly.appspot.com/iA/lists/memberships', 0L), ('http://informationarchitects.jp/', 0L)], created_uids:{0L: 27863L}
2011-03-06 14:09:55,818-ERROR-TBD! lowering the confidence of the problem urls: ['http://informationarchitects.jp/']
2011-03-06 14:09:55,818-DEBUG-lowering confidence for alias http://informationarchitects.jp/, query: UPDATE aliases SET confidence=confidence/2 WHERE name="http://informationarchitects.jp/"
2011-03-06 14:09:56,232-INFO-creating entities: [('http://xiaodoupao.yinjian.apigee.com/iA', 0L), ('http://xiaodoupao.yinjian.apigee.com/iA/followers', 0L), ('http://xiaodoupao.yinjian.apigee.com/iA/following', 0L), ('http://xiaodoupao.yinjian.apigee.com/iA/lists/memberships', 0L)]
2011-03-06 14:09:56,437-INFO-got a new entity id: 23617
2011-03-06 14:09:56,437-DEBUG-adding a new entity: id:23617, provider: 0, name: http://xiaodoupao.yinjian.apigee.com/iA
2011-03-06 14:09:56,439-DEBUG-adding alias for existing entity: id:23617, provider: 0, new alias: http://xiaodoupao.yinjian.apigee.com/iA/followers
2011-03-06 14:09:56,439-DEBUG-adding alias for existing entity: id:23617, provider: 0, new alias: http://xiaodoupao.yinjian.apigee.com/iA/following
2011-03-06 14:09:56,440-DEBUG-adding alias for existing entity: id:23617, provider: 0, new alias: http://xiaodoupao.yinjian.apigee.com/iA/lists/memberships
2011-03-06 14:09:56,440-DEBUG-created entities: names:[('http://xiaodoupao.yinjian.apigee.com/iA', 0L), ('http://xiaodoupao.yinjian.apigee.com/iA/followers', 0L), ('http://xiaodoupao.yinjian.apigee.com/iA/following', 0L), ('http://xiaodoupao.yinjian.apigee.com/iA/lists/memberships', 0L)], created_uids:{0L: 27864L}
2011-03-06 14:09:56,440-INFO-creating entities: []
2011-03-06 14:09:56,441-INFO-got a new entity id: 23618
2011-03-06 14:09:56,441-DEBUG-created entities: names:[], created_uids:{}
2011-03-06 14:09:56,441-DEBUG-Got []as candidates to add, but all of them already existed
2011-03-06 14:09:56,441-INFO-creating entities: [('http://explosivebigbang.appspot.com/iA/followers', 0L), ('http://explosivebigbang.appspot.com/iA/following', 0L), ('http://explosivebigbang.appspot.com/iA/lists/memberships', 0L), ('http://informationarchitects.jp/', 0L)]
2011-03-06 14:09:56,632-DEBUG-Finding entity by name http://informationarchitects.jp/, resulted in 26919
2011-03-06 14:09:56,632-WARNING-Notice! found a problematic provider: http://informationarchitects.jp/
2011-03-06 14:09:56,633-DEBUG- since there is an unknown provider with the id, creating a new entity id: 23618 (next msg is misleading)
2011-03-06 14:09:56,633-DEBUG- using an already existing entity_id: 23618
2011-03-06 14:09:56,634-DEBUG-adding a new entity: id:23618, provider: 0, name: http://explosivebigbang.appspot.com/iA/followers
2011-03-06 14:09:56,635-DEBUG-adding alias for existing entity: id:23618, provider: 0, new alias: http://explosivebigbang.appspot.com/iA/following
2011-03-06 14:09:56,636-DEBUG-adding alias for existing entity: id:23618, provider: 0, new alias: http://explosivebigbang.appspot.com/iA/lists/memberships
2011-03-06 14:09:56,636-DEBUG-adding alias for existing entity: id:23618, provider: 0, new alias: http://informationarchitects.jp/
2011-03-06 14:09:56,637-DEBUG-created entities: names:[('http://explosivebigbang.appspot.com/iA/followers', 0L), ('http://explosivebigbang.appspot.com/iA/following', 0L), ('http://explosivebigbang.appspot.com/iA/lists/memberships', 0L), ('http://informationarchitects.jp/', 0L)], created_uids:{0L: 27865L}
2011-03-06 14:09:56,637-ERROR-TBD! lowering the confidence of the problem urls: ['http://informationarchitects.jp/']
2011-03-06 14:09:56,637-DEBUG-lowering confidence for alias http://informationarchitects.jp/, query: UPDATE aliases SET confidence=confidence/2 WHERE name="http://informationarchitects.jp/"
2011-03-06 14:09:57,051-INFO-creating entities: []
2011-03-06 14:09:57,052-INFO-got a new entity id: 23619
2011-03-06 14:09:57,052-DEBUG-created entities: names:[], created_uids:{}
2011-03-06 14:09:57,052-DEBUG-Got []as candidates to add, but all of them already existed
2011-03-06 14:09:57,052-INFO-creating entities: [('http://assets.chencheng.org/iA/followers', 0L), ('http://assets.chencheng.org/iA/following', 0L), ('http://assets.chencheng.org/iA/lists/memberships', 0L), ('http://informationarchitects.jp/', 0L)]
2011-03-06 14:09:57,270-DEBUG-Finding entity by name http://informationarchitects.jp/, resulted in 26919
2011-03-06 14:09:57,271-WARNING-Notice! found a problematic provider: http://informationarchitects.jp/
2011-03-06 14:09:57,271-DEBUG- since there is an unknown provider with the id, creating a new entity id: 23619 (next msg is misleading)
2011-03-06 14:09:57,271-DEBUG- using an already existing entity_id: 23619
2011-03-06 14:09:57,272-DEBUG-adding a new entity: id:23619, provider: 0, name: http://assets.chencheng.org/iA/followers
2011-03-06 14:09:57,273-DEBUG-adding alias for existing entity: id:23619, provider: 0, new alias: http://assets.chencheng.org/iA/following
2011-03-06 14:09:57,274-DEBUG-adding alias for existing entity: id:23619, provider: 0, new alias: http://assets.chencheng.org/iA/lists/memberships
2011-03-06 14:09:57,275-DEBUG-adding alias for existing entity: id:23619, provider: 0, new alias: http://informationarchitects.jp/
2011-03-06 14:09:57,275-DEBUG-created entities: names:[('http://assets.chencheng.org/iA/followers', 0L), ('http://assets.chencheng.org/iA/following', 0L), ('http://assets.chencheng.org/iA/lists/memberships', 0L), ('http://informationarchitects.jp/', 0L)], created_uids:{0L: 27866L}
2011-03-06 14:09:57,275-ERROR-TBD! lowering the confidence of the problem urls: ['http://informationarchitects.jp/']
2011-03-06 14:09:57,275-DEBUG-lowering confidence for alias http://informationarchitects.jp/, query: UPDATE aliases SET confidence=confidence/2 WHERE name="http://informationarchitects.jp/"
2011-03-06 14:09:57,706-INFO-creating entities: []
2011-03-06 14:09:57,707-INFO-got a new entity id: 23620
2011-03-06 14:09:57,707-DEBUG-created entities: names:[], created_uids:{}
2011-03-06 14:09:57,707-DEBUG-Got []as candidates to add, but all of them already existed
2011-03-06 14:09:57,708-INFO-creating entities: [('http://informationarchitects.jp/', 0L), ('http://twitter.com/account/redirect_by_id?id=2087371', 1L), ('http://twitter.com/ia', 1L), ('http://www.quora.com/oliver-reichenstein', 3L), ('http://www.facebook.com/profile.php?id=500470034', 18L), ('http://www.facebook.com/reichenstein', 18L)]
2011-03-06 14:09:57,754-DEBUG-Finding entity by name http://informationarchitects.jp/, resulted in 26919
2011-03-06 14:09:57,809-DEBUG-Finding entity by name http://twitter.com/account/redirect_by_id?id=2087371, resulted in 26914
2011-03-06 14:09:57,862-DEBUG-Finding entity by name http://twitter.com/ia, resulted in 26914
2011-03-06 14:09:57,912-DEBUG-Finding entity by name http://www.quora.com/oliver-reichenstein, resulted in 26915
2011-03-06 14:09:57,962-DEBUG-Finding entity by name http://www.facebook.com/profile.php?id=500470034, resulted in 26916
2011-03-06 14:09:58,011-DEBUG-Finding entity by name http://www.facebook.com/reichenstein, resulted in 26916
2011-03-06 14:09:58,012-WARNING-Notice! found a problematic provider: http://informationarchitects.jp/
2011-03-06 14:09:58,014-DEBUG- using an already existing entity_id: 23207
2011-03-06 14:09:58,014-DEBUG-adding alias for existing entity: id:23207, provider: 0, new alias: http://informationarchitects.jp/
2011-03-06 14:09:58,015-DEBUG-created entities: names:[('http://informationarchitects.jp/', 0L), ('http://twitter.com/account/redirect_by_id?id=2087371', 1L), ('http://twitter.com/ia', 1L), ('http://www.quora.com/oliver-reichenstein', 3L), ('http://www.facebook.com/profile.php?id=500470034', 18L), ('http://www.facebook.com/reichenstein', 18L)], created_uids:{0L: 26919L}
2011-03-06 14:09:58,015-INFO-creating entities: [('http://informationarchitects.jp/', 0L), ('http://www.google.com/reader/shared/03307759438856562429', 0L), ('http://www.qwerly.com/profiles/reichenstein/contacts', 0L), ('http://www.qwerly.com/twitter/reichenstein', 0L), ('http://twitter.com/account/redirect_by_id?id=2087371', 1L), ('http://twitter.com/ia', 1L), ('http://twitter.com/reichenstein', 1L), ('http://www.linkedin.com/in/informationarchitect', 2L), ('http://www.quora.com/oliver-reichenstein', 3L), ('http://friendfeed.com/reichenstein', 6L), ('http://www.flickr.com/photos/35521318@N00/', 9L), ('http://www.flickr.com/photos/formforce/', 9L), ('http://www.facebook.com/profile.php?id=500470034', 18L), ('http://www.facebook.com/reichenstein', 18L)]
2011-03-06 14:09:58,061-DEBUG-Finding entity by name http://informationarchitects.jp/, resulted in 26919
2011-03-06 14:09:58,113-DEBUG-Finding entity by name http://www.google.com/reader/shared/03307759438856562429, resulted in 26919
2011-03-06 14:09:58,287-DEBUG-Finding entity by name http://twitter.com/account/redirect_by_id?id=2087371, resulted in 26914
2011-03-06 14:09:58,343-DEBUG-Finding entity by name http://twitter.com/ia, resulted in 26914
2011-03-06 14:09:58,399-DEBUG-Finding entity by name http://twitter.com/reichenstein, resulted in 26917
2011-03-06 14:09:58,448-DEBUG-Finding entity by name http://www.linkedin.com/in/informationarchitect, resulted in 26920
2011-03-06 14:09:58,500-DEBUG-Finding entity by name http://www.quora.com/oliver-reichenstein, resulted in 26915
2011-03-06 14:09:58,555-DEBUG-Finding entity by name http://friendfeed.com/reichenstein, resulted in 26921
2011-03-06 14:09:58,614-DEBUG-Finding entity by name http://www.flickr.com/photos/35521318@N00/, resulted in 26922
2011-03-06 14:09:58,668-DEBUG-Finding entity by name http://www.flickr.com/photos/formforce/, resulted in 26922
2011-03-06 14:09:58,726-DEBUG-Finding entity by name http://www.facebook.com/profile.php?id=500470034, resulted in 26916
2011-03-06 14:09:58,783-DEBUG-Finding entity by name http://www.facebook.com/reichenstein, resulted in 26916
2011-03-06 14:09:58,783-WARNING-Notice! found a problematic provider: http://informationarchitects.jp/
2011-03-06 14:09:58,784-WARNING-Notice! found a problematic provider: http://www.google.com/reader/shared/03307759438856562429
connected to db.
connected to db.
for key url, preferring value http://twitter.com/kenloojp over http://twitter.com/account/redirect_by_id?id=55843629
TBD! string has a strange locale...cannot insert attr fn:DoDGeR The Best ALL Or NothinG"
TBD! string has a strange locale...cannot insert attr fn:DoDGeR The Best ALL Or NothinG"
for key url, preferring value http://twitter.com/april301995 over http://twitter.com/account/redirect_by_id?id=36037779
for key url, preferring value http://twitter.com/navarrowwright over http://twitter.com/account/redirect_by_id?id=24753139
TBD! string has a strange locale...cannot insert attr fn:"Little Soya" Premium Soy Sauce
for key url, preferring value http://twitter.com/electronreviews over http://twitter.com/account/redirect_by_id?id=145846911
for key url, preferring value http://twitter.com/economistjane over http://twitter.com/account/redirect_by_id?id=132567722
for key url, preferring value http://twitter.com/philbuechler over http://twitter.com/account/redirect_by_id?id=14838584
Traceback (most recent call last):
File "server_parse.py", line 692, in <module>
main()
File "server_parse.py", line 688, in main
if not process(options, args):
File "server_parse.py", line 651, in process
process_entities()
File "server_parse.py", line 575, in process_entities
created_uids = create_entity( names, is_explicit=1 )
File "server_parse.py", line 272, in create_entity
entity_id = merge_entities_and_get_entity_id( entities_found )
File "server_parse.py", line 162, in merge_entities_and_get_entity_id
raise Exception("merge_entities_and_get_entity_id() not implemented yet!! called with: "+str(entities_found))
Exception: merge_entities_and_get_entity_id() not implemented yet!! called with: [('http://twitter.com/account/redirect_by_id?id=2087371', 1L, 26914L, 23207L, 50), ('http://twitter.com/ia', 1L, 26914L, 23207L, 50), ('http://twitter.com/reichenstein', 1L, 26917L, 23208L, 50), ('http://www.linkedin.com/in/informationarchitect', 2L, 26920L, 23207L, 50), ('http://www.quora.com/oliver-reichenstein', 3L, 26915L, 23207L, 50), ('http://friendfeed.com/reichenstein', 6L, 26921L, 23207L, 50), ('http://www.flickr.com/photos/35521318@N00/', 9L, 26922L, 23207L, 50), ('http://www.flickr.com/photos/formforce/', 9L, 26922L, 23207L, 50), ('http://www.facebook.com/profile.php?id=500470034', 18L, 26916L, 23207L, 50), ('http://www.facebook.com/reichenstein', 18L, 26916L, 23207L, 50)]
# clear __builtin__._
# clear sys.path
# clear sys.argv
# clear sys.ps1
# clear sys.ps2
# clear sys.exitfunc
# clear sys.exc_type
# clear sys.exc_value
# clear sys.exc_traceback
# clear sys.last_type
# clear sys.last_value
# clear sys.last_traceback
# clear sys.path_hooks
# clear sys.path_importer_cache
# clear sys.meta_path
# clear sys.flags
# clear sys.float_info
# restore sys.stdin
# restore sys.stdout
# restore sys.stderr
# cleanup __main__
# cleanup[1] _json
# cleanup[1] collections
# cleanup[1] zipimport
# cleanup[1] signal
# cleanup[1] decimal
# cleanup[1] abc
# cleanup[1] urllib
# cleanup[1] optparse
# cleanup[1] exceptions
# cleanup[1] _functools
# cleanup[1] _locale
# cleanup[1] socket
# cleanup[1] weakref
# cleanup[1] __future__
# cleanup[1] _collections
# cleanup[1] _socket
# cleanup[1] json
# cleanup[1] _warnings
# cleanup[1] _codecs
# cleanup[1] _struct
# cleanup[1] keyword
# cleanup[1] zlib
# cleanup[1] posix
# cleanup[1] numbers
# cleanup[1] json.decoder
# cleanup[1] site
# cleanup[1] _mysql_exceptions
# cleanup[1] strop
# cleanup[1] gettext
# cleanup[1] _sqlite3
# cleanup[1] sitecustomize
# cleanup[1] database
# cleanup[1] _weakref
# cleanup[1] urlparse
# cleanup[1] ssl
# cleanup[1] json.encoder
# cleanup[1] locale
# cleanup[1] MySQLdb
# cleanup[1] encodings
# cleanup[1] math
# cleanup[1] json.scanner
# cleanup[1] logging
# cleanup[1] thread
# cleanup[1] traceback
# cleanup[1] itertools
# cleanup[1] operator
# cleanup[1] encodings.unicode_escape
# cleanup[1] sre_constants
# cleanup[1] MySQLdb.converters
# cleanup[1] copy
# cleanup[1] atexit
# cleanup[1] encodings.aliases
# cleanup[1] MySQLdb.release
# cleanup[1] sqlite3
# cleanup[1] _ssl
# cleanup[1] textwrap
# cleanup[1] MySQLdb.constants
# cleanup[1] MySQLdb.constants.CLIENT
# cleanup[1] MySQLdb.constants.FIELD_TYPE
# cleanup[1] functools
# cleanup[1] base64
# cleanup[1] MySQLdb.times
# cleanup[1] string
# cleanup[1] encodings.utf_8
# cleanup[1] MySQLdb.connections
# cleanup[1] threading
# cleanup[1] cStringIO
# cleanup[1] codecs
# cleanup[1] MySQLdb.constants.FLAG
# cleanup[1] sqlite3.dbapi2
# cleanup[1] array
# cleanup[1] binascii
# cleanup[1] MySQLdb.cursors
# cleanup[1] _mysql
# cleanup[1] time
# cleanup[1] datetime
# cleanup[1] struct
# cleanup[1] re
# cleanup[1] sre_compile
# cleanup[1] _sre
# cleanup[1] sre_parse
# cleanup[2] UserDict
# cleanup[2] os
# cleanup[2] posixpath
# cleanup[2] errno
# cleanup[2] os.path
# cleanup[2] copy_reg
# cleanup[2] _abcoll
# cleanup[2] genericpath
# cleanup[2] stat
# cleanup[2] warnings
# cleanup[2] types
# cleanup[2] linecache
# cleanup sys
# cleanup __builtin__
# cleanup ints: 48 unfreed ints
# cleanup floats: 5 unfreed floats
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment