-
Everyone is trying to get work done for Havana-2
-
Attended and introduced myself as jd's intern. Made known my interest in learning how OpenStack can be used for science.
To escape a special character like #, use the backslash \. So in Markdown,
# should be typed as \#.
Reference: http://docs.moodle.org/25/en/Advanced_use_of_Markdown#Using_Special_Characters
-
Following jpich's suggestion, I tried using
printstatements to debug while runningtoxinstead of usingassert. Theprintstatements would allow be to see the value of a variable without stopping the executation of a program. -
The print statements don't appear in my terminal window when I run
tox. It probably has something to with the fact thatprint, by default, directs its output tosys.stdout -
Even a simple
print "hello"doesn't show up on the screen.
-
The bug report is here: https://bugs.launchpad.net/ceilometer/+bug/1193906
-
Successfully reproduced the error in the bug report:
OperationFailure: database error: too much data for sort() with no index. add an index or specify a smaller limit -
Attempted to fix the bug by creating an index for timestamp, but so far, it hasn't worked.
-
My patch so far: https://review.openstack.org/#/c/36159/
-
Debugging shows that the index is created when the MongoDB connection is initialized in
__init__(), but whenget_samples()is called, it doesn't see the indexes. Strangely enough, if I ran./stack.shand got a fresh database, I could see the indexes inget_samples(). -
Tried jd's patch https://review.openstack.org/#/c/33290/ which uses a real MongoDB instance to run the unit tests, but that didn't work either.
- After applying jd's patch, I had weird connection errors when I ran the tests. My quick hack was to manually change the port number each time I ran tests.
I tried explicitly specifying an descending index, even though the MongoDB documentation says this isn't necessary.
self.db.meter.create_index([('timestamp', pymongo.DESCENDING)], name='timestamp_idx')
Unfortunately, nothing changed. I still got the sorting error:
OperationFailure: database error: too much data for sort() with no index. add an index or specify a smaller limit
jd looked at PyMongo and tried to find some leads for what could be wrong.
I found that if you try to create indexes on a inexistent database, mongo doesn't tell you so and nods
so I'm checking if we don't do that by mistake
but it seems not so far :/
-
jd had another idea for what might be wrong
long story short, I think there's a race condition between the tests; test #1 starts, and then calls dropDatabase("ceilometer") which returns immediately; test #2 starts and creates the index, but NOW MongoDB starts dropping the database that test #1 requested… and now test#2 is going to call get_samples, but the db is empty, and the indexes are gone :(
so adding a sleep() should prove I'm correct since that should leave enough time for mongodb to drop the database for real
a correct fix would be to block dropDatabase() until the db is actually dropped
-
jd suggested making a change to the code in
ceilometer/storage/impl_mongodb.pyTo implement the sleep command, add these lines of code after
self.conn.drop_database(self.db)inceilometer/storage/impl_mongodb.pyimport time time.sleep(5)which causes execution to be suspended for five seconds after dropping the database.
See jd's gist: https://gist.github.com/jd/5977213
Reference for
sleep(): http://docs.python.org/2/library/time.html#time.sleep -
Unfortunately, this doesn't work either. The connection errors (e.g. https://gist.github.com/terriyu/5973022) go away sometimes, but this is only an intermittent fix. Most importantly, I still get the sorting error:
OperationFailure: database error: too much data for sort() with no index. add an index or specify a smaller limit -
Try increasing the sleep time to 50 seconds. Doesn't change anything. I recorded the connection errors for a sequence of tox runs:
oooXooooXo= okay,X= connection error