Journal for OpenStack Ceilometer work -- 10 Jul 2013

diary-10jul2013.md

10 Jul 2013

Simultaneously viewing output on screen and saving it to file

• A useful thing to do is to run a command and output stdout+stderr to screen while simultaneously writing the same stdout+stderr to a file.

  <command> 2>&1 | tee <file>
  

  Reference: http://www.skorks.com/2009/09/using-bash-to-output-to-screen-and-file-at-the-same-time/

Debugging while running tox

• Ceilometer uses the tox Python testing library to run its tests. It's not clear how to debug the code while running tox.

• So far, I've been throwing in assert statements. The simplest way to get the value of a variable is to put in a line of code

  assert False, variable
  

• jpich suggested a few other ways of doing it

  • print statements

  • Pdb - the standard Python Debugger

    Where you want the test to stop, you can type "import pdb ; pdb.set_trace()"

• jpich mentions some common useful commands in Pdb

  print my_variable # shows what's in the variable

  l / list # shows the code around the line you're currently running, can be used multiple times

  s / step # move into the next executed function / line

  n / next # go to the next line down (to avoid stepping inside a function, e.g. avoiding accidentally going into Python internals like getattr() if you only care about the function result)

  c / continue # leave pdb and continue running the test normally (or until the next breakpoint)

  Reference for Pdb commands: http://docs.python.org/2/library/pdb.html#debugger-commands

• vkmc has written a blog post about debugging in OpenStack: http://vmartinezdelacruz.com/logging-and-debugging-in-openstack/

Bug I'm working on, "unable to sort data with MongoDB"

Status

• The bug report is here: https://bugs.launchpad.net/ceilometer/+bug/1193906

• Successfully reproduced the error in the bug report:

  OperationFailure: database error: too much data for sort() with no index. add an index or specify a smaller limit
  

• Attempted to fix the bug by creating an index for timestamp, but so far, it hasn't worked.

• Debugging shows that the index is created when the MongoDB connection is initialized in __init__(), but when get_samples() is called, it doesn't see the indexes.

• My patch so far: https://review.openstack.org/#/c/36159/

Further debugging

• jd said that trying to debug indexes in MIM is pointless because MIM doesn't use real indexes.

• jd has no idea why the index isn't working. He speculated that it could have to do with get_sample() using a different MongoDB connection than the one that has the indexes.

• jd suggested going into the Mongo shell and seeing if it can see the indexes.

  • In the Mongo shell, you can use the method getIndexes() to get a list of the existing indexes on a collection. Reference: http://docs.mongodb.org/manual/reference/method/db.collection.getIndexes/

  • If I run my test in tox and go into the Mongo shell afterwards, I don't see the indexes in the shell: https://gist.github.com/terriyu/6fd393a783813edc2006#file-1-after-running-tox-the-indexes-aren-t-there

  • If I run ./stack.sh and get a fresh database and then run the Mongo shell, I do see the indexes in the shell: https://gist.github.com/terriyu/6fd393a783813edc2006#file-2-after-running-stack-sh-we-get-a-fresh-database-and-the-indexes-come-back

  • After having run ./stack.sh, if I then run tox again, then the indexes disappear when I look in the Mongo shell: https://gist.github.com/terriyu/6fd393a783813edc2006#file-3-run-tox-after-running-stack-sh-and-notice-the-indexes-disappear-again

Trying jd's patch: "use a real MongoDB instance to run unit tests"

• jd suggested that I try using his patch: https://review.openstack.org/#/c/33290/

  The advantage of this patch is that spawns a brand new MongoDB instance each time you run tox.

• I cherrypicked his patch and put my patch on top of it. Unfortunately, if I use the same port more than once (e.g. "29000"), then I get a connection error. My temporary hack is to edit the file run-tests.sh to use a different port for each tox run.

  My ugly procedure was something like: 1) I used port 29000, 2) Oops, used port 29000, now make it port 29001, 3) Ran a test and used port 29001, now let's change it to port 29002, 4) Run another test... and keep incrementing the port number by +1 each time I run a test.

  Example of a connection failure: https://gist.github.com/terriyu/5973022

• I used assert statements to find out the value of db.meter.index_information()

  • Inside __init__(), the value was

    {u'_id_': {u'key': [(u'_id', 1)], u'v': 1}, u'timestamp_idx': {u'key': [(u'timestamp', 1)], u'v': 1}, u'meter_idx': {u'key': [(u'resource_id', 1), (u'user_id', 1), (u'counter_name', 1), (u'timestamp', 1), (u'source', 1)], u'v': 1}}
    

  • Inside get_samples(), the value was:

    {u'_id_': {u'key': [(u'_id', 1)], u'v': 1}}
    

  Nothing has changed. Again, the user-defined indexes are not visible to get_samples().

• You can't use the Mongo shell to access this connection. I think this is because the connection is killed after the tests in tox are completed.

• As expected, if you run the test with a large database, using

  timestamps_for_test_samples_default_order = timestamps_for_test_samples_default_order*15000
  

  in tests/storage/base.py, then running the tests with

  `$ tox -e py27 -- tests.storage.test_impl_mongodb.RawSampleTest.test_get_samples_in_default_order` and record the results.
  

  gives the sorting error again:

  OperationFailure: database error: too much data for sort() with no index.  add an index or specify a smaller limit
  

  The entire output from tox is here: https://gist.github.com/terriyu/5972794