Journal for OpenStack Ceilometer work -- 12 Jul 2013

diary-12jul2013.md

12 Jul 2013

Blog

• Todo: add my blog to http://planet.openstack.org/ through this link: https://wiki.openstack.org/wiki/AddingYourBlog

  (Setup an OpenStack feed for my blog, too.)

• Todo: brainstorm OpenStack-related topics for my blog

  Some links to look at for ideas --

  • http://www.openstack.org/blog/2013/06/lessons-learning-and-long-views-for-internship-program/
  • http://anteaya.info/
  • http://ladquin.dreamwidth.org/
  • http://vmartinezdelacruz.com/

Travel support program

• The new OpenStack travel support program has been announced: http://www.openstack.org/blog/2013/07/the-all-new-openstack-travel-support-program/

• Todo: Find out more about the selection criteria

• List of PTLs: https://wiki.openstack.org/wiki/PTL_Elections_Spring_2013

• Maybe ask jd to recommend me for the program?

Dependencies installed by hand for Devstack and Ceilometer

I noticed that when I was installing Devstack and Ceilometer, I had to install the following dependencies manually:

• MongoDB

  $ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10
  $ echo 'deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen' | sudo tee /etc/apt/sources.list.d/10gen.list
  $ sudo apt-get update
  $ sudo apt-get install mongodb-10gen
  

• Dependencies for Spidermonkey

  $ sudo apt-get install pkg-config libnspr4-dev
  

• The dependency libxslt

  $ sudo apt-get install pkg-config libnspr4-dev
  
  $ sudo apt-get install libxml2-dev libxslt1-dev
  

  The libxml2-dev isn't strictly necessary, but I put it in there just in case. I haven't had to install libxslt1-dev before. This is a new error: https://gist.github.com/terriyu/5989746

• (optional) Dependency to fix GnomeKeyring errors

  $ sudo apt-get install gir1.2-gnomekeyring-1.0
  

General advice for working on stubborn bugs

jpich's advice:

You can beat them into submission with a bigger hammer, go work on something else for a while, try a completely different approach, ask for help/other people's ideas

It's ok to work on multiple things in parallel. Sometimes you get an epiphany will solving a completely unrelated problem

when a problem become too frustrating, completing something else, before getting back to it can feel very rewarding (and good for morale!)

Bug I'm working on, "unable to sort data with MongoDB"

Status

• The bug report is here: https://bugs.launchpad.net/ceilometer/+bug/1193906

• Successfully reproduced the error in the bug report:

  OperationFailure: database error: too much data for sort() with no index. add an index or specify a smaller limit
  

• Attempted to fix the bug by creating an index for timestamp, but so far, it hasn't worked.

• My patch so far: https://review.openstack.org/#/c/36159/

• Debugging shows that the index is created when the MongoDB connection is initialized in __init__(), but when get_samples() is called, it doesn't see the indexes. Strangely enough, if I ran ./stack.sh and got a fresh database, I could see the indexes in get_samples().

• Tried jd's patch https://review.openstack.org/#/c/33290/ which uses a real MongoDB instance to run the unit tests, but that didn't work either.

  • After applying jd's patch, I had weird connection errors when I ran the tests. My quick hack was to manually change the port number each time I ran tests.
  • jd then suggested I try adding a command for Python to sleep for 5 seconds for clear() in ceilometer/storage/impl_mongodb.py. He thought that if there is a race condition between tests, the sleep command would fix this. Unfortunately, even with the addition of sleep(), the sorting error still occurs, but the connection errors intermittently disappear. Increasing the sleep time to 50 seconds doesn't seem to change anything.
  • Bottom line: the patch hasn't fixed the sorting error bug.

jd's new idea for why the bug is occurring

jd thinks that in the setUp() function for the test database class defined in ceilometer/tests/db.py, the setup is destroying the MongoDB indexes in the last line self.conn.clear()

if you check https://github.com/openstack/ceilometer/blob/master/ceilometer/tests/db.py#L34

that's the base test class for all database tests

what it does is that it gets a connection, then upgrade() it so its schema or whatever is installed

and then it clear() everything inside the database

upgrade() is a no-op in mongodb (no schema), but clear destroys *everything* (including the indexes)

"no operation", does noting

so what happens is: get_connection() calls __init__ from impl_mongodb, which creates the index, and then upgrade() is called, it does nothing, and then clear() is called, and it destroys everything and the indexes

I asked jd why you would "do nothing" and then "destroy everything"

actually that was designed for SQL mainly

SQL needs to upgrade() (= install the tables schema) and then clear() (drop everything from the tables)

you can DELETE FROM in SQL if you have no tables

so we make sure we have tables installed via upgrade() before we clear()

(if tables are already intalled, upgrade() does nothing)

This base test database class was written for all types of databases (currently Ceilometer supports SQLAlchemy, HBase, and MongoDB), so that's why we have a generic function upgrade() which installs the schema in SQL, but does nothing in MongoDB.

jd's patch for the base test database class

To fix this possible (but unconfirmed) bug where setUp() for the test database class in ceilometer/tests/db.py destroys the MongoDB indexes, jd made a new patch "storage: fix clear/upgrade order": https://review.openstack.org/#/c/36854/

I've also changed upgrade() so it is now responsible for creating the indexes in impl_mongodb

that's why it now helps :)

so now it does "create a connection", "create index", "destroy everything", "create index" in the case of MongoDB

I pointed out to jd that calling upgrade() twice in setUp() seems inefficient. jd says:

not really, but since it's our unit test case, we don't really care about efficiency -- the production workflow is different, there's no call for clear nor upgrade

and since we always start from scratch, creating an index on an empty collection takes no time

Putting all the patches together

In order to see if jd's new patch for the test database class ("storage: fix clear/upgrade order") fixes the sorting error, I have to put together several patches. This is what I tried:

1. Create a new git branch from the master branch
2. Cherry pick the fix "clear/upgrade order" patch: https://review.openstack.org/#/c/36854/
3. Cherry pick the "use a real MongoDB instance to run unit tests" patch: https://review.openstack.org/#/c/33290/
4. Cherry pick my "add index for db.meter by descending timestamp" patch: https://review.openstack.org/#/c/36159/
5. Fix merging conflicts

This is the history of commands I used:

$ git checkout master
$ git checkout -b bug/1193906
$ git fetch https://review.openstack.org/openstack/ceilometer refs/changes/54/36854/2 && git cherry-pick FETCH_HEAD
$ git fetch https://review.openstack.org/openstack/ceilometer refs/changes/90/33290/21 && git cherry-pick FETCH_HEAD
$ git fetch https://review.openstack.org/openstack/ceilometer refs/changes/59/36159/2 && git cherry-pick FETCH_HEAD

Sorting error

Unfortunately, the sorting error is still present.

OperationFailure: database error: too much data for sort() with no index. add an index or specify a smaller limit

If I put in an assert False, self.db.meter.index_information() statement in the get_samples() function inside ceilometer/storage/impl_mongodb.py, I see that the value of index_information() is still wrong and doesn't contain any of the user-defined indexes:

{u'_id_': {u'key': [(u'_id', 1)], u'v': 1}}