Skip to content

Instantly share code, notes, and snippets.

@terriyu
Created July 23, 2013 20:02
Show Gist options
  • Select an option

  • Save terriyu/6065653 to your computer and use it in GitHub Desktop.

Select an option

Save terriyu/6065653 to your computer and use it in GitHub Desktop.
Journal for OpenStack Ceilometer work -- 16 Jul 2013

16 Jul 2013

OPW meeting

Attended the OPW meeting and introduced myself. Brought up my idea to have all the interns play an online game together and it seemed well-received. There will probably be 3 more meetings after this one.

Blog

Modified my blog to create an atom feed for posts tagged with openstack and submitted this feed http://terriyu.info/blog/feeds/tag/openstack.atom.xml to Planet OpenStack http://planet.openstack.org/

Using the . or source command

Read a little about using . versus source to run a shell script

  • This stackoverflow post explains that . is an alias for source: http://stackoverflow.com/questions/11027782/virtualenv-venv-bin-activate-vs-source-venv-bin-activate

    Also, it explains that both . <file> and source <file> run the script within the shell which is different than running ./path/to/file which runs the script in a subshell.

    The command runs the contents of the script within the current shell, and this is important in the case of activate because one of the things that the script does is exports and modifies environment variables within your current shell.

    If you were to run it using ./path/to/activate, the script will be run within a subshell and all environment variables that are set will be lost once the script completes and the subshell terminates.

  • Executing shell scripts in general: http://www.thegeekstuff.com/2010/07/execute-shell-script/

Code reviews

anteaya suggested doing code reviews:

https://review.openstack.org/#/q/status:open+project:openstack/ceilometer,n,z

jpich says this view is sorted by priority:

http://status.openstack.org/reviews/#ceilometer

Bug I'm working on, "unable to sort data with MongoDB"

Status

  • The bug report is here: https://bugs.launchpad.net/ceilometer/+bug/1193906

  • Successfully reproduced the error in the bug report:

    OperationFailure: database error: too much data for sort() with no index. add an index or specify a smaller limit
    
  • Attempted to fix the bug by creating an index for timestamp, but so far, it hasn't worked.

  • My patch so far: https://review.openstack.org/#/c/36159/

  • Debugging shows that the index is created when the MongoDB connection is initialized in __init__(), but when get_samples() is called, it doesn't see the indexes. Strangely enough, if I ran ./stack.sh and got a fresh database, I could see the indexes in get_samples().

  • Pulled newly merged changes, including jd's patch https://review.openstack.org/#/c/33290/ which uses a real MongoDB instance to run the unit tests, and applied jd's patch which tries to make sure the indexes are created in setUp() for ceilometer/tests/db.py https://review.openstack.org/#/c/36854/

    • When I ran the tests, I didn't get the sorting error anymore but small tests runs failed due to connection errors like

      so I had to run the full test suite. jd said this is because of a race condition between tests.

    • Now I don't get the sorting error bug anymore, but I also am unable to reproduce the same sorting error.

Brainstorming about how to reproduce sorting error

I asked people for some suggestions on how to reproduce the MongoDB sorting error which I had previously been able to generate, but not after all the new code I pulled down.

This is what jd said:

I don't think you'll be able to write a correct test for that bug, because anyway you'll have to flood the system with so much data that you're going to overload the tests for other backends too

at this point I think we have sufficient knowledge to just write the correct fix without more tests and move on

I tld jd that the test no longer fail with the same size database that used to generate an error and his response was:

another good reason to not try to provide a test that could start failling as soon as MongoDB changes its behaviour

This is what jpich said:

Hm... since I remember a discussion about changing the set up steps to make sure the index is created, could it be the index actually exists?

I'd probably try to make the db much bigger and see if that triggers it (maybe some other optimisation was merged in meaning the index requirement now comes later)

Did you try reverting to the commit where you could reproduce the error consistently and see if you can reproduce it again?

If it works, then you can try to figure out which commit introduce the change in behaviour and look at it to understand what caused it (there's a handy command called git bisect that can help with this)

New patchset

Even though I didn't find a way to rigorously test my sorting index patch, jd told me to submit my patch anyways.

I submitted a new patchset for the sorting error bug since jd said it was urgently needed for the Havana-2 milestone. For this patchset, I removed changes to the tests which check for MongoDB sorting errors (since they would either not work or be unreliable) and changed the call from create_index() to ensure_index() since the latter is more efficient: https://review.openstack.org/#/c/36159/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment