Last active
August 7, 2017 20:27
-
-
Save johntbush/b5d6a23c8def09be2bcd4c1dbecf9b7b to your computer and use it in GitHub Desktop.
storm rate metrics logic
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| if (type = source or type is null) and queue = rate-metrics and msg is not from CDC | |
| write log in cassandra | |
| call rates API to apply patterns | |
| write doc in elasticsearch | |
| catch failure | |
| if retries not exhausted | |
| modify message to set retry flag and update num of retries | |
| publish back on rate-metrics queue | |
| ack message | |
| else | |
| write to failure log | |
| ack message | |
| if (type = source or type is null) and queue = rate-metrics and msg is from CDC | |
| lookup last source doc for this fbid in cassandra | |
| if none then stop | |
| call rates API to apply patterns | |
| verify lastpublished date in doc coming back from rates api is still most currect | |
| if not then stop | |
| write doc to elasticsearch | |
| catch failure | |
| if retries not exhausted | |
| modify message to set retry flag and update num of retries | |
| publish back on rate-metrics-update queue | |
| ack message | |
| else | |
| write to failure log | |
| ack message | |
| if type = update or queue = rate-metrics-update | |
| extract fbid for either update or cdc style message | |
| lookup last source doc for this fbid in cassandra | |
| if none then stop | |
| call rates API to apply patterns | |
| verify lastpublished date in doc coming back from rates api is still most currect | |
| if not then stop | |
| write doc to elasticsearch | |
| catch failure | |
| if retries not exhausted | |
| modify message to set retry flag and update num of retries | |
| publish back on rate-metrics-update queue | |
| ack message | |
| else | |
| write to failure log | |
| ack message | |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment