- An event is used to trigger the creation of a compliance resource update which is used to track details about the Spaces and Accounts which are to be used and the result of the process.
- There are N Compliance Resource worker nodes in the system which process resource updates for an account being updated. This is fanned out and each node will update all resources for a specific account.
- The resource update will query the aggregated AWS Config resources for the account, transform the data, and create or update a resource record in the system for each resources in the account.
- Creating or updated a resource record emits an event which triggers a process to review the ingested AWS config rules evaluation results for the resource and create, update, or delete violations for NON_COMPLIANT violations.
- Some violations in QA were showing incorrect compliance rule information which indicated the lack of a description, however the rule itself is queryable and has a description.
- The resource record shows the evaluated results and the transformed results which has the correct rule information and description.
- Violations with incorrect rule information should be updated to correc the rule information based on the most recent data in the ingested resource record, but they are not.
- These violations have
updatedOntimestamps which are several days old.
QA
space-compliancex2- 64 vCPU
- 256mb RAM
space-resourcex4- 64 vCPU
- 256mb RAM
Prod
space-compliancex2- 64 vCPU
- 1024mb RAM
space-resourcex4- 64 vCPU
- 256mb RAM
QA
Prod
There are no associated errors which would indicate any form of resource contention or message/event delivery failure in QA. The only thing I can conclude at this time is that there is some form of contention that is silently failing behind the scenes, perhaps related to the difference in memory for the compliance nodes which handle compliance violations, reports, and notifications.