So, most of the code that sets text-properties needs to be careful to reset the modified-p flag afterwards. The best way to do that is usually by wrapping the code that sets the properties inside a with-silent-modification.
One way try to track down the culprit is by trying to undo the modification (e.g. with C-z), but of course, if the modification is not visible, undoing it won't be visible either. So instead you may want to look at C-h v buffer-undo-list RET which is the internal data used to keep track of the modifications. With luck, not only was the modified-p set but the undo-list as well, and that list will tell you what was changed. For example, that list could look like (nil (nil face nil 12345708 . 12345713)) which means that the change was to set the face property to a new value between positions 12345708 and 12345713 and that the old value of that property was nil (that's the 3rd nil in the above). Sometimes looking at the affected positions with M-: (goto-char 12345708) RET is sufficient to figure out who's to blame. Othertimes looking at M-: (get-text-property 12345708 'face) RET, which gives you the new value that was set, is more useful.