Well, ultimately music is always subjective. You can't really argue with someone who likes some particular music – they like what they like, and it would be dogmatism to condemn some composition just because it violates the theoretical frameworks† that you happen to know.
That said, I personally would probably agree with your assessment. I don't think much of music that's just too arbitrary – not because it's bad to violate the standard theory, but because music should have a purpose. This is regardless of whether that purpose can be explained within Common Practice theory, with deliberately violating the common rules, with providing a completely new theoretical framework, or something else.
In other words, “Why it couldn't go there anyways?” is just not the right question to ask to rebut criticism. The right rebuttal is to give a reason why it would go there.
If he picked that particular harmony for a reason – whatever reason – then there's that. In doubt, the reason could simply be “I liked that particular sound”, however that's not a particularly useful reason since music is more than individual sounds – it's bringing the sounds together. Again, that is possible without “knowing theory” in the sense of being able to name roman numerals etc. – in particular for melodies / solo improvisation, simple intuition often works very well. (It could be argued that this typically also follows some theoretical rules, even if the musician doesn't know the terms, but this isn't important.) It is certainly possible by toying around with short snippets, changing notes in a passage until the whole thing “works”, musically. However, if nothing else, knowing some theory allows to dramatically reduce the size of the search space. Most sequences of random tones will sound just... well, random. If everything sounds random, it is hard to convey any expression. Even if one wants to “break out” from the theory, this is much more effectful if she largely follows the theory only to then deliberately “violate” it at a specific time.
†Note that there are many completely different such frameworks – many musicians think “theory” is exclusively [Common Practice](https://en.wikipedia.org/wiki/Common_practice_period), but that's actually a very limited mindset.