qdbus org.kde.KWin /Compositor org.kde.kwin.Compositing.active
Gives you true or false depending on whether it is currently enabled or not
You can enable compositing by calling
qdbus org.kde.KWin /Compositor org.kde.kwin.Compositing.resume
and disable by calling