This is a backup to the great anwser that I found on Unity Answers.
@slippdouglas provided a class that enables us to detect raycasts without the extra draw call inside the UI
Source: https://answers.unity.com/questions/1091618/ui-panel-without-image-component-as-raycast-target.html