configure: Set libdrm flags correctly if only XORG is enabled

This fixes modesetting driver build failure which can be triggered with
the following configure options:

$ ./configure --disable-dri --disable-dri2 --disable-dri3
--disable-config-udev --enable-xorg

Bugzilla: https://bugs.gentoo.org/689768


Signed-off-by: Alexander Tsoy <alexander@tsoy.me>
Reviewed-by: Emil Velikov <emil.velikov@collabora.com>
(cherry picked from commit 334f1107)