Try harder to avoid 24bpp

We're preferring depth 24 even if that means 24bpp. Newer servers don't
support 24bpp anymore, and even for old servers it's suboptimal. Change
the heuristic to only try 24bpp if the alternative is pseudocolor.

Fixes: xorg/driver/xf86-video-vesa#3
(cherry picked from commit 634dae74)