fboverlay: move bpp checks above malloc

Avoids having to free the malloced object.

Pointed out by coverity.

Signed-off-by: Dave Airlie <airlied@redhat.com>
(cherry picked from commit c7fa6a0a)