st/nine: Fix non inversible matrix check

There was a missing absolute value when
checking if the determinant was big enough.

Fixes: https://github.com/iXit/Mesa-3D/issues/292



Signed-off-by: Axel Davy <davyaxel0@gmail.com>
Reviewed-by: Patrick Rudolph <siro@das-labor.org>
Tested-by: Dieter Nützel <Dieter@nuetzel-hh.de>

CC: "17.3 18.0" <mesa-stable@lists.freedesktop.org>
(cherry picked from commit dbc24835)