test: replace the double assert macros with proper checks

Instead of value * 256 which makes for bad debug messages, expand it to a full
double test with a 1/256 epsilon.

Signed-off-by: Peter Hutterer <peter.hutterer@who-t.net>