Only use -latomic with the intended architectures, instead of all of them.

This matches what was suggested in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=907106#5