libxsimd-dev: include only public header

libxsimd-dev contains a public header which includes architecture-specific implementations based on compiler defines. See /usr/include/xsimd/config/xsimd_config.hpp

The only relevant architecture for arm is neon and it is disabled for non 64bit arm:

#ifdef __aarch64__
#define XSIMD_WITH_NEON64 1
#else
#define XSIMD_WITH_NEON64 0
#endif
#else
#define XSIMD_WITH_NEON 0
#define XSIMD_WITH_NEON64 0
#endif

I have tried to exclude arch headers but lost some wildcard in the merge.

Probably the better option is to include only public header as it will automatically include relevant parts which would be compiled when no relevant architecture is found.