core: set fs.file-max sysctl to LONG_MAX rather than ULONG_MAX

Since kernel 5.2 (but also stable kernels like 4.19.53) the kernel
thankfully returns proper errors when we write a value out of range to
the sysctl. Which however breaks writing ULONG_MAX to request the
maximum value. Hence let's write the new maximum value instead,
LONG_MAX.

Closes: #945018