core: check size before mmap

core: check size before mmap

The data type off_t can be 64 on 32 bit systems if they have large
file support. Since mmap expects a size_t with 32 bits as second
argument truncation could occur. At worst these huge files could
lead to mmaps smaller than the previous check for small files.

This in turn shouldn't have a lot of impact because mmap allocates
at page size boundaries. This also made the PAGE_ALIGN call in
open_mmap unneeded. In fact it was neither in sync with other mmap
calls nor with its own munmap counterpart in error path.

If such large files are encountered, which is very unlikely in these
code paths, treat them with the same error as if they are too small.

(cherry picked from commit 1a823cdeb9faea3849843e0b3dae0fbdd607e8b7)