btrfs-util,tmpfiles: cleanup use of ERRNO_IS_NOT_SUPPORTED()

btrfs-util,tmpfiles: cleanup use of ERRNO_IS_NOT_SUPPORTED()

Given that ERRNO_IS_NOT_SUPPORTED() also matches positive values,
make sure this macro is not called with arguments that do not have
errno semantics.

In this case the arguments passed to ERRNO_IS_NOT_SUPPORTED() are the
values returned by btrfs_subvol_make_fd() which is not expected to
return any positive values, but let's be consistent anyway and move
ERRNO_IS_NOT_SUPPORTED() invocations to the branches where
the return values are known to be negative.