core/bpf: tighten handling of return values, improve messages

core/bpf: tighten handling of return values, improve messages

The code was written unidiomatically, using r as a boolean value, and
confusing errno and r in some places. AFAICS, there wasn't any actual
problem: even in the one place where errno was used instead of r, it would
almost certainly be initialized.

It seems that some libbpf functions set errno, while others return the
error, possibly encoded. Since there are almost no docs, the only way to
know is to read the code of the function. To make matters worse, there is
a global libbpf_mode which can be set to change the convention. With
LIBBPF_STRICT_DIRECT_ERRS in libbpf_mode, some functions set errno while others
return a negative error, and the only way to know is to read the code, except
that the split is now different. We currently don't set
LIBBPF_STRICT_DIRECT_ERRS, but even the possibility makes everything harder
to grok.

This is all very error-prone. Let's at least add some asserts to make sure that
the returned values are as expected.

(cherry picked from commit b7cba81553d0d958f23182ba9ab1739842ff9f5a)