quic_bpf failed

[LyraWang@…]

when "quic_bpf on;"
there comes "nginx: [alert] failed to create BPF map (22: Invalid argument)" when ngx_bpf_map_create.

[Maxim Dounin]

Could you please check if the kernel header files used during nginx-quic compilation match the kernel being used?

[LyraWang@…]

Actually they may not match. I couldn't find BPF_MAP_TYPE_SOCKHASH in /usr/include/linux/bpf.h, But I found it in "/usr/src/linux-hwe-5.4-headers-5.4.0-126/include/uapi/linux/bpf.h". I was confused why are they not matched. I copied /usr/src/linux-hwe-5.4-headers-5.4.0-126/include/uapi/linux/bpf.h to /usr/include/linux/bpf.h, so my compilation passed. But invalid arguement seems told that the kernel can't recognize it. So do I need to recompile the kernel with any bpf related option or what other things could be done to make it work?

[Maxim Dounin]

As long as the 5.4.0 kernel is actually installed, BPF_MAP_TYPE_SOCKHASH should be recognized properly. But overwriting just a single file in /usr/include/linux with a newer version might not be a good idea, and might result in arbitrary unexpected issues, including the one you are seeing.

I don't think there is an easy way to compile nginx with linux kernel headers different from the ones provided in /usr/include/linux by the libc. A better approach might be to actually upgrade to an OS version which provides BPF_MAP_TYPE_SOCKHASH.

[LyraWang@…]

Yes, when the OS version upgrade to ubuntu2204, problem solved. But it can't even work in ubuntu2104. BPF_MAP_TYPE_SOCKHASH has been supported in kernel 4.18, while kernel of ubuntu1804 is 5.4 and kernel of ubuntu2104 if 5.11. I'm so confused, is operating system do any thing to the kernel? And Is there anything could be done to make the lower OS work?

[Maxim Dounin]

Appropriate headers are shipped by Ubuntu in the linux-libc-dev package. These match the GA kernel as shipped in the particular OS version, and not updated with ​HWE kernels. In particular, in Ubuntu 18.04 these headers are from ​kernel 4.15, so BPF_MAP_TYPE_SOCKHASH is not expected to work there.

As for Ubuntu 21.04, not sure what goes wrong there. Note though that this Ubuntu version is EOLed and no longer supported.

On Ubuntu 20.04 LTS it compiles fine, but only works with HWE kernel 5.15. With GA kernel 5.4 it results in the EINVAL errors from bpf(BPF_MAP_TYPE_SOCKHASH), similarly to what you've seen on Ubuntu 18.04 with the same kernel, as originally reported. Looks like an issue in the kernel and/or some incompatibility in the headers, ​this commit (appeared in kernel 5.6) might be related.

[LyraWang@…]

Thank you very much. I've solved the problem by recompiling linux kernel. It seems original installed OS missed some BPF configs when installed. I almost turned on every bpf related config while compiling kernel, since I don't know which one really make impact.

[vl]

The EINVAL comes from passing 64-bit for values.
This was not allowed till this commit:

commit c1cdf65da060a8e047a9f4433306fd6dac1f51a6
Author: Jakub Sitnicki <jakub@…>
Date: Tue Feb 18 17:10:18 2020 +0000

bpf, sockmap: Return socket cookie on lookup from syscall

Tooling that populates the SOCK{MAP,HASH} with sockets from user-space
needs a way to inspect its contents. Returning the struct sock * that the
map holds to user-space is neither safe nor useful. An approach established
by REUSEPORT_SOCKARRAY is to return a socket cookie (a unique identifier)
instead.

Since socket cookies are u64 values, SOCK{MAP,HASH} need to support such a
value size for lookup to be possible. This requires special handling on
update, though. Attempts to do a lookup on a map holding u32 values will be
met with ENOSPC error.

The corresponding kernel version is 5.7

nginx uses sockhash map to deal with socket cookies, i.e. exactly the case described.

This explains why it doesn't work with 5.4 and works with later 5.15.