Docker: macvlan crashing

I have been using macvlan for a very long time, and have had no problems what so ever.

Now suddenly I started having an issue which would lock up my box (I'm running +25 containers), the box would run for everything from 15 minutes to 24 hours (sometimes longer) but it would lock up.

The only notification I would get would be an exception like this in dmesg:

Oct 21 00:51:10 <box> kernel: [22600.504145] ------------[ cut here ]------------ Oct 21 00:51:10 <box> kernel: [22600.504183] WARNING: CPU: 0 PID: 303960 at net/netfilter/nf_conntrack_core.c:1123 __nf_conntrack_confirm+0x3ff/0x480 [nf_conntrack] Oct 21 00:51:10 <box> kernel: [22600.504186] Modules linked in: tcp_diag udp_diag inet_diag xt_nat xt_tcpudp veth macvlan xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user nft_counter xt_addrtype nft_compat nf_tables nfnetlink br_netfilter overlay bridge stp llc mlx5_core(OE) mlxfw(OE) mlxdevm(OE) auxiliary(OE) mlx_compat(OE) psample pci_hyperv_intf ipmi_ssif intel_rapl_msr intel_rapl_common pnd2_edac x86_pkg_temp_thermal intel_powerclamp coretemp xfs kvm_intel kvm irqbypass ghash_clmulni_intel cp210x aesni_intel libaes usbserial crypto_simd cryptd glue_helper rapl intel_cstate wdat_wdt ee1004 watchdog pcspkr efi_pstore joydev nls_ascii nls_cp437 ast vfat fat drm_vram_helper drm_ttm_helper evdev ttm qat_c3xxx drm_kms_helper acpi_ipmi ipmi_si intel_qat cec sg i2c_algo_bit ipmi_devintf authenc ipmi_msghandler button acpi_cpufreq nfsd drm fuse auth_rpcgss nfs_acl lockd grace configfs sunrpc efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 Oct 21 00:51:10 <box> kernel: [22600.504403] mlx4_ib ib_uverbs raid10 raid1 raid0 multipath ib_core linear mlx4_en dm_mod raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c crc32c_generic md_mod hid_generic usbhid hid sd_mod ahci libahci xhci_pci xhci_hcd libata usbcore nvme nvme_core t10_pi crc_t10dif crct10dif_generic crc32_pclmul ixgbe mlx4_core crct10dif_pclmul crct10dif_common crc32c_intel scsi_mod xfrm_algo i2c_i801 dca mdio_devres i2c_smbus libphy usb_common ptp pps_core mdio i2c_ismt Oct 21 00:51:10 <box> kernel: [22600.504526] CPU: 0 PID: 303960 Comm: kworker/0:0 Tainted: G OE 5.10.0-19-amd64 #1 Debian 5.10.149-1 Oct 21 00:51:10 <box> kernel: [22600.504530] Hardware name: Supermicro Super Server/A2SDi-4C-HLN4F, BIOS 1.6 10/27/2021 Oct 21 00:51:10 <box> kernel: [22600.504541] Workqueue: events macvlan_process_broadcast [macvlan] Oct 21 00:51:10 <box> kernel: [22600.504565] RIP: 0010:__nf_conntrack_confirm+0x3ff/0x480 [nf_conntrack] Oct 21 00:51:10 <box> kernel: [22600.504572] Code: 86 90 00 00 00 48 8b 80 80 0d 00 00 48 85 c0 0f 84 72 fe ff ff 49 8b 86 b8 00 00 00 48 85 c0 0f 84 62 fe ff ff e9 3d fe ff ff <0f> 0b 44 89 ee 89 df e8 a5 fa ff ff be 00 02 00 00 48 c7 c7 4f a5 Oct 21 00:51:10 <box> kernel: [22600.504576] RSP: 0018:ffffa2aec0003dd8 EFLAGS: 00010202 Oct 21 00:51:10 <box> kernel: [22600.504582] RAX: 0000000000000188 RBX: 0000000000005252 RCX: 0000000049fde36b Oct 21 00:51:10 <box> kernel: [22600.504586] RDX: 0000000000000000 RSI: 0000000000000221 RDI: ffffffffc12c7c88 Oct 21 00:51:10 <box> kernel: [22600.504590] RBP: ffffffff8d614ec0 R08: 0000000003235663 R09: ffff8e1f130a6140 Oct 21 00:51:10 <box> kernel: [22600.504593] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000000 Oct 21 00:51:10 <box> kernel: [22600.504597] R13: 0000000000007a21 R14: ffff8e1f130a6140 R15: ffff8e1f130a6198 Oct 21 00:51:10 <box> kernel: [22600.504601] FS: 0000000000000000(0000) GS:ffff8e216fc00000(0000) knlGS:0000000000000000 Oct 21 00:51:10 <box> kernel: [22600.504605] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Oct 21 00:51:10 <box> kernel: [22600.504609] CR2: 000000c00034d000 CR3: 0000000107776000 CR4: 00000000003506f0 Oct 21 00:51:10 <box> kernel: [22600.504613] Call Trace: Oct 21 00:51:10 <box> kernel: [22600.504620] <IRQ> Oct 21 00:51:10 <box> kernel: [22600.504651] nf_confirm+0xe3/0x100 [nf_conntrack] Oct 21 00:51:10 <box> kernel: [22600.504664] nf_hook_slow+0x3f/0xb0 Oct 21 00:51:10 <box> kernel: [22600.504671] ip_local_deliver+0xac/0x110 Oct 21 00:51:10 <box> kernel: [22600.504679] ? ip_protocol_deliver_rcu+0x1c0/0x1c0 Oct 21 00:51:10 <box> kernel: [22600.504687] __netif_receive_skb_one_core+0x8b/0xa0 Oct 21 00:51:10 <box> kernel: [22600.504695] process_backlog+0x96/0x160 Oct 21 00:51:10 <box> kernel: [22600.504702] net_rx_action+0x145/0x3e0 Oct 21 00:51:10 <box> kernel: [22600.504712] __do_softirq+0xc5/0x279 Oct 21 00:51:10 <box> kernel: [22600.504719] asm_call_irq_on_stack+0x12/0x20 Oct 21 00:51:10 <box> kernel: [22600.504723] </IRQ> Oct 21 00:51:10 <box> kernel: [22600.504732] do_softirq_own_stack+0x37/0x50 Oct 21 00:51:10 <box> kernel: [22600.504740] do_softirq+0x66/0x80 Oct 21 00:51:10 <box> kernel: [22600.504747] netif_rx_ni+0x9d/0xa0 Oct 21 00:51:10 <box> kernel: [22600.504756] macvlan_broadcast+0x6b/0x170 [macvlan] Oct 21 00:51:10 <box> kernel: [22600.504765] macvlan_process_broadcast+0x179/0x180 [macvlan] Oct 21 00:51:10 <box> kernel: [22600.504775] process_one_work+0x1b6/0x350 Oct 21 00:51:10 <box> kernel: [22600.504782] worker_thread+0x53/0x3e0 Oct 21 00:51:10 <box> kernel: [22600.504789] ? process_one_work+0x350/0x350 Oct 21 00:51:10 <box> kernel: [22600.504795] kthread+0x11b/0x140 Oct 21 00:51:10 <box> kernel: [22600.504801] ? __kthread_bind_mask+0x60/0x60 Oct 21 00:51:10 <box> kernel: [22600.504809] ret_from_fork+0x22/0x30 Oct 21 00:51:10 <box> kernel: [22600.504817] ---[ end trace b92981306b6007e1 ]---

Notice the 'nf_' (__nf_conntrack_), this is a known problem and have been around for some time; see here at Unraid.

As far as I can figure out there, are two (possibly) ways to solve the issue:

• move your macvlan to a non-bridge interface or to a bridged interface which is not the one you have been using until now.
• use ipvlan, there are pro/cons in regards to macvlan vs ipvlan - you'll have to decide for yourself.

I decided that the easiest way was to use ipvlan:

docker network create -d ipvlan \ --subnet 192.168.42.0/24 \ --ip-range=192.168.42.128/27 \ --gateway 192.168.42.1 \ -o parent=nif0 \ -o ipvlan_mode=l2 \ public 

As I use stacks, this only took a few minutes.