I think I solved the issue. It turns out that iOS14 implemented even more unannounced SSL/TLS security measures than in iOS13. Mainly, according to this guide, This is what changed when generating your own CA and certs within the csr config file:
"iOS 13 allowed both DNS names and IP addresses to be tagged with DNS entries. It appears that iOS 14 has hardened this verification, and you must really specify DNS names with DNS entries, and IP addresses with IP entries."
This is different than how I've been creating my CAs and I figured was the likely culprit. Apple thought my certs weren't good enough even after I told the devices to trust it. But they don't throw any errors to indicate such. They just let you install your certs without telling you that they won't actually be used.
After following the above guide to the letter, I was able to get iOS 14 to sync just fine to my local NextCloud server. Looks like it properly used the root cert that I installed on the phone.
However, to make things even stranger, now I'm getting an error on the desktop application: "SELF SIGNED CERTIFICATE IN CHAIN" which I haven't gotten before. This is even after specifying a local file path to my trusted cert, enabling it in keychain, etc. Luckily, with the desktop version I can just click the "ignore TLS errors" and be good to go.
So, bottom line, I can now sync iOS 14 to my local Nextcloud with no errors and I can sync the my Big Sur MacOS by ignoring the "SELF SIGNED CERTIFICATE IN CHAIN" error.
Phew, that was a fun few days.