Is it? Reddit is technically "public" too in the sense that you can view all the content without an account, yet Google and others pay for the data anyway. And for many years, people made stuff public and could reasonably expect it won't show up in any major search engines because Google, MS and others respected robot.txt. I know it was never legally binding. I'm also not naive, I know I give up control when I post publicly and there won't ever be a perfect solution to the AI crawler situation. But a lot is changing right now, regulatory and technologically.