📊 Quantitative test results for language: eng, year: 2023, size: 10K, paranoia level: 1:
🚀 Quantitative testing did not detect new false positives

Not attacks, but it does block quite a bit of parasitic traffic coming from LLMs, which often crawl in a very uncontrolled way.

IMO, this change is completely harmless and can only help reduce server load caused by corrupted traffic.

@touchweb-vincent

Not attacks, but it does block quite a bit of parasitic traffic coming from LLMs, which often crawl in a very uncontrolled way.

Not all users will want to block LLMs, in some cases it may be desirable for LLMs to crawl their site and others may not care. What do you mean by "parasitic traffic coming from LLMs"?

IMO, this change is completely harmless and can only help reduce server load caused by corrupted traffic.}

Makes sense, but what kind of traffic are we talking, do you mean LLMs?

Esad, I’m not talking about blocking LLM traffic. Here, we provide certain LLMs with the same network highways on our infrastructure as those granted to SERPs, given their contribution to traffic.

I’m only referring to corrupted requests issued by some LLMs, which incorrectly parse HTML and attempt to extract or crawl nonsensical elements, sometimes resulting in an unnecessary increase in server load.

For example, they encounter <a href="proxy.php?url=https%3A%2F%2Fwww.test2.fr"> in the HTML source of https://www.test.fr, and then incorrectly attempt to crawl https://www.test.fr/https://www.test2.fr

Yes i'm speaking about corrupted LLM trafic - mostly from Facebook’s LLM, which appears to be severely bugged.

@touchweb-vincent hm... ok. Can you add an test to make sure same origin domains are blocked?

@EsadCetiner done