GnoSight is a crawler that analyzes millions of public internet pages every month. We only crawl pages that are publicly accessible. Any page behind a login or any other authentication won't be crawled.
This crawler follows robots.txt and meta instructions. We make sure our crawler doesn't add any noticeable load on websites we visit — usually this means we'll never crawl more than one page at a time and one page every 2 seconds.
If you wish to ban this crawler, target the User Agent GnoSightCrawler in your robots.txt:
We pick up on this change within a few hours.
If you notice any issues or want to send feedback regarding the crawler, contact us at: engineering.gnosight@gmail.com