Report: Robots.txt
We are checking if the /robots.txt file is present and if it is valid on the SMM Panels Website
The robots.txt file is a standard protocol (RFC 9309) that tells search engines which pages to index and which to ignore. It helps control website visibility and prevents indexing of irrelevant pages. The file must be in the root directory and accessible via HTTP/HTTPS. While not a security measure, it guides well-behaved web crawlers on how to interact with your site.
Recommendations
- Create a robots.txt file and add it to the root of your website
- Add the sitemap.xml to the robots.txt file
- Specify user-agent directives to control crawler access
- Use Allow and Disallow directives to manage page access
- Include Crawl-delay directive for rate limiting
- Test your robots.txt using Google Search Console
Common issues
- The robots.txt file is not present
- The robots.txt file is present but empty
- The robots.txt file is present but invalid (see details)
Example robots txt
Example robots txt description
User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /auth/
Disallow: /auth/login
Disallow: /auth/register
Disallow: /auth/forgot-password
Disallow: /auth/reset-password