Understanding Robots.txt

Robots.txt is a text file that contains instructions to search engines like Google, telling them how to crawl pages on your website.

This type of file would tell the search engine not to index the page on your site that you use to log into BentoBox, because you don’t want website visitors to find that page while searching online. 

For example, you would not want your users to access your site_admin page or the /bentobox subfolder. 

BentoBox auto-generates robots.txt files for its customers. 

If you want to see the robots.txt file on your website, you can do that at any time by adding “/robots.txt” to the end of your website URL.

The contents of the robots.txt page are auto-generated based on:

• Whether a page is currently published (unpublished pages automatically tell search engines not to index them).

• Whether a page is classified as one that should never be indexed (like a Terms of Service page that should only be accessed through links from sources you send).

You can’t directly edit your robots.txt page via BentoBox. However, you can contact your Customer Success Manager or our support team at support@getbento.com if you believe there is something wrong with your robots.txt file.