# robots.txt

robots.txt is a text file on your website that tells search engine robots which parts of the site they are allowed to visit.

Source: https://ibenta.no/en/glossary/robots-txt
Publisher: Ibenta (https://ibenta.no)
Updated: 2026-09-27

---

The file always lives in the same place: straight under the domain, as in yourdomain.no/robots.txt. It consists of simple lines saying which robots a rule applies to, and which folders they should stay out of. [Google states](https://developers.google.com/search/docs/crawling-indexing/robots/intro) that the file is mainly used to avoid overloading your site with requests.

## It does not hide pages

A block in robots.txt looks like it keeps a page out of Google. It does not. According to Google, a blocked page can still appear in search results if other websites link to it. If you want to keep a page out, a noindex tag or a password is the right tool.

The most common mistake is the opposite one. A line blocking the whole site is left behind after development, and the site drops out of the [index](/en/glossary/indexing).

## Check yours in one minute

Open yourdomain.no/robots.txt in your browser. If it says "Disallow: /" right under "User-agent: *", the whole site is closed to search engines. Tell whoever runs your website straight away. If the file does not exist, that is not a problem. [Google treats it as if there are no restrictions](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt). If you see a line starting with "Sitemap:", it shows where your [sitemap](/en/glossary/sitemap) lives.