Website
Add web pages to your agent's knowledge: one page, the whole site, or pages you choose.
Where to find it
Sources › Website
Open this screen
Steps
- In the sidebar, open Sources › Website.
- If you have several AI agents, pick the agent in the list at the top right; sources belong to one agent.
- Enter the Page URL in Add website and choose how to read it (one page, the whole site, or chosen pages).
- Press Add page, Add & crawl or Read the chosen pages.
- Follow the crawl under Crawls; the pages read appear in the list.
Settings on this screen
| Setting | What it does | Default |
|---|---|---|
| Page URL | The address to read, https only. | — |
| Crawl all pages on this site | Follows links and the sitemap on the same site, up to 200 pages and 180 seconds. | On |
| Choose pages | Finds the site's pages and reads only the ones you tick (up to 200). | Off |
| Name (optional) | The page title is used when empty. | — |
| Crawls | Each crawl's progress and status, with Stop. | — |
| Data sources | This agent's sources by type (Files, Website, Text snippets, Q&A, Notion import), the organisation's storage meter, and Re-index agent. | — |
Troubleshooting
"A crawl is already running for this organisation. Wait for it to finish, or stop it."
Wait, or press Stop on the running crawl under Crawls.
Tips
- Pages the site's robots.txt asks crawlers not to read are left out, and the row says how many.
- When a banner says live calls do not use this knowledge base yet, what you add is stored and searchable but not read on a call until AI answering is configured for the service.
Good to know
- Storage is shared by every agent in the organisation: up to 3,000,000 characters in all, and up to 200,000 characters per source (a longer one is cut, and the cut is shown).
- Members can read sources; only workspace managers can add, change or delete them.
Related how-tos
- Add a page or crawl a website Read one page, the whole site, or only the pages you choose into the agent's knowledge.
- Follow or stop a website crawl See how far a crawl has got and stop it if needed.
- Preview and edit what the agent learned Read the text the agent learned from a source and fix or trim it.
- Knowledge sources: fixing common problems Fix the messages the Sources pages can show when adding files, pages and other sources.