Add a page or crawl a website
Read one page, the whole site, or only the pages you choose into the agent's knowledge.
Where to find it
Sources › Website
Open this screen
Steps
- Open Sources › Website and pick the agent.
- Type the Page URL, for example your help page.
- For one page only, untick Crawl all pages on this site and press Add page.
- For the whole site, leave Crawl all pages on this site ticked and press Add & crawl.
- To choose pages, tick the second box (Finds the pages linked from the front page…), press Find pages, tick the folders or pages you want, then press Read the chosen pages.
- Optional: type a Name (optional) before adding.
Settings on this screen
| Setting | What it does | Default |
|---|---|---|
| Crawl all pages on this site | Up to 200 pages and 180 seconds; other sites are not followed. | On |
| Find pages | Lists the pages linked from the front page and in the sitemap, by folder. Folders that look like another language's copy are not ticked. | — |
Troubleshooting
"Enter a web address starting with https://."
Use the https address of the page.
"Choose at least one page on this site."
Tick at least one page after Find pages.
Tips
- The counter shows how many of the 200 pages are chosen.
Good to know
- Members can read sources; only workspace managers can add, change or delete them.
- One crawl at a time per organisation.
Related how-tos
- Website Add web pages to your agent's knowledge: one page, the whole site, or pages you choose.
- Follow or stop a website crawl See how far a crawl has got and stop it if needed.
- Knowledge sources: fixing common problems Fix the messages the Sources pages can show when adding files, pages and other sources.