Crawling an internal website

Martin Kafka 0 Reputation points
2026-06-25T09:32:55.95+00:00

Hi,

I am doing a PoC for the RAG based agent in Foundry. One source is an internal hub where the agent shall crawl the webpage with the specific information for the user (region based) and shall be authenticated by Entra ID so German user does not see an information for the Swedish one. Hot to properly approach this in PoC and further apply it in PROD?

Thanks

Microsoft Foundry
Microsoft Foundry

A unified Azure platform for creating and managing AI models, agents, and applications with built‑in enterprise security, monitoring, and governance

0 comments

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Jubin Soni 0 Reputation points
    2026-06-27T22:15:28.8333333+00:00

    Hi @Martin Kafka

    For a PoC I'd skip crawling at index time, instead use a custom API tool that fetches the internal page at query time, authenticated as the calling user.

    Here's how I'd wire it up:

    1. User asks a question → agent calls your custom API tool with the user's Entra token
    2. API fetches the internal hub page via OBO (On-Behalf-Of) flow, Entra ID handles the region filtering naturally
    3. Content comes back as context to the agent

    This way you're not storing region-restricted content in a shared index, which gets complicated fast. For PROD I'd go with the same pattern but tighten up the OBO flow and add caching for content that isn't user-specific.

    If you really want to pre-index, look into Azure AI Search security trimming, tag each doc with allowed regions/groups and filter at query time using the user's claims. But honestly for a PoC the custom tool approach is simpler and gets the auth right out of the box.

    Please Upvote and accept the answer if it helps!!

    Was this answer helpful?

    0 comments
  2. Alex Burlachenko 23,250 Reputation points MVP Volunteer Moderator
    2026-06-25T13:04:57.31+00:00

    hi Martin Kafka & thx for sharing urs issue here at Q&A portal,

    For this PoC, don’t let the agent ‘freely crawl’ the internal site at answer time. That’s risky and hard to secure.

    crawl/index the site with a controlled backend process, store content in Azure AI Search, and add metadata like region, country, department, securityGroup, or allowedUsers. Then at query time, filter results based on the signed-in user’s Entra ID claims.

    user signs in with Entra ID > app gets user claims/groups > app calls the agent/search layer > search query includes a security filter -> agent only sees docs the user is allowed to see. German user gets filter like region eq 'DE'

    or group-based: allowedGroups/any(g: g eq 'group-id-for-germany')

    Don’t rely only on the prompt saying ‘don’t show Sweden data to German users’. That’s not security. The filter has to happen before docs are passed to the model.

    For PROD, I’d use ACL/security trimming with Azure AI Search and keep the crawler as a trusted service using managed identity. If the source site uses Entra auth, the crawler should either use app permissions approved by the content owner, or crawl per content partition w/ clear ownership. Don’t scrape w/ a random user token unless u want audit/compliance pain later.

    crawl/index centrally, tag content w/ access metadata, enforce filters in search, then let the agent answer only from filtered results. For PoC u can start w/ region filtering; for PROD move to group/user ACLs.

    rgds,

    Alex

    &

    If my answer was helpful pls mark it and additional thx if u follow me at Q&A portal

    and at my blog https://ctrlaltdel.blog/

     

    Was this answer helpful?

    0 comments

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.