Microsoft Foundry_ Production Architecture

Microsoft Foundry Production Architecture

Network Insight / Engineering Guide 10 / Production AI Architecture

Microsoft Foundry Architecture Explained: Inside the Enterprise AI Production Stack

Microsoft Foundry provides an environment for building, deploying, evaluating and operating AI applications and agents. Production implementations combine Foundry capabilities with model endpoints, identity systems, enterprise data, search services, network connectivity, security controls and operational monitoring.

A production AI application is not simply a model endpoint receiving prompts. It is a distributed system in which application requests move through authentication, policy enforcement, model inference, optional agent execution, tool integration and data retrieval before a response is returned.

Understanding these dependencies is essential when designing reliable AI platforms, controlling cost, enforcing security boundaries and investigating production failures.

ENTERPRISE AI APPLICATIONS Copilots · Web Apps · APIs IDENTITY & API GOVERNANCE Entra ID · Gateway · Policies MICROSOFT FOUNDRY Projects · Models · Agents · Tools MODEL INFERENCE Endpoints RAG & SEARCH Enterprise Data PRIVATE NETWORKING · RBAC · OBSERVABILITY · CI/CD

Architecture interpretation: The diagram represents logical responsibilities, not a fixed network topology. Model deployments, agents, search services and security components may use different resources, endpoints and deployment boundaries depending on the Foundry architecture selected.

The Six Layers of a Production Foundry Platform

01
Application Layer

Enterprise applications, copilots, APIs and user-facing AI experiences generate requests and consume model or agent responses.

02
Identity and Governance

Authentication, API access control, quotas, policies and workload identity govern application interactions.

03
Foundry Control Layer

Projects, model deployment management, agent configuration and evaluation support AI application engineering.

04
Inference and Agents

Model endpoints process inference requests while agents may coordinate tools and multi-step execution.

05
Enterprise Knowledge

Search indexes, retrieval pipelines, enterprise data and grounding services provide contextual information.

06
Infrastructure and Operations

Private networking, DNS, identity, telemetry, security and CI/CD support reliable production delivery.

Why Microsoft Foundry Requires Production Architecture Engineering

Building a working model demonstration is only the first stage of AI application engineering. Production platforms must manage request volume, authentication, access to sensitive data, dependency failures, latency, model quality, operational cost and change management.

A model endpoint can be healthy while the application fails because a tool cannot authenticate, a search index returns irrelevant documents or a private DNS configuration prevents connectivity to a dependent service.

Similarly, an agent may successfully execute its instructions but produce an incorrect result because of inadequate retrieval context, unsafe tool permissions or missing evaluation controls.

Control Plane vs Inference Data Plane

Control Plane

The control plane manages the configuration and lifecycle of projects, deployments, identities, policies and supporting resources. It is primarily concerned with provisioning, administration and governance.

Inference and Execution Plane

The execution plane processes application requests, model inference, agent operations, tool interactions and retrieval workloads. It is primarily concerned with runtime performance and behaviour.

Production Request Lifecycle

Stage Operation Engineering concern
1. Request Application submits an AI request Authentication and request validation
2. Governance Identity and policy controls evaluate access Authorisation, quotas and safety
3. Orchestration Application or agent selects execution steps Tool permissions and workflow state
4. Retrieval Optional enterprise knowledge retrieval Search quality and data access
5. Inference Model deployment processes the request Latency, throughput and token usage
6. Response Application returns the result Output validation and user experience
7. Operations Telemetry and evaluations capture execution evidence Reliability, quality, security and cost

Engineering distinction: Not every request traverses every stage. A direct model inference call may not involve an agent or retrieval system. Production architecture should be designed around the actual application execution path rather than assuming every Foundry capability is required.

What This Engineering Guide Will Teach

This guide develops the complete production architecture of Microsoft Foundry through nine engineering sections: platform fundamentals, model deployment, agent orchestration, enterprise RAG, private networking, identity and security, observability, CI/CD automation and production troubleshooting. The objective is to connect AI platform concepts with the infrastructure, networking and operational skills required to deploy reliable enterprise AI systems.

Network Insight · Engineering Architecture Navigator

Explore the Microsoft Foundry Architecture

Nine connected engineering disciplines, from Foundry fundamentals and model deployments to agent orchestration, RAG, networking, security, observability, CI/CD and production troubleshooting. Select any architecture component to navigate to its engineering section.

FOUNDRY AI PLATFORM 01 02 03 04 05 06 07 08 09
Architecture Navigator · Select a node to explore
Network Insight / Microsoft Foundry / Section 01

Microsoft Foundry Architecture Fundamentals: Resources, Projects, Models, Agents and Runtime Execution

Microsoft Foundry provides capabilities for organising AI development, deploying models, building agents, integrating tools and operating enterprise AI applications. Its production architecture is best understood as a set of related management, execution, identity and infrastructure responsibilities rather than a single monolithic AI service.

A Foundry solution typically connects an application to one or more model deployments, with optional agent orchestration, enterprise knowledge retrieval and external tool execution. Foundry resources and projects help organise these capabilities, while Azure identity, networking and monitoring services provide the surrounding production environment.

1.1 — Foundry Resource and Project Architecture

A Foundry resource provides an Azure management boundary for supported platform capabilities. Projects provide an organisational context for application development and related assets. Exact resource relationships, endpoint types and supported features depend on the selected Foundry experience and service configuration.

AZURE SUBSCRIPTION / RESOURCE GROUP FOUNDRY RESOURCE PROJECT A PROJECT B Application Configuration Agents / Tools Application Configuration Agents / Tools Shared Model Deployments · Identity · Service Connections

Architecture note: The diagram is conceptual. It illustrates how multiple projects may organise application work around Foundry capabilities. It should not be interpreted as a universal resource hierarchy or a guarantee that all assets are isolated per project.

1.2 — Core Foundry Architecture Components

R
Foundry Resources

Azure-managed resources supporting the configured Foundry capabilities, identities and service access.

P
Projects

Organisational contexts for developing and managing related AI application assets.

M
Model Deployments

Configured inference deployments with model-specific capabilities, capacity and endpoint behaviour.

A
Agents

Agent configurations and execution workflows coordinating model interactions and supported tools.

T
Tools and Connections

Integration paths to external services, APIs and enterprise information systems.

O
Operations

Monitoring, evaluations, security, access governance and production lifecycle management.

1.3 — Foundry Control Plane vs Runtime Execution

Production engineers must distinguish management operations from the runtime processing of application requests.

Control Plane

Management operations include resource provisioning, project administration, deployment configuration, role assignments and supported lifecycle operations.

These operations are governed through the applicable Azure management APIs and platform permissions.

Runtime / Data Plane

Runtime operations include model inference, agent execution, retrieval calls and interactions with supported tools or services.

They require the appropriate endpoint connectivity, authentication, authorisation and service capacity.

Engineering distinction: Permissions to manage an Azure resource do not automatically imply permission to perform every data-plane operation. Validate the required roles and scopes for the specific API and Foundry capability being used.

1.4 — Model Deployment and Endpoint Relationships

A model deployment represents a configured model-serving capability. Applications connect through supported endpoints and APIs, using deployment identifiers and credentials or identities appropriate to the service.

AI Application Request + Identity Prompt / Parameters Inference Endpoint Auth + API Contract Deployment Selection Model Deployment Model Execution Capacity + Latency

Endpoint architecture matters because an application may encounter authentication failures, invalid deployment identifiers, unsupported request parameters, throttling or model capacity constraints.

1.5 — Agents, Tools and Execution Responsibilities

Agent-based applications introduce additional runtime responsibilities beyond a direct model inference call.

Component Engineering responsibility Production concern
Agent configuration Instructions, model selection and supported tool definitions Consistency, safety and version control
Execution workflow Coordinates model calls and tool interactions Latency, retries and execution failures
Tool connection Provides access to an API or enterprise service Authentication and least privilege
Enterprise knowledge Supplies retrieved information when required Data access, freshness and relevance
Application boundary Validates requests and processes results Business rules and output handling

1.6 — Production Request Flow and Dependency Analysis

A typical enterprise request may pass through multiple services, each with its own identity, network and availability requirements.

Stage Operation Failure domain
1. Client Submits request to enterprise application Client connectivity or application availability
2. Authentication Validates caller identity and permissions Token, role or policy failure
3. Orchestration Determines direct model or agent workflow Invalid configuration or execution error
4. Retrieval Obtains relevant knowledge when needed Search or data service failure
5. Inference Processes model request Quota, latency or endpoint error
6. Tool execution Invokes approved external operations when required Permission, connectivity or tool failure
7. Response Returns processed result to application Output validation or downstream failure

Architecture principle: These stages represent possible responsibilities, not a mandatory sequence for every application. Direct model calls, retrieval-augmented workflows and agent-based systems have different execution paths.

1.7 — Production Design Decisions

Resource Boundaries

Choose project and resource boundaries based on operational ownership and isolation requirements.

Endpoint Strategy

Determine model access, deployment selection, authentication and capacity requirements.

Identity Architecture

Use appropriate managed identities, workload credentials and least-privilege access.

Network Connectivity

Validate private access, DNS resolution and dependent service connectivity.

Observability

Collect relevant request, dependency, latency and operational telemetry.

Lifecycle Management

Control configuration changes, deployment versions and production release processes.

Section 1 — Engineering Takeaway

Microsoft Foundry production architecture combines Azure-managed resources, projects, model deployments, agent execution, enterprise integrations, identity and infrastructure services. Engineers must understand the separation between management operations and runtime execution, identify service dependencies and design appropriate resource, security and operational boundaries. These fundamentals provide the foundation for model deployment, agent engineering and enterprise AI operations.

Network Insight · Engineering Lab 01 · Microsoft Foundry

Foundry Architecture Explorer

Configure a conceptual Microsoft Foundry application architecture. Select a deployment pattern, enable model and agent capabilities, attach enterprise knowledge services and validate the relationships between the project, inference endpoints, tools and operational controls.

Architecture configuration

↓ APPLICATION INTEGRATION ↓
↓ DEPENDENT SERVICES ↓
0 Enabled components
0 Architecture issues
Pending Readiness

Component engineering inspector

Foundry Project

Architecture validation console

Dependency validation and engineering findings

Architecture boundary: The Foundry project is shown alongside runtime components for architectural clarity; it is not a mandatory network hop for every inference request. Model and agent capabilities, private networking support and resource relationships depend on the chosen Foundry service configuration. Validation here checks educational architecture rules, not live Azure resources.
Network Insight / Microsoft Foundry / Section 02

Microsoft Foundry Model Deployment and Inference Engineering: Capacity, Latency, Throughput and Production Reliability

Model deployment engineering is the process of selecting an appropriate AI model, configuring its serving environment, allocating supported capacity and designing the application request path for predictable performance, reliability and cost.

In Microsoft Foundry, model availability, deployment methods, API capabilities, regional support and capacity controls vary by model and underlying service. Production engineers must therefore evaluate both the model's capabilities and the operational characteristics of the deployment that serves it.

2.1 — Production Model-Serving Architecture

A production inference path commonly includes the application, an optional API gateway, authentication and routing logic, a supported model endpoint and operational telemetry.

Application Request Generation API Gateway Identity + Policy Model Endpoint API + Deployment Routing Model Deployment Inference Execution Capacity Controls Quotas + Throughput Observability Latency + Tokens + Errors

Architecture interpretation: The diagram illustrates logical responsibilities rather than a mandatory internal request sequence. Gateways and observability services are optional architectural components; model capacity controls depend on the selected deployment type.

2.2 — Model Selection and Deployment Decisions

M
Model Capability

Evaluate supported modalities, context limits, reasoning capabilities, tool support and response quality.

D
Deployment Availability

Confirm supported deployment types, regions, model versions and service compatibility.

C
Capacity

Understand quota allocations, request limits and provisioned throughput where applicable.

L
Latency

Measure time to first token, generation speed and complete response duration.

$
Cost

Evaluate token-based charges, deployment capacity commitments and operational overhead.

R
Reliability

Design retries, timeouts, fallback behaviour and dependency monitoring.

2.3 — Understanding Model Deployment Types

Deployment options differ in how capacity is allocated, how workloads are served and which operational constraints apply.

Deployment approach Typical characteristic Engineering consideration
Standard / Consumption Usage-based serving subject to applicable limits Quota, latency variation and workload demand
Provisioned Throughput Reserved serving capacity for supported models and services Capacity sizing, utilisation and commitment cost
Global Deployment Serving capacity may be distributed across supported geographies Data processing requirements and deployment support
Data Zone Deployment Serving within a supported defined data zone Availability and applicable data processing boundaries
Regional Deployment Serving associated with a selected Azure region where supported Regional availability, capacity and residency needs

Important: These categories describe common deployment characteristics and are not a universal mutually exclusive list. Exact Foundry deployment choices vary by model, provider, service and region.

2.4 — Token Processing and Inference Lifecycle

Token processing is central to model performance engineering. The amount of input context, output generation and concurrent workload affects latency, throughput and cost.

Input Tokens Prompt + Context Prefill Context Processing Generation Output Tokens Response Result Delivery

For autoregressive language models, prefill processes the input context, while decoding generates output tokens over successive steps. The resulting performance depends on model architecture, serving infrastructure, batching and workload characteristics.

2.5 — Capacity Planning and Throughput Engineering

A useful starting point is estimating the workload's token demand. This is an approximation rather than a guarantee of provider capacity.

# Illustrative workload Requests per minute = 120 Average input tokens = 1,500 Average output tokens = 500 # Estimated token demand Tokens per request = 2,000 Estimated tokens per minute = 120 × 2,000 = 240,000 tokens/minute

This estimate does not account for peak bursts, concurrency, caching, model-specific quota accounting, provider throttling or differing processing costs between input and output tokens.

Capacity principle: Use real workload measurements to validate capacity assumptions. Provisioned throughput and consumption-based quotas may use different capacity and admission-control mechanisms.

2.6 — Latency and Performance Metrics

Metric Meaning Engineering use
End-to-End Latency Total application request duration Measure user experience
Time to First Token Delay before first streamed output token Evaluate responsiveness
Output Token Rate Generated output tokens per second Measure generation speed
Requests per Minute Request arrival or processing rate Capacity planning
Tokens per Minute Token workload or service accounting rate Quota and throughput analysis
Throttling Rate Requests rejected due to applicable limits Identify capacity pressure
Error Rate Failed requests over total requests Reliability monitoring

2.7 — Production Reliability and Request Control

Production applications should anticipate temporary failures, throttling, network delays and downstream dependency problems.

Timeouts

Bound request duration and prevent indefinite waiting on model operations.

Retries

Use bounded retries with backoff and jitter for appropriate transient failures.

Concurrency

Control simultaneous requests to avoid unnecessary overload and throttling.

Fallback

Design approved alternative behaviours when the preferred deployment is unavailable.

Monitoring

Track request errors, token usage, latency and capacity pressure.

Cost Controls

Apply workload budgets, usage monitoring and application-level limits.

2.8 — Production Model Deployment Troubleshooting

Symptom Possible cause Engineering investigation
HTTP 401 Missing or invalid authentication credentials Verify token, credential configuration and endpoint requirements
HTTP 403 Insufficient permission or access restriction Inspect effective authorisation and service policies
HTTP 404 Incorrect endpoint, deployment identifier or API route Verify endpoint URL, deployment name and supported API version
HTTP 429 Throttling or capacity limits Inspect quota, request rates and retry guidance
High latency Long context, generation workload, queueing or network delays Separate client, gateway, model and dependency latency
Unexpected cost High token volume, repeated calls or inefficient orchestration Review token usage, request patterns and deployment pricing

Operational warning: HTTP status codes provide useful diagnostic signals but are not sufficient by themselves to identify root cause. Inspect service error details, request identifiers, logs and the selected API contract.

Section 2 — Engineering Takeaway

Microsoft Foundry model deployment engineering combines model selection, endpoint architecture, capacity allocation, token processing, latency analysis, reliability controls and cost management. Production performance cannot be determined from model capability alone. Engineers must measure real workloads, understand deployment-specific limits and design applications that respond predictably to throttling, failures and changes in demand.

Network Insight · Engineering Lab 02 · Microsoft Foundry

Model Deployment and Capacity Planner

Estimate how model selection, deployment capacity, token demand and traffic concurrency affect a production AI inference workload. Compare illustrative deployment profiles, investigate quota pressure and identify potential throughput and latency constraints.

Inference workload configuration

↓ INFERENCE REQUEST FLOW ↓
0 Peak tokens/min
0 Illustrative capacity
0% Demand / capacity
0s Modeled latency
Estimated capacity utilisation

Deployment engineering inspector

Model Deployment

Capacity diagnostics console

Capacity findings and production recommendations

Modeling limitation: This calculator uses synthetic model and deployment profiles for engineering education. Real Microsoft Foundry capacity, quotas, provisioned throughput, token accounting, model latency, regional availability and deployment options vary by model, service, subscription and configuration. Batch deployments are asynchronous and should not be interpreted as interactive low-latency endpoints.
Network Insight · Engineering Lab 03 · Microsoft Foundry

Agent Execution and Tool-Call Trace

Trace a multi-step enterprise agent workflow from user request to model decision, tool selection, authorisation, execution and final response. Inject failures, inspect tool-call evidence and practise safe recovery and approval decisions.

Agent workflow configuration

Idle Execution state
0 Completed steps
0 Tool calls
0 Retries
Select a scenario and start the agent execution trace.

Agent engineering inspector

Agent Runtime

Agent execution console

Execution trace and tool-call evidence

Security and execution boundary: A model-generated tool call is a request, not an authorisation decision. Tool permissions must be enforced by the application or tool service. Human approval is required only when defined by the workflow's risk policy. This is a conceptual simulator, not a live Microsoft Foundry agent runtime or an actual API execution trace.
Network Insight / Microsoft Foundry / Section 03

Microsoft Foundry Agents and Tool Orchestration: Execution Architecture, State, Tools and Production Control

AI agents extend model inference by combining language-model reasoning with instructions, tools, execution state and external system interactions. In Microsoft Foundry, supported agent capabilities provide ways to construct workflows in which models can select or request tool operations and use their results to continue processing.

An enterprise agent is therefore not simply a prompt connected to a model. It is an execution system that must coordinate model requests, tool permissions, state management, network dependencies, retries, timeouts and application-level approval boundaries.

3.1 — Production Agent Orchestration Architecture

A production agent workflow connects the application to an agent execution layer, a model deployment and one or more supported tools.

Application User Request Agent Runtime Instructions + State Model Deployment Reasoning + Tool Selection Tool Gateway Authorisation + Validation Enterprise APIs Data + Business Actions Execution State Context + History

Architecture interpretation: The diagram represents logical responsibilities rather than a mandatory Foundry internal implementation. Tool execution may be handled by a managed capability, application code or another authorised runtime, depending on the tool type and supported agent architecture.

3.2 — Six Core Agent Engineering Components

I
Instructions

Define agent purpose, operational boundaries, expected behaviour and response requirements.

M
Model

Provides language understanding, reasoning and supported tool-selection capabilities.

T
Tools

Expose approved operations such as retrieval, API calls or enterprise actions.

S
Execution State

Maintains relevant conversation, workflow and tool-result context as designed.

P
Permissions

Restrict agent access to authorised data and external operations.

O
Observability

Records execution traces, latency, failures and relevant tool interactions.

3.3 — Agent Tool-Call Lifecycle

A tool call introduces a boundary between model-generated instructions and an external operation. The model may propose a tool call, but the execution system must validate and authorise the action.

Stage Operation Engineering control
1. Request Application submits task to agent Authenticate caller and validate request
2. Planning Model determines next supported action Constrain tool definitions and instructions
3. Tool proposal Model emits a structured tool request Validate name, arguments and schema
4. Authorisation Execution system evaluates permission Enforce policy and approval requirements
5. Execution Approved tool performs operation Apply timeouts, identity and network controls
6. Result Tool output returns to agent workflow Validate and treat output as untrusted data
7. Continuation Agent processes result and continues or responds Bound execution steps and capture telemetry

3.4 — Agent Instructions vs Enforcement Controls

Agent instructions can influence behaviour, but they are not a substitute for technical security enforcement.

Instruction-Level Controls

Define intended behaviour, approved workflows, response formatting and restrictions communicated to the model.

Execution-Level Controls

Enforce identity permissions, tool allowlists, parameter validation, network restrictions, approval requirements and audit logging outside model-generated text.

Security principle: A model instruction such as "do not delete production data" cannot reliably replace IAM restrictions, application authorisation or explicit approval gates for destructive operations.

3.5 — Agent State and Workflow Continuity

Multi-step agents often need to maintain relevant context between model calls and tool interactions.

State category Purpose Production consideration
Conversation context Preserves relevant user and assistant exchanges Retention, privacy and context limits
Tool results Supplies external operation results Data validation and sensitivity
Workflow progress Tracks completed and pending steps Recovery and duplicate execution
Approval state Records authorised high-impact operations Identity verification and audit evidence
Execution telemetry Supports diagnosis and performance analysis Log security and trace correlation

3.6 — Multi-Step Agent Execution Patterns

Production agent workflows may combine several operations before returning a final result.

# Illustrative enterprise agent workflow 1. Receive incident investigation request 2. Retrieve relevant monitoring data 3. Analyse observed failure indicators 4. Request additional diagnostic evidence 5. Correlate logs and service dependencies 6. Produce investigation findings 7. Request human approval before any high-impact remediation 8. Execute only approved operations 9. Record results and audit evidence

The application should control maximum execution steps, overall time budgets, retry behaviour and external operation permissions.

3.7 — Tool Integration and Enterprise API Security

Identity

Use an appropriate workload identity and scope access to required operations.

Input Validation

Validate tool arguments before invoking an external service.

Network Controls

Restrict connectivity to approved endpoints where applicable.

Approval Gates

Require explicit approval for sensitive or irreversible actions.

Timeouts and Retries

Prevent indefinite execution and uncontrolled repeated operations.

Auditability

Correlate user requests, tool invocations and resulting changes.

3.8 — Agent Reliability and Failure Domains

Failure Potential cause Engineering response
Agent fails to select expected tool Ambiguous instructions or tool descriptions Review tool schemas, instructions and evaluation results
Tool returns 401/403 Invalid identity or insufficient permission Inspect execution identity and effective access
Tool call times out Slow service, network failure or excessive workload Investigate dependency latency and timeout settings
Repeated tool calls Missing termination controls or ineffective workflow state Enforce step limits and inspect execution traces
Incorrect final answer Unreliable reasoning, incomplete evidence or bad tool output Review intermediate results and evaluation cases
Duplicate external action Retry after ambiguous operation completion Use idempotency controls and reconcile operation status

Operational caution: Retrying a read-only retrieval operation is different from retrying an action that changes infrastructure or business data. Side-effecting tools require explicit idempotency and recovery design.

3.9 — Production Agent Engineering Checklist

A production-ready agent architecture should establish clear operational boundaries before exposing tools to model-driven workflows.

Design Controls

Define agent responsibilities. Select supported models. Document tool contracts. Establish identity boundaries. Limit high-impact operations. Define approval requirements.

Operational Controls

Configure timeouts and retries. Bound execution steps. Capture trace correlation. Protect sensitive state. Evaluate tool behaviour. Test failure recovery.

Section 3 — Engineering Takeaway

Microsoft Foundry agent engineering combines model execution, instructions, tools, state, authorisation and workflow control. Production agents must be treated as distributed execution systems with external dependencies and potentially significant operational permissions. Reliable designs enforce security outside model instructions, validate tool requests, control side effects and capture sufficient telemetry to investigate every important execution step.

Network Insight / Microsoft Foundry / Section 04

Enterprise RAG Architecture: Azure AI Search, Vector Retrieval, Grounding and Knowledge Integration

Retrieval-Augmented Generation (RAG) connects generative AI models to external knowledge systems. Rather than relying exclusively on information represented in model parameters, a RAG application retrieves relevant enterprise content and supplies it as context for response generation.

In Microsoft Foundry architectures, RAG may combine a model deployment, application or agent orchestration, Azure AI Search, embedding models and enterprise data repositories. The retrieval workflow is responsible for finding useful evidence, while the generation workflow uses that evidence to construct an answer.

A production RAG implementation must address more than semantic similarity. Engineers must design ingestion pipelines, document security, index freshness, retrieval relevance, context limits, citation provenance, monitoring and failure recovery.

4.1 — Enterprise RAG Architecture: Two Connected Pipelines

A production RAG system has two distinct engineering workflows: an offline or asynchronous knowledge ingestion pipeline, and an online retrieval and generation pipeline.

KNOWLEDGE INGESTION PIPELINE ONLINE RETRIEVAL + GENERATION Enterprise Data Chunking Embeddings Azure AI Search PDFs · Wikis · Files Text + Metadata Vector Generation Searchable Index User Question Retrieval Grounding Model Inference Query + Identity Hybrid Search Evidence Context Answer + Citations Indexed Knowledge

Architecture distinction: The ingestion pipeline prepares knowledge for retrieval. The online pipeline retrieves and uses relevant content during an application request. Neither process automatically guarantees answer correctness or enforces document-level authorisation without appropriate security configuration.

4.2 — Six Core Components of Enterprise RAG

01
Data Sources

Documents, databases, knowledge repositories and enterprise content systems.

02
Ingestion

Extraction, normalisation, chunking, metadata processing and index updates.

03
Embeddings

Vector representations supporting similarity-based retrieval for compatible text and query inputs.

04
Search Index

Stores searchable fields, metadata and configured vector representations.

05
Retrieval Logic

Combines query processing, filters, ranking and context selection.

06
Grounded Generation

Supplies selected evidence to the model and preserves source provenance.

4.3 — Document Ingestion and Chunking Engineering

Enterprise documents are often too large or heterogeneous to index as single retrieval units. Chunking divides extracted content into smaller passages that can be searched and supplied to the model efficiently.

Chunk size influences retrieval precision, context completeness and index volume. Small chunks may retrieve specific facts but lose surrounding meaning. Large chunks preserve more context but can introduce irrelevant material and consume additional tokens.

Design parameter Purpose Engineering trade-off
Chunk size Controls passage length Precision versus context completeness
Chunk overlap Preserves continuity across boundaries Context retention versus duplication
Metadata Stores source, title, owner and other attributes Filtering and provenance requirements
Document identifiers Connect chunks to original records Update, deletion and citation correctness
Extraction quality Converts source documents into usable text Tables, layouts, scanned content and formatting
Index freshness Keeps search data aligned with source changes Ingestion frequency and update latency

Engineering caution: Chunking settings should be validated against real queries. There is no universal optimal chunk size. Document structure, language, retrieval method and expected question types all affect the result.

4.4 — Vector Search, Keyword Search and Hybrid Retrieval

Azure AI Search supports multiple retrieval techniques. A production system can combine lexical matching with vector similarity and optional semantic ranking where supported and configured.

Keyword and Lexical Search

Matches indexed text using search terms and lexical ranking techniques. It is useful for exact technical identifiers, product names, error codes and terminology.

Vector Search

Uses vector representations to retrieve content with similar semantic characteristics. It can help find relevant passages even when the question and document use different wording.

Hybrid search: Combining lexical and vector retrieval can improve coverage for enterprise technical content. Semantic reranking may further refine candidate results, but retrieval quality must still be measured against representative questions.

4.5 — Retrieval Ranking and Context Construction

Retrieval produces candidate documents or chunks. The application must decide which passages should become model context. This step may include filtering, reranking, deduplication and token-budget management.

Illustrative retrieval workflow 1. Receive authenticated user question 2. Determine applicable document filters 3. Generate query embedding if required 4. Execute lexical and/or vector retrieval 5. Apply supported ranking or reranking 6. Select relevant authorised passages 7. Construct model context with source IDs 8. Generate grounded response 9. Validate and display source citations

A retrieval score does not establish that a document is factually correct, current or authorised for the requesting user. Those properties require additional application and data-governance controls.

4.6 — Grounding, Citations and Answer Reliability

Grounding supplies retrieved evidence to the generation process. The model can then use that context when constructing an answer.

Citations should connect claims to identifiable source records or passages. Reliable implementations retain document identifiers, source locations and appropriate provenance metadata through retrieval and response processing.

Control Engineering purpose Failure if missing
Source IDs Track retrieved evidence Unverifiable citations
Context validation Detect incomplete or irrelevant evidence Unsupported answers
Freshness metadata Identify document age or revision Outdated guidance
Response evaluation Measure answer relevance and grounding Undetected quality regressions
Abstention policy Handle insufficient evidence Confident unsupported responses

4.7 — Enterprise RAG Security and Document Access

Enterprise retrieval systems must preserve the security boundaries of their underlying knowledge sources. A user should not receive restricted content simply because it appears in a shared search index.

Identity

Authenticate users and workloads accessing the retrieval application.

Authorisation

Apply effective document or record permissions before releasing content.

Metadata Filtering

Use appropriate security attributes to constrain retrieval results.

Private Connectivity

Protect supported service connections using approved network architecture.

Data Protection

Control sensitive content, logging, retention and data exposure.

Auditability

Record relevant retrieval access and application execution evidence.

Security principle: Search filters are only effective security controls when their values are derived from trusted identity and authorisation logic. Never trust a user-supplied filter as proof that the user may access a document.

4.8 — Production RAG Observability and Quality Metrics

Metric What it measures Engineering use
Retrieval latency Time required to obtain search results Identify search bottlenecks
Recall at K Relevant results found within top K Evaluate retrieval coverage
Precision at K Relevant proportion of top K results Evaluate retrieval relevance
Groundedness Degree to which answer claims follow supplied evidence Assess unsupported generation
Citation correctness Whether citations support associated claims Validate source attribution
Index freshness Delay between source changes and searchable updates Monitor ingestion reliability
Context token usage Tokens consumed by retrieved evidence Optimise cost and context budget

4.9 — Production RAG Troubleshooting

Symptom Likely investigation area Engineering action
Relevant document not found Ingestion, index fields, filters or ranking Inspect indexed content and retrieval queries
Incorrect answer despite relevant source Context selection or model generation Compare retrieved evidence with final prompt and response
Outdated answer Index freshness or stale source documents Verify ingestion updates and source revisions
Slow responses Search, reranking, model inference or network latency Measure each pipeline stage separately
Incorrect citations Source mapping or response attribution Trace document and chunk identifiers
Unauthorised document exposure Access-control enforcement failure Contain exposure and inspect identity, filters and logs
Embedding mismatch Incompatible vector dimensions or embedding configuration Verify index schema and embedding model compatibility

Investigation method: Separate ingestion failures, retrieval failures, authorisation failures and generation failures. A model cannot reliably use evidence that the retrieval pipeline never supplied, while a successful search request does not guarantee a correct answer.

Section 4 — Engineering Takeaway

Enterprise RAG engineering combines document ingestion, chunking, embeddings, Azure AI Search, hybrid retrieval, grounding, citations, security and observability. The objective is not merely to retrieve similar text. A reliable production RAG system must retrieve relevant and authorised evidence, preserve provenance, manage context efficiently and provide enough telemetry to explain why an answer was produced.

Network Insight · Engineering Lab 04 · Microsoft Foundry

RAG Retrieval Quality Workbench

Investigate how chunking, retrieval strategy, ranking and Top-K selection affect enterprise knowledge retrieval. Compare synthetic search results, inspect grounding evidence and diagnose missing or weak citations before a model generates its response.

Retrieval engineering configuration

0 Indexed chunks
0 Retrieved chunks
0% Evidence coverage
Pending Grounding assessment

Retrieval engineering inspector

Azure AI Search

Retrieval diagnostics console

Ranked retrieval results

Grounded answer assessment

Run retrieval to inspect evidence quality.
Engineering scope: This is a synthetic local retrieval workbench. Keyword scores, vector similarity, fusion ranking, evidence coverage and grounding labels are illustrative. It does not call Azure AI Search or generate a real model response. Actual Azure AI Search uses configurable analyzers, vector indexes, scoring profiles, hybrid search, optional semantic ranking and service-specific score interpretation. A retrieval score alone does not establish answer correctness or citation faithfulness.
Network Insight / Microsoft Foundry / Section 05

Microsoft Foundry Networking Architecture: Private Endpoints, VNets, DNS, Egress and Secure AI Connectivity

Enterprise Microsoft Foundry deployments depend on network connectivity between applications, model endpoints, agent execution environments, identity providers, retrieval systems and supporting Azure services.

Although AI platforms are often described in terms of models and agents, their production reliability depends heavily on familiar cloud networking principles: IP addressing, DNS resolution, routing, firewall policy, private connectivity and service dependency management.

A secure architecture must determine which components require private access, which operations need outbound connectivity and how traffic crosses trust boundaries. The exact network capabilities available depend on the selected Foundry resources, agent runtime, model deployment and supporting Azure services.

5.1 — Enterprise Foundry Hub-and-Spoke Network Architecture

A common Azure enterprise design uses a hub-and-spoke network topology. Shared security and connectivity services reside in the hub, while applications and private service endpoints are organised into spokes.

HUB VNET Azure Firewall Routing · Inspection · Egress APPLICATION SPOKE AI Application API / Workload Identity SERVICE SPOKE Private Endpoints Supported AI Services Search / Storage Enterprise Users / Clients Approved Network Entry Private DNS Resolution Private Zones · Resolver · Forwarding

Architecture interpretation: This is a conceptual enterprise network design, not a requirement that Foundry services are deployed inside customer VNets. Private endpoints expose supported Azure service interfaces through private IP addresses in a VNet. The underlying managed service remains operated by Azure. Actual packet paths depend on peering, routes, DNS, endpoint configuration and service-specific network support.

5.2 — Six Foundry Networking Building Blocks

V
Azure VNets

Provide private IP addressing, subnets and network boundaries for customer-managed workloads.

P
Private Endpoints

Expose supported Azure service interfaces using private IP addresses.

D
Private DNS

Resolves supported service names to the correct private endpoint addresses.

F
Firewall Policy

Controls permitted traffic between workloads and external destinations.

E
Controlled Egress

Restricts outbound connections to approved services and APIs.

I
Identity Connectivity

Supports token acquisition, authentication and access to protected service APIs.

5.3 — Private Endpoint Architecture and Traffic Flow

Azure Private Link allows supported Azure services to be accessed through private endpoints. A private endpoint creates a network interface with a private IP address inside a selected subnet.

Applications continue to use the service's supported DNS name, while private DNS configuration resolves that name to the appropriate private endpoint IP address.

Application Private DNS Private Endpoint Service FQDN Private IP Answer Private Service Access

Important: Private endpoint connectivity does not automatically disable public network access. Where supported, configure the service's public access settings separately and validate the effective security posture.

5.4 — Private DNS and Name Resolution Engineering

Private DNS is one of the most important dependencies in private Azure AI networking. An application can have correct routing and firewall permissions but still fail because its service hostname resolves to an unexpected public address.

In hybrid environments, DNS forwarding and Azure DNS Private Resolver may be used to connect on-premises DNS infrastructure with Azure private zones. The exact private DNS zone names depend on the Azure service and endpoint type.

# Illustrative DNS investigation nslookup <service-fqdn> # Check the answer from a specified resolver nslookup <service-fqdn> <dns-server> # Linux / macOS DNS inspection dig <service-fqdn> # Inspect the DNS answer and CNAME chain dig <service-fqdn> +noall +answer # Validate HTTPS reachability curl -v https://<service-fqdn>/

Engineering sequence: Confirm the hostname, inspect the DNS response, verify the expected private IP address, examine routing, then test TCP/TLS connectivity and application authentication. A successful DNS lookup does not prove that the service API is reachable.

5.5 — Foundry Network Dependency Matrix

Dependency Traffic purpose Engineering consideration
Model endpoint Inference requests and responses Endpoint type, DNS, authentication and network support
Foundry management APIs Resource and project administration Management-plane connectivity and permissions
Agent tools External tool operations Execution location, egress and identity
Azure AI Search Knowledge retrieval Private access, index API connectivity and authorisation
Azure Storage Documents, artifacts or supporting data Service endpoints, private access and DNS
Microsoft Entra ID Token acquisition and identity operations Required identity service connectivity
Monitoring services Logs, metrics and traces Telemetry ingestion paths and data protection

Not every deployment requires every dependency listed above. Engineers should document the actual services used by their application and agent execution architecture.

5.6 — Controlled Outbound Access and Tool Connectivity

AI agents may require outbound access to enterprise APIs, search services, databases or approved third-party systems. The network architecture must distinguish between traffic originating from the application and traffic originating from a managed or customer-hosted agent execution environment.

Application-Controlled Egress

Customer-managed applications can use appropriate subnet routing, network security groups, firewall policies and approved outbound connectivity.

Managed Runtime Egress

Managed agent or tool execution environments may have different network integration capabilities. Validate supported private networking and outbound controls for the selected runtime.

Security consideration: Do not assume that placing an application in a VNet automatically makes every agent tool call originate from that VNet. Identify the actual execution location and network boundary for each tool integration.

5.7 — Routing, Firewall and Network Security Controls

Route Tables

Control next-hop selection for supported workload subnet traffic.

Network Security Groups

Filter permitted traffic at applicable subnet or interface boundaries.

Azure Firewall

Provides centralised network and application traffic controls.

Private DNS Zones

Map supported private service names to private endpoint addresses.

Private Link

Enables private access to supported managed service interfaces.

Network Monitoring

Supports investigation of reachability, routing and filtering.

User-defined routes and centralised firewall insertion must be designed carefully. Private endpoint traffic can be subject to specific routing and network policy behaviour, depending on subnet configuration and supported Azure capabilities.

5.8 — Production Foundry Network Troubleshooting

Symptom Potential cause Investigation
Endpoint resolves to public IP Incorrect private DNS configuration Check zone records, VNet links and DNS forwarding
DNS resolves but connection times out Routing, filtering or endpoint connectivity issue Inspect routes, NSGs, firewall rules and endpoint state
HTTP 401/403 Identity, authorisation or access policy failure Verify credentials, roles and service access restrictions
Search works publicly but not privately Private endpoint or DNS mismatch Validate search endpoint configuration and private resolution
Agent tool cannot reach API Unsupported or restricted runtime egress Identify actual tool execution environment
Intermittent connectivity DNS inconsistency, transient service or network issues Correlate DNS answers, connection traces and timestamps
Telemetry missing Monitoring configuration or ingestion connectivity Verify diagnostic settings and telemetry destinations

Recommended troubleshooting order: Start with the service FQDN and expected network path. Validate DNS, IP reachability, TCP/TLS connectivity, service access policy, identity and finally application-level API behaviour. This avoids treating every connectivity problem as a model deployment failure.

5.9 — Production Network Architecture Checklist

Connectivity Design

Document service endpoints, DNS zones, private endpoint subnets, VNet relationships, routing paths and required outbound dependencies.

Security and Operations

Validate public access restrictions, firewall policy, identity boundaries, network monitoring, change controls and incident recovery procedures.

Section 5 — Engineering Takeaway

Microsoft Foundry networking combines Azure VNets, private endpoints, Private DNS, routing, firewall policy, identity connectivity and controlled service dependencies. Production engineers must understand where application and agent traffic originates, which endpoints support private access and how DNS and routing influence the actual request path. A secure AI deployment requires both application security and a validated network architecture.

Network Insight · Engineering Lab 05 · Microsoft Foundry

Private AI Network Path Investigator

Trace a conceptual application-to-Azure-service connection. Investigate private DNS resolution, private endpoint readiness, routing, security controls and service authorisation. Inject failures to identify exactly where an enterprise AI network path breaks.

Network investigation configuration

Idle Path status
0 / 6 Checks completed
Pending DNS status
— Result code
Select a destination and scenario, then start the investigation.

Network engineering inspector

Private Endpoint

Network diagnostics console

Hop-by-hop path investigation

Production networking note: The stages represent diagnostic checks, not literal packet-forwarding hops. DNS resolution and identity validation are separate processes from IP forwarding. A private endpoint provides private connectivity to a supported Azure service; it does not automatically configure all required DNS, routing, firewall or authorisation controls. Supported private connectivity patterns vary across Microsoft Foundry and its dependent services.
Network Insight / Microsoft Foundry / Section 06

Microsoft Foundry Security Architecture: Identity, RBAC, Agent Permissions, Data Protection and Zero Trust Governance

Enterprise AI security requires more than protecting model endpoints. Microsoft Foundry environments may connect applications, model deployments, agents, enterprise knowledge systems, external APIs and sensitive business workflows. Each connection introduces an identity, authorisation, network or data-trust boundary.

A production security architecture must enforce least privilege, protect credentials, restrict access to sensitive information, validate agent tool operations and establish appropriate governance over model and application behaviour.

Microsoft Entra ID, Azure role-based access control, managed identities, Key Vault, network security controls, content safety mechanisms and operational monitoring can contribute to these requirements. Their exact configuration depends on the Foundry services and deployment architecture being used.

6.1 — Zero Trust Security Architecture for Foundry

A useful design approach is to treat every interaction between the user, application, model, agent, tool and data service as an explicit trust-boundary crossing.

PROTECTED AI EXECUTION BOUNDARY Enterprise User Identity + Permissions Application Auth + Policy Agent Runtime Tools + State Model Endpoint Inference Tool Authorisation Validation + Approval Enterprise Data Protected Access

Security interpretation: The diagram illustrates logical trust boundaries. It does not imply that every Foundry component runs inside a single customer-controlled network. Managed services and agent execution environments may have separate identity, network and administrative boundaries.

6.2 — Six Core Security Controls

01
Authentication

Establish the identity of users, applications and service workloads.

02
Authorisation

Apply least-privilege permissions to resources, APIs, tools and data.

03
Network Security

Restrict supported service access and unnecessary connectivity.

04
Data Protection

Protect sensitive prompts, retrieved data, outputs and logs.

05
Agent Controls

Validate tool requests and require approval for sensitive actions.

06
Governance

Apply policy, monitoring, evaluations and change management to AI systems.

6.3 — Microsoft Entra ID and Managed Identity Architecture

Microsoft Entra ID provides identity capabilities used by Azure applications and services. Managed identities allow supported Azure workloads to obtain credentials without embedding long-lived secrets in application code.

The identity used by an application may differ from the identity used by an agent execution environment or external tool. Engineers must identify the actual security principal at every service boundary.

Identity type Typical use Security consideration
User identity Interactive user authentication User permissions and delegated access
System-assigned managed identity Identity associated with an Azure resource Lifecycle and scope of access
User-assigned managed identity Reusable identity assigned to supported resources Shared permission exposure and ownership
Service principal Application or automation identity Credential lifecycle and least privilege
API key Authentication for supported service APIs Storage, rotation and key exposure

Engineering principle: Managed identity removes the need to manage certain application credentials, but it does not automatically grant access to a service. Appropriate permissions must still be assigned to the correct principal.

6.4 — Azure RBAC and Permission Boundaries

Azure role-based access control assigns permissions to security principals at supported scopes. A production Foundry environment may require different permissions for resource administration, model inference, agent operations and access to enterprise data.

Boundary Example operation Control objective
Resource management Create or configure supported Azure resources Restrict administrative privileges
Model inference Invoke an authorised model endpoint Limit runtime model access
Agent execution Run approved agent workflows Constrain runtime capabilities
Search access Query protected enterprise indexes Enforce data access policy
Tool execution Invoke an external enterprise API Restrict permitted business operations
Monitoring Read operational telemetry Protect sensitive diagnostic information

Role names and supported permissions vary by service and API. Production configuration should be validated against the current permission model for each Foundry capability.

6.5 — Secrets and Credential Protection

Applications may require credentials for external APIs, databases or other enterprise systems. Secrets should not be embedded in prompts, source code, logs or agent instructions.

Key Vault

Centralise supported secret and certificate storage and access.

Credential Rotation

Establish procedures for replacing credentials without uncontrolled downtime.

Least Privilege

Restrict credential access to approved workload identities.

Log Redaction

Prevent secrets from entering traces, diagnostics and audit records.

Environment Separation

Use appropriate development, test and production credential boundaries.

Incident Recovery

Support revocation, rotation and investigation after credential exposure.

6.6 — Prompt Injection and Agent Tool Security

Prompt injection occurs when untrusted content attempts to influence the model or agent outside its intended instruction hierarchy. The content may arrive through a user message, retrieved document, web page or tool result.

The principal risk is that a model may treat untrusted source content as an instruction and attempt an unauthorised tool operation or disclose protected data.

Illustrative security boundary Trusted application policy | v Agent instructions | v Untrusted retrieved document | v Model-generated tool proposal | v Independent policy validation | +-- Deny unauthorised action | +-- Request human approval | +-- Execute approved operation

Critical control: Prompt filtering and agent instructions alone cannot guarantee protection against prompt injection. Enforce authorisation, tool restrictions, input validation and approval requirements outside model-generated instructions.

6.7 — Content Safety and Data Governance

Content safety controls can help identify or restrict categories of unsafe input and output where supported. However, content filtering is distinct from data authorisation, privacy controls and tool security.

Control area Purpose Engineering requirement
Input filtering Evaluate supported categories of risky requests Configure appropriate safety behaviour
Output filtering Evaluate generated responses Define handling of blocked or flagged outputs
Data classification Identify sensitive enterprise information Apply access and handling policies
Retention Control stored conversations, traces and artifacts Define lifecycle and deletion requirements
Data residency Meet applicable processing-location requirements Validate deployment-specific commitments
Audit logging Support accountability and investigation Protect logs and control access

6.8 — Governance and Policy Enforcement

Enterprise AI governance should apply to infrastructure, model access, agent tools, application releases, data handling and operational monitoring.

Infrastructure Governance

Azure Policy and infrastructure-as-code controls can help standardise supported resource configurations, network settings and security requirements.

AI Application Governance

Model evaluations, agent testing, approval workflows, release controls and runtime monitoring help govern application behaviour.

Governance distinction: Azure resource policies do not automatically validate the correctness, safety or reliability of model-generated answers and agent actions. Application-level evaluation and runtime enforcement remain necessary.

6.9 — Production Security Troubleshooting

Symptom Potential cause Investigation
Model request returns 401 Invalid or missing authentication Inspect token audience, credential and endpoint requirements
Model request returns 403 Insufficient permission or access restriction Inspect effective roles and service policies
Agent tool access denied Tool identity lacks required permissions Identify actual execution principal and target API authorisation
Unexpected document access Retrieval authorisation or filtering failure Inspect document permissions and trusted filter enforcement
Secret appears in logs Inadequate logging or redaction controls Contain exposure, rotate credential and review telemetry
Agent proposes unsafe action Prompt injection or insufficient tool controls Review tool policy, approval gates and execution trace
Deployment blocked Policy or permission enforcement Inspect deployment errors and applicable policy assignments

Incident response: For suspected data exposure or unauthorised tool execution, prioritise containment, credential revocation, preservation of audit evidence and investigation of the affected trust boundary.

Section 6 — Engineering Takeaway

Microsoft Foundry security engineering combines Microsoft Entra ID, managed identities, Azure RBAC, credential protection, network controls, agent tool authorisation, content safety, data governance and operational monitoring. Production security requires independent enforcement at every important trust boundary. Model instructions can guide behaviour, but identity permissions, application policies and explicit approval controls must determine what the system is actually allowed to do.

Network Insight · Engineering Lab 06 · Microsoft Foundry

AI Security Boundary Inspector

Investigate how user identity, managed identities, resource roles, tool permissions, retrieval access and data protection controls govern a production AI application. Configure security controls, inject policy failures and trace whether an operation should be permitted, denied or flagged for review.

Security investigation configuration

Trust boundary 01 · User and workload identity
Trust boundary 02 · Foundry runtime and authorisation
Trust boundary 03 · Enterprise data and secrets
Pending Security decision
0 Checks passed
0 Failed checks
— Exposure assessment

Security engineering inspector

Microsoft Entra ID

Security decision console

Security boundary findings

Security engineering scope: This lab uses synthetic policies and simulated access decisions. Real Microsoft Foundry deployments may use different identity, RBAC, network, agent and content protection mechanisms depending on service configuration. Azure RBAC, data-plane authorisation, retrieval security trimming and application-level tool permissions are distinct controls. The widget does not modify Azure resources or evaluate a real identity token.
Network Insight / Microsoft Foundry / Section 07

Microsoft Foundry Observability and Evaluation: Distributed Tracing, Model Performance, Agent Telemetry and Production Quality

Production AI observability extends traditional application monitoring by introducing model-specific and agent-specific execution behaviour. A single user request may trigger model inference, knowledge retrieval, multiple tool calls, agent orchestration and external service dependencies.

Engineers must be able to answer several operational questions: Where did latency occur? Which model deployment processed the request? How many tokens were consumed? Which tools were invoked? What evidence was retrieved? Did the response meet quality and safety expectations?

Microsoft Foundry monitoring and evaluation capabilities can be combined with Azure Monitor, Application Insights, OpenTelemetry-compatible instrumentation and application-specific quality evaluation workflows, depending on the selected services and runtime.

7.1 — End-to-End Production AI Telemetry Architecture

A useful observability design connects application traces, agent execution events, model request metrics, retrieval telemetry and tool dependency records into a correlated operational view.

AI APPLICATION EXECUTION Application Agent Retrieval Model Tools Request Trace Execution Events Search Metrics Tokens + Latency Dependency Calls Observability Collection Traces · Metrics · Logs · Evaluations Dashboards Quality Evaluation Alerting

Architecture interpretation: This diagram represents logical telemetry collection. Not every Foundry capability automatically emits every metric or trace shown. Production instrumentation must be designed and validated for the specific application, SDK, model API, agent runtime and supporting services.

7.2 — Six Observability Dimensions

01
Application Reliability

Request success, errors, availability and end-to-end latency.

02
Model Performance

Inference latency, token consumption, throughput and throttling.

03
Agent Execution

Workflow steps, tool calls, retries, failures and completion.

04
Retrieval Quality

Search latency, relevance, grounding and citation correctness.

05
Safety and Quality

Evaluation results, policy outcomes and response quality.

06
Cost and Capacity

Token demand, deployment utilisation and operational expenditure.

7.3 — Distributed Tracing Across AI Request Paths

Distributed tracing records the timing and relationships between operations involved in a request. A parent application span may contain child spans for agent execution, retrieval, model inference and tool calls.

Application Agent Retrieval Model Tool End-to-End Request Agent Execution Search Inference Tool Start Completion

The waterfall above is illustrative rather than measured telemetry. Real traces may contain parallel operations, multiple model calls, repeated tool execution and nested agent workflows.

Tracing principle: Capture correlation identifiers and span relationships, but avoid recording sensitive prompts, credentials, retrieved documents or personal information without an approved data-handling policy.

7.4 — Model Inference Performance Metrics

Metric Meaning Engineering use
End-to-end latency Total application response time Measure user experience
Time to first token Delay before first streamed output token Measure responsiveness
Output token rate Generated output tokens per second Evaluate generation performance
Input tokens Prompt and context consumption Analyse context efficiency
Output tokens Generated token consumption Analyse latency and cost
HTTP 429 rate Throttled requests Detect quota or capacity pressure
Error rate Failed requests over total requests Monitor reliability
Concurrency Simultaneous active operations Understand workload pressure

7.5 — Agent Execution Telemetry

Agent observability requires visibility into the workflow, not just the final answer. An agent may successfully return a response while performing unnecessary tool calls or consuming excessive model capacity.

Execution Steps

Number and order of workflow operations.

Tool Calls

Invoked tools, arguments and outcomes under approved logging rules.

Retries

Repeated model or dependency operations.

Execution Duration

Total time spent in agent workflows.

Failure Location

Specific operation that caused an error.

Approval Events

Human authorisation and high-impact operation decisions.

7.6 — Production AI Evaluation Framework

Traditional infrastructure monitoring measures whether a service is available and responsive. AI evaluation additionally measures whether the system produces useful, accurate and appropriate results.

Evaluation dimension Question Example method
Relevance Does the answer address the question? Reference datasets and evaluator scoring
Groundedness Are claims supported by supplied evidence? Compare response claims with retrieved context
Retrieval quality Were relevant documents returned? Precision, recall and labelled queries
Tool correctness Did the agent invoke appropriate tools? Execution trace evaluation
Safety Does output satisfy defined safety requirements? Policy tests and supported evaluators
Task completion Did the workflow achieve its objective? Scenario-based acceptance tests
Regression Did quality change after a release? Compare evaluation results across versions

Evaluation caution: Automated evaluator scores are estimates and may themselves be imperfect. For high-impact workflows, combine automated evaluation with representative datasets, human review and application-specific acceptance criteria.

7.7 — Service Level Objectives for AI Applications

Production teams can define service level objectives for reliability and performance, while separately tracking AI response quality.

Illustrative AI application SLOs Availability target: 99.9% successful service availability Latency target: 95% of requests complete within the defined workload-specific budget Error-rate target: Failed requests remain below an agreed operational threshold Quality target: Evaluation results meet the approved release baseline Cost target: Average cost per completed task remains within the agreed budget

These are example objectives, not Microsoft Foundry service guarantees. Appropriate thresholds depend on the workload, deployment type, business criticality and measured baseline.

7.8 — Token Usage, Capacity and Cost Observability

Token consumption affects both operating cost and serving capacity. Agent applications may make several model calls for a single user request, making per-task cost more useful than per-request model cost alone.

Illustrative task cost model Task cost = Model inference charges + Retrieval service charges + Tool execution charges + Supporting infrastructure costs Cost per completed task = Total attributable cost ----------------------- Number of completed tasks

Cost engineering: Pricing varies by model, deployment type, region, usage category and supporting services. Use current service billing data rather than assuming a fixed token price across all Foundry deployments.

7.9 — Production Observability Troubleshooting

Symptom Investigation area Engineering action
High end-to-end latency Application, retrieval, model or tools Inspect correlated spans and dependency durations
Unexpected token growth Prompt context or repeated model calls Review token accounting and agent execution steps
Missing trace spans Instrumentation or context propagation Validate telemetry configuration and trace correlation
Agent task fails intermittently Tool dependency or execution state Compare successful and failed execution traces
Quality regression Model, prompt, retrieval or tool changes Compare versioned evaluation datasets and outputs
Unexpected cost increase Usage growth, retries or capacity configuration Correlate billing, tokens and workload volume
Missing monitoring data Diagnostic configuration or telemetry ingestion Inspect instrumentation, permissions and ingestion paths
Section 7 — Engineering Takeaway

Production Microsoft Foundry observability combines distributed tracing, model performance metrics, agent execution telemetry, retrieval evaluation, response quality, safety monitoring, capacity analysis and cost engineering. A successful HTTP response does not necessarily mean an AI task succeeded. Engineers must observe both the infrastructure that executes the workload and the quality of the outcome it produces.

Network Insight · Engineering Lab 07 · Microsoft Foundry

AI Production Telemetry Analyzer

Investigate production AI health using request latency, token consumption, error rates, quality indicators and cost estimates. Compare incident scenarios, inspect individual traces and identify which signals reveal the likely operational failure domain.

Telemetry investigation configuration

0s Average latency
0% Request error rate
0 Total tokens
$0 Illustrative cost
Production telemetry timeline
Synthetic request traces

Request trace inspector

Production telemetry

Select a request trace to inspect its execution details.

Observability diagnostics console

Operational findings and incident hypotheses

Observability scope: This workbench generates deterministic synthetic telemetry. The token prices, latency values, error rates and quality assessments are illustrative rather than live Microsoft Foundry measurements. Real production analysis should combine distributed traces, service metrics, model and agent diagnostics, evaluations, application logs and actual billing data. Quality, safety and groundedness require dedicated evaluation methods; they cannot be inferred reliably from latency or token usage alone.
Network Insight / Microsoft Foundry / Section 08

Microsoft Foundry CI/CD and MLOps Architecture: Infrastructure as Code, Evaluation Gates, Release Pipelines and Rollback

Production AI applications require repeatable deployment processes, controlled changes, automated testing and reliable recovery procedures. Microsoft Foundry workloads introduce additional release dependencies beyond traditional application code, including model deployments, agent instructions, tool definitions, retrieval indexes and evaluation datasets.

A mature delivery pipeline must coordinate infrastructure configuration, application artifacts, model selection, security permissions, agent behaviour and operational readiness.

CI/CD and MLOps practices provide a framework for managing these dependencies through version control, automated validation, environment promotion, approval gates and production monitoring.

8.1 — Production AI Delivery Pipeline Architecture

An enterprise delivery workflow typically begins with a source-controlled change and progresses through build, validation, evaluation, approval and production release.

AI APPLICATION DELIVERY PIPELINE Git Repository Build + Test AI Evaluation Approval Gate Code + Config IaC + Security Quality + Safety Release Decision Staging Preproduction Validation Production Monitored Deployment Promote

Architecture interpretation: This diagram represents a logical delivery process. Azure DevOps, GitHub Actions or another approved CI/CD platform may implement the pipeline. Deployment steps and automation interfaces depend on the selected Foundry services, model providers and runtime capabilities.

8.2 — Six Components of Production AI Release Engineering

01
Version Control

Track application code, agent configuration, infrastructure and evaluation definitions.

02
Infrastructure as Code

Provision supported Azure resources through repeatable declarative configuration.

03
Automated Testing

Validate application logic, configuration, security and integrations.

04
AI Evaluations

Measure response quality, grounding, safety and task completion.

05
Release Gates

Require approval when changes exceed defined risk boundaries.

06
Rollback

Restore a validated configuration after a failed deployment.

8.3 — Infrastructure as Code for Foundry Environments

Infrastructure as code helps standardise Azure resource deployment, network configuration, identity assignments, monitoring and supported Foundry service configuration.

Bicep, ARM templates and Terraform can be used for supported Azure infrastructure resources. Not every Foundry runtime artifact or model operation is necessarily managed through the same infrastructure provider or deployment API.

Illustrative infrastructure repository infrastructure/ main.bicep modules/ network.bicep identity.bicep monitoring.bicep ai-resources.bicep application/ src/ tests/ agents/ instructions/ tools/ configuration/ evaluations/ datasets/ test-cases/ thresholds/ pipelines/ build.yml release.yml

Engineering caution: Infrastructure-as-code support changes over time. Validate current provider and API support before assuming every Foundry project, agent, model deployment or runtime configuration can be managed declaratively.

8.4 — Development, Staging and Production Environments

Environment separation reduces the risk that experimentation or testing affects production workloads.

Environment Purpose Engineering controls
Development Implement and experiment with changes Restricted test data and limited permissions
Integration Validate dependent service interactions API contracts, identity and network tests
Staging Exercise release candidate under representative conditions Performance, safety and regression evaluations
Production Serve approved business workloads Controlled access, monitoring and rollback

Separate environments may use different model deployments, capacity allocations and service endpoints. The pipeline should make these differences explicit and prevent unintended cross-environment access.

8.5 — Model, Prompt and Agent Version Management

AI release engineering must track more than application source code. A change in model version, system instructions, tool schema or retrieval configuration can alter application behaviour.

Versioned component Potential impact Release control
Application code Business logic and API behaviour Automated tests and artifact versioning
Model deployment Capabilities, latency and response behaviour Compatibility and evaluation testing
Agent instructions Workflow decisions and output structure Regression evaluation
Tool definitions Available actions and argument contracts Schema and permission validation
Retrieval configuration Knowledge selection and grounding Retrieval relevance testing
Evaluation dataset Quality acceptance criteria Dataset versioning and review

8.6 — AI Evaluation Gates in CI/CD

A conventional software pipeline may validate that code compiles and unit tests pass. An AI pipeline should also validate that important application behaviours remain within approved quality and safety thresholds.

Illustrative release decision Build successful? | +-- No → Block release | +-- Yes | v Security checks passed? | +-- No → Block release | +-- Yes | v AI evaluation baseline passed? | +-- No → Investigate regression | +-- Yes | v Approval requirements satisfied? | +-- No → Hold deployment | +-- Yes → Promote release

Evaluation principle: Quality gates should use representative and versioned test datasets. A model change that improves one evaluation score may still introduce regressions in other workload categories.

8.7 — Deployment Strategies and Controlled Rollout

Staged Deployment

Validate a release in lower environments before production.

Canary Release

Direct a limited portion of eligible traffic to a new version.

Blue-Green

Maintain separate serving environments where architecture supports it.

Feature Flags

Enable or disable application capabilities without a full redeployment.

Traffic Control

Use application or gateway routing where supported and appropriate.

Rollback

Return to a known-good application and configuration baseline.

Deployment limitation: Canary and blue-green strategies require appropriate application, gateway or serving architecture. They are not automatically available as native traffic-splitting features for every Foundry model deployment type.

8.8 — Rollback and Release Recovery Engineering

Rollback must account for dependencies between application code, model endpoints, agent configuration, retrieval indexes and identity permissions.

Application Rollback

Restore the previous validated application artifact and associated configuration while preserving required runtime compatibility.

AI Configuration Rollback

Restore approved model references, agent instructions, tool definitions and retrieval settings where previous versions remain supported.

Recovery principle: A rollback is not guaranteed merely because a previous configuration is stored in Git. The underlying model version, deployment capacity and dependent services must still be available and compatible.

8.9 — Production CI/CD Troubleshooting

Failure Potential cause Engineering investigation
Infrastructure deployment fails Invalid template, permission or policy Inspect deployment operation and policy errors
Model deployment unavailable Unsupported model, region or capacity Verify deployment support and current limits
Agent evaluation fails Prompt, tool or model behaviour regression Compare versioned traces and test cases
Staging works but production fails Environment-specific identity, DNS or configuration Compare effective environment settings
Release increases latency Model, context or dependency change Compare pre- and post-release telemetry
Rollback fails Missing artifact or incompatible dependency Inspect release manifests and dependency availability
Unexpected production behaviour Configuration drift or incomplete release validation Compare deployed state with approved release baseline
Section 8 — Engineering Takeaway

Microsoft Foundry CI/CD and MLOps engineering combine version control, infrastructure as code, automated testing, AI evaluations, environment promotion, release approval, deployment strategies and rollback planning. Production AI delivery must manage both deterministic software components and model-driven behaviour. A reliable release process validates infrastructure correctness, application security, AI quality and operational readiness before changes reach production.

Network Insight · Engineering Lab 08 · Microsoft Foundry

AI Release Pipeline Simulator

Simulate a production AI release from version-controlled configuration through Infrastructure as Code validation, model evaluation, security checks, staging deployment, approval, canary traffic and production promotion. Investigate release failures and practise rollback decisions.

Release engineering configuration

GIT Source Control Versioned release
IaC Infrastructure Validation Terraform / Bicep
EVAL Model Evaluation Quality gates
SEC Security Checks Policy / Secrets
STG Staging Deployment Integration tests
OK Release Approval Human gate
10% Canary Evaluation Traffic / Telemetry
PRD Production Promotion Release complete
Idle Pipeline state
0 / 8 Stages completed
v2.0 Production version
0 Passed gates
Select a release scenario and start the pipeline.

Release engineering inspector

Production Release

Select a pipeline stage to inspect its engineering purpose.

Release diagnostics console

Release audit trail

Release engineering scope: This is a conceptual CI/CD simulator. It does not execute Terraform, deploy Azure resources, call Microsoft Foundry APIs or change production traffic. Real production pipelines require service-specific deployment capabilities, immutable release artifacts, environment isolation, identity controls, automated evaluations, telemetry and tested rollback procedures. Model deployment rollback and application traffic rollback may require different mechanisms.
Network Insight / Microsoft Foundry / Section 09

Microsoft Foundry Production Troubleshooting: Model Failures, Agent Incidents, Network Diagnostics and Recovery Engineering

Production Microsoft Foundry applications combine multiple distributed components: client applications, identity services, API gateways, model deployments, agent runtimes, retrieval systems, enterprise tools and monitoring platforms.

When an AI application fails, the visible symptom may appear far from the actual cause. A model timeout may originate from network connectivity, an agent failure may be caused by an external API, and an incorrect answer may originate from stale retrieval data rather than the model itself.

Effective troubleshooting requires a structured method that separates failure domains, correlates evidence and validates recovery against a known-good baseline.

9.1 — Production Incident Investigation Architecture

The first engineering task is to map the affected request path and determine which component failed. A useful investigation follows the request from the application through identity, model inference, retrieval and tools.

PRODUCTION AI REQUEST PATH Application Identity Model Agent Request Entry Auth + RBAC Inference Orchestration Retrieval Azure AI Search Enterprise Tools External APIs Incident Telemetry Traces · Logs · Metrics

Architecture interpretation: The diagram represents logical dependencies. Actual request ordering varies by application, agent design and service configuration. Identity checks, retrieval and tool calls may occur at several points within one request.

9.2 — Six Production Failure Domains

01
Application

Invalid requests, timeouts, configuration errors and client failures.

02
Identity

Authentication, RBAC, token scope and permission failures.

03
Model

Deployment errors, throttling, capacity and inference latency.

04
Retrieval

Search failures, stale indexes and incorrect evidence.

05
Agent Tools

External API failures, execution loops and invalid tool calls.

06
Networking

DNS, private endpoints, routing and firewall restrictions.

9.3 — Structured Fault Isolation Method

A consistent investigation process reduces the risk of changing unrelated components during an incident.

Production investigation sequence 1. Confirm the user-visible symptom 2. Determine affected scope - All users or selected users? - All deployments or one endpoint? - All requests or one workflow? 3. Identify the last known-good state 4. Correlate request and trace identifiers 5. Locate the first failing dependency 6. Compare configuration and telemetry 7. Form a testable root-cause hypothesis 8. Apply a controlled mitigation 9. Validate recovery and monitor 10. Record incident findings

Engineering principle: Do not begin with random configuration changes. First establish the failure boundary and collect evidence from the affected request path.

9.4 — Model Endpoint and API Failure Investigation

Symptom Potential cause Investigation
HTTP 400 Invalid request, unsupported parameters or malformed payload Inspect request schema and API contract
HTTP 401 Missing or invalid authentication Verify credential, token and endpoint requirements
HTTP 403 Insufficient permission or access restriction Inspect effective roles and service policy
HTTP 404 Incorrect endpoint, route or deployment reference Verify endpoint URL and deployment configuration
HTTP 429 Quota or capacity throttling Review request rate, limits and retry guidance
HTTP 5xx Service or downstream failure Inspect service error details and dependency health
High latency Long context, generation workload, queueing or dependencies Compare inference and end-to-end timing

HTTP diagnostics: Status codes indicate failure categories, not definitive root causes. Capture safe request identifiers, response error details and timestamps for correlation. Avoid logging sensitive prompt content unnecessarily.

9.5 — HTTP 429 Throttling and Capacity Incidents

Throttling can occur when a workload exceeds applicable request, token or capacity limits. The exact limits and accounting behaviour depend on the selected model and deployment type.

Immediate Mitigation

Reduce unnecessary concurrency, respect supported retry guidance, apply bounded exponential backoff and investigate unexpected request bursts.

Long-Term Remediation

Measure peak workload demand, optimise token usage, review available quota or capacity and redesign traffic distribution where appropriate.

Illustrative retry policy IF response indicates throttling: 1. Inspect Retry-After if supplied 2. Wait for the advised interval or use bounded backoff with jitter 3. Retry only when safe 4. Stop after a defined attempt limit 5. Record the failed operation 6. Return a controlled error if the retry budget is exhausted

Reliability warning: Unbounded retries can increase system pressure and amplify incidents. For agent tool operations that modify external state, verify idempotency before repeating the operation.

9.6 — Agent Execution and Tool-Call Failures

Agent failures may occur because the model selects an inappropriate tool, a tool request fails validation, the runtime lacks permissions or an external API becomes unavailable.

Failure Investigation Control
Tool not invoked Inspect tool definitions and agent instructions Improve schemas and test scenarios
Tool returns 403 Identify execution identity and effective permissions Correct least-privilege access
Tool timeout Inspect dependency latency and network connectivity Apply timeouts and safe retries
Repeated tool calls Inspect workflow state and termination conditions Enforce execution limits
Incorrect final response Compare intermediate tool results with generated answer Evaluate grounding and task completion
Duplicate external action Inspect retries and operation completion status Use idempotency and reconciliation

9.7 — Azure AI Search and RAG Incident Investigation

RAG incidents often appear as model quality problems even when the actual failure occurs during document ingestion, indexing, retrieval or context construction.

Missing Documents

Verify source ingestion, index fields and query filters.

Stale Knowledge

Inspect index refresh and source update timestamps.

Low Relevance

Review query formulation, retrieval strategy and ranking.

Incorrect Citations

Trace chunk identifiers and source mapping.

Search Latency

Measure query execution and downstream dependency timing.

Access Failures

Validate search identity, permissions and document-level controls.

9.8 — Private Endpoint, DNS and Network Troubleshooting

Network incidents should be investigated from the perspective of the workload making the request. A private endpoint may be correctly configured while a client resolves the wrong address or follows an unexpected route.

# Illustrative diagnostic commands # Inspect DNS resolution nslookup <service-fqdn> # Inspect DNS answer chain dig <service-fqdn> +noall +answer # Test HTTPS connection and TLS negotiation curl -v https://<service-fqdn>/ # Inspect the local route to an IP on Linux ip route get <destination-ip> # macOS route inspection route -n get <destination-ip>
Observation Possible fault domain Next check
Unexpected public DNS answer Private DNS configuration Zone links and resolver forwarding
Private IP resolves but TCP fails Routing or filtering Routes, firewall, NSG and endpoint state
TLS handshake fails Certificate, proxy or TLS configuration Hostname, trust chain and intermediary policy
Connection succeeds but API returns 403 Identity or service access policy Authentication, roles and endpoint restrictions
Agent cannot reach a tool Runtime egress or tool network configuration Identify actual execution environment

Network investigation: Validate DNS, routing, TCP connectivity, TLS negotiation, service access policy and authentication as separate stages. A successful TCP connection does not prove that a request is authorised.

9.9 — Incident Timeline and Change Correlation

A production incident should be correlated with configuration changes, model updates, agent releases, identity modifications and service health events.

Deployment Error Spike Mitigation Recovery Change introduced Failures detected Controlled action Metrics validated

The timeline is illustrative. Real investigations should record exact timestamps, time zones, affected deployments, request IDs, changes and recovery verification evidence.

9.10 — Production Incident Recovery Framework

Phase Engineering objective Expected evidence
Detect Identify abnormal service behaviour Alerts, metrics and user reports
Triage Determine impact and affected scope Incident severity and service inventory
Isolate Identify failing dependency or change Correlated traces and configuration comparisons
Mitigate Reduce impact safely Approved operational action
Recover Restore expected service behaviour Successful requests and stable metrics
Validate Confirm technical and AI quality recovery Functional tests and evaluation results
Review Prevent recurrence Root-cause analysis and corrective actions

Recovery principle: Restoring HTTP availability is not sufficient if the AI application continues producing incorrect or unauthorised results. Validate both service health and application behaviour before closing the incident.

9.11 — Production Troubleshooting Checklist

Technical Investigation

Confirm endpoint and deployment configuration. Inspect DNS and network connectivity. Verify authentication and effective permissions. Correlate model, retrieval and agent execution traces. Review recent changes.

Recovery and Governance

Apply controlled mitigation. Validate model and agent behaviour. Monitor latency, errors and capacity. Preserve incident evidence. Document root cause and corrective actions.

Section 9 — Engineering Takeaway

Production Microsoft Foundry troubleshooting requires multi-layer fault isolation, model endpoint diagnostics, agent execution analysis, retrieval investigation, identity validation, network troubleshooting, distributed tracing and controlled incident recovery. The objective is to identify the first failing dependency, restore service safely and validate that the AI application once again produces reliable, authorised and operationally acceptable results.

Network Insight · Engineering Lab 09 · Microsoft Foundry

Production Incident Investigation Lab

Investigate simulated Microsoft Foundry production incidents using service telemetry, identity diagnostics, network checks, model capacity evidence, retrieval results and agent tool traces. Determine the root cause, select the correct remediation and verify recovery.

Incident investigation configuration

Idle Incident state
0 Evidence collected
0 Diagnostic commands
100 Investigation score
Select an incident and start the investigation.

Incident diagnostic terminal

Root-cause investigation and remediation

Collected investigation evidence

Engineering scope: The terminal uses simulated diagnostic commands rather than real Azure CLI commands. All incidents, outputs, remediation actions and scores are local to this widget. No Azure resources are modified. In production, verify incident hypotheses using actual service diagnostics, Azure Monitor, application traces, identity logs, network tests and supported Foundry management APIs.
☕
Support Network Insight
Help support the development of interactive networking labs, BGP simulations, and educational content.
Support the Project
Matt Conran
Latest posts by Matt Conran (see all)
Tags: No tags

Comments are closed.