Microsoft Foundry Architecture Explained

Microsoft Foundry Architecture Explained

Network Insight / Azure AI Engineering

Microsoft Foundry Architecture Explained

Microsoft Foundry provides a platform for developing, deploying, evaluating, and operating AI applications and agents using foundation models, connected tools, and enterprise governance controls.

Understanding its architecture requires more than knowing how to deploy a model. Engineers must understand the relationship between Foundry resources, projects, inference services, agent execution, identity, data access, evaluation, and production observability.

Architecture / Platform Overview

Inside a Microsoft Foundry AI Application

An enterprise application can connect to a deployed model directly or use an agent-based workflow that selects tools, retrieves information, and coordinates multiple operations.

Foundry projects organize application development resources and configurations. Model deployments provide inference capabilities, while supported agent services provide additional orchestration and tool integration.

NETWORK INSIGHT / CONCEPTUAL FOUNDRY ARCHITECTURE
Enterprise AI Application Web App · API · Copilot · Business Workflow
Microsoft Foundry Resource + Project Project Configuration · Identity · Connections · Access Control
Models and Inference Model Catalog · Deployments · Inference APIs · Token Processing
Agent Service Agent Runtime · Tool Invocation · State · Orchestration
Knowledge and Tools Azure AI Search · Data Sources · Functions · External Services
Evaluation and Testing Quality · Groundedness · Safety · Regression Evaluation
Security, Governance, and Observability Microsoft Entra ID · RBAC · Network Controls · Monitoring · Traces
Conceptual platform relationship, not a literal network topology or mandatory request path. Available capabilities depend on the resource, model, deployment type, and service configuration.
Architecture / Execution Journey

Five Stages of an Enterprise AI Request

A model-driven application follows an execution path that depends on whether it uses direct inference, retrieval augmentation, or agent orchestration.

The following journey illustrates a representative agent-enabled application that uses model inference and connected tools.

01
Request Application submits an authenticated user request
02
Context Application or agent prepares instructions and context
03
Inference Deployed model processes the input and generates output
04
Tools Optional agent workflow invokes authorized tools or retrieval
05
Response Application returns results and records telemetry
Engineering Detail

Agent workflows are not always linear. A tool result may be sent back to the model for additional inference, and some requests may require several model and tool execution cycles. Evaluation and monitoring are supporting operational processes rather than mandatory sequential inference stages.

Architecture / Responsibility Boundaries

Three Layers Engineers Must Understand

Platform and Control

Foundry resources, projects, access management, model deployment configuration, and service-level governance.

Application and Execution

Prompts, model selection, application logic, agent workflows, tool permissions, and inference behavior.

Operations and Quality

Monitoring, tracing, evaluations, release controls, cost management, incident response, and model quality.

Separating these layers helps engineers identify where a failure originates. A model deployment can be healthy while an agent tool call fails. Similarly, an application can return successful responses while producing incorrect or poorly grounded answers.

Architecture / Engineering Perspective

Why Microsoft Foundry Matters for Production AI

Enterprise AI systems combine infrastructure, application logic, model execution, and operational governance. The complexity increases when applications use external knowledge sources, multiple tools, and agent-driven execution.

Microsoft Foundry provides a common environment for supported AI development and operational workflows, but engineering teams remain responsible for designing appropriate security boundaries, validating outputs, managing dependencies, and establishing reliable production processes.

Core Engineering Takeaway

Microsoft Foundry should be understood as an enterprise AI application platform rather than simply a model deployment portal. Its architecture connects projects, inference, agent execution, knowledge integration, evaluations, and governance into a broader production AI engineering workflow.

Network Insight / Architecture Navigation

Explore the Microsoft Foundry Architecture

Navigate nine engineering layers of Microsoft Foundry, from platform fundamentals and model deployment to AI agents, enterprise security, and production operations. Select any orbiting circle to jump directly to its section.

Section 01 / Microsoft Foundry Fundamentals

Microsoft Foundry Architecture Fundamentals

Microsoft Foundry provides a platform for building and operating AI applications using model deployments, project configurations, agent capabilities, connected services, and enterprise governance controls.

The architecture combines a management and configuration layer with runtime services that execute inference, agent workflows, and connected operations.

1.1 / Platform Architecture

Understanding the Microsoft Foundry Platform

Microsoft Foundry is designed to support the development and operation of AI applications across multiple models, tools, and service configurations.

Engineers interact with platform resources to configure projects, manage deployments, integrate external services, and establish access controls.

01
Foundry Resource

An Azure resource that provides the parent service context for supported Foundry project and platform capabilities.

02
Foundry Project

A project context for organizing AI application development, configurations, connections, and supported resources.

03
Model Deployment

A configured model serving capability used by applications to submit inference requests through supported APIs.

04
AI Application

Application logic that uses deployed models, retrieval systems, or agent workflows to deliver AI-enabled functionality.

1.2 / Layered Architecture

How the Foundry Architecture Fits Together

Foundry applications use multiple architectural layers. The application consumes model or agent capabilities, while the platform provides resource management, deployment configuration, and supporting services.

NETWORK INSIGHT / FOUNDRY LAYERED ARCHITECTURE
Enterprise AI Applications Web Apps · APIs · Copilots · Enterprise Workflows
Microsoft Foundry Resource and Project Context Configuration · Identity · Access · Connections
Models and Inference Model Catalog · Deployments · Model APIs · Token Processing
Agent Execution Agent Runtime · Tools · State · Orchestration
Connected Services Search · Data Sources · Functions · External APIs
Security, Governance, and Observability Entra ID · RBAC · Networking · Evaluation · Traces · Monitoring
Conceptual service architecture. Model deployment, agent runtime, and connected-service configurations are not necessarily hosted within the same execution environment.
1.3 / Resource Hierarchy

Foundry Resources, Projects, and Deployment Relationships

A Foundry resource and its associated projects provide an organizational foundation for supported AI development workflows.

Projects can provide a working context for application development, evaluation, and service integration. However, not every connected service or model deployment is exclusively owned by an individual project.

Component Architectural Role Engineering Consideration
Azure Subscription Azure billing and governance scope Quotas, policy, billing, and access
Resource Group Azure resource management boundary Lifecycle, permissions, and organization
Foundry Resource Parent resource for supported Foundry capabilities Region, access, network, and service configuration
Foundry Project Application development and configuration context Project identity, connections, and collaboration
Model Deployment Configured inference capability Model availability, capacity, authentication, and quotas
External Services Data retrieval, tools, and business integration Identity, network reachability, and data permissions
Important Architecture Boundary

Foundry project organization does not automatically create a dedicated network, compute environment, or isolated model-serving infrastructure for every project. Isolation requirements must be evaluated at the relevant resource, identity, deployment, and network layers.

1.4 / Control Plane and Data Plane

Separating Platform Management from Inference Execution

A useful engineering distinction is the separation between control-plane operations and data-plane operations.

Control-plane operations manage resources and configurations. Data-plane operations execute the actual AI application workload, such as model inference or agent tool execution.

Dimension Control Plane Data Plane
Primary Role Manage resources and configurations Execute workload requests
Typical Operations Create resources, configure deployments, assign permissions Run inference, process tool results, return model outputs
Typical Consumers Platform engineers, administrators, automation Applications, agents, authorized API clients
Failure Examples Deployment configuration rejected, missing role assignment Inference timeout, tool failure, model response error
Operational Focus Governance, resource state, configuration integrity Latency, throughput, quality, availability
Operational Implication

A successful deployment configuration does not prove that inference requests will succeed. Conversely, an existing deployment may continue serving requests even when certain management operations are temporarily unavailable. The exact behavior depends on the affected service and failure domain.

1.5 / Inference Request Journey

Tracing a Request Through a Foundry AI Application

Consider an enterprise application that sends a user request to a deployed foundation model through a supported inference API.

The application must identify the intended deployment, authenticate appropriately, construct the request payload, and handle the model response.

01 / Client Application prepares the request
02 / Authenticate Credentials or tokens are validated
03 / Route Request targets the configured deployment
04 / Infer Model processes input and generates output
05 / Return Application receives and handles results

Agent-based workflows can introduce additional steps, including tool selection, retrieval, state management, and repeated model inference cycles.

1.6 / Identity and Trust

Security Boundaries in Foundry Architecture

Enterprise Foundry deployments must account for identities that manage platform resources and identities that access runtime services.

01
Management Identity

Users or automation principals authorized to configure Foundry resources and deployments.

02
Application Identity

The calling application authenticates to supported model or agent APIs.

03
Tool Identity

Agent tools and integrations require appropriate authorization to connected services.

04
Data Access Boundary

Search indexes, databases, and external resources require their own access controls.

Zero-Trust Principle

Authorization to invoke a model does not automatically authorize access to every data source or external tool used by an AI application. Evaluate each trust boundary independently.

1.7 / Engineering Summary

Foundry as an Enterprise AI Application Platform

Microsoft Foundry connects resource management, project organization, model deployments, agent capabilities, and supporting evaluation and governance workflows.

The architectural challenge is understanding how these services relate without assuming that they all share the same execution boundary or security context.

Key Engineering Takeaway

Microsoft Foundry architecture should be understood through its resource hierarchy, management and runtime boundaries, deployment relationships, application identities, and connected services. These foundations are essential for designing secure, scalable, and observable enterprise AI applications.

Network Insight · Engineering Lab 01

Microsoft Foundry Architecture Explorer

Investigate the platform resource hierarchy, inspect control-plane and data-plane responsibilities, and trace a simulated enterprise AI inference request.

Configuration Runtime External dependency
MICROSOFT FOUNDRY RESOURCE
FOUNDRY PROJECT
ARCHITECTURE COMPONENT

Foundry Resource

Select a component in the architecture to investigate its engineering responsibilities.

Engineering responsibility Platform organization and governance
Operational dependency Azure identity, resource configuration and access
Failure investigation Inspect permissions, resource state and configuration.

Request Flow Simulator

Run a conceptual model inference request through the application, authorization, deployment and model runtime.

0Requests
0/5Steps
IdleStatus

Engineering Event Trace

READY · Select a component or run the request.
Section 02 / Projects and Resource Architecture

Microsoft Foundry Projects and Resource Architecture

Microsoft Foundry projects provide an organizational context for AI application development, while Foundry resources and connected Azure services supply the underlying platform capabilities.

Designing an enterprise Foundry environment requires careful attention to resource hierarchy, project organization, identity, connections, isolation, and governance boundaries.

2.1 / Resource Hierarchy

How Foundry Resources and Projects Are Organized

Foundry environments are deployed within Azure subscriptions and resource groups. A Foundry resource provides a parent context for supported platform capabilities, while associated projects organize application-level development activities.

A single Foundry resource may support multiple projects, allowing teams to organize separate applications or development initiatives under a shared resource context.

NETWORK INSIGHT / RESOURCE HIERARCHY
Azure Subscription and Resource Group Billing · Azure Policy · RBAC · Resource Lifecycle
Microsoft Foundry Resource Parent Resource · Service Configuration · Access Controls
Project A Customer Support AI Application Configuration
Project B Document Intelligence Knowledge Integration
Project C Operations Assistant Agent Workflows
Model Deployments Inference Capacity
Azure AI Search Knowledge Retrieval
Storage Application Data
External APIs Tool Integration
Conceptual enterprise resource organization. Connected services can be shared, independently deployed, or separately governed; the diagram does not imply that they are child resources of each Foundry project.
2.2 / Project Design

What a Foundry Project Represents

A Foundry project is a working context for supported AI application development activities. Projects help teams organize configurations, collaboration, and application-related resources.

Project capabilities and their resource relationships depend on the Foundry resource type, available services, and current platform features.

01
Application Organization

Group related AI application configurations and development activities into a defined project context.

02
Connections

Configure supported connections to external resources such as search services, data sources, and other platform dependencies.

03
Identity

Use supported project and workload identity configurations for authorized service access.

04
Development Workflow

Support model experimentation, agent development, evaluation, and application integration within the available project capabilities.

2.3 / Resource Dependencies

Understanding External Service Relationships

Enterprise Foundry applications often depend on services outside the Foundry project itself.

A retrieval-based application may connect to Azure AI Search, while an agent may invoke external APIs or business services. Each dependency introduces operational and security considerations.

Dependency Architectural Role Engineering Consideration
Model Deployment Provides supported model inference capability Region, quota, deployment configuration, and authentication
Azure AI Search Indexes and retrieves enterprise knowledge Index permissions, network access, and retrieval quality
Azure Storage Stores application documents or related data Data-plane permissions, encryption, and lifecycle policies
Azure Key Vault Stores supported secrets and cryptographic material Access controls and secret rotation
Azure Functions or APIs Provides application or agent tool capabilities Execution identity, authorization, and timeout behavior
Azure Monitor Collects supported operational telemetry Diagnostic configuration, access, and retention
Dependency Engineering Principle

Creating a Foundry connection does not automatically guarantee that a workload can reach or access the connected service. Network reachability, identity, authorization, service configuration, and runtime behavior must be validated independently.

2.4 / Identity Architecture

Managed Identities and Access Boundaries

Foundry applications can involve multiple identities: administrators managing resources, developers working in projects, applications invoking models, and workloads accessing external services.

Microsoft Entra ID, Azure RBAC, and supported managed identities provide important building blocks for enterprise access design.

Identity Context Purpose Security Check
Platform Administrator Manage Azure resources and configuration Scope privileged roles appropriately
Project Developer Develop supported AI application assets Grant only required project and resource access
Application Caller Invoke supported model or agent APIs Validate authentication and inference authorization
Workload Identity Access supported connected Azure services Check managed identity assignment and permissions
External Tool Identity Authorize access to downstream APIs Validate tool-specific authorization and data access
Authorization Boundary

Azure management permissions do not necessarily provide data-plane access to model APIs, storage objects, search indexes, or external services. Each permission boundary must be evaluated separately.

2.5 / Enterprise Isolation

Designing Development, Staging, and Production Environments

Enterprise Foundry architectures commonly separate development, staging, and production workflows to reduce deployment risk and control access to sensitive resources.

Project separation can help organize workloads, but stronger isolation may require separate Foundry resources, resource groups, subscriptions, identities, or network configurations.

Development Model experimentation Agent prototyping Developer access Non-production data
Staging Integration testing Evaluation gates Deployment validation Controlled release approval
Production Restricted permissions Approved model deployments Monitoring and alerting Incident recovery procedures
Isolation Design Principle

Separate projects should not be assumed to provide complete security or infrastructure isolation. Select isolation boundaries based on regulatory, operational, data protection, and workload requirements.

2.6 / Governance Architecture

Managing Projects Across Enterprise Teams

Governance becomes increasingly important as organizations deploy multiple Foundry projects, model endpoints, and agent applications.

01
Resource Naming

Establish consistent naming and tagging conventions across projects, deployments, and connected services.

02
Access Governance

Review privileged roles, project membership, managed identities, and service permissions.

03
Deployment Governance

Define approved model versions, release workflows, evaluation requirements, and change controls.

04
Cost Governance

Monitor model consumption, service usage, quota allocation, and associated Azure resource costs.

2.7 / Troubleshooting

Investigating Foundry Project and Resource Failures

Project-level problems can originate from missing permissions, unavailable dependencies, invalid configuration, or connectivity failures.

Symptom Potential Cause Investigation
Project access denied Missing role assignment or incorrect identity Inspect principal, role scope, and project access
Model invocation fails Deployment, authentication, or quota issue Check endpoint configuration and request errors
Search connection fails Network, identity, or index permissions Test service reachability and data-plane access
Agent tool fails Downstream API authorization or runtime error Inspect tool configuration and execution traces
Resource creation fails Policy, regional availability, quota, or permissions Inspect Azure deployment errors and resource constraints
2.8 / Engineering Summary

Building a Governed Foundry Resource Architecture

Foundry projects help organize enterprise AI development, but production architecture depends on more than project structure.

Engineers must account for resource ownership, identity, network connectivity, external dependencies, deployment configuration, and operational governance.

Key Engineering Takeaway

A Microsoft Foundry project is an application development and configuration context, not automatically a complete infrastructure isolation boundary. Reliable enterprise design requires explicit decisions about resource hierarchy, shared dependencies, managed identities, network controls, environment separation, and governance responsibilities.

Network Insight · Engineering Lab 02

Foundry Project & Resource Dependency Lab

Repair a misconfigured enterprise AI project. Investigate identity, resource access, model deployment and external service dependencies before validating operational readiness.

Engineering Incident Scenario Select an incident to investigate.

Resource Dependency Graph

Click a resource to inspect its role. Red nodes indicate a configuration fault in the selected scenario.

Azure Subscription Billing · Resource governance · RBAC scopes
Microsoft Foundry Resource Resource boundary · Platform configuration
Enterprise AI Project Project identity · Connections · Assets
Identity Entra ID · RBAC
Access Policy Inference authorization
Model Deployment Endpoint · Capacity
Model Service Inference execution
Connection External integration
AI Search Knowledge retrieval

Dependency Inspection

Select a resource

Inspect resource relationships and identify the source of the failure.

Engineering Event Console

READY · Dependency lab initialized.

Project Dependency Configuration

Adjust the settings to restore the required service relationships.

Project Managed Identity Identity
Inference Role Assignment Authorization
Model Deployment Model endpoint
AI Search Connection Knowledge integration
Knowledge Access Permission Data authorization

Architecture Validation

Run validation to inspect project dependencies.

0 Passed
0 Failed
— Readiness
Awaiting Validation Configure dependencies and run the checks.
Section 03 / Model Catalog and Selection

Microsoft Foundry Model Catalog and Model Selection

The Microsoft Foundry model catalog helps engineering teams discover and evaluate foundation models for enterprise AI applications. Selecting a model requires understanding its capabilities, supported deployment options, operational limits, and suitability for the intended workload.

Model selection is an architecture decision. It affects response quality, latency, throughput, cost, governance, and the reliability of production AI applications.

3.1 / Model Catalog Architecture

Understanding the Foundry Model Catalog

Microsoft Foundry provides access to a catalog of supported AI models. Available models can differ in provider, capability, licensing, deployment requirements, and regional availability.

The model catalog is a discovery and selection experience. Choosing a model does not by itself provision serving capacity or guarantee that the model is deployable in every Azure region or subscription.

NETWORK INSIGHT / MODEL SELECTION ARCHITECTURE
Enterprise AI Application Requirements Accuracy · Latency · Cost · Context · Security · Modality
Microsoft Foundry Model Catalog Model Discovery · Capability Assessment · Deployment Compatibility
Language Models Text Generation · Summarization · Question Answering
Reasoning Models Multi-Step Analysis · Planning · Complex Problem Solving
Multimodal Models Supported Text · Image · Audio Processing
Embedding Models Vector Representation · Semantic Retrieval
Supported Deployment Configuration Model Version · Region · Deployment Type · Capacity · Authentication
Production Inference Application API Requests · Monitoring · Evaluation · Cost Management
Conceptual model selection process. Model families describe broad capabilities; individual model versions may support different inputs, APIs, tools, and deployment options.
3.2 / Model Families

Understanding Different Model Capabilities

Not all AI models are designed for the same workload. Engineers should identify the required capability before comparing model performance.

01
General-Purpose Language Models

Support conversational interfaces, document summarization, extraction, and many general text-generation tasks.

02
Reasoning-Oriented Models

Designed for tasks that benefit from deeper analysis, structured reasoning, and complex problem solving.

03
Multimodal Models

Process supported combinations of text, images, audio, or other modalities, depending on the model.

04
Embedding Models

Convert supported input content into vector representations used for semantic search, similarity, and retrieval systems.

Model Capability Principle

Model family labels are useful for initial screening but are not a substitute for checking the exact model version, supported APIs, context limits, tool capabilities, and deployment documentation.

3.3 / Capability Assessment

Comparing Models Against Application Requirements

Enterprise model selection requires evaluating multiple technical dimensions rather than relying on a single benchmark score.

Capability Engineering Question Architectural Impact
Task Quality Does the model produce acceptable results on representative workloads? Determines application usefulness and reliability
Context Window How much input and output context is supported? Influences prompt construction and document processing
Tool Support Does the selected API and model support required tool interactions? Affects agent and workflow architecture
Multimodal Support Which input and output modalities are available? Determines document, image, and audio capabilities
Latency How quickly does the model return usable output? Influences user experience and service-level objectives
Throughput Can the deployment sustain expected workload demand? Determines capacity and scaling requirements
Cost What are the applicable usage or provisioned-capacity costs? Influences application economics
Deployment Availability Is the model supported in the required region and deployment type? Constrains architecture and rollout options
3.4 / Context Engineering

Context Windows, Tokens, and Prompt Design

Model context windows define limits on the information that can be processed within a request or supported interaction.

Engineers must account for system instructions, conversation history, retrieved documents, tool outputs, and expected model output.

Input Tokens

Include user input, instructions, history, and other supported contextual information.

Output Tokens

Represent generated content and may be subject to separate model output limits.

Retrieved Context

External knowledge added to a request can improve grounding but consumes context capacity.

Context Management

Truncation, summarization, retrieval selection, and application state management can control context growth.

Important Context Limitation

A larger context window does not automatically guarantee better retrieval, reasoning, or answer quality. Models should be tested using realistic context lengths and document distributions.

3.5 / Deployment Compatibility

From Model Selection to Deployable Inference

Selecting a model from the catalog is only one stage of deployment planning.

Engineers must confirm the supported deployment options, applicable region, subscription eligibility, quotas, and runtime APIs.

01 / Requirements Define application workload
02 / Shortlist Compare model capabilities
03 / Validate Check regional and API support
04 / Deploy Configure supported serving capacity
05 / Evaluate Test quality, latency, and cost
Deployment Engineering Principle

Availability in a model catalog does not guarantee immediate deployment availability for a particular region, subscription, or deployment configuration. Validate the selected model and deployment type before finalizing application architecture.

3.6 / Performance and Cost

Balancing Model Quality, Latency, and Economics

Production model selection involves tradeoffs between quality, performance, and cost.

A model that performs well on complex reasoning tasks may not be the most efficient option for high-volume, straightforward classification or extraction workloads.

Workload Primary Priority Selection Strategy
Customer Support Latency, quality, cost Benchmark representative conversations and escalation cases
Complex Analysis Reasoning quality and reliability Evaluate difficult tasks and acceptable response times
Document Extraction Accuracy and structured output Test real document formats and validation rules
Enterprise Search Retrieval relevance and embedding cost Measure retrieval metrics on representative queries
High-Volume Automation Throughput and unit economics Test sustained concurrency and cost per completed task
3.7 / Model Evaluation

Why Benchmark Scores Are Not Enough

Public benchmarks can help identify promising models, but they may not represent an organization's actual workload.

Engineering teams should create representative evaluation datasets and test models against real application requirements.

Functional Accuracy

Evaluate whether responses satisfy the expected task requirements.

Safety and Policy

Test relevant safety constraints, policy compliance, and undesirable output behavior.

Operational Performance

Measure latency, throughput, errors, and resource consumption.

Regression Testing

Compare model versions and prompt changes before production rollout.

Production Model Selection Risk

Selecting a model solely because it has a higher benchmark score can produce unnecessary cost, latency, or operational complexity. Use workload-specific evaluations to justify model selection decisions.

3.8 / Engineering Summary

Choosing the Right Model for the Architecture

Microsoft Foundry model selection requires a combination of capability assessment, deployment validation, performance measurement, and enterprise governance.

The best model is not necessarily the largest, most expensive, or highest-ranked model on a public benchmark.

Key Engineering Takeaway

Select Microsoft Foundry models according to application requirements, supported deployment configurations, measurable quality, latency, throughput, cost, and governance. Model selection should be validated through realistic evaluation workloads before production deployment.

Network Insight · Engineering Lab 03

Model Selection Engineering Lab

Design an enterprise AI workload, evaluate candidate model architectures and investigate the engineering trade-offs behind a model deployment decision.

Enterprise Workload Requirements

3.0 s
Illustrative per-request latency estimate, not a service-level guarantee.
$8.00
Synthetic blended cost used for educational comparisons.

Deployment Constraints

Engineering objective

Find a model that satisfies every mandatory constraint. Then compare suitable models using the selected optimization priority. A high-capability model may still be rejected for cost, latency or deployment incompatibility.

Selection Analysis

5 Candidates
— Qualified
— Rejected

Candidate Model Comparison

All model profiles are fictional. Context, latency, cost, capabilities and availability are synthetic lab inputs.

Architecture Decision

Awaiting Model Analysis

Configure the workload and run the model-selection engine.

Model Engineering Inspection

Select Inspect Model to examine capabilities, limitations and deployment trade-offs.

Selection Engine Event Trace

READY · Model catalog profiles loaded.
Section 04 / Deployment and Inference Architecture

Microsoft Foundry Model Deployment and Inference Architecture

Model deployment transforms a selected foundation model into a callable inference capability. Engineers must understand deployment configuration, API routing, token processing, capacity, latency, throughput, and operational limits.

A production AI application is only as reliable as its inference architecture. Model quality alone cannot compensate for insufficient capacity, unstable connectivity, authentication failures, or poorly managed request concurrency.

4.1 / Deployment Architecture

From Model Catalog to Inference Endpoint

A model catalog entry describes an available model and its supported capabilities. A model deployment provides the configured serving capability that applications invoke through supported APIs.

Deployment options vary according to model, provider, region, subscription eligibility, and available serving infrastructure.

NETWORK INSIGHT / MODEL INFERENCE ARCHITECTURE
Enterprise AI Application Web App · API Client · Copilot · Agent Workflow
Authenticated Inference API Request Endpoint · Deployment Identifier · Request Payload · Credentials
Microsoft Foundry Model Serving Supported Deployment · Capacity · Runtime Processing
Request Admission Authentication · Validation · Quota and Rate Controls
Model Execution Input Processing · Model Computation · Output Generation
Response Delivery Generated Output · Usage Metadata · Error Handling
Application Response and Telemetry Response Handling · Latency · Errors · Usage Monitoring
Conceptual inference request flow. The actual serving infrastructure and supported API features depend on the selected model and deployment type.
4.2 / Deployment Types

Understanding Model Serving Options

Microsoft Foundry supports different model deployment arrangements. Their availability depends on the selected model, service, region, and subscription.

Deployment Approach Characteristics Engineering Consideration
Standard Consumption-Based Usage-based serving with applicable shared-capacity limits Rate limits, regional support, variable demand
Provisioned Throughput Reserved serving capacity for supported models Capacity planning, utilization, and commitment cost
Global Deployment Supported serving that can use capacity across eligible geographies Data processing requirements and availability
Data Zone Deployment Supported serving within a defined data zone Data processing boundaries and regional eligibility
Regional Deployment Serving tied to a supported Azure region Regional capacity and resilience design
Managed Compute Hosting Supported models hosted on configured compute resources Compute sizing, scaling, and operational responsibility
Deployment Terminology

Deployment type names and availability differ across model providers and Foundry services. Global, data-zone, regional, standard, and provisioned options are not interchangeable dimensions or universally supported choices. Confirm the exact model's current deployment documentation.

4.3 / Request Processing

How an Inference Request Is Processed

An application constructs a request containing supported input content, instructions, generation settings, and deployment information.

The inference service authenticates and validates the request, applies applicable capacity controls, performs model computation, and returns the result or an error.

01 / Prepare Construct request payload
02 / Authorize Validate credentials and access
03 / Admit Check request and capacity limits
04 / Execute Process model input and output
05 / Respond Return result and usage data
Streaming Versus Non-Streaming

Supported models and APIs may return a completed response or stream partial output. Streaming can improve perceived responsiveness, but it does not necessarily reduce total computation time or the cost of generating the response.

4.4 / Token Processing

Input Tokens, Output Tokens, and Inference Workload

Token processing is a major driver of inference performance and cost for many language models.

The total workload depends on input size, generated output length, model behavior, request volume, and applicable service limits.

01
Prompt Processing

Instructions, user input, retrieved documents, and supported context contribute to input processing requirements.

02
Output Generation

Generated tokens contribute to response duration and usage, with model-specific processing behavior.

03
Concurrency

Multiple simultaneous requests increase aggregate serving demand and may affect latency or admission.

04
Capacity Controls

Applicable request, token, and provisioned capacity limits determine sustainable workload rates.

Token Engineering Detail

Token accounting can vary across model families and modalities. Some models also perform additional internal computation. Use the model's supported usage metadata and service documentation rather than assuming every request has identical token behavior.

4.5 / Performance Engineering

Latency, Throughput, and Concurrency

Production inference architecture must balance response latency with aggregate workload throughput.

Performance is affected by model characteristics, prompt length, output length, serving capacity, network conditions, and concurrent demand.

Metric Meaning Engineering Use
End-to-End Latency Elapsed time from application request to completed response Measure user-facing performance
Time to First Token Time until the first streamed output token is received Evaluate perceived responsiveness
Output Token Rate Generated output tokens per unit time Measure generation performance
Request Throughput Completed requests per unit time Plan application capacity
Concurrency Requests active simultaneously Assess parallel workload demand
Throttling Rate Requests rejected or limited by service capacity controls Identify capacity or quota pressure
Error Rate Proportion of requests that fail Track availability and reliability
Performance Engineering Principle

Average latency alone can hide production problems. Measure percentile latency, request volume, throttling, errors, and workload mix together to understand inference behavior under realistic demand.

4.6 / Capacity and Scaling

Designing Inference Capacity for Production

Capacity planning begins with the application's expected request rate, input and output token distribution, concurrency, and latency objectives.

Engineers should test sustained workloads and traffic bursts rather than relying exclusively on individual successful requests.

Baseline Demand

Establish normal request rates, token usage, and concurrency profiles.

Peak Demand

Evaluate traffic bursts, seasonal demand, and simultaneous application activity.

Scaling Strategy

Choose supported capacity and deployment arrangements based on workload requirements.

Resilience

Design retries, backoff, timeouts, and supported fallback behavior for service failures.

Retry Engineering Risk

Aggressive retries can amplify load during throttling or service incidents. Use bounded retries, exponential backoff, jitter, and appropriate timeout budgets. Respect service retry guidance when available.

4.7 / Production Troubleshooting

Diagnosing Model Deployment and Inference Failures

Inference failures can originate from authentication, deployment configuration, quota limits, network connectivity, model availability, or application logic.

Symptom Potential Cause Investigation
401 Unauthorized Invalid, expired, or missing credentials Inspect authentication method and token audience
403 Forbidden Insufficient authorization or access restriction Check role assignments and applicable service policies
404 Not Found Incorrect endpoint, deployment name, or API route Validate endpoint URL and deployment configuration
429 Too Many Requests Rate limit or capacity restriction Inspect quota, traffic patterns, and retry behavior
High Latency Large prompts, long output, concurrency, or capacity pressure Measure token usage, request timing, and workload mix
Timeout Slow execution, network issue, or client timeout Correlate client traces with service telemetry
5xx Service Error Transient service or serving infrastructure failure Inspect response details, service health, and retry guidance
Incident Investigation Sequence

Start with the failing request's timestamp, endpoint, deployment, HTTP status, request identifier, and application trace. Determine whether the failure occurs before inference admission, during model execution, or while returning the response.

4.8 / Engineering Summary

Designing Reliable Foundry Inference Architecture

Microsoft Foundry model deployment architecture connects model selection, serving configuration, API invocation, capacity management, and operational monitoring.

Production success depends on understanding the complete request lifecycle rather than treating a successful model deployment as proof of application readiness.

Key Engineering Takeaway

Reliable Microsoft Foundry inference requires the correct model and deployment configuration, authenticated API access, sufficient serving capacity, measured latency and throughput, controlled retries, and operational visibility into request failures and usage.

Network Insight · Engineering Lab 04

Inference Request Flow Simulator

Generate model inference traffic, investigate token demand, deployment capacity, request concurrency, queueing and throttling. Observe how infrastructure constraints affect application performance.

Inference Deployment Configuration

Synthetic deployment profiles for exploring capacity planning. They do not reproduce provider admission rules.
4 req/s
1000
500
8
120K TPM

Incident Presets

Load a traffic scenario and investigate the resulting behavior.

Capacity Estimate

360K Demand TPM
120K Capacity TPM
300% Demand / Capacity

Demand is calculated from request rate × (prompt tokens + expected output tokens) × 60. Actual providers may use different rate-limit accounting.

Live Inference Architecture

▣ AI Application Request generation
◇ Inference Endpoint API admission
▤ Capacity Control Tokens · Concurrency
✦ Model Runtime Token generation

Animated conceptual inference path · Request → Admission → Runtime → Response

Live Deployment Telemetry

0 Received
0 Completed
0 429 / Rejected
0 Active
0 Queue Depth
— Avg Latency
Token Capacity Utilization 0%

Engineering Diagnosis

Awaiting Traffic

Start the simulation to investigate deployment behavior.

Inference Event Trace

READY · Inference simulator initialized.
Section 05 / AI Agents and Tool Integration

Microsoft Foundry AI Agents and Tool Integration

AI agents extend model inference into workflows that can interpret requests, select supported tools, retrieve information, and coordinate multiple execution steps.

Microsoft Foundry Agent Service provides capabilities for building and operating supported agent applications. Engineers must understand how the agent runtime, model deployments, tools, application state, and external services interact.

5.1 / Agent Architecture

Understanding the Agent Execution Model

A traditional model inference request usually produces a response from the supplied input and context. An agent-enabled application can introduce a workflow in which model output influences subsequent tool invocations or application actions.

The agent runtime and application logic coordinate supported execution steps, while connected tools perform operations outside the model itself.

NETWORK INSIGHT / AGENT EXECUTION ARCHITECTURE
Enterprise Application or User Request Question · Task · Workflow · Business Operation
Microsoft Foundry Agent Execution Agent Configuration · Runtime · Model Integration
Agent Instructions Role · Task Constraints · Tool Configuration
Model Inference Request Interpretation · Response Generation · Tool Selection
Execution Coordination Tool Requests · State Handling · Execution Control
Knowledge Retrieval Search · Documents · Grounding Context
Application Functions Business Logic · Calculations · APIs
Enterprise Systems Approved Tools · External Services
Result Validation and Application Response Tool Results · Model Output · Application Controls · Telemetry
Conceptual agent architecture. Tool execution may be managed by the service or by application code, depending on the tool type and supported integration pattern.
5.2 / Agent Runtime

What Happens During an Agent Run?

An agent execution begins when an application submits a task or input to a configured agent workflow.

The agent may use model inference to determine whether it can respond directly or requires additional information or tool execution.

01 / Receive Accept application input and context
02 / Interpret Process instructions and task requirements
03 / Select Determine whether tools are needed
04 / Execute Invoke supported tools or functions
05 / Continue Process tool results and additional context
06 / Return Produce a response or failure outcome
Execution Is Not Necessarily Linear

An agent may perform several model and tool execution cycles. Some tasks complete without tools, while others require repeated retrieval, validation, or application-side processing.

5.3 / Tool Integration

How Agents Interact with External Systems

Tool integration allows agent applications to access information or execute approved operations beyond the model's generated text.

Tools may expose search functionality, application functions, data retrieval, or external service operations.

01
Knowledge Tools

Retrieve relevant documents or records from authorized knowledge sources.

02
Function Tools

Request application-defined functions for calculations, validation, or business operations.

03
Connected Services

Integrate with supported Azure services and external APIs through configured connections.

04
Workflow Tools

Coordinate approved operations that require structured inputs, execution checks, or downstream actions.

Tool Execution Boundary

A model's decision to request a tool does not automatically grant permission to execute it. Tool execution must respect the authorization, validation, and runtime controls of the executing application or service.

5.4 / Agent State

Conversation State, Execution Context, and Memory

Agent applications often need to maintain context across multiple steps or user interactions.

State management can involve conversation history, tool results, application data, and persisted workflow information.

State Type Purpose Engineering Concern
Conversation Context Preserve relevant dialogue information Context limits, privacy, and relevance
Tool Results Provide outputs from external operations Validation, freshness, and provenance
Execution State Track progress through supported workflow steps Timeouts, retries, and recovery
Application State Maintain business-specific information Consistency and authorization
Persistent Memory Retain selected information across interactions Retention, access control, and data lifecycle
State Management Principle

Agent state is not equivalent to unrestricted model memory. The persistence, retrieval, and retention of information depend on the agent service, application architecture, and configured storage or memory capabilities.

5.5 / Orchestration Patterns

Designing Multi-Step Agent Workflows

Agent orchestration defines how an application coordinates model inference, tool execution, and task progression.

Single-Agent Workflow

One configured agent coordinates the task using supported tools and model interactions.

Application-Orchestrated Workflow

Application code controls the execution sequence and determines when models or tools are invoked.

Multi-Agent Workflow

Multiple specialized agents collaborate through an explicit coordination pattern, where supported.

Human Approval Workflow

Sensitive actions require human review or approval before the application permits execution.

Architecture Tradeoff

More agents do not automatically produce better outcomes. Multi-agent systems introduce additional model calls, state coordination, latency, cost, and failure modes. Use them only when the workload justifies the complexity.

5.6 / Security Architecture

Identity, Permissions, and Tool Trust Boundaries

Agent applications introduce security boundaries beyond ordinary model inference.

A model may suggest an action, but the application or tool service must decide whether that action is authorized.

Boundary Risk Engineering Control
User to Agent Unauthorized requests or misuse Authentication, authorization, input controls
Agent to Model Incorrect or unsafe generated decisions Instructions, validation, evaluations, application safeguards
Agent to Tool Unauthorized tool invocation Tool allowlists, permission checks, schema validation
Tool to Data Unauthorized information access Least privilege, scoped identity, data permissions
External Content to Agent Prompt injection or malicious instructions Trust separation, output validation, restricted actions
Agent to Business Action Unintended changes to enterprise systems Approval gates, idempotency, audit logging
Zero-Trust Agent Design

Treat model-generated tool requests and externally retrieved content as untrusted inputs. Enforce authorization in application code or the downstream service, not through prompt instructions alone.

5.7 / Production Reliability

Latency, Failure Handling, and Agent Observability

Agent workflows can require multiple model calls and external operations. Each additional step introduces latency and a potential failure point.

Production systems should capture trace information that identifies model calls, tool execution, failures, and overall task outcomes.

Symptom Potential Cause Investigation
Agent produces no useful response Insufficient context, model behavior, or tool configuration Inspect prompts, tool availability, and execution traces
Tool call fails Authorization, invalid arguments, or downstream service failure Inspect tool inputs, identity, and service response
Repeated tool calls Poor termination conditions or repeated model decisions Review orchestration logic and execution limits
High end-to-end latency Multiple model calls or slow external tools Break down latency by execution step
Unexpected data access Overprivileged tool identity or missing authorization Audit identity scopes and downstream permissions
Workflow times out Long-running tools, retries, or orchestration delays Inspect timeouts, retry budgets, and trace durations
Reliability Engineering Principle

Set explicit limits on tool execution, model-call budgets, workflow duration, and retry behavior. Use human approval for sensitive actions and ensure operations are safe to retry where appropriate.

5.8 / Engineering Summary

Building Reliable Microsoft Foundry Agent Systems

Microsoft Foundry agent architecture combines model inference with orchestration, tool integration, state management, and operational controls.

Engineers must distinguish between a model proposing an action and an authorized system executing that action. This separation is essential for secure enterprise automation.

Key Engineering Takeaway

Production AI agents require more than a capable foundation model. Reliable architecture depends on controlled tool execution, scoped identities, validated inputs and outputs, explicit workflow limits, state management, and end-to-end tracing.

Network Insight · Engineering Lab 05

Agent Tool Execution Trace Lab

Investigate an enterprise AI agent as it plans an action, selects a tool, crosses an authorization boundary, executes a service call and produces a final response.

Agent Execution Architecture

Enterprise User Request Natural language instruction · Application context
MICROSOFT FOUNDRY · AGENT EXECUTION
Agent Planning / Model Decision Intent interpretation · Tool selection
Identity & Tool Authorization Credential context · Permission boundary
Tool Execution External service request · Structured result
AI Search Knowledge retrieval
Business API Enterprise operation
Function Tool Custom execution
Tool Result & Agent Observation Result validation · Context update
Final Agent Response Answer generation · Application delivery

Execution Timeline

Agent Runtime Configuration

Retries apply only to the modeled transient timeout and service-unavailable faults.

Runtime Telemetry

0/6 Steps
0 Tool Calls
0 Retries

Engineering Diagnosis

Awaiting Execution

Run the agent to investigate tool execution and failure handling.

Agent Execution Event Trace

READY · Agent runtime initialized.
Section 06 / Knowledge Integration and RAG

Microsoft Foundry Knowledge Integration and RAG Architecture

Retrieval-Augmented Generation connects foundation models to external knowledge sources, allowing AI applications to retrieve relevant information before generating a response.

In Microsoft Foundry architectures, RAG applications can integrate with Azure AI Search and other supported knowledge services to provide enterprise-specific context.

Reliable RAG engineering requires more than attaching documents to a model. Teams must design ingestion pipelines, chunking strategies, embedding workflows, retrieval logic, access controls, and evaluation processes.

6.1 / RAG Fundamentals

Why Retrieval-Augmented Generation Matters

Foundation models generate responses using information available through their trained capabilities and the context supplied during inference.

Enterprise applications often require information that is private, frequently updated, or specific to an organization's systems.

RAG addresses this requirement by retrieving relevant content from external knowledge sources and incorporating it into the model's inference context.

01
External Knowledge

Connect AI applications to approved documents, records, knowledge bases, and enterprise data.

02
Retrieval

Identify relevant information using keyword, vector, or hybrid retrieval techniques.

03
Grounding

Supply retrieved evidence to the model to support responses based on available source material.

04
Evaluation

Measure retrieval relevance, answer correctness, groundedness, and operational performance.

Important Distinction

RAG does not normally modify the underlying foundation model's trained parameters. It provides additional context at inference time. Fine-tuning and RAG solve different engineering problems and may be combined where appropriate.

6.2 / RAG System Architecture

Separating Indexing from Online Retrieval

A production RAG architecture typically contains two distinct processing paths: an ingestion and indexing pipeline, and an online retrieval and generation pipeline.

The indexing pipeline prepares searchable knowledge. The online pipeline uses that knowledge to answer application requests.

NETWORK INSIGHT / ENTERPRISE RAG ARCHITECTURE
Offline Knowledge Ingestion and Indexing Source Processing · Chunking · Embeddings · Index Updates
Data Sources Documents · Storage · Approved Enterprise Data
Preparation Parsing · Cleaning · Chunking · Metadata
Embeddings Supported Vector Representations
Search Index Text · Vectors · Metadata · Filters
Searchable Knowledge Available to Runtime
Enterprise AI Application Request User Question · Identity · Application Context
Online Retrieval and Inference Query Processing · Retrieval · Grounding · Generation
Query User Intent · Query Preparation
Retrieve Azure AI Search · Relevant Chunks
Ground Context Assembly · Source Evidence
Generate Model Inference · Answer Construction
Application Response Generated Answer · Source References · Validation · Telemetry
Conceptual architecture. Azure AI Search can support multiple indexing and retrieval patterns. Embedding generation, query orchestration, and model inference may be performed by different services or application components.
Architecture Boundary

Azure AI Search is a retrieval and indexing service, not the foundation model inference engine. The AI application or supported orchestration service coordinates retrieval and generation.

6.3 / Document Ingestion

Preparing Enterprise Knowledge for Retrieval

Before documents can support RAG, their content must be transformed into a searchable representation.

Ingestion pipelines may include parsing, document normalization, chunking, metadata extraction, embedding generation, and index updates.

01 / Acquire Read approved source documents
02 / Parse Extract usable text and structure
03 / Chunk Divide content into retrieval units
04 / Embed Generate supported vector representations
05 / Index Store searchable fields and metadata
06 / Refresh Update changed or deleted content

The quality of document preparation directly influences retrieval performance. Poor parsing, missing metadata, or inappropriate chunk boundaries can cause relevant information to be omitted from results.

Engineering Consideration

Some Azure AI Search ingestion workflows support integrated enrichment and vectorization. Other architectures perform preparation and embedding generation in application-managed pipelines. Select the approach that matches data formats, control requirements, and operational needs.

6.4 / Chunking and Embeddings

Designing Searchable Knowledge Representations

Chunking divides documents into smaller units that can be independently retrieved.

Embedding models transform supported text or other content into vector representations that enable similarity-based retrieval.

Chunk Size

Influences how much information is returned per result and how precisely content can be matched.

Chunk Overlap

Can preserve context across chunk boundaries, but excessive overlap increases index size and redundant retrieval.

Embedding Consistency

Query and indexed content must use compatible vector representations and index configurations.

Metadata Preservation

Store useful source identifiers, document titles, timestamps, categories, and access-control attributes.

Embedding Compatibility

Vector dimensions and embedding representations must match the configured vector fields and retrieval workflow. Changing embedding models can require re-embedding indexed content and validating retrieval behavior.

6.5 / Azure AI Search Retrieval

Keyword, Vector, and Hybrid Search

Azure AI Search supports multiple retrieval approaches. The appropriate method depends on the structure of the content and the types of questions users ask.

Retrieval Method How It Works Engineering Use
Keyword Search Uses text search and lexical relevance Exact terminology, identifiers, product names, technical keywords
Vector Search Finds similar vector representations Semantic similarity and concept-oriented retrieval
Hybrid Search Combines text and vector retrieval signals Workloads requiring both lexical and semantic matching
Semantic Ranking Uses supported semantic ranking capabilities to refine eligible search results Improve relevance for supported query and index configurations
Metadata Filtering Restricts results using configured filterable fields Tenant, document category, date, or access constraints

Hybrid retrieval can be valuable for technical enterprise content where exact identifiers and conceptual similarity are both important.

Example: Network Troubleshooting Knowledge

A query about a BGP session stuck in Active may benefit from exact matching of protocol terminology and semantic retrieval of troubleshooting procedures. Hybrid retrieval can combine these signals, subject to the configured search pipeline.

6.6 / Grounding Architecture

How Retrieved Evidence Becomes Model Context

Retrieval identifies candidate source material. Grounding is the process of providing relevant evidence to the model so that it can generate a response informed by that material.

Application logic may select, filter, rerank, truncate, or format retrieved passages before including them in the inference request.

01
Evidence Selection

Choose relevant source passages from the retrieval results.

02
Context Assembly

Combine retrieved content with task instructions and user input.

03
Answer Generation

Invoke the model using the selected context and application constraints.

04
Source Validation

Preserve source references and assess whether generated claims are supported.

Grounding Is Not a Guarantee

Supplying retrieved documents does not guarantee factual correctness. Models may misunderstand, omit, or misrepresent source material. Groundedness evaluation, citation validation, and application-level safeguards remain necessary.

6.7 / Enterprise Security

Permission-Aware Retrieval and Data Access

Enterprise RAG applications must ensure that users receive only information they are authorized to access.

Authentication to a search service does not automatically enforce every document-level permission inherited from the original data source.

Security Boundary Risk Engineering Control
Source Ingestion Unauthorized documents indexed Approved source selection and ingestion permissions
Index Access Overprivileged search clients Scoped identities, Azure RBAC, and supported access controls
Document Permissions Users retrieve content outside their authorization scope Permission-aware retrieval, supported ACL mechanisms, or enforced security filters
Tenant Isolation Cross-tenant information exposure Tenant-scoped indexes or rigorously enforced filters and access boundaries
Prompt Injection Malicious instructions embedded in retrieved documents Trust separation, restricted tools, validation, and defense-in-depth
Source References Unauthorized content exposed through citations or snippets Apply authorization to both retrieved content and displayed references
Critical Security Principle

Retrieval authorization must be enforced before sensitive source content is passed into model context. Do not rely on instructions telling the model to ignore unauthorized information.

6.8 / RAG Operations and Troubleshooting

Diagnosing Retrieval and Grounding Failures

A RAG application may successfully invoke a foundation model while still producing poor answers because retrieval failed to supply the correct evidence.

Troubleshooting should separate indexing failures, retrieval failures, context assembly problems, and generation errors.

Symptom Potential Cause Investigation
Relevant document missing Ingestion failure, indexing delay, or missing content Check ingestion runs, source changes, and index contents
Irrelevant retrieval results Chunking, query construction, embedding, or ranking issues Inspect retrieved chunks, scores, filters, and query configuration
Search returns no results Incorrect index, restrictive filters, or query mismatch Validate index name, searchable fields, and filters
Answer ignores evidence Weak grounding instructions or unsuitable context Inspect assembled prompts and compare output with source passages
Outdated answers Stale index or delayed refresh Check source timestamps, indexing schedules, and document versions
High response latency Slow retrieval, reranking, large context, or model inference Trace retrieval and generation latency separately
Unauthorized content exposed Missing or incorrect permission enforcement Stop exposure, inspect access filters and identities, review affected queries
Production Incident Principle

When an answer is incorrect, first determine whether the correct evidence was retrieved. If it was not retrieved, investigate indexing and search. If it was retrieved but misused, investigate grounding, context assembly, and model generation.

6.9 / Engineering Summary

Building Reliable Enterprise RAG with Microsoft Foundry

Microsoft Foundry can provide the model and application development capabilities used in enterprise RAG systems, while Azure AI Search and connected knowledge services supply retrieval functionality.

Reliable RAG architecture requires coordinated ingestion, indexing, retrieval, grounding, security, evaluation, and observability.

Key Engineering Takeaway

Enterprise RAG should be engineered as two connected systems: an offline knowledge preparation pipeline and an online retrieval-and-inference pipeline. The quality and security of the final answer depend on both, not simply on the selected foundation model.

Network Insight · Engineering Lab 06

Retrieval & Grounding Investigation Lab

Investigate enterprise knowledge retrieval, inspect document relevance and permissions, diagnose stale indexes and determine whether an AI response is supported by retrieved evidence.

Retrieval Investigation Configuration

0.65
3

Disabling access enforcement demonstrates an insecure retrieval design. It is not recommended for production.

Retrieval Pipeline

QueryUser intent
SearchIndex lookup
AccessPermission check
RankingTop-K selection
GroundingEvidence check

Retrieval Metrics

0 Candidates
0 Retrieved
0 Access Denied

Azure AI Search — Document Chunk Inspection

Inspect synthetic document chunks, relevance scores, permission decisions and index freshness.

Generated Answer & Grounding Assessment

Awaiting Retrieval

Run the investigation to retrieve evidence and assess the answer.

Evidence Comparison

Run retrieval and select Compare Evidence to inspect whether the proposed answer is supported.

Retrieval Diagnostic Console

READY · Knowledge investigation initialized.
Section 07 / Evaluations, Testing and Observability

Microsoft Foundry Evaluations, Testing, and Observability

Production AI systems require more than successful model inference. Engineers must measure answer quality, safety, groundedness, reliability, latency, and operational behavior.

Microsoft Foundry provides evaluation capabilities for supported AI workflows, while Azure monitoring and tracing integrations can help teams investigate runtime behavior.

Effective AI operations combine offline evaluation, pre-production regression testing, and continuous production observability.

7.1 / Evaluation Fundamentals

Why AI Applications Need Dedicated Evaluation

Traditional software tests often verify deterministic behavior: given a known input, the application should produce a specific expected result.

Generative AI applications can produce different valid responses to similar inputs. Evaluation therefore requires a combination of objective checks, reference comparisons, model-based scoring, and human review.

01
Response Quality

Determine whether generated answers are useful, relevant, accurate, and aligned with task requirements.

02
Groundedness

Evaluate whether generated claims are supported by the supplied source evidence.

03
Safety

Assess supported risk categories, policy violations, and undesirable application behavior.

04
Operational Reliability

Measure errors, latency, throughput, tool failures, and service availability.

Engineering Principle

A technically successful inference request can still produce an incorrect or unsafe answer. API success and application quality must be measured separately.

7.2 / Evaluation Architecture

Connecting Offline Testing to Production Monitoring

A mature AI evaluation architecture separates development experiments from production monitoring.

Offline evaluations compare candidate models, prompts, retrieval configurations, or agent workflows using representative datasets.

Production observability investigates actual runtime requests, failures, performance, and selected quality signals.

NETWORK INSIGHT / AI EVALUATION AND OBSERVABILITY
AI Application Development Models · Prompts · RAG Pipelines · Agents · Tools
Offline Evaluation Pipeline Test Datasets · Evaluators · Quality Measurements
Dataset Inputs · Expected Behavior · References
Execution Candidate Model · Prompt · Workflow
Evaluation Quality · Safety · Groundedness
Decision Compare Results · Review · Release Gate
Approved Configuration Released
Production Observability Request Tracing · Metrics · Operational Investigation
Application User Requests · Workload Activity
Telemetry Traces · Metrics · Error Events
Analysis Latency · Failures · Selected Quality Signals
Improvement Incident Review · New Test Cases
Conceptual evaluation lifecycle. Evaluator availability, tracing features, monitoring integrations, and supported metrics depend on the selected Foundry capabilities and configuration.
Important Architecture Boundary

Offline evaluation does not automatically occur on every production request. Continuous quality assessment requires an explicitly designed monitoring and evaluation workflow, with appropriate privacy controls.

7.3 / Evaluation Datasets

Building Representative AI Test Cases

Evaluation datasets provide repeatable inputs for comparing application behavior across changes.

A useful dataset should reflect realistic user requests, difficult scenarios, edge cases, and known failure conditions.

01 / Collect Gather approved representative tasks
02 / Label Define expected behavior and evidence
03 / Execute Run candidate application versions
04 / Score Apply appropriate evaluation methods
05 / Review Investigate failures and release decisions
Normal Cases

Representative requests expected during ordinary application usage.

Edge Cases

Ambiguous queries, unusual inputs, long contexts, and incomplete information.

Adversarial Cases

Approved tests involving malicious instructions, unsafe requests, or attempted policy bypasses.

Regression Cases

Previously identified failures retained as repeatable tests for future releases.

Dataset Quality Principle

Evaluation results are only as representative as the underlying test cases. A model can perform well on a small curated dataset while failing on real production workloads.

7.4 / Evaluation Metrics

Measuring Quality, Groundedness, and Safety

Different AI workloads require different evaluation criteria.

For example, a RAG application may prioritize groundedness and retrieval relevance, while an agent workflow may require successful tool execution and correct task completion.

Metric What It Measures Typical Application
Relevance Whether the response addresses the requested task Question answering and enterprise assistants
Groundedness Whether claims are supported by supplied evidence RAG and document-based question answering
Correctness Whether the response meets factual or task-specific expectations Extraction, analysis, and structured tasks
Retrieval Quality Whether the search pipeline retrieves useful evidence RAG indexing and retrieval systems
Safety Whether outputs violate defined safety requirements Customer-facing and sensitive applications
Tool Success Whether required tool operations complete correctly Agentic applications
Task Completion Whether the overall workflow achieves its objective Multi-step agent workflows
Latency Elapsed time required to complete a request or task Interactive production applications
Evaluator Limitations

Model-based evaluators can produce imperfect or variable judgments. Calibrate evaluation criteria against human-reviewed examples, use appropriate deterministic checks, and investigate significant disagreements.

7.5 / Regression Testing

Preventing Quality Regressions During AI Releases

AI application behavior can change when engineers modify the model, prompt, retrieval pipeline, tool configuration, or application code.

Regression testing compares a candidate configuration against an established baseline using consistent evaluation scenarios.

01
Model Version Changes

Compare output quality, latency, and compatibility before changing production models.

02
Prompt Changes

Validate that updated instructions improve intended behavior without breaking existing use cases.

03
RAG Changes

Test indexing, chunking, embeddings, retrieval configuration, and grounding quality.

04
Agent Changes

Verify tool selection, authorization, execution order, task completion, and failure handling.

Release Engineering Principle

Define acceptable evaluation thresholds before release. A candidate version should not be approved solely because its average quality score improved. Review critical failures, safety regressions, latency, and operational cost.

7.6 / Distributed Tracing

Observing Model Calls, Retrieval, and Tool Execution

AI applications frequently involve multiple services. A single user request may trigger retrieval, model inference, tool execution, and additional model calls.

Distributed tracing helps engineers reconstruct the execution path and identify where latency or failures occur.

Trace Component Useful Information Engineering Purpose
Application Request Request identifier, timestamp, duration Correlate the complete transaction
Retrieval Operation Search duration, index, result metadata Investigate retrieval latency and relevance
Model Invocation Deployment, duration, token usage, status Measure inference performance and errors
Tool Invocation Tool identifier, execution duration, outcome Diagnose agent tool failures
Application Processing Validation, transformation, and business logic timing Identify non-model bottlenecks
Final Response Completion status and total latency Measure end-to-end user experience
Telemetry Privacy

Prompts, retrieved documents, tool arguments, and model responses may contain sensitive information. Configure tracing, sampling, redaction, retention, and access controls according to organizational requirements.

7.7 / Production Monitoring

Monitoring AI Quality and Infrastructure Health

Production AI observability should combine conventional service reliability measurements with selected AI-specific quality indicators.

A healthy API does not guarantee high-quality answers, and a high-quality model cannot compensate for persistent infrastructure failures.

Availability

Request success rates, service errors, endpoint health, and operational incidents.

Performance

End-to-end latency, model execution time, tool latency, and throughput.

Quality Signals

Selected evaluation results, user feedback, task success, and groundedness checks.

Consumption

Token usage, request volume, service consumption, and associated operational costs.

Monitoring Architecture

Azure Monitor, Application Insights, and supported OpenTelemetry-based instrumentation can contribute to observability architectures. The available telemetry depends on service support, instrumentation, and configuration.

7.8 / Troubleshooting and Incident Analysis

Diagnosing Evaluation and Observability Failures

AI production incidents may involve infrastructure failures, degraded answer quality, incorrect retrieval, or unexpected agent behavior.

Engineers should use evaluation results and execution traces together to identify the affected component.

Symptom Potential Cause Investigation
Quality score decreases Model, prompt, retrieval, or dataset change Compare configurations and inspect failing examples
Groundedness failures increase Incorrect evidence or unsupported generated claims Inspect retrieved context and response evidence
Evaluation results vary unexpectedly Non-deterministic outputs or evaluator variability Repeat tests and review scoring consistency
Traces are missing Instrumentation, sampling, export, or permission issue Inspect telemetry configuration and ingestion pipeline
Agent latency increases Additional model calls or slow tool execution Compare trace spans and execution counts
API succeeds but task fails Incorrect output or incomplete workflow Review task-level evaluations and application validation
Production quality drifts Changing user requests, data, or model behavior Compare current workloads with baseline evaluation distributions
Incident Investigation Principle

Separate operational failures from semantic quality failures. Use request traces to understand execution behavior and evaluation evidence to understand whether the output met application requirements.

7.9 / Engineering Summary

Building an Observable and Testable Foundry Architecture

Microsoft Foundry evaluations help engineering teams assess supported AI application behaviors before and during the development lifecycle.

Production observability provides visibility into execution, latency, errors, and operational performance.

Together, these practices establish a feedback loop between development, testing, deployment, incident analysis, and continuous improvement.

Key Engineering Takeaway

Reliable enterprise AI systems require both quality evaluation and operational observability. Use representative datasets, explicit release criteria, regression testing, distributed tracing, and privacy-aware production monitoring to detect failures that ordinary API health checks cannot reveal.

Network Insight · Engineering Lab 07

AI Evaluation & Regression Lab

Compare baseline and candidate AI application versions, run synthetic evaluation datasets, inspect quality regressions and enforce release gates before production promotion.

Evaluation Configuration

85%
85%
95%
4.0s

Evaluation Summary

0 Test Cases
0 Passed
0 Regressions

Baseline vs Candidate Evaluation

Run an evaluation to compare release metrics.

Release Gate Validation

Apply release gates after running the evaluation.

Production Promotion Decision

Awaiting Evaluation

Evaluate the candidate release before making a deployment decision.

Evaluation Case Investigation

Failed or regressed test cases will appear here.

Evaluation Event Console

READY · Evaluation environment initialized.
Section 08 / Enterprise Security and Governance

Microsoft Foundry Enterprise Security and Governance

Enterprise AI platforms must protect identities, model endpoints, connected data sources, tool integrations, and operational telemetry.

Microsoft Foundry security architecture combines Azure identity, resource authorization, supported network isolation, data protection, AI safety controls, and governance processes.

Secure deployment requires engineers to understand the boundaries between Foundry resources, projects, model services, agents, and downstream enterprise systems.

8.1 / Security Fundamentals

Applying Defense in Depth to Enterprise AI

AI applications introduce multiple trust boundaries. Users access applications, applications invoke models, agents request tools, and tools may access sensitive enterprise data.

Each boundary requires explicit controls rather than relying on the model's instructions to enforce security.

01
Identity

Authenticate users, services, workloads, and supported machine identities.

02
Authorization

Apply least-privilege permissions to resources, deployments, projects, and connected services.

03
Network Protection

Restrict supported service connectivity and private access paths where required.

04
Governance

Define deployment policies, audit requirements, data controls, and operational accountability.

Security Architecture Principle

Model instructions are not a substitute for access control. Authorization must be enforced by the application, service, or downstream system performing the protected operation.

8.2 / Security Architecture

Understanding Foundry Trust Boundaries

A secure Foundry architecture separates identity and management controls from runtime inference and downstream data access.

NETWORK INSIGHT / FOUNDRY SECURITY ARCHITECTURE
Enterprise Users and Applications Human Identities · Application Workloads · Service Principals
Identity and Access Boundary Microsoft Entra ID · Azure RBAC · Resource Permissions
Authentication User and Workload Identity
Authorization Roles · Scopes · Service Permissions
Credential Management Managed Identity · Approved Secrets
Foundry Resource and Runtime Boundary Project Configuration · Model Services · Agent Capabilities
Projects Configuration · Resource Access
Model Inference Authorized Endpoints · Deployment Controls
Agent Execution Tool Permissions · Execution Controls
Connected Enterprise Services Search · Storage · APIs · Approved Knowledge Sources
Knowledge Search Indexes · Document Permissions
Data Services Storage · Databases · Approved Connections
Enterprise Tools API Authorization · Business Controls
Logical trust-boundary diagram, not a literal network routing path. Project management, inference, and downstream access may use separate endpoints and identities. Available controls depend on the service and deployment type.
Important Network Distinction

A Foundry project is not automatically a network security boundary. Network isolation requires supported service configurations, connectivity controls, and validation of every dependent service endpoint.

8.3 / Microsoft Entra ID

Authenticating Users and AI Workloads

Microsoft Entra ID provides identity capabilities used across Azure services and supported Foundry integrations.

Applications can use supported token-based authentication instead of embedding long-lived service credentials.

Human Identities

Developers, administrators, operators, and application users authenticate through supported identity workflows.

Service Principals

Applications and automation workflows can authenticate using configured workload identities.

Managed Identities

Supported Azure workloads can obtain tokens without maintaining application-managed credentials.

Credential Governance

Where credentials are required, protect them through approved secret management and rotation.

Managed Identity Principle

Managed identities simplify credential handling but do not automatically grant access. The identity still requires appropriate role assignments and service permissions.

8.4 / Azure RBAC

Controlling Resource and Project Permissions

Azure role-based access control assigns permissions to identities at supported resource scopes.

Foundry deployments may require different permissions for management operations, project configuration, model invocation, and connected data services.

Access Area Typical Operation Security Consideration
Resource Management Create or configure supported Azure resources Restrict privileged management permissions
Foundry Projects Manage supported project assets and configuration Use documented roles and scopes
Model Deployment Create, update, or manage deployments Separate deployment administration from application use
Model Inference Invoke supported model endpoints Grant only required inference permissions
Knowledge Retrieval Read permitted search indexes or documents Enforce service and document-level access
Agent Tools Execute approved downstream operations Validate identity and authorization at each tool
RBAC Scope Matters

Azure management permissions and service data-plane permissions are not always equivalent. Confirm the exact documented role, scope, and API authorization requirements for each service.

8.5 / Private Networking

Protecting AI Service Connectivity

Enterprise environments may require private connectivity between applications, model endpoints, search services, storage, and other dependencies.

Azure Private Link, private endpoints, network restrictions, and approved connectivity designs can help reduce exposure for supported services.

01 / Identify Inventory all service endpoints
02 / Restrict Apply supported network controls
03 / Resolve Validate private DNS resolution
04 / Connect Test application and service paths
05 / Monitor Audit connectivity and configuration
Network Engineering Consideration

Private endpoint deployment alone does not guarantee a fully private AI architecture. Validate DNS, routing, public access settings, dependent services, and any outbound connections used by the workload.

8.6 / Data Protection

Securing Prompts, Responses, and Enterprise Knowledge

AI applications can process sensitive information through prompts, retrieved documents, model responses, tool arguments, and telemetry.

Data protection requires controls across the complete application lifecycle.

Data Classification

Identify sensitive data categories and determine permitted AI processing scenarios.

Encryption

Apply supported encryption controls for data in transit and at rest.

Retention

Understand service data handling, application storage, logging, and deletion requirements.

Telemetry Protection

Avoid unnecessary collection of sensitive prompts, retrieved content, and tool results.

Data Residency and Processing

Data processing boundaries vary by model provider, deployment type, region, and service capability. Verify current service-specific data handling commitments before processing regulated workloads.

8.7 / AI Safety and Governance

Controlling Model and Agent Behavior

AI governance extends beyond infrastructure security. Organizations must define acceptable use, safety requirements, evaluation standards, and operational accountability.

Supported content filtering and safety capabilities can help manage selected categories of unsafe input and output.

Governance Area Objective Engineering Practice
Model Approval Control which models are permitted Maintain approved model and deployment inventories
Safety Controls Reduce harmful or prohibited outputs Use supported safety features and application safeguards
Prompt Injection Defense Limit influence of untrusted content Separate trust levels and restrict privileged actions
Agent Permissions Prevent unauthorized business operations Tool allowlists, authorization checks, and approval gates
Evaluation Verify behavior before release Run representative quality and safety tests
Auditability Support incident review and accountability Capture relevant configuration and execution records
Change Control Prevent uncontrolled production modifications Version and review models, prompts, tools, and policies
Governance Principle

Content filtering is only one layer of AI risk management. It does not replace authorization, application validation, human review, or ongoing quality evaluation.

8.8 / Security Troubleshooting

Diagnosing Foundry Security and Access Failures

Security incidents and access failures should be investigated by identifying the exact boundary at which the operation fails.

Symptom Potential Cause Investigation
401 Unauthorized Invalid or missing authentication token Check identity, token audience, and credential acquisition
403 Forbidden Insufficient role or service authorization Inspect role assignments, scopes, and applicable service policies
Private endpoint unreachable DNS, routing, firewall, or endpoint configuration issue Validate name resolution, connectivity, and public access settings
Agent tool denied Missing downstream permissions Identify the executing identity and verify tool authorization
Knowledge access failure Search service or document-level permission restriction Inspect search identity, index permissions, and security filters
Content filtered Applicable safety or content filtering policy Review supported filter outcomes and application handling
Unexpected data exposure Overprivileged identity, missing filters, or unsafe logging Contain exposure, audit access, and review data paths
Security Incident Principle

For suspected unauthorized data access, prioritize containment and evidence preservation. Investigate the identity, resource, authorization scope, network path, and downstream service involved.

8.9 / Engineering Summary

Designing Secure Microsoft Foundry Environments

Microsoft Foundry security architecture requires coordinated controls across identity, authorization, network connectivity, model inference, agent tools, connected data, and governance.

A secure enterprise deployment must validate each service boundary independently and ensure that operational controls remain effective as applications evolve.

Key Engineering Takeaway

Secure Foundry architecture depends on least-privilege identity, explicit authorization, supported private connectivity, protected data handling, controlled agent execution, safety evaluation, and auditable governance. Security must be enforced throughout the application, not delegated to the model.

Network Insight · Engineering Lab 08

Foundry Security Boundary Lab

Trace enterprise AI requests across identity, authorization, private networking, model access, knowledge permissions and governance controls. Inject faults and investigate access decisions.

Enterprise Security Boundary Map

The simulated request passes through six sequential controls.

01 · Microsoft Entra Identity Workload authentication · Managed identity
PENDING
02 · Azure RBAC Authorization Required inference permission · Role scope
PENDING
03 · Network Access Boundary Private endpoint · DNS · Connectivity
PENDING
04 · Model Inference Access Deployment endpoint · Service permissions
PENDING
05 · Knowledge Data Permissions Search index · Document authorization
PENDING
06 · Governance & Audit Policy enforcement · Telemetry · Accountability
PENDING

Security Investigation Console

READY · Security boundary lab initialized.

Security Incident Configuration

Security Control Configuration

Security Assessment Metrics

0 Passed
0 Failed
— Posture

Engineering Security Decision

Awaiting Assessment

Trace the request or run a full security posture assessment.

Section 09 / Production Operations and Troubleshooting

Microsoft Foundry Production Operations and Troubleshooting

Production AI applications require reliable inference, secure connectivity, controlled agent execution, accurate retrieval, observability, and disciplined incident response.

Microsoft Foundry applications can depend on several interconnected services. A single user request may involve identity, model deployments, retrieval systems, agent tools, and external APIs.

Troubleshooting requires engineers to identify which component failed, correlate telemetry across service boundaries, and distinguish infrastructure incidents from AI quality problems.

9.1 / Production Architecture

Understanding the Complete Application Dependency Chain

Production AI systems are distributed applications. Their reliability depends on more than the availability of a single model endpoint.

Engineers should map the dependencies between the application, identity provider, Foundry resources, model inference, knowledge retrieval, and agent tools.

NETWORK INSIGHT / PRODUCTION DEPENDENCY ARCHITECTURE
Production AI Application Web Interface · API · Enterprise Assistant · Agent Client
Application Access and Runtime Authentication · Authorization · Request Coordination
Identity Microsoft Entra ID · Credentials
Application Logic Input Validation · Request Routing
Foundry Integration Project Configuration · Service Connections
Operational Controls Timeouts · Retry Limits · Telemetry
AI Execution Dependencies Inference · Retrieval · Agents · External Tools
Model Deployment Endpoint · Capacity · Token Processing
Knowledge Retrieval Azure AI Search · Indexes · Filters
Agent Runtime Model Calls · Tool Execution
Enterprise Services APIs · Storage · Business Systems
Response Validation and Operational Telemetry Request Outcome · Quality Signals · Traces · Metrics · Logs
Conceptual dependency diagram. Individual applications may use only a subset of these services. Retrieval and tool calls can occur in parallel or in multiple execution cycles.
Dependency Engineering Principle

Document every service required to complete a user request. Include authentication, DNS, network access, deployment configuration, external tools, and telemetry dependencies.

9.2 / Incident Classification

Separating Infrastructure, Application, and AI Quality Failures

AI incidents should be classified according to their observable symptoms and affected components.

An API failure is different from an incorrect model response, even when both result in a failed user task.

01
Infrastructure Failures

Connectivity, service availability, endpoint configuration, DNS, and authentication problems.

02
Capacity Failures

Rate limiting, insufficient serving capacity, excessive concurrency, and request timeouts.

03
AI Quality Failures

Incorrect responses, unsupported claims, poor retrieval, or task completion failures.

04
Security Incidents

Unauthorized access, unexpected tool execution, sensitive data exposure, or policy violations.

Incident Classification Principle

Do not assume that every failed AI task is a model failure. Determine whether the problem originated in infrastructure, retrieval, orchestration, model generation, or application validation.

9.3 / Deployment Troubleshooting

Diagnosing Endpoint and Configuration Failures

Deployment-related failures can occur when an application uses an incorrect endpoint, unsupported API, missing deployment, or incompatible configuration.

Symptom Potential Cause Investigation
Deployment unavailable Unsupported model, region, or deployment configuration Check model compatibility, region, and subscription eligibility
404 Not Found Incorrect deployment identifier or API route Verify endpoint, deployment name, and supported API
401 Unauthorized Invalid or missing authentication credentials Validate token acquisition and authentication configuration
403 Forbidden Missing permissions or applicable access restrictions Inspect identity, RBAC scope, and service access policies
Connection timeout Network, DNS, firewall, or service connectivity issue Test endpoint resolution and network connectivity
Unexpected model behavior Incorrect model version or configuration change Compare deployment metadata and evaluation baselines
Deployment Investigation Sequence

Verify the application endpoint, deployment identifier, API compatibility, identity, permissions, and network connectivity before investigating model inference behavior.

9.4 / Inference Capacity

Troubleshooting Throttling, Latency, and Timeouts

Model inference performance depends on workload characteristics, deployment capacity, service limits, and concurrent request demand.

High latency may originate from large prompts, long outputs, capacity pressure, or upstream application processing.

Rate Limiting

Investigate 429 responses, applicable quotas, request volume, and retry behavior.

Token Consumption

Inspect input and output token distributions and their effect on serving demand.

Concurrency

Measure simultaneous requests and sustained workload pressure against available capacity.

Timeout Budgets

Align client, application, and downstream timeout settings with expected execution behavior.

Retry Storm Risk

Repeated immediate retries can worsen capacity incidents. Apply bounded retries, exponential backoff, jitter, and appropriate request limits. Respect retry guidance from the affected service.

9.5 / Agent and Tool Failures

Investigating Multi-Step Execution Problems

Agent workflows introduce additional failure points because a task may require multiple model calls, tool invocations, and application decisions.

Symptom Potential Cause Investigation
Tool call rejected Missing authorization or invalid tool arguments Inspect tool schema, identity, and downstream response
Agent repeats operations Weak termination logic or repeated model decisions Review execution limits and orchestration traces
Agent returns incomplete answer Tool failure, insufficient context, or interrupted workflow Trace model calls, tool outcomes, and state transitions
Agent latency increases Additional inference cycles or slow external tools Measure per-step execution duration
Unexpected tool action Insufficient validation or authorization controls Contain the action and review tool execution permissions
Workflow state inconsistent Interrupted operations or unsafe retry behavior Inspect state persistence and idempotency controls
Agent Troubleshooting Principle

Reconstruct the execution sequence: user request, model decision, tool request, authorization, tool result, and final response. Identify the first step where actual behavior diverged from expected behavior.

9.6 / RAG Troubleshooting

Diagnosing Retrieval and Grounding Degradation

RAG failures can produce inaccurate responses even when the inference endpoint is operating normally.

Engineers should investigate whether the correct evidence was indexed, retrieved, authorized, and supplied to the model.

Indexing Failures

Missing documents, stale content, parsing errors, or incomplete index updates.

Retrieval Failures

Poor query matching, unsuitable chunking, incorrect filters, or embedding incompatibility.

Grounding Failures

Relevant evidence is retrieved but omitted, misinterpreted, or not used correctly.

Permission Failures

Incorrect document access controls, missing filters, or unauthorized source exposure.

RAG Diagnostic Rule

If the correct evidence was not retrieved, investigate ingestion and search. If the correct evidence was retrieved but the answer is incorrect, investigate context assembly and model generation.

9.7 / Incident Response

A Structured Production Investigation Workflow

Incident response should follow a repeatable process that prioritizes user impact, containment, evidence collection, and service recovery.

01 / Detect Identify symptoms and affected users
02 / Classify Determine incident type and severity
03 / Correlate Collect traces, metrics, and logs
04 / Isolate Locate the failing dependency
05 / Recover Apply controlled remediation
06 / Improve Record causes and prevention
Example: Enterprise AI Assistant Incident

Users report slow responses and intermittent failures. Engineers correlate request timestamps with model deployment telemetry and discover increased throttling during a traffic spike. They control retries, reduce unnecessary concurrency, and evaluate supported capacity adjustments before confirming recovery.

Incident Evidence

Preserve relevant request identifiers, timestamps, deployment configuration, service responses, and execution traces. Protect sensitive content when collecting or sharing diagnostic information.

9.8 / Production Readiness

Microsoft Foundry Production Engineering Checklist

Before releasing an enterprise AI application, engineering teams should validate its critical operational and security requirements.

✓
Deployment Validation Confirm model version, endpoint, API compatibility, and deployment configuration.
✓
Identity and Access Validate workload identity, role assignments, and downstream permissions.
✓
Network Connectivity Test DNS, routing, private endpoints, and required service access.
✓
Capacity Testing Evaluate realistic concurrency, token demand, latency, and throttling.
✓
Quality Evaluation Run representative correctness, safety, and groundedness evaluations.
✓
Agent Safeguards Validate tool authorization, execution limits, approval requirements, and retry behavior.
✓
Observability Configure appropriate metrics, tracing, logging, and alert thresholds.
✓
Recovery Planning Document rollback, incident escalation, fallback behavior, and operational ownership.
Production Readiness Principle

A successful test inference request does not establish production readiness. Validate sustained workload behavior, security boundaries, failure recovery, quality requirements, and operational monitoring.

9.9 / Engineering Summary

Operating Reliable Microsoft Foundry Applications

Microsoft Foundry production operations require visibility across the complete application dependency chain.

Engineers must distinguish between endpoint failures, capacity problems, retrieval degradation, agent execution errors, and model quality issues.

Structured incident response, representative evaluations, distributed tracing, secure configuration, and capacity planning provide the foundation for reliable enterprise AI operations.

Key Engineering Takeaway

Reliable Microsoft Foundry applications are engineered as distributed production systems. Successful operations depend on validated dependencies, secure access, measured capacity, controlled agent execution, trustworthy retrieval, quality evaluation, end-to-end telemetry, and disciplined incident response.

Network Insight · Engineering Lab 09

Foundry Production Incident Simulator

Investigate an enterprise AI production incident using telemetry, dependency health, diagnostic investigations and operational evidence. Identify the root cause, apply a remediation and verify service recovery.

Production Incident Overview

Incident Not Started

Select an incident scenario and start the investigation.

Production Telemetry

— Error Rate
— P95 Latency
— 429 Rate
— Availability

Application Dependency Map

AI Application Client traffic
Foundry Endpoint Inference API
Model Deployment Runtime capacity
↓     ↓     ↓
Entra Identity Authentication
AI Search Knowledge retrieval
Private Network DNS · Connectivity

Logical dependency overview, not a literal packet-forwarding topology.

Incident Timeline

Incident events will appear here.

Production Event Console

READY · Production incident simulator initialized.

Incident Scenario

Diagnostic Investigations

Investigation Findings

Select a diagnostic investigation to inspect production evidence.

Root-Cause Identification

Remediation Action

Incident Resolution

Investigation Pending

Identify the fault, apply remediation and verify recovery.

Engineering Performance

100 Investigation score / 100
0 Diagnostic checks
☕
Support Network Insight
Help support the development of interactive networking labs, BGP simulations, and educational content.
Support the Project
Matt Conran
Latest posts by Matt Conran (see all)
Tags: No tags

Comments are closed.