Azure AI Search Custom Skills for Document Intelligence

Azure Cognitive Search (now Azure AI Search) has transformed how organizations extract insights from unstructured documents. While built-in cognitive skills handle common tasks like OCR and entity recognition, custom skills unlock the full potential of AI-powered document processing. By integrating Azure Document Intelligence (formerly Form Recognizer) through custom skills, developers can build sophisticated enrichment pipelines that extract structured data from invoices, contracts, forms, and complex documents at scale.

This guide explores how to build production-ready custom skills that bridge Azure AI Search with Document Intelligence, enabling intelligent document processing workflows that go far beyond basic text extraction.

What is Azure Cognitive Search Custom Skills

Azure Cognitive Search custom skills are developer-defined web services that extend the AI enrichment pipeline during document indexing. Unlike built-in cognitive skills (such as language detection, key phrase extraction, or image analysis), custom skills allow you to integrate external AI models, APIs, or business logic directly into your search indexing workflow.

A custom skill is essentially a REST endpoint that:

• Receives a batch of documents from the indexing pipeline
• Performs custom AI processing or enrichment
• Returns structured results that get merged back into the search index

Custom skills are defined within a skillset—a collection of both built-in and custom skills executed sequentially during document ingestion. Each skill in the skillset has:

• Input mappings: Specify which document fields to process
• Output mappings: Define where enriched data should be stored
• Context: Determines the scope of execution (per document, per page, per sentence, etc.)

When you integrate Document Intelligence as a custom skill, you can extract:

• Form fields from structured documents (invoices, receipts, ID cards)
• Table data with cell-level precision
• Layout information including reading order and text hierarchy
• Pre-built model outputs for common document types
• Custom trained model results specific to your organization's documents

The power of custom skills lies in their flexibility—you can chain multiple AI services, apply business rules, call external databases, or implement complex document understanding logic that's impossible with built-in skills alone.

Why it Matters

Organizations drowning in document-heavy workflows can't afford manual processing. Custom skills with Document Intelligence deliver transformative business value:

Automated Data Extraction at Scale: Process thousands of invoices, contracts, or forms per hour with consistent accuracy. What once took days of manual data entry now happens in minutes, freeing staff for higher-value work.

Intelligent Search Beyond Keywords: Enable semantic search over extracted form fields. Users can query "show me all invoices over $10,000 from December" even when those values are buried in scanned PDFs with varying layouts.

Compliance and Audit Readiness: Automatically extract and index compliance-critical fields (dates, signatures, amounts, terms) from legal documents. Build audit trails showing exactly what was extracted when and by which AI model version.

Multi-Modal Document Understanding: Combine OCR, layout analysis, entity recognition, and form understanding in a single pipeline. Process documents with mixed content—text, tables, images, checkboxes—and maintain relationships between elements.

Cost Efficiency: Pay only for what you process. Azure's consumption-based pricing means you're not maintaining expensive on-premises infrastructure or paying for idle capacity.

Developer Productivity: Build once, reuse everywhere. A well-designed custom skill becomes a reusable component across multiple indexers and applications. Your Document Intelligence integration works the same whether processing legal contracts or medical records.

For enterprises with document-centric operations—legal firms, insurance companies, healthcare providers, financial services—custom skills aren't just a technical feature. They're a competitive advantage that directly impacts revenue through faster processing, reduced errors, and better customer experiences.

Architecture / How it Works

The custom skill architecture creates a powerful feedback loop between Azure AI Search and Document Intelligence during document indexing:

Core Components

1. Indexer: Orchestrates the entire process, pulling documents from a data source, executing the skillset, and writing enriched results to the search index.

2. Skillset: Defines the enrichment pipeline as a directed acyclic graph (DAG) of skills. Each skill has inputs, outputs, and execution context.

3. Custom Skill Web Service: A REST endpoint (typically Azure Function, Azure Web App, or Azure Container Instance) that receives document content and returns enriched data.

4. Document Intelligence Service: Azure's AI for document understanding, accessed via SDK or REST API from your custom skill code.

5. Knowledge Store (Optional): Persists enriched document structure to Azure Storage for downstream analytics or training data collection.

Data Flow

Execution Context

Custom skills execute at different granularities:

• /document: Once per document (ideal for Document Intelligence on entire forms)
• /document/pages/*: Once per page (for multi-page form processing)
• /document/normalized_images/*: Once per extracted image

The indexer automatically batches requests to your custom skill for efficiency, sending multiple documents in a single HTTP call. Your skill implementation must handle batch processing and return results in the expected format.

Scaling Considerations

• Indexers run with configurable batch sizes (max 1000 items per batch)
• Custom skills should implement async processing for large documents
• Document Intelligence has throttling limits; implement exponential backoff
• Use Parallel Execution in skillsets for CPU-bound enrichments
• Monitor with Application Insights for latency and failure patterns

Step-by-Step Example

Let's build a custom skill that extracts invoice data using Document Intelligence and surfaces it in Azure AI Search.

Step 1: Create the Custom Skill Azure Function

import json
import logging
import azure.functions as func
from azure.core.credentials import AzureKeyCredential
from azure.ai.formrecognizer import DocumentAnalysisClient

# Initialize Document Intelligence client
endpoint = "https://YOUR_FORM_RECOGNIZER.cognitiveservices.azure.com/"
credential = AzureKeyCredential("YOUR_API_KEY")
document_analysis_client = DocumentAnalysisClient(endpoint, credential)

def main(req: func.HttpRequest) -> func.HttpResponse:
    logging.info('Invoice extraction custom skill triggered.')
    
    try:
        # Parse incoming request from Azure AI Search
        request_body = req.get_json()
        values = request_body.get('values', [])
        
        results = []
        for record in values:
            record_id = record.get('recordId')
            data = record.get('data', {})
            
            # Get document URL or base64 content
            document_url = data.get('documentUrl')
            document_sas_token = data.get('documentSasToken')
            full_url = f"{document_url}?{document_sas_token}"
            
            try:
                # Analyze invoice using prebuilt model
                poller = document_analysis_client.begin_analyze_document_from_url(
                    "prebuilt-invoice", 
                    full_url
                )
                result = poller.result()
                
                # Extract invoice fields
                invoice_data = {}
                if result.documents:
                    doc = result.documents[0]
                    fields = doc.fields
                    
                    invoice_data = {
                        "invoiceId": fields.get("InvoiceId").value if fields.get("InvoiceId") else None,
                        "invoiceDate": fields.get("InvoiceDate").value if fields.get("InvoiceDate") else None,
                        "vendorName": fields.get("VendorName").value if fields.get("VendorName") else None,
                        "customerName": fields.get("CustomerName").value if fields.get("CustomerName") else None,
                        "invoiceTotal": fields.get("InvoiceTotal").value if fields.get("InvoiceTotal") else None,
                        "amountDue": fields.get("AmountDue").value if fields.get("AmountDue") else None,
                        "items": [
                            {
                                "description": item.value.get("Description").value if item.value.get("Description") else "",
                                "quantity": item.value.get("Quantity").value if item.value.get("Quantity") else 0,
                                "unitPrice": item.value.get("UnitPrice").value if item.value.get("UnitPrice") else 0,
                                "amount": item.value.get("Amount").value if item.value.get("Amount") else 0
                            }
                            for item in fields.get("Items").value
                        ] if fields.get("Items") else []
                    }
                
                results.append({
                    "recordId": record_id,
                    "data": invoice_data,
                    "errors": [],
                    "warnings": []
                })
                
            except Exception as e:
                results.append({
                    "recordId": record_id,
                    "data": {},
                    "errors": [{"message": str(e)}],
                    "warnings": []
                })
        
        return func.HttpResponse(
            json.dumps({"values": results}),
            mimetype="application/json",
            status_code=200
        )
        
    except Exception as e:
        logging.error(f"Error processing request: {str(e)}")
        return func.HttpResponse(
            json.dumps({"error": str(e)}),
            status_code=500
        )

Step 2: Define the Custom Skill in Skillset

{
  "@odata.type": "#Microsoft.Skills.Custom.WebApiSkill",
  "name": "InvoiceExtractor",
  "description": "Extracts structured data from invoices using Document Intelligence",
  "uri": "https://YOUR_FUNCTION_APP.azurewebsites.net/api/ExtractInvoice?code=YOUR_FUNCTION_KEY",
  "httpMethod": "POST",
  "timeout": "PT230S",
  "batchSize": 1,
  "degreeOfParallelism": 1,
  "context": "/document",
  "inputs": [
    {
      "name": "documentUrl",
      "source": "/document/metadata_storage_path"
    },
    {
      "name": "documentSasToken",
      "source": "/document/metadata_storage_sas_token"
    }
  ],
  "outputs": [
    {
      "name": "invoiceId",
      "targetName": "invoiceId"
    },
    {
      "name": "invoiceDate",
      "targetName": "invoiceDate"
    },
    {
      "name": "vendorName",
      "targetName": "vendorName"
    },
    {
      "name": "customerName",
      "targetName": "customerName"
    },
    {
      "name": "invoiceTotal",
      "targetName": "invoiceTotal"
    },
    {
      "name": "amountDue",
      "targetName": "amountDue"
    },
    {
      "name": "items",
      "targetName": "lineItems"
    }
  ],
  "httpHeaders": {}
}

Step 3: Map Outputs to Index Fields

In your index definition, create fields that match the custom skill outputs:

{
  "fields": [
    {"name": "id", "type": "Edm.String", "key": true},
    {"name": "content", "type": "Edm.String", "searchable": true},
    {"name": "invoiceId", "type": "Edm.String", "filterable": true, "sortable": true},
    {"name": "invoiceDate", "type": "Edm.DateTimeOffset", "filterable": true, "sortable": true},
    {"name": "vendorName", "type": "Edm.String", "searchable": true, "filterable": true, "facetable": true},
    {"name": "customerName", "type": "Edm.String", "searchable": true, "filterable": true},
    {"name": "invoiceTotal", "type": "Edm.Double", "filterable": true, "sortable": true},
    {"name": "amountDue", "type": "Edm.Double", "filterable": true, "sortable": true},
    {"name": "lineItems", "type": "Collection(Edm.ComplexType)", "fields": [
      {"name": "description", "type": "Edm.String", "searchable": true},
      {"name": "quantity", "type": "Edm.Double"},
      {"name": "unitPrice", "type": "Edm.Double"},
      {"name": "amount", "type": "Edm.Double"}
    ]}
  ]
}

Tip: Always enable Application Insights on your Azure Function to track custom skill performance, failures, and Document Intelligence API latencies. Set up alerts for error rates above 5% and p95 latency spikes.

Best Practices

Design for Idempotency: Your custom skill may be called multiple times for the same document due to indexer resets or failures. Ensure repeated invocations produce identical results and don't cause side effects.

Implement Robust Error Handling: Return partial results when possible rather than failing the entire batch. Use the errors and warnings arrays in the response to communicate issues back to the indexer.

Optimize Batch Processing: Process multiple documents in parallel within your custom skill using async/await patterns. Don't make Document Intelligence calls sequentially—use asyncio.gather() or Task.WhenAll().

Cache Expensive Operations: If your custom skill performs lookups or preprocessing that doesn't change often, implement caching using Azure Redis or in-memory caches to reduce latency.

Version Your Models: When using custom Document Intelligence models, include model version metadata in enrichment outputs. This enables auditability and easier debugging when model behavior changes.

Monitor Token Consumption: Document Intelligence charges per page analyzed. Log page counts and monitor costs, especially during development when you may reindex frequently.

Use Managed Identity: Replace API keys with Managed Identity authentication between Azure Function and Document Intelligence for better security and automated credential rotation.

Implement Circuit Breakers: When Document Intelligence service experiences transient failures, implement exponential backoff with jitter to prevent overwhelming the service during recovery.

Test with Diverse Documents: Create a test corpus with varied layouts, languages, quality levels, and edge cases. Don't just test with pristine samples—include skewed scans, mixed languages, and damaged documents.

Design for Extensibility: Structure your custom skill to accept configuration parameters. This allows reusing the same function for different document types by varying skill definitions in the skillset.

Common Mistakes

Forgetting SAS Token Expiration: Custom skills accessing documents in Blob Storage via URL need SAS tokens. If tokens expire during long-running indexing jobs, processing fails. Set expiration to at least 24 hours.

Ignoring Skillset Execution Order: Skills execute in dependency order based on input/output mappings. If skill B depends on skill A's output, ensure proper source mapping. Circular dependencies cause indexer failures.

Returning Incorrect Response Schema: Azure AI Search expects a specific JSON structure with values, recordId, data, errors, and warnings. Deviating from this schema causes silent failures or cryptic errors.

Overloading Complex Types: While nested complex types in index fields are powerful, deeply nested structures (more than 3 levels) hurt query performance and complicate relevance scoring. Flatten when possible.

Not Handling Unicode Properly: Documents contain characters from many languages. Ensure your custom skill properly decodes UTF-8, handles right-to-left text, and doesn't corrupt special characters.

Hardcoding Document Intelligence Model IDs: Model IDs change between API versions and when you retrain custom models. Pass model IDs as skill parameters rather than hardcoding in function code.

Skipping Input Validation: Don't assume inputs are always present or correctly formatted. Check for null values, validate URLs, and handle missing or corrupt document content gracefully.

Deploying Without Load Testing: Custom skills become bottlenecks during indexing if not properly scaled. Load test your Azure Function with realistic batch sizes before indexing production document sets.

Mixing Synchronous and Asynchronous Patterns: In async frameworks (Python asyncio, C# async/await), mixing blocking calls causes thread pool exhaustion. Always use async HTTP clients for Document Intelligence API calls.

Neglecting Timeout Configuration: Default Azure Function timeouts (5 minutes consumption plan) may be insufficient for large documents. Adjust timeout settings and ensure Document Intelligence calls have appropriate timeouts.

Real-World Use Cases

Legal Contract Analysis: A multinational law firm processes 50,000+ contracts annually. Custom skills extract parties, effective dates, termination clauses, and financial terms from MSAs, NDAs, and SOWs. Document Intelligence identifies standard clause variations across jurisdictions. Lawyers search by contract type, party name, or date range, reducing contract review time by 70%.

Insurance Claims Processing: An insurance carrier automates claims intake by extracting accident dates, policy numbers, claim amounts, and supporting documentation from forms, photos, and medical records. Custom skills validate extracted data against policy databases in real-time, flagging discrepancies for human review. Processing time dropped from 5 days to 4 hours.

Healthcare Records Management: A hospital network ingests patient records, lab results, prescriptions, and imaging reports. Custom skills with Document Intelligence extract patient identifiers, visit dates, diagnoses codes (ICD-10), procedures, and medications. Physicians search across millions of historical records during patient consultations, with PHI properly secured and audit-logged.

Financial Document Compliance: A fintech company processes loan applications containing bank statements, tax returns, pay stubs, and identity documents. Custom skills extract income verification data, calculate debt-to-income ratios, and flag inconsistencies. Compliance officers search for applications missing required disclosures or containing suspicious patterns.

Supply Chain Invoice Reconciliation: A manufacturing company receives 100,000+ supplier invoices monthly in various formats and languages. Custom skills extract line items, match against purchase orders in SAP, and flag discrepancies (price mismatches, quantity overages). Accounts payable teams resolve only the 5% of flagged invoices, rather than manually reviewing everything.

Government Document Digitization: A government agency digitizes 50 years of paper archives. Custom skills extract dates, case numbers, petitioner names, and decision outcomes from court filings. Researchers perform full-text searches across historical records that were previously accessible only through manual archive visits.

Conclusion

Custom skills transform Azure Cognitive Search from a basic text indexer into an intelligent document processing platform. By integrating Document Intelligence, you can extract structured data from unstructured documents at scale, enabling search experiences that understand not just keywords but the semantic meaning and structure of your content.

The architecture is flexible—start with prebuilt Document Intelligence models for common document types, then layer in custom trained models as your needs evolve. Your custom skill becomes a reusable component across multiple indexers and applications, providing consistent AI behaviors organization-wide.

Success requires attention to error handling, performance optimization, and testing with real-world document variability. But the investment pays dividends: organizations processing document-heavy workflows see dramatic reductions in manual effort, faster processing times, and better compliance outcomes.

"Azure AI Search custom skills with Document Intelligence have reduced our contract review cycle from weeks to days. We're now extracting insights from documents we never had time to properly analyze before." — Legal Operations Director, Fortune 500 Company

Start small with a pilot project on a single document type. Prove the value, learn the patterns, then expand. The combination of Azure AI Search's powerful indexing and query capabilities with Document Intelligence's advanced document understanding creates a foundation for intelligent automation that scales with your organization's needs.

Next Steps: Deploy the invoice extraction example above, test with your documents, and monitor performance in Application Insights. Then extend the custom skill to handle multiple document types using Document Intelligence's prebuilt models or train custom models for organization-specific forms. Your journey to intelligent document automation starts now.