Glean Drawing Index Pipeline

Sargent & Lundy · primary developer

Native connectors could not index the firm's engineering drawings at all, so a corpus of tens of thousands of sheets was effectively invisible to search.

I am the primary developer on the pipeline that enriches and indexes them, and am productionizing it.

What I built

Designed an incremental extraction approach starting from deterministic OCR, with validation layers stacked on top, producing a structured record of contextual metadata per drawing sheet rather than relying solely on a vision-language model.
Moved the pipeline from a 104-cell Microsoft Fabric notebook into Git-tracked Python modules while keeping Fabric as the execution runtime, so the logic is reviewable, testable, and version-controlled.
Architected for scale from the start: fan-out across thousands of sheets, API rate-limit handling, cost ceilings, and secret management instead of inline credentials.
Integrated the Egnyte API as the document source and pushed enriched records into Glean through its Indexing API, with a CRUD verification plan validating that source-side file changes propagate correctly into the index.

Stack

PythonMicrosoft Fabric notebooksAzure OpenAIOCR and vision-language extractionEgnyte APIGlean Indexing APIGit