ProcSight

ProcSight is an AI-powered procurement investigation agent that analyzes supplier performance, purchasing activity, inventory risk, and procurement documents through natural-language questions.

Instead of asking an LLM to calculate everything itself, ProcSight gives the agent a set of deterministic analytics and retrieval tools. The agent decides which tools to use, combines the evidence, and produces a grounded answer.

---

Demo

Investigation workspace

ProcSight starts with a simple investigation interface and suggested questions.

Agent investigation

The agent can combine operational data and business documents in the same investigation and expose the tools it used.

---

What can ProcSight investigate?

Examples include:

• Which suppliers require the most attention?
• Is Apex Electronics missing its delivery target?
• Which supplier had the largest price increase?
• Which supplier had the largest spend growth?
• Where is critical inventory most exposed?
• What price increase triggers a commercial review?
• Does the available evidence support taking action against a supplier?

ProcSight can reason across multiple sources instead of treating procurement as a document-only or SQL-only problem.

---

How it works

User question
      │
      ▼
ProcSight Agent
      │
      ├── Supplier Performance
      ├── Supplier Exposure
      ├── Portfolio Comparison
      └── Procurement Document Search
              │
              ▼
     Structured data + documents
              │
              ▼
        Grounded answer

The LLM handles reasoning and tool selection.

ProcSight's Python and SQL tools handle deterministic calculations such as:

• on-time delivery rate
• average delivery delay
• supplier spend
• spend growth
• weighted unit-price change
• inventory coverage
• critical inventory exposure
• supplier priority ranking

This keeps numerical calculations outside of the language model.

For questions involving policies, SLAs, supplier agreements, or other business documents, the agent can retrieve the relevant evidence separately and combine it with operational metrics.

---

Example investigation

A question such as:

Does Apex Electronics appear to be violating its delivery target?

can require two different sources of evidence.

ProcSight can:

1. calculate Apex's actual delivery performance from purchase-order data,
2. retrieve the applicable delivery target from the supplier agreement,
3. compare the observed performance against the documented requirement,
4. return an answer grounded in both sources.

That same architecture works for investigations involving pricing, inventory exposure, supplier risk, or procurement policy.

---

Tech stack

AI / Agent Frameworks

• OpenAI Agents SDK — agent orchestration and tool calling
• OpenAI API — language-model reasoning
• LlamaIndex — procurement document indexing and retrieval
• OpenAI embeddings — semantic document search

Backend / Analytics

• Python
• DuckDB — local analytical database
• Pandas — data transformation and analysis
• Pydantic — structured models and validation

Data / Cloud

• Azure SQL Database
• Microsoft WideWorldImporters
• Canonical CSV procurement data model
• Source adapters for external datasets

Application

• Streamlit — interactive investigation interface
• python-dotenv — environment configuration

Testing / Validation

• pytest
• deterministic unit tests
• agent evaluation harness
• independent ground-truth evaluation
• regression tests
• SHA-256 code-freeze validation

---

Source-independent data architecture

ProcSight does not depend on one specific database schema.

External data is first converted into a canonical procurement model:

External Source
      │
      ▼
Source Adapter
      │
      ▼
Canonical Procurement Schema
      │
      ▼
ProcSight Analytics + Agent

The canonical model contains entities such as:

Suppliers
Products
Purchase Orders
Purchase Order Lines
Inventory

This allows new enterprise data sources to be integrated through adapters without rewriting ProcSight's analytics or agent tools.

The data layer also validates required fields, relationships, dates, IDs, and allowed values before analytics run.

---

Document retrieval

ProcSight also supports supporting procurement documents such as:

• contracts
• supplier agreements
• SLAs
• procurement policies
• procedures
• supplier terms
• other business documents

Documents are indexed through LlamaIndex and exposed to the agent through a generic procurement-document search tool.

ProcSight also distinguishes between different kinds of evidence.

For example:

• an operational metric can prove poor delivery performance,
• a contract can establish a contractual obligation,
• a policy can establish an internal review requirement,

but one should not automatically be treated as proof of another.

---

External validation with Azure SQL

The initial ProcSight environment uses controlled procurement fixtures so expected behavior can be tested precisely.

To test whether the architecture generalized beyond that environment, I integrated Microsoft WideWorldImporters OLTP hosted in Azure SQL Database.

The external validation pipeline is:

Azure SQL
    │
    ▼
WideWorldImporters
    │
    ▼
WWI Source Adapter
    │
    ▼
ProcSight Canonical Schema
    │
    ▼
DuckDB
    │
    ▼
ProcSight Agent
    │
    ▼
Independent Ground Truth

The adapter mapped the external database into:

• 13 suppliers
• 227 products
• 2,072 purchase orders
• 8,358 purchase-order lines
• 227 inventory records

Fields that do not exist in the source are not invented. For example, WideWorldImporters does not provide ProcSight's business-criticality classification, so those values remain explicitly unknown.

---

Validation results

Automated tests       40 passed
Controlled agent eval 10 / 10
WWI agent eval         4 / 4
Cold validation        PASS

The WWI ground-truth answers are generated independently rather than by calling ProcSight's own analytics functions.

A final cold validation rebuilds the external-data path from Azure SQL through the adapter and then runs the agent evaluation.

Azure SQL
→ adapter
→ canonical data
→ analytics
→ independent ground truth
→ agent evaluation

The final pipeline completed successfully without modifying the frozen ProcSight logic.

---

Bugs found through external validation

Using a second dataset also exposed assumptions that were not visible in the controlled fixtures.

Multi-line purchase orders

The original fixtures had one line per purchase order.

WideWorldImporters contains orders with multiple lines, which exposed a join that could duplicate order-level delivery observations.

The analytics were corrected so:

• delivery performance is calculated at the purchase-order level
• spend and pricing are calculated at the purchase-order-line level

A regression test was added for the scenario.

Spend-growth comparison

External evaluation also exposed that spend growth was calculated but was not available as an explicit portfolio-ranking criterion.

A deterministic spend_growth comparison path and regression test were added.

This external validation step was useful specifically because it forced the system to operate on data that was not designed around ProcSight.

---

Project structure

procsight/
│
├── app.py
│
├── procsight/
│   ├── agent.py
│   ├── analytics/
│   ├── tools/
│   ├── retrieval/
│   └── adapters/
│
├── data/
├── evals/
├── scripts/
├── tests/
└── validation/

The main components are:

• agent.py — agent orchestration
• analytics/ — deterministic procurement analytics
• tools/ — agent-facing tools
• retrieval/ — document search
• adapters/ — external source integration
• evals/ — agent evaluation suites
• tests/ — unit and regression tests
• validation/ — external ground truth and validation artifacts

---

Run locally

Install

pip install -r requirements.txt

Configure

Create a .env file:

OPENAI_API_KEY=your_key
OPENAI_MODEL=your_model

Start the application

python -m streamlit run app.py

Run tests

python -m pytest -q

Run the controlled agent evaluations

python -m evals.run_evals

---

Design goals

ProcSight was built around a few principles:

Use LLMs for reasoning, not arithmetic.
Important procurement metrics are calculated by deterministic tools.

Keep evidence traceable.
Operational data and retrieved documents remain distinct sources of evidence.

Do not fabricate missing enterprise data.
Unavailable source concepts remain unknown.

Make data sources replaceable.
Source adapters isolate external schemas from the core application.

Test the agent as a system.
In addition to unit tests, ProcSight is evaluated on complete natural-language investigations against known ground truth.