A purchasing assistant that compares supplier quotes can save a buyer time. Give that assistant permission to send emails, change supplier records or place orders, and the business decision becomes more complicated.
AI agents can use connected tools to carry out tasks across several steps. In procurement, that could mean gathering quotations, identifying missing information and preparing a purchase request. The useful question is how much responsibility to give the system—and what evidence would justify giving it more.
For small and medium-sized businesses, our recommendation is to begin with a narrow purchasing task and clear operating limits. These seven checks can help you assess a proposed solution before it becomes part of daily operations.
1. Check whether the task needs an agent
Start with the work your team wants to improve. “Compare these three quotations and highlight missing delivery dates” is a clearer requirement than “automate procurement”.
A quotation comparison might need document extraction and a fixed review process. A system that decides which information is missing, selects a tool and follows up across several steps may need more flexibility.
Anthropic’s guidance on building agents recommends starting with the simplest effective approach, because additional autonomy can increase cost and complexity.
Ask the provider to demonstrate your actual task. Check whether a simpler workflow can deliver the same result before paying for capabilities your team will not use.
2. Check exactly what the system can access
Ask for a permission list covering every connected system. Distinguish reading quotations from editing supplier records, sending messages and releasing purchase orders.
For an initial quotation comparison, access to an approved document folder may be enough. Access to payment details or unrelated contracts should have a specific justification.
OWASP’s guidance on excessive agency recommends restricting tools and permissions to what the task requires. It also recommends enforcing authorisation in connected systems, rather than relying on the language model to decide what is allowed.
Have your IT team verify those restrictions in the configured product.
3. Check where human approval is required
Decide which actions need approval before they happen. For an initial pilot, our recommendation is to retain buyer approval for supplier communications and purchase commitments.
The reviewer should see the proposed recipient, order details, quantity, price and supporting quotation. Approval should apply to that specific action; a material change should trigger another review.
Test the boundary during the demonstration. Reject a proposed action and confirm that it stops. Change the order value and confirm that the earlier approval cannot authorise the revised purchase.
This turns “human oversight” into a behaviour the team can verify.
4. Check how the agent handles instructions inside supplier documents
Supplier emails and attachments are information the agent needs to examine. They should not be able to redefine its purchasing rules.
An external document might contain text that attempts to make the system ignore instructions, favour a quotation or disclose information. OWASP describes this type of manipulation through external content as indirect prompt injection.
Ask the provider how it separates document content from authorised instructions and limits the consequences when the model makes a mistake.
Include a controlled test document containing an instruction that conflicts with your policy. Check the resulting comparison and any attempted actions. A successful test is useful evidence, but it does not establish protection against every possible attack.
5. Check whether you can reconstruct what happened
If an incorrect order appears, your team needs a record of the information used, the action requested, the approval given and the result returned by the purchasing system.
Ask the provider to demonstrate those records using a completed test. Check who can access them, how long they remain available and whether they can be exported.
Do not assume an “audit log” contains the full interaction. For example, Microsoft’s Copilot Studio documentationexplains that Purview audit events and conversation transcripts are stored separately; the audit event itself does not contain the full conversation text.
Also name the person who can pause the workflow and establish how staff will handle outstanding requests while it is unavailable.
6. Check difficult cases before testing live purchasing
A clean demonstration with three complete quotations tells you little about how the system handles everyday exceptions.
Build a test set from the problems your buyers actually encounter. Useful examples include:
- A price quoted per box when the request specifies individual units.
- A revised quotation that conflicts with an earlier version.
- An attachment with missing delivery charges.
- A supplier response that leaves the delivery date unclear.
- A connection failure after a purchase request has been submitted.
Agree the expected response for each case. Sometimes success means completing the comparison. Sometimes it means identifying missing information and asking a buyer to decide.
For interrupted submissions, check that retrying cannot create duplicate requests. Keep these tests in a controlled environment until the team understands the results.
7. Check the value after review and correction time
Measure the complete purchasing task, including the time people spend checking and correcting the agent’s work.
Consider a hypothetical team processing 100 quotation comparisons per month. At 12 minutes each, the existing process takes 20 hours.
If an assisted process takes seven minutes per comparison, including buyer review, that falls to 11 hours and 40 minutes. Add two hours of monthly maintenance, and the capacity released is six hours and 20 minutes.
That is useful, but it is not automatically a cash saving. The business still needs to account for software, setup, integration and training costs—and decide how the released time will be used.
Track quality alongside time. Incorrect prices, missed conditions and repeated supplier follow-ups can outweigh faster document processing.
Start with one purchasing decision
Choose one purchasing category, one buyer and a small set of representative quotations. Ask the proposed system to prepare a comparison that a person can verify against the original documents.
Use the pilot to establish whether the output is useful, whether the operating limits hold and whether the whole process improves. Expand its responsibility only when the evidence supports that decision.
An effective procurement agent should make purchasing easier to manage. Clear permissions, useful records and measurable results give your team a practical basis for deciding where it belongs.
Leave a Reply