Assessing Real v. Fake AI

12 Questions to Ask Your CLM Solution Vendor

Many CLM solutions claim to have AI. But assessments by Forrester and others reveal that most CLM solutions are still in preschool when it comes to AI capabilities.

Designed for procurement leaders to share with their vendor selection team, this handy 12-point checklist helps you more easily assess the true AI capabilities of each solution you're considering.


  1. Does your product automatically read in all contract data from all types of files?

    Real AI enables automated ingestion of all contract languanges from historical contracts and templates in any format.

  2. Does your product automatically extract any data in a contract and automatically tag clauses for use in NLP?

    “Automated capture and metadata clause tagging of legacy or third-party contracts” is one of the AI-powered CLM solution features that deliver business value, according to Forrester.

  3. What NLP technology are you using?

    The NLP technology used should enable a semantic similarity engine, which is another of the CLM solution features Forrester identified to deliver business value: “Semantic analysis AI to identify new issues in contracts and apply new metadata tags.”

  4. What search technology does your product use?

    With context-based search, real AI will find contracts and provisions without you having to search the specific term.

  5. Does your platform leverage ontologies (or knowledge graphs) to comprehend language?

    Real AI leverages ontologies (or knowledge graphs) to enable human-like contract comprehension. The system can compare clauses to identify missing terms and pre-requisites, such as negotiation boundaries, policy transgressions, material changes, clause and template similarity, and obligation exposure – even if the words it is comparing are not the same.

  6. Can we derive insights from all contract language or just a subset of key data extracted?

    Real AI can analyze all historical contracts, templates, clauses and supplier contracting behavior (for example: which clauses were most changed by suppliers; which suppliers made the most changes to contracts; which contracts have the highest risk) and present insights accordingly.

  7. Do you have to build reports as part of the implementation or are dashboards dynamic based on search questions/criteria?

    Dashboards should be dynamic, enabling deeper analysis based on insight revealed through your search questions. This is important because risk is often lurking in hidden places you don’t even know about. (If you knew where the risk lay, you’d have addressed it already.)

  8. Can we see contract data integrated with other systems’ data to gain further insights?

    An AI-powered CLM solution should enable integration and cross-analysis of contract data with data from other systems – for example, accounting. Then you can see how risk is created through the interrelationships between functions, like contracting and AP, as just one example.

  9. Does your system leverage analytics to improve the contract language, improve the contracting processes, and reduce risk?

    This is another of the CLM solution features that Forrester identified among six that deliver business value: “Linking contracts to results to improve contract language and contracting processes.”

  10. How is machine learning being used to improve areas such as contract creation, risk mitigation and contract management?

    The major source of learning initially is the analysis of historical contracts. The system should learn from what you have signed before to make template recommendations and initial risk ratings without heavy manually intervention.

  11. How easy is it for us as end users to train the system or do we need data analysts or developers to improve the system?

    Ongoing, the system should learn when you 1) alter a risk rating or the risk points assigned to different clauses and free-form a reason why; and 2) manually tag a clause. These training abilities should be a seamless part of the application that every user can easily take advantage of.

  12. How does supplier feedback get incorporated into the process?

    Suppliers’ contract edits – or in ContractAI’s case, supplier choices such as which clauses are edited the most – should be taken into consideration for AI models to improve the process. Only with supplier feedback can the efficiency of the process be optimized. den places you don’t even know about. (If you knew where the risk lay, you’d have addressed it already.)

Ready to Talk Through These Questions with ContractAI?

Schedule an introductory demo to see the ways in which ContractAI leverages real AI to accelerate your time to contract and reduce your risk.

Schedule a Demo
© 2021 - App Orchid. All rights reserved