This tool evaluates the factual correctness of AI-generated summaries using structured fact representations in the form of triples (subject, predicate, object). It converts both the source document and the generated summary into triples and then compares them through triple overlap. This makes the evaluation process more transparent and easier to interpret than purely black-box factuality metrics.
For example, the sentence “Machine A produced 120 units on Line 3” can be represented using triples such as (Machine A, produced, 120 units) and (Machine A, located on, Line 3).
https://github.com/AFigaro/triple_based_fceThe repository contains the research code and supporting materials for the framework.
The tool addresses factual correctness evaluation (FCE) for AI-generated summaries by converting both the source document and the generated summary into explicit structured fact representations and comparing them directly.
Input:
Output:
Key features:
Possible extracted triples:
(Machine A, produced, 120 units), (Machine A, located on, Line 3)(Machine A, produced, 120 units)In this case, the shared triple indicates factual agreement on the production claim, while the missing location triple reflects omitted information rather than a contradiction.
The tool operates in the following stages:
Depending on the configuration, the framework can operate with a single triple representation per text or explore multiple representations to account for extraction variability.
This tool is intended for:
Within the AIMS5.0 context, the tool is especially relevant for scenarios where large volumes of textual records or logs are summarized automatically and require efficient factual verification before downstream use.
If you use this tool in research or development, please cite the corresponding paper.
@article{latipov2025triple,
title={Triple-based Factual Correctness Evaluation of AI-Generated Summaries},
author={Latipov, Insan-Aleksandr and Holenderski, Mike and Meratnia, Nirvana},
journal={IFAC-PapersOnLine},
volume={59},
number={27},
pages={226--231},
year={2025},
publisher={Elsevier}
}