Seeing Where the Answer Comes From: Evidence Provenance, Calibrated Reliance, and Decision Quality in AI-Assisted Construction Project Risk Assessment
Keywords:
Evidence provenance, calibrated reliance, generative artificial intelligence, retrieval-augmented generation, construction project management, risk assessment, decision quality, AI literacy, emotional intelligence.Abstract
Generative artificial intelligence is increasingly being explored for construction project risk assessment, including the preparation of risk registers, contract summaries, progress reports, and project documentation. However, retrieval-augmented generation does not guarantee that generated claims are fully supported by retrieved evidence. Unsupported claims can consequently complicate professional judgment, particularly when project managers must evaluate multiple sources of information under time pressure. Although technical approaches to factuality improvement have received growing attention, the influence of evidence provenance visibility on human reliance and decision quality remains an important area for empirical investigation. This study proposes a between-subjects scenario experiment examining whether provenance labels improve calibrated reliance and decision quality in AI-assisted construction project risk assessment. Drawing on trust-in-automation theory, research on retrieval-augmented generation, and emerging approaches to evidence-bound factual repair, the study investigates the effects of provenance visibility on unsupported-claim detection, calibrated reliance, and risk decision quality. It also examines the potential moderating roles of AI literacy and project managers’ emotional intelligence. Construction and infrastructure professionals in Khyber Pakhtunkhwa, Pakistan, will evaluate AI-generated risk assessments under conditions with or without provenance labels. The planned analysis uses between-group comparisons and claim-level mixed-effects models, with mediation and moderation analyses treated as secondary and conditional on adequate statistical power. No empirical findings are claimed because data have not yet been collected. The study provides a framework for investigating whether making the evidential status of AI-generated claims visible can support more appropriate reliance on generative AI in construction project management.
References
1. Asai, A., Wu, Z., Wang, Y., Sil, A., & Hajishirzi, H. (2024). Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR 2024).
2. Endsley, M. R. (2017). From here to autonomy: Lessons learned from human–automation research. Human Factors, 59(1), 5–27. https://doi.org/10.1177/0018720816681350
3. Glikson, E., & Woolley, A. W. (2020). Human trust in artificial intelligence: Review of empirical research. Academy of Management Annals, 14(2), 627–660. https://doi.org/10.5465/annals.2018.0057
4. Hoff, K. A., & Bashir, M. (2015). Trust in automation: Integrating empirical evidence on factors that influence trust. Human Factors, 57(3), 407–434. https://doi.org/10.1177/0018720814547570
5. Lee, J. D., & See, K. A. (2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80. https://doi.org/10.1518/hfes.46.1.50_30392
6. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Kuttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems, 33, 9459–9474.
7. Lintner, T. (2024). A systematic review of AI literacy scales. npj Science of Learning, 9, Article 50. https://doi.org/10.1038/s41539-024-00264-4
8. Lv, B., Feng, A., & Xie, C. (2024). Improving factuality by contrastive decoding with factual and hallucination prompts. Sensors, 24(21), Article 7097. https://doi.org/10.3390/s24217097
9. Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-T., Koh, P. W., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023). FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 12076–12100.
10. Niu, C., Wu, Y., Zhu, J., Xu, S., Shum, K.-S., Zhong, R., Song, J., & Zhang, T. (2024). RAGTruth: A hallucination corpus for developing and benchmarking large language models in retrieval-augmented generation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 10862–10878). Association for Computational Linguistics.
11. Parasuraman, R., & Riley, V. (1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39(2), 230–253. https://doi.org/10.1518/001872097778543886
12. Rajendran, S., & Singaravelu, D. (2026). Evidence-bound factual repair in retrieval-augmented LLM answers: Separating semantic localization from deterministic realization (Version 1) [Preprint]. Preprints.org. https://doi.org/10.20944/preprints202609.0490.v1
13. Zhang, Q., Hao, S., & Chung, K. S. K. (2022). The impact of project manager’s emotional intelligence on project performance: A meta-analysis. Chinese Management Studies, 17(3), 461–487. https://doi.org/10.1108/CMS-05-2021-0195
14. Zhang, W., & Zhang, J. (2025). Hallucination mitigation for retrieval-augmented large language models: A review. Mathematics, 13(5), Article 856. https://doi.org/10.3390/math13050856

