New cyber software can verify how much knowledge AI really knows

With a growing interest in generative artificial intelligence (AI) systems worldwide, researchers at the University of Surrey have created software that is able to verify how much information an AI farmed from an organisation’s digital database.

Surrey’s verification software can be used as part of a company’s online security protocol, helping an organisation understand whether an AI has learned too much or even accessed sensitive data.

The software is also capable of identifying whether AI has identified and is capable of exploiting flaws in software code. For example, in an online gaming context, it could identify whether an AI has learned to always win in online poker by exploiting a coding fault.

Dr Solofomampionona Fortunat Rajaona is Research Fellow in formal verification of privacy at the University of Surrey and the lead author of the paper. He said:

“In many applications, AI systems interact with each other or with humans, such as self-driving cars in a highway or hospital robots. Working out what an intelligent AI data system knows is an ongoing problem which we have taken years to find a working solution for.

"Our verification software can deduce how much AI can learn from their interaction, whether they have enough knowledge that enable successful cooperation, and whether they have too much knowledge that will break privacy. Through the ability to verify what AI has learned, we can give organisations the confidence to safely unleash the power of AI into secure settings.”

The study about Surrey’s software won the best paper award at the 25th International Symposium on Formal Methods.

Professor Adrian Hilton, Director of the Institute for People-Centred AI at the University of Surrey, said:

“Over the past few months there has been a huge surge of public and industry interest in generative AI models fuelled by advances in large language models such as ChatGPT. Creation of tools that can verify the performance of generative AI is essential to underpin their safe and responsible deployment. This research is an important step towards is an important step towards maintaining the privacy and integrity of datasets used in training.”

The full paper can be read here.

###

Notes for Editors

The University of Surrey is a research-intensive university, producing world-leading research and delivering innovation in teaching to transform lives and change the world for the better. The University of Surrey’s Institute for People-Centred AI combines over 30 years of technical excellence in the field of machine learning with multi-disciplinary research to answer the technical, ethical and governance questions that will enable the future of AI to be truly people-centred. A focus on research that makes a difference to the world has contributed to Surrey being ranked 55th in the world in the Times Higher Education (THE) University Impact Rankings 2022, which assesses more than 1,400 universities' performance against the United Nations’ Sustainable Development Goals (SDGs).