Evaluating the trustworthiness of AI applications – Lessons learned from an audit

3 Application of FH catalogue 

According to the FH Catalogue, not all six dimensions are necessarily equally relevant for the use case. For this reason, a risk analysis (named as “protection requirement analysis” in the catalogue) is first made for each dimension in order to determine the protection requirements of each dimension. Depending on potential damages (e.g., due to malfunctions or violations of requirements), the protection requirement is to be set as „low“, „medium“ or „high“. If a low protection requirement is determined for a dimension, the dimension does not have to be considered further, since no significant risks exist (with the exception of the dimension Reliability, which always needs to be considered, independently of the determined risk).

In the present study, only the dimension Fairness was evaluated as low in need of protection requirements. This is primarily because neither personal data was processed nor are human users affected by the results of the AI application. For the other five dimensions, the need for protection was assessed as at least medium, which is why these dimensions were examined in greater depth by first conducting a detailed risk analysis for each dimension based on the risk areas. This resulted in target specifications. A series of qualitative and quantitative criteria were then defined, which were used to assess whether the targets had been achieved. At the end, a summary consideration of the dimension was made.

The challenges from the different perspectives that occurred in the course of the study are described below.

3.1 AI developer perspective 

The FH Catalogue requires comprehensive documentation on the AI application. Since we selected the use case for auditing after go-live, some of the requirements of the catalog had to be documented retrospectively. Standard documentation was already created during development. However, this was not adequate in scope and depth to meet the requirements of the FH catalog. Therefore, some documentation had to be created from scratch.

In addition, it was notable that some points were repeated or divided thematically in different dimensions. For example, in requirement “[TR-R-NB-MA-01] Training and Test Data”, general requirements were given for training data. However, data quality was to be documented in another dimension under requirement “[VE-R-RE-KR-03] Quality of training and test data”. 

Furthermore, some items in the audit catalog concerned requirements that fell within the domain of the end user. This related, for example, to training or IT security measures (e.g. “[SI-R-BD-MA-01] Training and sensitization of employees” or “[SI-R-FS-MA-01] Security guidelines and instructions for use”). Therefore, another challenge was to constantly coordinate between AI developer and end user. This also required additional time and therefore some deadlines needed to be extended. Besides this, any risk analysis needs to describe the impact or damage, which is why input form the end user was needed to provide meaningful answers. If no concrete answers were possible, the team needed to estimate the consequences.

As mentioned before, some points appear repeatedly in the catalogue. This is due to the structuring of the catalogue along the risk dimensions. This makes the documentation requirements rather long, as those points need to be addressed again and again. We asked ourselves whether an alternative structuring along ML-topics would be more useful, e.g.: 

ML-model and transparency along all risk dimensions

AI application (the application as a whole, not the ML-model) along all risk dimensions 

Systems architecture of the underlying system along all risk dimensions 

Quantity, quality and resources of training data 

3.2 Auditor perspective 

The main challenge from the auditor side is that there is always some room for interpretation. This is made more difficult by the fact that AI auditing is very new. Even though the FH Catalogue is the most comprehensive one, still, at many points, there is a lot of scope for interpretation, as there are not many hard or quantitative requirements.

The audit was set up in steps, guided by the risk dimensions of the catalogue. The first two dimensions covered were “Autonomy and Control” and “Fairness”. Here, multiple iterations between AI developer and auditors were necessary. These rounds consisted of the AI developer providing documents, the auditors giving feedback, and then subsequently the AI developer providing improved documents.

For the dimensions covered towards the end of the process, much fewer iterations were needed.

The dimension where the process was most different to the others was Reliability. Here, the auditors needed to request feedback to a large number of small points (evaluation metrics, how exactly something was done, etc.). Given the nature of the underlying topic, this might not be highly surprising. Still, a clear outcome of this test-case was that most time was spent on the topic of reliability, even though it was done at a later stage, where there was already a more efficient workflow established between AI developer and auditors.

Due to the lack of  widely accepted standards or metrics for evaluating certain requirements of the catalog the result is often subject to auditors judgement which can lead to inconsistencies.

Considering the simplicity of the target AI application, the effort for evaluating all the requirements is quite high. Going from the protection requirement low to medium is a tremendous step with respect to the number of requirements.

Additionally, the auditors were working with the developer of the AI component. Many requirements are addressing the integration and usage of the AI application which can only be handled by the end user. Ideally all parties are involved in the evaluation process and a lot of work will be repeated for every AI application using a similar AI component.  

3.3 Dimension reliability 

Reliability is arguably the most important dimension and is the only one that according to the audit catalogue always needs to be addressed, independent of the risk analysis. It is also one of the most challenging ones, as a high level of AI expertise is required for the auditing party. The AI developer also had to first gather information internally from the machine learning experts.

A characteristic of this dimension is that there can be in principle a large number of quantitative approaches involved (such as computing accuracy metrics).  While, at first glance this makes the task seemingly easy, the lack of generalizable thresholds/baselines makes it actually quite hard in practice. This is particularly important, however, because accuracy is a crucial factor in the selection of a machine learning model. The training and test data must also adequately (quantitatively and qualitatively) cover the application area.

From the auditor’s perspective, the AI developer needs to provide convincing arguments on why certain metrics were chosen, show which values they have for the application, as well as why these values are good enough. From the AI developer‘s point of view, however, it was challenging to argue in detail why certain metrics or tests were not considered. For efficiency reasons, short answers were often given that the requirements were not relevant.

To understand the evaluation done by the AI developer, many – often small – clarification questions were necessary. This is caused by the fact that while many evaluation metrics are well-known and standardized, still, from application to application small details can vary (such as on which data exactly they were computed, and similar).

Example of a difficult case: The requirement “[RE-R-SC-ME-06] AI application real-world tests” (“Real-world test of AI-application“) requires the application be tested in a real-world setting. This was not done with the AI application described in this study. The AI developer  argued – in a convincing way – that a real-world test is not necessary, given the nature of the application and the data used. Moreover, for the real-world-test, the end user would have had to agree and grant the auditors access to the IT system. If taken literally, however, the catalogue does not really leave the possibility of not doing a real-world test. Therefore, this was perceived as a borderline case by the auditors. 

3.4 Dimension safety and security 

This dimension includes functional security and IT security. In this dimension, the general issue that it is unclear whether the AI developer or the end user company is responsible for certain requirements became especially apparent.  

For example, requirement “[S-R-IA-ME-11] Logging and monitoring” requires that violations of integrity or availability (of the system) are logged. Here it is unclear whether implementation of this logging is the task of the AI developer or the end user company, and even more unclear is who is required to present the necessary evidence during the audit. Another example would be „[S-R-FS-ME-01] Safety guidelines and instructions for use“, after which own security policies and usage instructions must be defined. Here, the AI developer can only help in an advisory capacity and at the same time must comply with the end user‘s internal guidelines. 

The same applies for the requirements on data integrity and confidentiality of data. Here, from the auditors’ perspective, several details were missing in the documentation provided by the AI developer, especially how integrity is guaranteed in the long term. These are, however, questions where the responsibility in terms of documentation is not clear from the FH Catalogue. One reason why the AI developer could not disclose all the information in detail was that a strict confidentiality agreement was signed between the AI developer and the end user. Confidential information (such as infrastructure details, audit-logs or training data) could therefore not be disclosed to the auditor. It would have been necessary to execute a separate confidentiality agreement for this purpose. However, this was not done for this study.

Another point that came up was that the auditors had difficulties in understanding the bigger picture of how the system is implemented on the end user’s side.  The AI application itself – as implemented by the AI developer – was well described in the documentation. For several safety and security requirements it is, however, essential to additionally understand the embedding in the larger system at the end user’s side. This was – at least in the beginning of the exercise – missing from the documentation, which, naturally, was focused on the AI application itself. 

For this dimension it was especially apparent that a full application of the audit catalogue requires all parties involved to contribute to the evaluation, end users as well as all suppliers. Considering that the AI developer has multiple customers who might want to go through such an evaluation, a more efficient approach is desired than repeating the same analysis multiple times.

4 Conclusions 

If it is possible to develop secure and trustworthy AI applications, the willingness of companies to integrate such AI applications into their processes and to benefit from the advantages of this technology will increase. Independent testing and auditing by third parties within the framework of certification is one way to establishing security and trust in AI applications. The creation of a trustworthy “AI made in Europe” brand can create an international competitive advantage. 

The FH Catalogue is very extensive and therefore, plenty of time and human resources must be planned for documentation and auditing. This makes it well suitable for AI applications that either have highly sophisticated parts (e.g. consisting of deep neural networks) or being part of a larger complicated system. For simpler applications – which presumably most applications will be (such as the use case considered here) – the catalogue seems, at least in parts, too detailed. The fulfillment of these details requires a lot of work both by the AI developers as well as by the auditors, and thus might be an obstacle for choosing to do an audit in cases where it is not legally required. For such cases, it would be beneficiary to have some kind of an “audit-lite” approach. Here, only certain aspects, such as the training of the AI component, would be audited. This would also be beneficial for the end user, as an application with an “audit-lite” would still be perceived more trustworthy than if no audit is performed at all.

However, in our opinion, the six dimensions are sufficient to cover all aspects of trustworthy AI. They are also largely consistent with the requirements of other catalogs (e.g. the European Union‘s Ethics Guidelines for Trustworthy AI). As the catalog has a risk-based approach, the risk analyses are a central component. Since sometimes consequences of the AI application are difficult to assess, the elaboration of the risk analyses is also challenging. Besides that, also a risk analysis of the dimension as medium or high means that the requirements of the whole dimension have to be documented. Thus, there is no further distinction between these two risk levels. This leads to the fact that a dimension with a risk analysis medium has the same effort as the risk level high.

The specific structure of the catalogue along risk dimensions made both writing the documentation, and checking by the auditors, cumbersome at some times. This is because many points that are from a technical perspective the same or at least very similar – e.g. questions regarding the training data – occur in multiple risk dimensions, and thus need to be addressed multiple times.

Furthermore, many requirements need the involvement of the end user and hence, a full audit is only possible if all parties are part of the process. Consequently, this means that the AI developer most likely will have additional effort if a similar or the same component is used in other applications. The possibility of independently auditing an AI component would benefit the process greatly. Many unclarities during the audit are ideally resolved by having an AI management system in place defining a clear process for all relevant aspects of the lifecycle. Defining a shared responsibility model like the one used for cloud service provider would further clarify the responsibility of certain requirements.

Additionally, due to the fact that the catalog is so extensive, and many topics are covered in different dimensions, it is easy to lose the overview. However, a certain basic knowledge of the catalog and its contents is necessary in order to be able to make meaningful references, for example. Therefore, it was very helpful for the AI developer that the knowledge about the content and structure of the catalog was gathered in one person and that this person distributed the tasks internally in the company.

Highly useful from both sides’ view would be further guidance and template documents supporting the documentation and audit process. This would decrease some of the spent efforts and to some extent standardize the audit process. 

Finally, throughout the study we noticed the similarity to the cybersecurity evaluation of IT systems. In cybersecurity we face very similar challenges, like required expertise, constantly changing systems or supply chain risks. This makes it worthwhile to explore the applicability of methodologies used in the cybersecurity evaluations to the evaluation of AI systems.