Art. 54a Testing of High-Risk AI Systems in Real World Conditions Outside AI Regulatory Sandboxes

Article 54a

Testing of High-Risk AI Systems in Real World Conditions Outside AI Regulatory Sandboxes

‍Updated on Feb 6^th 2024 based on the version endorsed by the Coreper I on Feb 2^nd

1. Testing of AI systems in real world conditions outside AI regulatory sandboxes may be conducted by providers or prospective providers of high-risk AI systems listed in Annex III, in accordance with the provisions of this Article and the real-world testing plan referred to in this Article, without prejudice to the prohibitions under Article 5.

The detailed elements of the real world testing plan shall be specified in implementing acts adopted by the Commission in accordance with the examination procedure referred to in Article 74(2).

This provision shall be without prejudice to Union or national law for the testing in real world conditions of high-risk AI systems related to products covered by legislation listed in Annex II.

2. Providers or prospective providers may conduct testing of high-risk AI systems referred to in Annex III in real world conditions at any time before the placing on the market or putting into service of the AI system on their own or in partnership with one or more prospective deployers.

3. The testing of high-risk AI systems in real world conditions under this Article shall be without prejudice to ethical review that may be required by national or Union law.

4. Providers or prospective providers may conduct the testing in real world conditions only where all of the following conditions are met:

the provider or prospective provider has drawn up a real world testing plan and submitted it to the market surveillance authority in the Member State(s) where the testing in real world conditions is to be conducted;
the market surveillance authority in the Member State(s) where the testing in real world conditions is to be conducted has approved the testing in real world conditions and the real world testing plan. Where the market surveillance authority in that Member State has not provided with an answer in 30 days, the testing in real world conditions and the real world testing plan shall be understood as approved. In cases where national law does not foresee a tacit approval, the testing in real world conditions shall be subject to an authorisation;
the provider or prospective provider with the exception of high-risk AI systems referred to in Annex III, points 1, 6 and 7 in the areas of law enforcement, migration, asylum and border control management, and high risk AI systems referred to in Annex III point 2, has registered the testing in real world conditions in the non-public part of the EU database referred to in Article 60(3) with a Union-wide unique single identification number and the information specified in Annex VIIIa;
the provider or prospective provider conducting the testing in real world conditions is established in the Union or it has appointed a legal representative who is established in the Union;
data collected and processed for the purpose of the testing in real world conditions shall only be transferred to third countries outside the Union provided appropriate and applicable safeguards under Union law are implemented;
the testing in real world conditions does not last longer than necessary to achieve its objectives and in any case not longer than 6 months, which may be extended for an additional amount of 6 months, subject to prior notification by the provider to the market surveillance authority, accompanied by an explanation on the need for such time extension;
persons belonging to vulnerable groups due to their age, physical or mental disability are appropriately protected;
where a provider or prospective provider organises the testing in real world conditions in cooperation with one or more prospective deployers, the latter have been informed of all aspects of the testing that are relevant to their decision to participate, and given the relevant instructions on how to use the AI system referred to in Article 13; the provider or prospective provider and the deployer(s) shall conclude an agreement specifying their roles and responsibilities with a view to ensuring compliance with the provisions for testing in real world conditions under this Regulation and other applicable Union and Member States legislation;
the subjects of the testing in real world conditions have given informed consent in accordance with Article 54b, or in the case of law enforcement, where the seeking of informed consent would prevent the AI system from being tested, the testing itself and the outcome of the testing in the real world conditions shall not have any negative effect on the subject and his or her personal data shall be deleted after the test is performed;
the testing in real world conditions is effectively overseen by the provider or prospective provider and deployer(s) with persons who are suitably qualified in the relevant field and have the necessary capacity, training and authority to perform their tasks;
the predictions, recommendations or decisions of the AI system can be effectively reversed and disregarded.

5 Any subject of the testing in real world conditions, or his or her legally designated representative, as appropriate, may, without any resulting detriment and without having to provide any justification, withdraw from the testing at any time by revoking his or her informed consent and request the immediate and permanent deletion of their personal data. The withdrawal of the informed consent shall not affect the activities already carried out.

5a. In accordance with Article 63a, Member States shall confer their market surveillance authorities the powers of requiring providers and prospective providers information, of carrying out unannounced remote or on-site inspections and on performing checks on the development of the testing in real world conditions and the related products. Market surveillance authorities shall use these powers to ensure a safe development of these tests.

6. Any serious incident identified in the course of the testing in real world conditions shall be reported to the national market surveillance authority in accordance with Article 62 of this Regulation. The provider or prospective provider shall adopt immediate mitigation measures or, failing that, suspend the testing in real world conditions until such mitigation takes place or otherwise terminate it. The provider or prospective provider shall establish a procedure for the prompt recall of the AI system upon such termination of the testing in real world conditions.

7. Providers or prospective providers shall notify the national market surveillance authority in the Member State(s) where the testing in real world conditions is to be conducted of the suspension or termination of the testing in real world conditions and the final outcomes.

8. The provider and prospective provider shall be liable under applicable Union and Member States liability legislation for any damage caused in the course of their participation in the testing in real world conditions.

‍

[Previous version]

The detailed elements of the real-world testing plan shall be specified in implementing acts adopted by the Commission in accordance with the examination procedure referred to in Article 74(2).

This provision shall be without prejudice to Union or Member State legislation for the testing in real world conditions of high-risk AI systems related to products covered by legislation listed in Annex II.

3. The testing of high-risk AI systems in real world conditions under this Article shall be without prejudice to ethical review that may be required by national or Union law.

4. Providers or prospective providers may conduct the testing in real world conditions only where all of the following conditions are met:

the provider or prospective provider has drawn up a real-world testing plan and submitted it to the market surveillance authority in the Member State(s) where the testing in real world conditions is to be conducted;

the market surveillance authority in the Member State(s) where the testing in real world conditions is to be conducted have not objected to the testing within 30 days after its submission;

the provider or prospective provider with the exception of high-risk AI systems referred to in Annex III, points 1, 6 and 7 in the areas of law enforcement, migration, asylum and border control management, and high risk AI systems referred to in Annex III point 2, has registered the testing in real world conditions in the EU database referred to in Article 60(5a) with a Union-wide unique single identification number and the information specified in Annex VIIIa;

the provider or prospective provider conducting the testing in real world conditions is established in the Union or it has appointed a legal representative for the purpose of the testing in real world conditions who is established in the Union;

data collected and processed for the purpose of the testing in real world conditions shall not be transferred to countries outside the Union, unless the transfer and the processing provides equivalent safeguards to those provided under Union law;

the testing in real world conditions does not last longer than necessary to achieve its objectives and in any case not longer than 12 months;

persons belonging to vulnerable groups due to their age, physical or mental disability are appropriately protected;

[deleted]

where a provider or prospective provider organises the testing in real world conditions in cooperation with one or more prospective users, the latter have been informed of all aspects of the testing that are relevant to their decision to participate, and given the relevant instructions on how to use the AI system referred to in Article 13; the provider or prospective provider and the user(s) shall conclude an agreement specifying their roles and responsibilities with a view to ensuring compliance with the provisions for testing in real world conditions under this Regulation and other applicable Union and Member States legislation;

the subjects of the testing in real world conditions have given informed consent in accordance with Article 54b, or in the case of law enforcement, where the seeking of informed consent would prevent the AI system from being tested, the testing itself and the outcome of the testing in the real world conditions shall not have a negative effect on the subject;

the testing in real world conditions is effectively overseen by the provider or prospective provider and user(s) with persons who are suitably qualified in the relevant field and have the necessary capacity, training and authority to perform their tasks;

the predictions, recommendations or decisions of the AI system can be effectively reversed or disregarded.

5. Any subject of the testing in real world conditions, or his or her legally designated representative, as appropriate, may, without any resulting detriment and without having to provide any justification, withdraw from the testing at any time by revoking his or her informed consent. The withdrawal of the informed consent shall not affect the activities already carried out and the use of data obtained based on the informed consent before its withdrawal.

Suitable Recitals