RePAIR: the research project for the reassembly of fragmented frescoes using an “intelligent robot” comes to an end
The research project RePAIR (Reconstructing the Past: Artificial Intelligence and Robotics Meet Cultural Heritage), funded by the European Union, is coming to an end and is presenting its main results: the creation of a robotic infrastructure guided by artificial intelligence and the use of algorithms to reassemble fragmented Pompeian frescoes like a “puzzle”. The entire prototype was validated through initial experimental tests conducted on-site at the Archaeological Park of Pompeii, demonstrating that robotics and artificial intelligence may, in the future, facilitate the work of archaeologists.
The research focused on two iconic examples of large frescoes belonging to the world’s cultural heritage, which are currently in a fragmentary state and preserved in the storerooms of the Archaeological Park of Pompeii. These are the ceiling frescoes from rooms in the House of the Painters at Work in the Insula of the Chaste Lovers, damaged during the eruption of AD 79 and later shattered by bombing during the Second World War, and the frescoes of the Schola Armaturarum, which collapsed in 2010 and are still partially unassembled.
In the first case, a group of experts in wall paintings from the University of Lausanne, led by Professor Michel E. Fuchs, had already been working on the context since 2018 with a study and manual reassembly programme, with whom the RePAIR researchers engaged in dialogue.
Launched in September 2021, the project was coordinated by University of Venice Ca’ Foscari and involved universities and research institutes across Europe and Italy, including the Italian Institute of Technology (IIT) and the Archaeological Park of Pompeii, which hosted and provided the experimental testing ground for the project. International partners included the Ben-Gurion University of the Negev(Israel), the Associacao do Instituto Superior Tecnico Para a Investigacao e Desenvolvimento (Portugal), and the Rheinische Friedrich Wilhelms Universität Bonn (Germany). The project received funding from the European Union’s Horizon 2020 research and innovation programme under Grant Agreement No. 964854. https://www.repairproject.eu/
“After four years of work, the European RePAIR project has come to an end — an international collaboration bringing together seemingly distant worlds: the most advanced techniques in artificial intelligence and robotics with archaeology and the preservation of cultural heritage, a shared legacy of humankind. It represents a first, pioneering step toward an ambitious goal: eliminating one of the most laborious and frustrating activities in archaeological research, thus allowing precious energy and expertise to be channeled into activities that are more strictly scientific and creative,” comments Marcello Pelillo, Professor at University of Venice and project coordinator.
The robotic infrastructure has been installed in the Casina Rustica, a state-owned building within the Archaeological Park of Pompeii that has been restored and repurposed to house the necessary technological equipment.
While the robotics teams were responsible for designing and building the system, experts in artificial intelligence and machine learning developed and defined algorithms for reassembling the frescoes.
To study and further preserve the artefacts, the research group also created artificial replicas of the fragments after their digitalization, so that the robotic system could manipulate non-authentic pieces during testing phases.
“After acquiring and digitalizing images of the individual fragments, the system attempts to solve the ‘puzzle’, and the solution found is sent to the hardware platform which, using two robotic arms equipped with ‘soft hands’, automatically places the fragments in the desired position. This is an extremely complex puzzle, made up of hundreds or thousands of fragments that are often worn or severely damaged, without knowing in advance what the result should look like,” explains Prof. Pelillo. “In other words, the image on the box that would guide the work is missing. Moreover, the recovered pieces often represent only a portion of the original work, making large or numerous gaps in the reconstruction inevitable. To complicate matters further, establishing their actual provenance is difficult, since fragments from different works are frequently mixed together. To tackle this formidable problem, we used sophisticated artificial intelligence techniques and developed an interface that allows archaeologists to interact with the system.”
“The future of archaeology presupposes the ethically sound use of artificial intelligence. It is a great challenge,” says the Director of the Archaeological Park, Gabriel Zuchtriegel. “Reassembling the immense quantity of fragments — such as those damaged during the bombing of Pompeii in 1943 — should be possible thanks to the distinctive shape and decoration of each element. But no human being could accomplish this alone. This is where artificial intelligence comes into play, helping us address the complexity of archaeological materials, and it will have a central role in archaeology in the future, especially considering the vast amounts of data emerging from preventive archaeology excavations at worksites across Italy. However, shared skills and values are needed to ensure AI is used in a scientifically and ethically correct manner, and Pompeii is contributing to this global development.”
THE ROBOTIC PLATFORM AND THE DIGITALISATION SYSTEM
The Italian Institute of Technology (IIT) developed the prototype of the robotic platform and the digitalization system, with the contribution of four research groups based at the IIT Centres in Genoa and Venice, led by Antonio Bicchi, Alessio Del Bue, Arianna Traviglia and Nikolaos Tsagarakis.
The system consists of two identical robotic arms connected by a torso equipped with vision sensors, and two hands with compliant structures capable of delicate grip. The system is installed on an aluminium workbench featuring a linear guide along which the robot can move horizontally, sliding into optimal positions for manipulating each fragment.
The robotic hands are two SoftHands — soft robotic hands that easily adapt to different shapes and objects — available in two different sizes to manipulate fragments of various dimensions. For heavy and large objects, the system can also operate bimanually, coordinating the movements of both hands to ensure a stable grip. To achieve a firm yet delicate grip that does not damage the surface of the frescoes, the fingertips of the robotic hands have been fitted with soft, high-friction pads. Cameras for object recognition and for planning the robotic system’s movements have been integrated at wrist level. The system is also capable of detecting whether an object has been successfully grasped or not, repositioning itself to immediately try again.
To assist archaeologists in classifying artefacts — which in the specific case of frescoes can be a laborious and slow process — IIT researchers developed a portable device for 3D scanning so that digitalization can be carried out easily and affordably on site. The device consists of a lighting box, a rotating plate, a high-resolution RGB camera and a 3D scanning sensor. With this device, the research team obtained digital images of about 2,000 fragments, grouped into 117 different sets and characterized by various decorations identified by archaeologists, such as stucco or eroded surfaces.
In addition to 3D digitalization, the IIT research group also conducted hyperspectral analyses of the wall paintings. The data acquired can be used to support artificial intelligence algorithms in reconstructing the fragment “puzzles”, facilitating the assembly of the pieces. The hyperspectral approach allows for the identification of the pigments used, providing additional and detailed information about features no longer visible to the naked eye, thus contributing to a better understanding of the original decoration.
THE PARTNERS’ ACTIVITIES
Ca’ Foscari University and the other European partners developed artificial intelligence techniques. After acquiring and digitalizing images of the individual fragments, the system attempts to solve the “puzzle”, and the solution is then sent to the hardware platform which, using the two robotic arms, automatically places the fragments in their correct positions.
The Archaeological Park of Pompeii, besides hosting the project and providing the experimental testing ground, contributed by sharing know-how in archaeological sciences and conservation, and simultaneously created an extensive dataset of fresco images forming the knowledge base for the artificial intelligence system’s iconographic recognition — a key element for reassembling the frescoes stored in the Park’s deposits.
