Shadow Sense – Visual foundation model for capability enhancement PhD

This PhD provides the future digital workforce with an adaptive AR visualisation guided by the state-of-the-art Foundation Model (FM) (Large Language Model (LLM)) AI methods. The research seeks to develop a novel approach that can identify elements of a task; observe and understand state; and intervene with corrective guidance.

In complex engineering tasks, an operator must perform a physical action in response to a work instruction. For situations where the conventional work instruction is unavailable (e.g., it may be tacit knowledge), an FM-AR system can substitute the absent instruction in the form of visual overlay in the region of interest.

During the epistemic action of task completion, a novel FM-AR system can assess semantic scene understanding, operator intent, action context and intervention response. This method forms part of a system integration between an engineering system, visual overlay, and data processing.

Aim

The aim of the PhD is to develop an FM-AR solution that focuses on assisting the digital engineers of the future, improving operator efficiency, and providing consistent results. The solution is expected to support a wide range of scenarios that a digital engineer would be involved in such as:

• • Maintenance – inspection, adjustment, servicing
• • Repair – removal, installation, replacement
• • Overhaul – upgrade, modification, rebuild.

The research is expected to include many concepts and technologies such as: FM-AR, Data analytics. Consideration should be given to these concepts and the reliability of these to enhance the productivity of digital engineers. The project should consider what information or data needs to be provided within the context of Maintenance, Repair and Overhaul (MRO) scenarios.

The AR solution will provide real-time support to the digital engineer in various areas such as: a) information overlay, b) process integration, c) task awareness and d) intervention. Each of these areas provide distinct benefits to the operator and demonstrate the art of the possible with artificial intelligence and augmented reality.

Objectives

• 1.Conduct literature review to identify state-of-the-art in FM-AR
• 2. Identify suitable use cases for the solution.
• 3. Construct a solution for the next generation digital engineers.
• 4. Develop a framework for FM-AR assisted intervention for digital engineers.

At Cranfield, the candidate will be based at the Centre for Digital Engineering and Manufacturing, which hosts cutting-edge simulation and visualisation facilities. The student will have access to high-end computers for constructing the proposed solution. There will be relevant opportunities throughout the PhD to develop and demonstrate the research.