AI systems and Human operators

 𝘗𝘳𝘦𝘢𝘮𝘣𝘭𝘦

𝘛𝘩𝘪𝘴 𝘢𝘳𝘵𝘪𝘤𝘭𝘦 𝘥𝘳𝘢𝘸𝘴 𝘪𝘯𝘴𝘱𝘪𝘳𝘢𝘵𝘪𝘰𝘯 𝘧𝘳𝘰𝘮 𝘵𝘩𝘦 𝘱𝘶𝘣𝘭𝘪𝘤𝘭𝘺 𝘴𝘩𝘢𝘳𝘦𝘥 𝘪𝘯𝘴𝘪𝘨𝘩𝘵𝘴 𝘪𝘯 𝘵𝘩𝘦 𝘷𝘪𝘥𝘦𝘰 "𝘛𝘩𝘦 𝘑𝘰𝘶𝘳𝘯𝘦𝘺 𝘧𝘳𝘰𝘮 𝘚𝘶𝘴𝘵𝘢𝘪𝘯𝘢𝘣𝘪𝘭𝘪𝘵𝘺 𝘵𝘰 𝘈𝘶𝘵𝘰𝘯𝘰𝘮𝘺" 𝘣𝘺 𝘒𝘦𝘭𝘷𝘪𝘯.𝘢𝘪. 𝘛𝘩𝘦 𝘥𝘪𝘴𝘤𝘶𝘴𝘴𝘪𝘰𝘯𝘴 𝘱𝘳𝘦𝘴𝘦𝘯𝘵𝘦𝘥 𝘩𝘦𝘳𝘦 𝘦𝘹𝘱𝘭𝘰𝘳𝘦 𝘤𝘳𝘪𝘵𝘪𝘤𝘢𝘭 𝘴𝘢𝘧𝘦𝘵𝘺 𝘢𝘯𝘥 𝘩𝘶𝘮𝘢𝘯 𝘧𝘢𝘤𝘵𝘰𝘳𝘴 𝘤𝘩𝘢𝘭𝘭𝘦𝘯𝘨𝘦𝘴 𝘪𝘯 𝘵𝘩𝘦 𝘥𝘦𝘱𝘭𝘰𝘺𝘮𝘦𝘯𝘵 𝘰𝘧 𝘈𝘐 𝘴𝘺𝘴𝘵𝘦𝘮𝘴 𝘪𝘯 𝘴𝘢𝘧𝘦𝘵𝘺-𝘤𝘳𝘪𝘵𝘪𝘤𝘢𝘭 𝘥𝘰𝘮𝘢𝘪𝘯𝘴.

𝘛𝘩𝘦 𝘤𝘰𝘯𝘵𝘦𝘯𝘵 𝘸𝘢𝘴 𝘥𝘦𝘷𝘦𝘭𝘰𝘱𝘦𝘥 𝘶𝘴𝘪𝘯𝘨 𝘖𝘱𝘦𝘯𝘈𝘐 𝘊𝘩𝘢𝘵𝘎𝘗𝘛, 𝘸𝘪𝘵𝘩 𝘈𝘯𝘥𝘳𝘪𝘺 𝘒𝘰𝘴𝘵𝘺𝘶𝘬 𝘭𝘦𝘢𝘥𝘪𝘯𝘨 𝘵𝘩𝘦 𝘱𝘳𝘰𝘤𝘦𝘴𝘴 𝘵𝘩𝘳𝘰𝘶𝘨𝘩 𝘳𝘦𝘭𝘦𝘷𝘢𝘯𝘵 𝘲𝘶𝘦𝘳𝘪𝘦𝘴 𝘢𝘯𝘥 𝘪𝘵𝘦𝘳𝘢𝘵𝘪𝘷𝘦 𝘱𝘳𝘰𝘮𝘱𝘵𝘴 𝘰𝘷𝘦𝘳 𝘢𝘯 𝘦𝘹𝘵𝘦𝘯𝘥𝘦𝘥 𝘥𝘪𝘴𝘤𝘶𝘴𝘴𝘪𝘰𝘯. 𝘞𝘩𝘪𝘭𝘦 𝘈𝘐 𝘸𝘢𝘴 𝘪𝘯𝘴𝘵𝘳𝘶𝘮𝘦𝘯𝘵𝘢𝘭 𝘪𝘯 𝘰𝘳𝘨𝘢𝘯𝘪𝘴𝘪𝘯𝘨 𝘢𝘯𝘥 𝘳𝘦𝘧𝘪𝘯𝘪𝘯𝘨 𝘵𝘩𝘦 𝘪𝘥𝘦𝘢𝘴, 𝘵𝘩𝘪𝘴 𝘢𝘳𝘵𝘪𝘤𝘭𝘦 𝘪𝘴 𝘵𝘩𝘦 𝘳𝘦𝘴𝘶𝘭𝘵 𝘰𝘧 𝘢 𝘤𝘰𝘭𝘭𝘢𝘣𝘰𝘳𝘢𝘵𝘪𝘰𝘯 𝘸𝘩𝘦𝘳𝘦 𝘵𝘩𝘦 𝘱𝘳𝘪𝘮𝘢𝘳𝘺 𝘪𝘥𝘦𝘢𝘵𝘪𝘰𝘯, 𝘴𝘺𝘯𝘵𝘩𝘦𝘴𝘪𝘴, 𝘢𝘯𝘥 𝘧𝘳𝘢𝘮𝘪𝘯𝘨 𝘸𝘦𝘳𝘦 𝘨𝘶𝘪𝘥𝘦𝘥 𝘣𝘺 𝘵𝘩𝘦 𝘩𝘶𝘮𝘢𝘯 𝘢𝘶𝘵𝘩𝘰𝘳.

𝘉𝘺 𝘣𝘭𝘦𝘯𝘥𝘪𝘯𝘨 𝘩𝘶𝘮𝘢𝘯 𝘦𝘹𝘱𝘦𝘳𝘵𝘪𝘴𝘦 𝘢𝘯𝘥 𝘈𝘐 𝘢𝘴𝘴𝘪𝘴𝘵𝘢𝘯𝘤𝘦, 𝘵𝘩𝘦 𝘢𝘳𝘵𝘪𝘤𝘭𝘦 𝘢𝘪𝘮𝘴 𝘵𝘰 𝘱𝘳𝘰𝘷𝘪𝘥𝘦 𝘢 𝘤𝘰𝘮𝘱𝘳𝘦𝘩𝘦𝘯𝘴𝘪𝘷𝘦 𝘢𝘯𝘥 𝘵𝘩𝘰𝘶𝘨𝘩𝘵𝘧𝘶𝘭 𝘱𝘦𝘳𝘴𝘱𝘦𝘤𝘵𝘪𝘷𝘦 𝘰𝘯 𝘵𝘩𝘦 𝘳𝘪𝘴𝘬𝘴 𝘢𝘯𝘥 𝘤𝘰𝘮𝘱𝘭𝘦𝘹𝘪𝘵𝘪𝘦𝘴 𝘰𝘧 𝘪𝘯𝘵𝘦𝘨𝘳𝘢𝘵𝘪𝘯𝘨 𝘈𝘐 𝘪𝘯𝘵𝘰 𝘴𝘢𝘧𝘦𝘵𝘺-𝘤𝘳𝘪𝘵𝘪𝘤𝘢𝘭 𝘰𝘱𝘦𝘳𝘢𝘵𝘪𝘰𝘯𝘴.

🚨 𝗣𝗿𝗲𝗺𝗮𝘁𝘂𝗿𝗲 𝗔𝗜 𝗗𝗲𝗽𝗹𝗼𝘆𝗺𝗲𝗻𝘁 𝗶𝗻 𝗦𝗮𝗳𝗲𝘁𝘆-𝗖𝗿𝗶𝘁𝗶𝗰𝗮𝗹 𝗦𝘆𝘀𝘁𝗲𝗺𝘀: 𝗥𝗶𝘀𝗸𝘀 𝗮𝗻𝗱 𝗛𝘂𝗺𝗮𝗻 𝗙𝗮𝗰𝘁𝗼𝗿𝘀 𝗖𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗲𝘀 🚨

As safety engineers, we often hear about the promises of AI revolutionising operations in safety-critical domains like transportation, energy, and industrial systems. While AI holds tremendous potential, we must pause and ask: Are we rushing into replacing human operators without fully engineering the foundation these systems rely on?

𝗧𝗵𝗲 𝗥𝗶𝘀𝗸𝘀 𝗼𝗳 𝗣𝗿𝗲𝗺𝗮𝘁𝘂𝗿𝗲 𝗔𝗜 𝗜𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻

Too often, AI is introduced as a solution to fill gaps in under-engineered systems. This can lead to several risks:

• 𝘐𝘯𝘢𝘥𝘦𝘲𝘶𝘢𝘵𝘦 𝘊𝘰𝘳𝘦 𝘌𝘯𝘨𝘪𝘯𝘦𝘦𝘳𝘪𝘯𝘨: Foundational issues like handling uncertainties and edge cases may be overlooked, leaving AI to compensate for gaps it cannot fully address.
• 𝘖𝘷𝘦𝘳-𝘖𝘱𝘵𝘪𝘮𝘪𝘴𝘵𝘪𝘤 𝘈𝘶𝘵𝘰𝘯𝘰𝘮𝘺: AI is expected to perform flawlessly across all scenarios, even when data or system boundaries limit its situational awareness.
• 𝘋𝘪𝘴𝘦𝘯𝘨𝘢𝘨𝘦𝘥 𝘖𝘱𝘦𝘳𝘢𝘵𝘰𝘳𝘴: Human operators, relegated to a passive "monitoring" role, may lose situational awareness, reducing their ability to intervene effectively in emergencies.
• 𝘖𝘱𝘢𝘲𝘶𝘦 𝘋𝘦𝘤𝘪𝘴𝘪𝘰𝘯-𝘔𝘢𝘬𝘪𝘯𝘨: Unlike deterministic logic systems, AI often operates as a black box, making it difficult to understand, trust, or audit its decisions.

𝗟𝗲𝘀𝘀𝗼𝗻𝘀 𝗳𝗿𝗼𝗺 𝗣𝗿𝗼𝘃𝗲𝗻 𝗦𝘆𝘀𝘁𝗲𝗺𝘀

Take ETCS Level 3 (European Train Control System) as an example. It relies on robust, deterministic logic where all known scenarios are explicitly engineered, leaving room for human operators to manage unforeseen situations. There's no AI—just well-defined logic that ensures safety, transparency, and collaboration. This balance of automation and human oversight highlights the value of sound engineering over novelty.

𝗛𝘂𝗺𝗮𝗻 𝗙𝗮𝗰𝘁𝗼𝗿𝘀 𝗖𝗼𝗻𝘀𝗶𝗱𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝘀

Replacing operators too quickly introduces serious human factors challenges:

• Operators may feel alienated or worry about being blamed if they override AI recommendations.
• In emergencies, fear of liability may push operators to follow AI advice, even when their intuition suggests otherwise.
• AI's bounded situational awareness cannot match human flexibility, especially when operators can coordinate with peers or access adjacent systems beyond AI's scope.

𝗕𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝘁𝗵𝗲 𝗥𝗶𝗴𝗵𝘁 𝗕𝗮𝗹𝗮𝗻𝗰𝗲

To ensure safety and effectiveness, we must:

• 𝘌𝘯𝘨𝘪𝘯𝘦𝘦𝘳 𝘍𝘪𝘳𝘴𝘵, 𝘈𝘶𝘵𝘰𝘮𝘢𝘵𝘦 𝘓𝘢𝘵𝘦𝘳: Fully address uncertainties and edge cases with deterministic logic before introducing AI.
• 𝘋𝘦𝘧𝘪𝘯𝘦 𝘈𝘐’𝘴 𝘙𝘰𝘭𝘦 𝘊𝘭𝘦𝘢𝘳𝘭𝘺: Use AI to augment decision-making, not replace core safety functions.
• 𝘌𝘮𝘱𝘰𝘸𝘦𝘳 𝘏𝘶𝘮𝘢𝘯 𝘖𝘱𝘦𝘳𝘢𝘵𝘰𝘳𝘴: Design systems where humans remain engaged and have the authority to intervene without fear of liability.
• 𝘌𝘯𝘴𝘶𝘳𝘦 𝘌𝘹𝘱𝘭𝘢𝘪𝘯𝘢𝘣𝘪𝘭𝘪𝘵𝘺: Avoid black-box AI; prioritise transparency and auditability.
• 𝘛𝘦𝘴𝘵 𝘜𝘯𝘥𝘦𝘳 𝘙𝘦𝘢𝘭𝘪𝘴𝘵𝘪𝘤 𝘊𝘰𝘯𝘥𝘪𝘵𝘪𝘰𝘯𝘴: Validate combined AI-human systems extensively before deployment.

𝗧𝗵𝗲 𝗕𝗼𝘁𝘁𝗼𝗺 𝗟𝗶𝗻𝗲

AI is not a shortcut to safety—it’s a tool to enhance robust, well-engineered systems. Let’s prioritise core engineering and human-centric design to ensure that the integration of AI doesn’t come at the cost of reliability, safety, and trust.

________________________________________________

𝗔𝗽𝗽𝗲𝗻𝗱𝗶𝘅: 𝗦𝘂𝗺𝗺𝗮𝗿𝘆 𝗼𝗳 𝗛𝗮𝘇𝗮𝗿𝗱𝘀 𝗳𝗿𝗼𝗺 𝗶𝗻𝘁𝗲𝗿𝗳𝗮𝗰𝗶𝗻𝗴 𝗔𝗜 𝘀𝘆𝘀𝘁𝗲𝗺𝘀 𝘄𝗶𝘁𝗵 𝘀𝗮𝗳𝗲𝘁𝘆 𝗰𝗿𝗶𝘁𝗶𝗰𝗮𝗹 𝘀𝘆𝘀𝘁𝗲𝗺𝘀 𝗮𝗻𝗱 𝗼𝗿𝗴𝗮𝗻𝗶𝘀𝗮𝘁𝗶𝗼𝗻𝘀

1. 𝘐𝘯𝘤𝘰𝘮𝘱𝘭𝘦𝘵𝘦 𝘊𝘰𝘷𝘦𝘳𝘢𝘨𝘦 𝘰𝘧 𝘖𝘱𝘦𝘳𝘢𝘵𝘪𝘰𝘯𝘢𝘭 𝘚𝘤𝘦𝘯𝘢𝘳𝘪𝘰𝘴:

The AI system is deployed without fully addressing all known uncertainties, edge cases, and failure scenarios, resulting in unpredictable behaviour during unmodeled events.

2. 𝘓𝘰𝘴𝘴 𝘰𝘧 𝘖𝘱𝘦𝘳𝘢𝘵𝘰𝘳 𝘚𝘪𝘵𝘶𝘢𝘵𝘪𝘰𝘯𝘢𝘭 𝘈𝘸𝘢𝘳𝘦𝘯𝘦𝘴𝘴:

Human operators become disengaged or overly reliant on AI, reducing their ability to effectively intervene in emergencies.

3. 𝘌𝘹𝘤𝘦𝘴𝘴𝘪𝘷𝘦 𝘛𝘳𝘶𝘴𝘵 𝘪𝘯 𝘈𝘐 𝘙𝘦𝘤𝘰𝘮𝘮𝘦𝘯𝘥𝘢𝘵𝘪𝘰𝘯𝘴:

Operators may blindly follow AI suggestions due to the system's perceived authority, even when the recommendations are flawed or lack sufficient context (automation bias).

4. 𝘓𝘦𝘨𝘢𝘭 𝘢𝘯𝘥 𝘈𝘤𝘤𝘰𝘶𝘯𝘵𝘢𝘣𝘪𝘭𝘪𝘵𝘺 𝘗𝘳𝘦𝘴𝘴𝘶𝘳𝘦𝘴 𝘰𝘯 𝘖𝘱𝘦𝘳𝘢𝘵𝘰𝘳𝘴:

Fear of liability or legal consequences pushes operators to comply with AI advice, even when their intuition or expertise suggests an alternative action.

5. 𝘖𝘱𝘢𝘲𝘶𝘦 𝘈𝘐 𝘋𝘦𝘤𝘪𝘴𝘪𝘰𝘯-𝘔𝘢𝘬𝘪𝘯𝘨:

AI operates as a black box, preventing operators and engineers from understanding or validating the reasoning behind its recommendations.

6. 𝘔𝘪𝘴𝘮𝘢𝘵𝘤𝘩𝘦𝘥 𝘚𝘤𝘰𝘱𝘦 𝘰𝘧 𝘈𝘐 𝘚𝘺𝘴𝘵𝘦𝘮 𝘈𝘸𝘢𝘳𝘦𝘯𝘦𝘴𝘴:

The AI system has limited situational awareness, particularly at the boundaries of its defined domain, leading to errors when adjacent systems, underlying or external conditions affect safety.

 7. 𝘖𝘷𝘦𝘳𝘭𝘰𝘢𝘥𝘪𝘯𝘨 𝘖𝘱𝘦𝘳𝘢𝘵𝘰𝘳𝘴 𝘸𝘪𝘵𝘩 𝘓𝘰𝘸-𝘝𝘢𝘭𝘶𝘦 𝘈𝘭𝘦𝘳𝘵𝘴:

AI systems generate nuisance workload by recommending trivial or redundant actions, leading to frustration and potential system disengagement by operators.

8. 𝘖𝘱𝘦𝘳𝘢𝘵𝘰𝘳 𝘈𝘭𝘪𝘦𝘯𝘢𝘵𝘪𝘰𝘯:

Premature introduction of AI reduces operator trust and engagement, as they feel their role is being diminished or undervalued.

9. 𝘊𝘰𝘯𝘧𝘭𝘪𝘤𝘵𝘴 𝘋𝘶𝘳𝘪𝘯𝘨 𝘌𝘮𝘦𝘳𝘨𝘦𝘯𝘤𝘺 𝘚𝘤𝘦𝘯𝘢𝘳𝘪𝘰𝘴:

In high-pressure situations, AI recommendations may conflict with operator intuition, causing delays or errors in decision-making due to hesitation or fear of overriding the system.

10. 𝘐𝘮𝘣𝘢𝘭𝘢𝘯𝘤𝘦𝘥 𝘗𝘰𝘴𝘵-𝘐𝘯𝘤𝘪𝘥𝘦𝘯𝘵 𝘈𝘤𝘤𝘰𝘶𝘯𝘵𝘢𝘣𝘪𝘭𝘪𝘵𝘺:

AI recommendations are automatically recorded, while human decisions are not, creating an asymmetry in post-incident reviews that unfairly places blame on operators.

 11. 𝘜𝘯𝘥𝘦𝘳-𝘌𝘯𝘨𝘪𝘯𝘦𝘦𝘳𝘦𝘥 𝘚𝘺𝘴𝘵𝘦𝘮 𝘍𝘰𝘶𝘯𝘥𝘢𝘵𝘪𝘰𝘯𝘴:

Deployment of AI before addressing fundamental engineering challenges leads to a fragile system dependent on probabilistic logic rather than deterministic safety measures.

12. 𝘐𝘯𝘢𝘥𝘦𝘲𝘶𝘢𝘵𝘦 𝘐𝘯𝘵𝘦𝘳𝘰𝘱𝘦𝘳𝘢𝘣𝘪𝘭𝘪𝘵𝘺 𝘈𝘤𝘳𝘰𝘴𝘴 𝘚𝘺𝘴𝘵𝘦𝘮𝘴:

AI systems are siloed and unable to communicate with adjacent systems or integrate data effectively, leading to blind spots in safety-critical operations.

13. 𝘗𝘳𝘦𝘮𝘢𝘵𝘶𝘳𝘦 𝘌𝘹𝘤𝘭𝘶𝘴𝘪𝘰𝘯 𝘰𝘧 𝘏𝘶𝘮𝘢𝘯 𝘖𝘱𝘦𝘳𝘢𝘵𝘰𝘳𝘴:

Attempts to eliminate human operators entirely for the sake of novelty or cost reduction result in the loss of essential human flexibility and expertise in unforeseen scenarios.

14. 𝘔𝘪𝘴𝘢𝘭𝘪𝘨𝘯𝘮𝘦𝘯𝘵 𝘰𝘧 𝘈𝘐 𝘸𝘪𝘵𝘩 𝘏𝘶𝘮𝘢𝘯 𝘋𝘦𝘤𝘪𝘴𝘪𝘰𝘯 𝘗𝘳𝘰𝘤𝘦𝘴𝘴𝘦𝘴:

AI systems fail to adapt to operator needs, producing irrelevant or non-contextual recommendations that conflict with human workflows, e.g., due to bias inherent from synthetic or real data (statistical, sampling, group attribution, confirmation, implicit or human cognitive bias).

15. 𝘋𝘦𝘨𝘳𝘢𝘥𝘢𝘵𝘪𝘰𝘯 𝘰𝘧 𝘛𝘦𝘢𝘮 𝘊𝘰𝘮𝘮𝘶𝘯𝘪𝘤𝘢𝘵𝘪𝘰𝘯 𝘢𝘯𝘥 𝘊𝘰𝘭𝘭𝘢𝘣𝘰𝘳𝘢𝘵𝘪𝘰𝘯:

Operators lose access to informal or peer-based information-sharing mechanisms as AI systems cannot replicate human-to-human communication in complex scenarios.

16. 𝘖𝘷𝘦𝘳-𝘖𝘱𝘵𝘪𝘮𝘪𝘴𝘵𝘪𝘤 𝘋𝘦𝘱𝘭𝘰𝘺𝘮𝘦𝘯𝘵 𝘰𝘧 𝘈𝘐 𝘈𝘶𝘵𝘰𝘯𝘰𝘮𝘺:

AI systems are expected to function autonomously without sufficient validation or operational testing under real-world conditions, increasing the risk of failures.

17. 𝘉𝘪𝘢𝘴 𝘛𝘰𝘸𝘢𝘳𝘥 𝘕𝘰𝘷𝘦𝘭𝘵𝘺 𝘖𝘷𝘦𝘳 𝘚𝘢𝘧𝘦𝘵𝘺:

Organizational pressure to adopt AI for innovation undermines traditional safety engineering practices, compromising system reliability.

18. 𝘍𝘢𝘪𝘭𝘶𝘳𝘦 𝘵𝘰 𝘙𝘦𝘤𝘰𝘨𝘯𝘪s𝘦 𝘈𝘐 𝘚𝘺𝘴𝘵𝘦𝘮 𝘓𝘪𝘮𝘪𝘵𝘢𝘵𝘪𝘰𝘯𝘴:

AI systems do not adequately flag or defer to operators when faced with uncertainties or gaps in data, leading to unsafe actions or delays.

19. 𝘐𝘮𝘣𝘢𝘭𝘢𝘯𝘤𝘦 𝘪𝘯 𝘙𝘦𝘴𝘰𝘶𝘳𝘤𝘦 𝘈𝘭𝘭𝘰𝘤𝘢𝘵𝘪𝘰𝘯:

Over-investment in AI development diverts resources from improving deterministic safety logic or operator training, creating a suboptimal hybrid system.

 20. 𝘓𝘰𝘴𝘴 𝘰𝘧 𝘙𝘦𝘥𝘶𝘯𝘥𝘢𝘯𝘤𝘺 𝘪𝘯 𝘋𝘦𝘤𝘪𝘴𝘪𝘰𝘯-𝘔𝘢𝘬𝘪𝘯𝘨:

Removing the operator as an active decision-maker eliminates a critical layer of redundancy, increasing the likelihood of single-point failures in AI-driven systems.