在自动驾驶汽车研发与测试中，开发商通常会采用公共道路“影子驾驶”模式，也就是说车上会配备一名人类驾驶员，但这位驾驶员并不操控方向盘，仅观察被测系统的运转情况。但这一作法实际有两个大前提，第一是车辆已经学会如何适当地应对驾驶过程中的各种事件；第二是人类驾驶员（观察者）的反应足够快，可以阻止任何不利结果的发生。

由于各种各样的原因，看起来“影子驾驶”模式似乎是目前完全自动驾驶汽车研发与测试中不可或缺的组成部分。

事实上，自动驾驶汽车的测试并不轻松，每家制造商均需累积大约一万亿英里的自动驾驶里程，并覆盖甚至多次覆盖所有可能的场景。根据作者的保守估计，假设配置 23.4万 辆汽车，每天以时速 50 英里 24 小时运行，那么累计一万亿英里自动驾驶里程需要 10 年，成本高达 3000 亿美元以上。

除了成本，“影子驾驶员”还存在安全方面的隐患。举个例子，为了训练人工智能和 SAE L3级自动驾驶及控制交接功能，厂商必须让车辆经历一系列真实事故场景。小至剐蹭追尾，大到其他更加危险、更加复杂的场景，整个训练过程可能涉及成千上万个事故场景，极有可能造成不同程度的伤害，甚至伤亡。另外，此类事故场景还必须加上公共道路测试，这也会让整个测试过程更加危险。无论是一些正在借助“影子驾驶员”进行研发测试的系统，还是已经投入使用的 SAE L3 级公共自动驾驶汽车，目前这些系统根本不可能留给“影子驾驶员”充裕的反应时间，以充分了解当下的驾驶环境，重新取得对车辆的有效控制，并保证安全。

正如最近几起自动驾驶事故所掀起的风波一样，“影子驾驶员”出现在公共道路上，有可能削弱消费者对自动驾驶汽车的信心与支持，甚至引来铺天盖地的负面媒体报道，催生更加严格的监管，招致无休止的诉讼，还会造成投资者的信任丧失，给自动驾驶汽车的发展带来沉重的后果，并最终将这项有潜力挽救成数十万民众的技术扼杀在摇篮之中。

值得庆幸的是，我们还有一个解决方案，那就是使用“完整模拟”完成主要验证过程，降低行业对“影子驾驶员”的依赖。该解决方案采用“全系统工程”方法设计，以客户的用户需求和设计流程为基础，并包含“最终状态场景”。

合情合理的模拟手段

目前，汽车行业使用的模拟系统远未达到航空航天行业的水平和复杂度（即美国联邦航空管理局FAA的D 级规定），也没有采用适当的实时架构。现阶段，车辆、轮胎和道路模拟的模型均不够精确，特别是在模拟一些恶劣条件时。人工智能看起来似乎已经做好了学习准备，但事实并非如此。更可怕的是，直至真实场景发生之前，此类情况通常都很难发现，而一旦发生，则会给项目进展带来沉重的时间和执行压力，甚至直接让项目叫停。

注意，这些问题并不会暴露在一些常规测试场景下，只有当面对一些非常复杂，或对时间要求很高的场景，迫使车辆达到甚至超过性能极限时才会出现，而这通常也正是问题的开始。

假如不配备全动系统(full motion systems)，一些驾驶员在环（DiL）模拟器可能会让开发人员的信心爆棚，但真实情况并非如此乐观。运动系统设备可以配合真人模拟器使用，用于模拟自动驾驶行程，并允许开发人员评估车上人员的晕车感、舒适感及对自动驾驶汽车的信任水平。除了配备合适的运动系统外，开发人员还可以借助“航空航天/DoD/FAA”的仿真技术、最佳做法和测试方法，解决汽车行业面临的自动驾驶模拟挑战。这是因为，一些国防城市战争游戏的游戏场景与很多复杂驾驶场景高度重合，另外还有一些采用了专业模型并提供实时保真的效果，尤其可以发挥重要作用。

如图所示，大多数自动驾驶汽车开发商均无法履行之前的承诺，即在有限范围内推出真正具备SAE L4-L5级自动驾驶功能的自动驾驶汽车。(数据来源：Eric Paul Dennis/Center for Automotive Research)

数据方法论至关重要

并预先定义和构建最困难的场景，整个项目可能将最终远超预计时间，才能做到所有模拟场景的执行，包括在当下及未来无休止地重复修补这些复杂场景。

如果采用敏捷开发流程，可能浪费的时间将难以预估，而且从历史经验来看，一些较为复杂的元素通常很难完成，这只能为日后开发埋下隐患。此外，除非遭遇到一些最复杂和最困难的场景，否则这些设计缺陷通常很难暴露。最终，项目可能不得不进行“修修补补”，大量返工，而不是在一开始就在许多常见场景中设置妥当。

目前，“边缘场景”和“角落场景”经常用于描述事故情景。但事实上，事故情景与任何其他情景并无差别，只是结果是没有人想看到而已。一些真正的“边缘场景”或“角落场景”是在任何情况下都不应该，也不可能发生的 — 例如要求搜索引擎寻找一张猫的图像，但最终得到的是一张垃圾桶的图像。工程师通常不会覆盖所有可能的事故场景，也就是被他们划分在“核心场景”之外的“边缘场景”或“角落场景”。也正因如此，人们有理由进行必要的尽职调查。

模拟仿真的目标应集中在为 AI 堆栈提供可用于辨别不同物体的数字表达式，采用相同的输入速率，并具备相同的模糊度，从而找到妨碍 AI 堆栈做出正确决定的问题。这些数据集中，最难实现的部分常被称为“边缘”或“角落”场景；然而，这些场景才是判断 AI 堆栈是否具备成功决策能力的关键案例。为了清晰定义这些案例，并明确每个案例的预期结果，我们需要一种条理分明的可管理式递归数据方法。

最终状态场景矩阵

除了提供影响上述系统工程方法的场景数据之外，所有各方（包括政策制定机构、验证机构、保险公司和制造商等）都需要尽早了解模拟目标，也就是“项目完工”的定义。只有具备对实时变化的支持能力（从而及时修正任何 AI 感知错误），场景数据集才能真正称得上全面，但相应的工作量几乎与为仿真过程清晰定义“整合’’与“系统模型”一样可观。

从地理围栏到 SAE L4 级和 L5级自动驾驶汽车，要成功实现这些目标，该测试数据集的建立需要依赖众多数据源和数据域；要求全球汽车开发界开展最高水平的尽职调查；必须确保达到必要的安全水平并能够证明这一点；还必须映射到上文提到的仿真系统，并与之同步。

在目前的 AV 测试范例做出改变之前，汽车行业永远不会迎来可以挽救万千生命的 SAE L4 级自动驾驶汽车，也不会迎来真正的全自动驾驶汽车。

Autonomous vehicle developers are widely using public “shadow” driving which involves a human in the driver’s seat letting go of the steering wheel and ceding control to the system under test to observe how it performs. The fundamental premise of this process is that the vehicle has learned the proper management of possible events which may occur during the maneuvers, and the human observer can react fast enough to stop any negative results from occurring.

It is a myth that public shadow driving is the best or only solution to create a fully autonomous vehicle, for several reasons.

To complete such an effort would require each AV maker to accumulate roughly one trillion miles in driving and re-driving all the potential scenarios. The estimated cost of such programs is over $300 billion [based on the author’s conservative calculation of 234,000 vehicles operating at an average of 50 mph, every day all day for 10 years, to arrive at one trillion miles].

Other problems with shadow driving involve safety, including the running of actual accident scenarios to train the AI and SAE Level 3/handover. The process of accident-scenario “training” has potential to cause thousands of accidents, injuries and casualties when efforts to train and test the AI move from the benign scenarios to more complex and dangerous ones. Thousands of accident scenarios will have to be driven multiple times on the public streets driving scenarios—whether in a system under development using public shadow driving or an SAE Level 3 vehicle in use by the public, it is impossible to provide the driver with a sufficient margin of time to regain situational awareness for safely executing effective vehicle control.

As we’ve seen in the aftermath of recent accidents, public shadow driving can weaken consumers’ support of AVs while bringing negative media coverage, increased regulations, endless litigation, and loss of investor trust. As a result, the industry could lose the opportunity to deliver true autonomous vehicles and thus save tens of thousands of lives and avoid hundreds of thousands of injuries.

There is a solution, however. It is to replace most of the AV public shadow driving as the primary validation process with complete simulation. Such a solution would be designed from full systems engineering, driven by a requirements definition and design process, augmented by an end-state scenario.

A proposal for proper simulation

The systems currently in use by the auto industry are inadequate, and nowhere near aerospace complexity, or FAA Level D competency. They do not have proper real-time architectures. The models being used for vehicles, tires and roads, are not precise enough, especially in degraded conditions. The AI will appear to have learned properly, when in actuality it has not. This is often not discovered until analogous real-world scenarios are experienced—and in the process, expose critical timing and execution gaps.

Keep in mind that the benign scenarios being run now will not encounter these problems. It is not until you run complex or time-critical scenarios that the performance envelope of the vehicle, tire or road models is reached. That’s typically when the problems start.

Driver-in-the-loop (DiL) simulators without full motion systems can cause a significant level of false confidence. A motion system device used in manned simulators would be used for the simulated autonomous driving. Motion systems permit evaluation of motion sickness and passengers’ feeling of comfort and trust with the autonomous vehicle management. In addition to having a proper motion system, simulation issues can be resolved by leveraging aerospace/DoD/FAA simulation technology, practices and test methodologies. Especially useful are those relating to DoD urban war games, which are directly analogous to complex driving scenarios, as well as those employing proper model and real-time fidelity.

Data methodology is key 

Utilization of Agile processes, or a bottom-up engineering approach, is an inefficient if not debilitating process when it is employed in complex systems. Too much time is wasted time, by not developing components in parallel, as well as not defining and building to the most difficult scenarios up front. All of which require simulation to execute, including the immediate and endless repletion of these scenarios.

If the agile approach is taken the time lost will be extreme, and historically the less complex elements will be completed, leaving the more complex configurations for “later”. Also, flawed design assumptions will not likely be exposed until the most complex and difficult scenarios are encountered. That will usually drive a design and execution change that will need to be employed for many benign scenarios—with significant rework as a result.

Two popular and very flawed terms—“edge case” and “corner case”—are widely used today to describe accident scenarios. Accident scenarios are like any other scenario, but with outcomes that no one desires. A true edge or corner case is a scenario that should not happen in any possible scenario—such as asking a search engine to find an image of a cat, and then receiving the image of a garbage can. Engineers typically will not search out all the possible accident scenarios because they are deemed on the edges or corners of the core set. This then gives people an excuse to do the required due diligence.

The purpose of the simulation should be focused on presenting the AI stack the same digital representations it might experience with the same level of items to be discriminated, at the same input rate, with the same level of ambiguity to allow for determining where the AI stack has issues in making proper decision. The very hardest of these data sets to achieve have been called “edge” or “corner” cases; however, these are the key cases which define success for the AI stacks decision process. Defining those cases, along with defining the desired results of those cases, requires a structured, recursive and manageable data methodology.

End-state Scenario Matrix 

Beyond providing the scenario data to affect the systems engineering approach mentioned above is the need for all parties—including policy makers, validators, insurance companies and manufacturers—to know what “done” looks like as early as possible. The scenario set would indeed be comprehensive, with support for real time variations, which will confirm any AI perception errors. The effort to do this needs to be just as significant as the definition of the integration and systemic models for the simulation.

From legitimate geofencing to legitimate SAE Level 4 and 5, this test set would need to be informed from a multitude of data sources and domains. It would need to reflect the highest level of due diligence the global vehicle-development community can muster. It must ensure that the requisite levels of safety are attained and proven. Finally, it would have to be mapped to and synced with the simulation/simulator system noted above.

The industry will never save nearly the lives it hopes to with Level 4 vehicles, nor will it get close to having a true autonomous vehicle, until the current paradigm of AV testing is changed.

Author: Michael DeKort
Source: SAE Aerospace Engineering Magazine