SAE 4 级和 5 级自动驾驶汽车的验证挑战, 一直是驾驶模拟专家 rFpro 公司的主要关注点，fFpro技术总监 Chris Hoyle 表示，网联自动驾驶汽车的出现为汽车行业带来了一系列新的未知数。未来，汽车制造商必将建立包含数千个模拟测试场景的数据库，以应对汽车验证的挑战。

Hoyle 认为，目前必须回答的关键问题包括：我们如何才能确定网联自动驾驶汽车可以在任何条件下都可安全运行？我们该如何保证测试的全面性和严谨性，但又同时满足对测试周期和成本要求？我们是否有方法在进入验证阶段前，加速自动驾驶车辆的开发，但又同时避免给公共道路用户带来危险？

Hoyle 表示自己经常遇到相关讨论。他认为，与现实世界中的真实测试相比，模拟仿真测试具有覆盖范围大、测试周期短等优势，但前提是必须加以正确应用。

新的模拟仿真平台

rFpro 公司声称已经推出了世上首款针对自动驾驶汽车模拟仿真、训练及开发的商用平台，可以在“各种可以想象到的环境下”对自动驾驶汽车进行测试。据称，该平台的一个关键特点是“以非常高的精度，精确复制现实世界的测试环境”。通过一项为期 3 年的项目，rFpro 公司已经利用高精度扫描技术（更多信息，请点击这里）将大量真实道路转换为模拟测试场景，并最终创建了一个“模拟测试库”。用户可在库中选择各种不同的模拟测试场景，并控制从“天气”到“行人”等各种变量输入。目前，这项技术已被 2 家大型OEM、3 家自动驾驶汽车开发商及 1 条无人驾驶赛车产品线所采用，但出于商业机密方面的考虑，rFpro 公司并未透露更多细节。  

凭借一组全天候工作的计算机，制造商每月可累计数百万英里的模拟测试里程。Hoyle 解释说：“从统计学角度而言，人类驾驶员平均每行驶 1 亿英里，就会发生一起交通致死事故（来源：NHTSA数据），但我们在自动驾驶汽车的模拟测试中很难真正累计到这种级别的测试里程数。事实上，人类驾驶员的“数据”如此优秀是因为：在日常驾驶中，我们的绝大多数里程都是“无事发生”的。正因如此，我们可以消除这部分“无事故”里程，并通过模拟技术让网联自动驾驶汽车每隔几秒钟即遇到一次“千年一遇”的事件，进而大幅缩短模拟测试周期。未来，汽车制造商将建立包含数千个模拟测试场景的数据库，而自动驾驶汽车必须成功通过这些测试场景，才能通过验证。”

每次遇到测试失败的情况，模拟测试场景库中都会新增几项针对这一场景的标准测试。此外，车辆不仅必须成功通过每一项测试，而且还必须保证性能的稳定。

Hoyle 表示，这种采用回归逻辑不断重复运行的测试库，可以确保任何对自动驾驶汽车的新改进均不会影响现有功能。“为了实现这个目的，rFpro 不仅可以在一组设备上并行运行多个实验，而且还可在多个 CPU 和 GPU 上对一项实验进行扩展，从而应对由于自动驾驶汽车数据来源多样（包括多部摄像头、激光雷达、雷达传感器等）而引入的复杂性。”

标准化模拟

程度如此密集的测试需要一定时间才能达到预期的结果。但 Hoyle 预计，未来五年中，新测试场景的增加速度将降至在统计学角度低于人类驾驶员出错率的程度。这时，现实世界中的真实物理验证流程就可以启动了。

Hoyle 认为，如果进展理想，汽车行业将开发一套全球性的标准化测试场景库，任何自动驾驶车型一旦通过该标准库的验证，即可进入下一阶段测试——即从库里抽样选择一部分场景，进行现实世界中的真实测试。

但这就带来了另一个问题：我们如何才能保证这个场景库的全面性和严谨性？ Hoyle 表示，讽刺的是，人类非常善于测试自动驾驶汽车，“因为人类具有随机且不可预知的特点，从来不会重复；人类会犯错，而且表现也会随着情绪和疲劳程度的不同而改变。”目前，单个模拟测试中可以支持的人类驾驶员数量已经增加至 50 个。我们可以测试自动驾驶汽车在人口或道路用户密集的城市中心地区的表现，而且无需承担人员伤亡的风险。

rFpro 公司预计，到今年晚些时候，加入单独测试的人类驾驶员数量将增加至 250 名，负责对一辆或多辆网联自动驾驶汽车的测试。

人工智能 (AI) 系统的高效开发，离不开在启动重新测试前从失败中进行学习、总结、改进的能力。Hoyle 强调说，我们还会将频繁遇到的边缘案例（其中一个参数超过系统限制）或极端案例（其中两个或两个以上参数超过系统限制）反馈至系统中，从而不断丰富系统的知识库。

我们可以建立训练数据库，用以展示失败测试中本应表现出来的正确行为，且每个数据库均应包含来自所有传感器模型的数据，包括虚拟摄像头、激光雷达和雷达。Hoyle 表示，每一帧训练数据均应与“地面实况”数据相关联，包括语义段、实例段、光流算法、深度和标记目标数据，“在这种逻辑下，我们的模型可以通过监控下的学习，针对每一种新的失败模型，进行相应的改进和适应。”

然而Hoyle 也补充道，尽管如此，我们还有一个至关重要的因素必须始终铭记于心， “人类是无偏差输入的最佳来源，因为即使在相同天气条件下的同一条道路上，人类驾驶员也不会以完全相同的方式驾驶。此外，他们可以发现一些异常、激怒或意外事件，并有可能因为其他道路用户的行为而作出不当反应！”

The challenges of validating autonomous vehicles designed to operate at SAE Level 4 and 5 are a major focus of driving-simulation specialist rFpro. The advent of connected autonomous vehicles (CAVs) presents the auto industry with a broad new set of unknowns that will see automakers establish "libraries" of thousands of test scenarios, said the company’s technical director, Chris Hoyle.

He presents key questions that have to be answered: “How will we know that a CAV is safe to operate under all conditions? How can we ensure that testing is sufficiently comprehensive and rigorous, yet timely and cost-effective? Even before we reach the validation stage, is there a way to accelerate the development of autonomous vehicles without the risks associated with exposing them to public road users?”

Hoyle said he is routinely exposed to this debate and believes that simulation can provide greater scope and shorter timelines than physical testing—but that it must be applied correctly.

New simulation platform

The company has launched what it claims to be the world’s first commercially available platform to train and develop autonomous vehicles in simulation and test their systems in “every scenario imaginable.” A key aspect of the platform is claimed to be the level of simulation accuracy achieved replicating the real world. A three-year program has seen the company build a library of real roads via high-precision scanning technology (see previous article). Users have control of a wide range of variables, from weather to pedestrians. The technology has been adopted by two large OEMs, three autonomous-vehicle developers and a driverless motorsport series. For commercial confidential reasons, rFpro is unable to give further details.  

By using a cluster of computers 24/7, manufacturers can achieve millions of simulated miles of testing every month. Explains Hoyle: “Human drivers average one fatality in 100 million miles driven (source: www-fars.nhtsa.dot.gov/Main/index.aspx), but we cannot realistically attempt to accumulate this sort of mileage with a CAV before declaring a test to be complete. The reason a human driver scores so well is because much of the distance is uneventful; by eliminating this ‘dead’ mileage and subjecting—via simulation—the CAV to a “once-in-a-1000 year” event every few seconds, we can massively compress the timescale. Vehicle manufacturers will build libraries of thousands of simulated test scenarios, which autonomous models will have to successfully pass before they will be considered ready for validation.”

Any failed experiment typically results in several more standard tests being added to the library of simulated scenarios, each of which must be reliably passed with consistency.

Hoyle said the libraries of tests, run continuously using a regression process, would ensure that any new developments to an autonomous model do not break existing functionality. “To enable this, rFpro not only scales across a cluster of machines to run multiple experiments in parallel, but it also allows each experiment to scale across multiple CPUs and GPUs to cope with the complexity of autonomous models fed by multiple camera, LiDAR and radar sensors.”

Standardizing simulation

Testing so intensively will take time to achieve required results. But Hoyle anticipates that over the next five years, the rate at which new test scenarios are identified will fall to the point at which it can be statistically proven to be below the error rate for human drivers. At that stage, the physical validation and verification process could begin.

He expects—perhaps hopefully—that the auto industry will develop a global standardized library of test scenarios which, once the model validates them, will then move forward to the next stage: a statistical sample of those tests will be selected and expanded for physical testing in the real world.

But along comes another question: how to build a library of tests that is sufficiently comprehensive and rigorous. Ironically, Hoyle stated that humans are very good at testing autonomous vehicles: “Humans are random, unpredictable, never the same twice; we make mistakes and our performance changes with mood and fatigue level.” At present, up to 50 human drivers can be added into a single simulated experiment, piloting vehicles with the autonomous model tested in densely-populated, simulated urban environments surrounded by other road users and pedestrians, without any risk of death or injury.

By late this year, rFpro anticipates this will be scaled up to 250 human test drivers entering a single experiment, shared by one or more CAVs.

Efficient development of artificial-intelligence (AI) systems requires the ability to learn from failures and improve the functionality before re-testing, stressed Hoyle, saying edge cases (where one parameter exceeds system limits) or corner cases (where a combination of two or more parameters exceeds system limits) frequently will be encountered and fed back into the system, increasing its knowledge base.

Training datasets can be established to demonstrate correct behavior for failed experiments, each comprising all the data that is fed to sensor models for virtual cameras, LiDAR and radar. Every frame of training data is associated with “ground-truth” data, comprising semantic segmentation, instance segmentation, optical flow, depth and labelled object data: “In that way, through supervised learning, the models improve and adapt to each new failure mode,” Hoyle said.

But through all this, a vital factor must always be remembered and appreciated, he added: “Humans are the best source of unbiased inputs because they never drive in an identical manner, even when repeating a journey on the same road in the same weather conditions. Also, they can identify behavior which is unusual, irritating or unexpected and likely to promote adverse reaction from other road users!”

Author: Stuart Birch
Source: SAE Automotive Engineering Magazine