AI in the Car: A Look to the Future

Jack Melling (Host):

Hello and welcome to the latest on Viewpoints podcast. I’m your host Jack Melling, Senior Editorial Manager at Arm. Today we’re going to be talking about AI in a car.

My guests today are Masashige Muzuyama, who serves as representative director, executive vice president and chief technology officer at Panasonic Automotive Systems,

Since joining Panasonic over 30 years ago, Mishiyama -san has been involved in wide array of endeavors, including operating system technology development and product and technology development, with roles including head of technology for the automotive division’s entertainment business and head of advanced technology development at the automotive business CTO’s office. Mishiyama -san’s exceptional technical prowess and leadership have earned him higher claim within and beyond the industry as he continually strives for new innovations. Great introduction there!

From Arm, we have Simon Teng. He is the Senior Director of Go -To -Market for Automotive at Arm. He leads the team in the APAC region to build partnerships in automotive ecosystem and to promote Arm products, technologies, and solutions as part of the partners’ innovations. He has nearly 12 years professional experience in automotive industry. And prior to joining Arm, he worked as the Assistant General Manager of the automotive business at MediaTek, leading a function of strategic planning, product management and business development.

As you can see, we have a lot of experience on the panel today. So, first question I have to you, Mizuyama-san, is many people still see AI as a recent phenomenon, but this isn’t necessarily the case for the automotive industry. What are the best examples of AI powered automotive applications in the car today?

Masashige Mizuyama (Panasonic Automotive Systems):

Yes, AI is obviously not a future thing, but the technology currently in practical use. One leading example is of course AD/ADAS applications, but for this application there are still significant technical constraints due to the less explainability of AI. The other is a more unconstrained one.

It’s cockpit Human Machine Interface. Voice recognition used to interface with the driver who should have eyes on the road, hands on the steering wheel, and for passengers as well, it is employed as an easy-to-use, friendly interface, and is rapidly evolving and being deployed.

Simon Teng (Arm):

But I also see that there’s a of fusion of all these kind of sensor data using the AI to process that and the ADAS and also the main machine interface. And the more important, especially in the vehicle, is to fusion all these kind of information, provide meaningful information for the driver and also for the passenger in the future. Not only just for the efficiency of getting the information, but also improving the safety. It’s not only just one individual function, but also every function will connect to using the AI technology.

Jack Melling:

So how long has AI been in the car for? As I mentioned in the first question, it seems to me it’s something that has just suddenly sprung upon us from nowhere. But I imagine it’s been in development in the car for quite a while now, possibly over the past decade. So, from your perspective, how long has this been in the works for?

Masashige Mizuyama:

Actually, it has been well over a decade. Maybe 20 years or more.

Simon Teng:

But I just give one example. Do you remember about the modern 10 or 15 years ago, we have some voice command, You follow the voice, but you have to say specific words, and then the system process this voice signal to the command. And the comment can be executed in the electronic system.

And now, you fast forward to today, we are talking about it is not a comment. It’s kind of a very natural intuition to speak what I want to play this person’s music. And at the same time, can you tune the voice a little higher for the bass? So, this is a very, very complicated kind of instruction from the driver or the passenger. And this type of the message from the human or the driver and the passenger now is easily being processed by the AI capability in the vehicle.

Masashige Mizuyama (04:55)

As Simon said, the voice interface in the very early stage, was as simple as recognizing dozens of predefined commands. This was done using statistical method or simpler machine learning method.

In the next stage, the AI model evolved into a so called Deep Learning, specifically, in the form of deep neural network or recurrent neural network, while semiconductor performance became greatly improved. As a result, voice interface has evolved dramatically. In other words, it supports interfaces in which unspecified speakers speak sentences continuously and environmental robustness has also been significantly improved. At this point, it has finally became practical to give instructions to the car by voice, and it has been accepted by many users.

Furthermore, what will be interesting car in the near future is LLM or other transformer-based generative AI technologies. LLMs make it possible for drivers or passengers to communicate their intentions to the car through a conversation with a very high degree of freedom. In the past, people talked in a limited way that their car can understand. In the future, you will be able to communicate pretty much in similar way with talking to a person, and your car infers your intention like your buddy. Moreover, the communication will be not only verbal, but also multimodal like with gesture.

Jack Melling:

So we touched upon this concept of the man-machine interface through both of your examples of AI in a car. I just want to delve a little bit deeper into that. Firstly, what is man-machine interface in the car?

And secondly, why is it so important, particularly in today’s modern vehicles? I’d like to go to Mizuyama-san.

Masashige Mizuyama:

So, in contrast to traditional vehicles, where HMI consisted only of mechanical means such as steering, shift knobs, pedals and needle meters, modern cars have incomparably more complex functions and by which safety and comfort is enhanced. So…

Sophisticated HMIs are needed to manipulate and understand the information that are far more complex and richer than in the past. Here are some examples.

For example, since such a modern car cannot be handled sufficiently with simple control levers and pedals, voice interfaces have been developed and graphical expression using displays have become frequently used to present information that cannot be expressed with needle meters and simple lamps. That’s why instrument cluster is replaced from needle meters to displays.

Furthermore, in recent years, information presentation has been evolving to be intuitive by AR, augmented reality using head-up display.

In addition, voice interfaces have become more flexible and sophisticated in the form of understanding complex contexts and intentions with inference, as I mentioned, so that driver can handle such complex functions and information.

And, in addition to all those, an even greater evolution is emerging. It is an infrared interface in which car senses the state of the driver and passenger, and the car autonomously adapts to the person without any instruction.

For example, car detects the driver’s drowsiness by in-cabin camera, then issues an alert or even changes the air conditioning to keep the driver awake.

Simon Teng:

in summary, I think we can imagine that there is a different phase of the MMI.

In the early stage, it’s just like Mizuyama mentioned, it’s mechanical centric, right? So, this is just the mechanical parts. The people, the driver need to fit into this mechanical, right?

The second stage is like a hardware and like an electronic component centric. So, it’s still the hardware, but moving from the mechanical to the electronics. And you still have to fit into that, but the response time and there will be a more variety of these kind of a component or the device that can communicate or to have interactive with the driver or the passenger.

The third phase, what I will see is become the software centric. It’s just like what Mizuyama said, of the display, the graphic, the rendering and the graphics. So, the software will talk to and have interactive action with the specific role in the vehicle, like a passenger, like a driver.

I see the fourth stage is the AI centric. The AI centric is convert these kind of role based into personalized. Just like in the animation there. Okay, I can specifically for this person to customize because the AI can recognize and the learning your behavior and then to have more interactivity. So I will see that the main machine interface with these four different kinds of evolution.

Jack Melling:

So that’s great, Simon, that you actually mentioned AI there and the role of AI in man-machine interface, because that leads me very nicely onto my next question, which is just, I would like both of you to talk me through the role of AI in man-machine interface and why it’s so important.

Masashige Mizuyama:

So, to answer the question, I think there are two points. One is the aspect of perception.

To realize the Human Machine Interface in a way that is more natural for humans, like people communicating with people through the cognitive functions of AI.

The other is smartness, by inferring the intent of the driver or passenger through a simpler communication or active sensing, the car responses or even makes suggestions autonomously. In a nutshell, the concept of AI agent which has been repeatedly mentioned in the past is going to be real.

For example, if you say, “I’m feeling like good coffee” to your car, it will recommend a destination and set a navigation route by considering your current location, your favorite coffee shop, public evaluation of nearby ones, congestion, etc.

If possible, it will place an online order automatically with estimated arrival time. As such, with the destruction minimized within the constraints of driver being driving, it understands the driver’s intention and provides maximum support.

This would not be possible without Generative AI technology.

Simon Teng:

My view, if you look at it from the driver or the behavior, why do people use the car? Because people stay in the car for transportation, maybe 10 minutes to one hour transportation to the walking to the office, right? So, some people stay to quite long a day in the vehicle. This is one thing. people are expecting not just to focus on the driving situation on the road. So, if the people can take this kind of long hours in the state in the vehicle, can do something extra creative or relax even more. So that is from the human-beings demand, we can utilize this kind of a time when we are driving. So, this is one aspect.

And the other aspect the vehicle is really a complicated system. You have a lot of, moving parts. It’s a power train, there’s a lot of signal and interacting with the other vehicles. So, the vehicle is really the complicated system. So, if you want to achieve the safety and also the comfortability and the creativity, all these kinds of things, you need to process a lot of this kind of information. The only kind of technology we believe that can realize that these kind of requirements are on demand when we’re the car as the tool for us and utilizing the time is the AI.

Using the AI is probably the most promising way to create really, really when people driving the car, using the car, people can really utilize the time and also be safe. So that’s why the main machine interface and the AI is very, very important in this industry.

Jack Melling:

So just hearing all about man -machine interface and everything it does. It seems to me that it requires quite a bit of computing resources. So, I know at Arm we love getting into the tech and understanding what these applications need. What computing resources does man-machine interface need? Can it run on a CPU or GPU alone? Or does it need AI accelerators to perform these tasks?

Masashige Mizuyama:

So, in my opinion, heterogeneous computing which consists of CPU, GPU as well as NPU would be optimal for HMI in car because there will be more displays, more sophisticated AI as I mentioned. Heterogeneous computing is suitable. Especially in the future, more AI agents will be used to operate cars.

As the interface between cars and people becomes more dependent on it, there will be less and less something like a control lever from a car. In that case, even in a situation where there is no network connection, it will be a serious problem if the AI-based interface does not work.

In addition to that, AI algorithm research has been evolved to make inferencing sufficiently performable with fixed-point operations. This makes edge side AI quite feasible and richer in functionality.

So, for AI at the edge, a relatively large processing unit will with low degree of parallelism which CPU generally has, will be inefficient and insufficient in performance.

However, in the case of GPUs, memory traffic will become a bottleneck.

Therefore, I think that domain-specific architecture, that is NPU in this case, is a realistic solution for AI in the edge.

And, most of the NPUs installed in many current SoCs are suitable for CNN or RNNs, and I think they need to be optimized for generative AI workload for the future.

There is no doubt that something like the Edge Cloud Collaborative AI will be developed, and more advanced AI agents for automobiles will be developed. Privacy also needs to be given due consideration to use cloud.

Simon Teng:

I would like to make a comment on that. Yes, there will be a CPU, GPU, MPU. if you look at why we need to MPU, what is the difference, just like I’m using that. Different type of the processor technology has their strengths and also the weakness.

It really depends on what is that because the AI is really the data driven, right? So depend on the data and the training data set or the inference data set. For example, voice data, if you want to do the recognition or the voice, the voice data compare with the vision data. For example, a megapixel, HBS image, the size of the raw data is quite different.

Depending on the data itself, you can use different type of the processor. For example, like a voice, it would be suitable for the CPU, could be, or the GPU. Using the NPU, maybe the utilization, because the data, amount of the data would be small, the utilization of the NPU would be low. So, it depends on how this kind of data would be used. So, we believe because in the vehicle, there are so many different types of data set and the data. So, we believe that there will be a heterogeneous processor that adapt to the AI. It’s not necessary that we only using the NPU for AI, but the CPU for the sequential workload and the GPU just for the certain level parallelization of the workload. So, we believe that it should be depend on the data, among other data. This is one point, looking at the processor.

The second point is the safety. Currently from the process of technology point of view, it would be much easier to get the safety certification for CPU and GPU, but we need to develop some other safety mechanisms for the NPU in the future. So if you are running the AI which is you need a safety critical workload, it would be better running on the CPU and the CPU because that is the safety certified, easily safe certified.

The third aspect is the programming model. Because currently, NPU is the… This kind of proprietary, there’s no standard, even the defective system. So even there’s an open source, but it’s very, very difficult for the people to really easily to program effectively using the NPU. There’s a gap, we need to work on that. But there’s some defective standard for the programming model in the CPU, but you need to do the manually compiler interpreter or this kind of a graphic compiling. And so from the programming models point of view, the CPU is most natural one.

Jack Melling:

Thank you. That’s a really fascinating look at the technology and how different workloads can be processed across different processes. So really, really great deep dive into that. We’ve talked about the tech. Let’s think about the vision for the future. So looking ahead, how will AI advances improve the in-vehicle experience?

Masashige Mizuyama:

In summary, comfortability, hospitality, that kind of thing will we greatly improve using AI.

We should not necessarily say everything in detail by step by step to instruct cars. Cars in inference, do inference what the drivers or passengers want. And then, proactively offers such a control or some kind of proposal to the passengers or the drivers.

Simon Teng:

If a vehicle is like a place that you can simulate or emulate whatever environment you want to have. So for example, if you want to leverage the time that you are driving to become the office life, like a moving office, you can say to the vehicles, I want to give me the background and all these kind of connections, I want to have a conference call, want to have all these kind of supposedly, you can get the vehicle to transform the small space to the moving office. If you want to take a relax, you can tell the vehicle, say, I want to be relaxed, and give me the relaxed environment. You can create the different kinds of environment in the vehicle through the AI and the vehicle will provide to you. So it’s just a simple as that. So I think that would be perfect AI’s application and the people would love and enjoy the time in the car.

Jack Melling:

What a lovely vision for the future. I like the idea of relaxing as you’re in your car! Focusing more on Arm and Panasonic as companies, could you just talk me through your latest announcements, technology developments in relation to the vehicle? Plus, how are Arm and Panasonic working together.

Masashige Mizuyama:

So, this is not a technology, but one of the most exciting things on our side is our announcement, recent announcement of company’s vision to become the best Joy in Motion design company in the world. We aim to become a company that provides the best technology solutions and services that eliminate pains when you are moving, including the cockpit and cabin environment, and make the total experience the best for comfortable mobility, just like the comfort of home or comfort of clothes you wear. To that end, we are working on the development of various user experience technologies, including HMI, AI, etc.

One specific instance of these technologies is unified HMI, which we are working on in collaboration with ARM, especially with Simon-san. There is an industry trend in which more and more various displays, including head-up-displays, are arranged in the cockpit and pursuing optimal information presentation.

Unified HMI is a virtualization technology that integrates physical individual displays into a single large virtual screen. This makes it possible to develop the optimal user experience in an agile manner.

Simon Teng:

Yeah, so especially for the automotive industry, we have a dedicated product line. So, we know that the trend is the system become very complicated. And because the more friendly user experience need to be axed by the industry, and also the more safety function, automated function, and the electrification.

So, we need to provide more advanced processing capabilities to IP. So, this is our core business. We already just in the early of this year, we launched the newest AE23 automotive IP portfolio. And this is the first time we adopted server-level IP, the processing capability, into the automotive with the functional safety support certified. But we believe that it’s not enough. We think the IP need to be evolved into the solution, which means that because the system getting more and more complicated. So, Arm has a lot of technology. If we can put the IP become a subsystem to combine with the multiple IP and create a subsystem level solution to the industry, that will save all these kind of development and make it more easier and reduce the barrier to adopt the newest technology in the industry.

But not only that, we also recognize the only the hardware and the IT is not enough, you have to have some software solution. So that’s why we also initiated for example like a SOAFEE, to become the software, as a foundation of the software-defined vehicle solution. And we also think even moving this kind of a software solution into the cloud, like a digital train and developing the cloud.

We are also invest and work with the industrial partner, the ecosystem partner to provide the, we call the virtual prototyping solution together. So, this is the Arm’s investment we think we can contribute. And we hope that this is an advance and the solution for the industry.

Jack Melling:

So, I guess in conclusion, there’s a lot of fantastic work happening by both Arm and Panasonic in the automotive market. So really great to see and actually really looking forward to some of these solutions hitting the market and seeing how they can influence the software defined vehicles of the future. So, the final question, what are your personal favorite features or applications in the car which are powered by AI and perhaps more importantly how is it Arm-powered or maybe even Panasonic-powered. So we’ll go to Mizuyama-san first.

Masashige Mizuyama:

Okay, so, favorite feature, that’s difficult question for me because any ADAS, human interfaces and smart backend services are all evolving, from my perspective, in a super exciting way by AI.

So I can’t choose only one for all of them.

It’s so exciting in the vehicle industry. So, and if I had to say one thing, it would be that if you look back at the history of computers, just as evolved from central computing to distributed computing, personal computing, and then ubiquitous computing with Internet of Things, AI also will become more ubiquitous, and eventually AI will be behind everything.

I am very much looking forward to a future where all the user experience in the car will be with comfortable hospitality and the wonderful “joy in motion” or comfortable moving will be realized by that.

So, I bet that Arm is one of the most eligible companies who bring computing platform to realize such ubiquitous intelligence.

And using Arm’s powers, we will be the world’s best “joy in motion” design company.

That’s our vision.

Simon Teng:

For me, I would like to have my car to understand where I am. So, when I use my mobile phone, I need you so the car can come to me and then pick me up. And as I want to go home and the car.

We’re all the route for the traffic jam, and then they can detour to the other. So, I don’t need to worry about it, because currently we need to worry about, I need to look at the map. So, this is the thing. If I can have this function, and also when I sit into the car, I can read all the information through whatever display, for example, the transparent display in the window, the door window or the header display.

So, there will be a display all the place. And this kind of user interface that I can enjoy the reading during my stay in the vehicle. So that’s the dream feature that I want to have. And without AI, it cannot be come true and be realized. And also, that’s the reason why we work with Panasonic. We think the display will be one of the key elements in the main machine interface. Also Panasonic is the leading company in the automotive industry to provide a system level solution to the OEM. I think the cooperation between the Arm as a semiconductor or the solution provider together with the system levels company like Panasonic, this kind of a strategic relationship will really bring some innovation to the industry and the benefit for everyone.

Jack Melling:

Well, that’s a fantastic way to end. I love your vision for the future, Simon. More importantly, do you see it being powered by Arm?

Simon Teng:

Yes, yes, yes. And together with Panasonic.

Jack Melling:

Of course. This podcast discussion has been absolutely fascinating. I’ve certainly learnt a lot. I hope the listeners have learnt a lot too. Really covered a lot of ground around AI in a car. And thank you very much for your time. It’s been a wonderful discussion and we look forward to people joining us for the next Arm Viewpoints podcast in the future.