"Why America's EV chargers keep breaking" - Article

[tchavei]

I've got absolutely no insight into what causes this – but I'm happy to share my "hunch". We know that PCM 6.0 is based on a docker architecture – Porsche has been reasonably public and proud of this in their Porsche Engineering magazine. Each "app" or tile in the PCM is a docker container.

For Porsche, this solves two big problems: (a) each app can be packaged (and in theory updated) without having any impact on other apps and (b) there doesn't have to be a totally unified approach to each app.

They run in their own protected memory and process space and especially they can run their own dependent processes without impacting other apps. This is a totally different approach to how iOS or Android works on billions of mobile devices. It is in fact closer to how many cloud servers work than consumer devices.

The fact that they feel they can't update apps or have multiple apps running in the same process and address space (or on one single OS) like smartphones do – is a window both into the chaos of automotive software and old fashioned approaches to keeping apps updated.

On the surface a Docker based PCM sounds modern – but is in fact a result of not being able to emulate how Apple or Google have deployed complex software to most of the planet.

However – and this is pure speculation on my part – the approach could have worked – but got caught up in traditional penny pinching in the automotive industry.

Docker is relatively CPU friendly – it doesn't add too much pressure on a processor – but where it is resource intensive is in its use of memory.

Running each app as its own "OS" with its own processes is not memory efficient – which is fine in server (or large desktop) situations where memory is usually plentiful.

Every time you tap a "app" or tile in the PCM – its booting a complete Docker container. You can see this in the "loading" screen with the spinning circle. If you hit an app you rarely use such as "smart service" or "info" you'll almost certainly see the loading screen.

For example, if you tap on "Weather" and haven't used it in a while – you'll see the loading spinner – which again is the docker container loading. After it has loaded, you'll be presented with the Weather app – which is auto run through something called Docker Entrypoint on the containers Dockerfile configuration. Anyways, you'll then see "data loading" in the app as it then needs to get its data from the weather service API it uses to show content.

Most apps need data to be useful so you'll see this dual loading which is a characteristic of this One App One Docker container approach they have taken.

One thing Docker will do – automatically – is that it will start to shutdown Docker containers if its running low on memory. I do not know what strategy Porsche uses here – but one reasonable guess is it will shut down the oldest, least used containers that are not currently showing on the screen.

The navigation is just a "app" and will also be shutdown in a memory constrained environment. If you've ever been using an app – such as Navigation – and suddenly it goes blank and you see the "Loading" spinner again – maybe even while driving – that is happening due to a low memory situation across the PCM – otherwise known as the Docker host.

You'll almost immediately see the "loading" spinner again as the PCM realises it has had to shutdown a container that was being used. But still even that – will free up processes and memory – and the PCM wouldn't do it if it had no choice.

I do not know how much physical memory the PCM has – but in my opinion it is not enough.

The one thing I would do if I was at Porsche is get more physical memory into the PCM – and I bet you they'd consider the €50/unit increase in cost as way too much – in what was for me a €200k car.

I'd bet my bottom dollar what @whitex is experiencing is an extreme low memory situation – and because of the "auto restore" to where you were – even on reboot – when this happens rebooting isn't working as its re-loading back to the problem.

You'll notice for example, if you have an active navigation route and you do the "two finger" reset – go into engineering mode and then restart back out of it – the PCM will restart with the current navigation route still "live". This is window into how it wants to get back to where it was pre-reboot.

While sometimes the "two finger" reset is enough of a reboot to fix problems – the lock car and walk away - seems to cause the PCM to do a deeper reset which solves this problem.

The second thing I'd do at Porsche – is just acknowledge that the PCM sometimes gets itself into problems and allow users access the "full/deep reset" function within the "advanced settings" menu option. Stop pretending its not required.

My Macbook that has a battle-hardened macOS with a UNIX linage right back to the original AT&T Unix of the 1970s – has a full restart computer option.

What makes the PCM so special ? Nothing.

Great information

So now all we need to know is how to type:

sudo docker-compose up -d force-recreate --build

directly into the pcm and we're golden.

Sponsored

 

[snstevens]

I've got absolutely no insight into what causes this – but I'm happy to share my "hunch". We know that PCM 6.0 is based on a docker architecture – Porsche has been reasonably public and proud of this in their Porsche Engineering magazine. Each "app" or tile in the PCM is a docker container.

For Porsche, this solves two big problems: (a) each app can be packaged (and in theory updated) without having any impact on other apps and (b) there doesn't have to be a totally unified approach to each app.

They run in their own protected memory and process space and especially they can run their own dependent processes without impacting other apps. This is a totally different approach to how iOS or Android works on billions of mobile devices. It is in fact closer to how many cloud servers work than consumer devices.

The fact that they feel they can't update apps or have multiple apps running in the same process and address space (or on one single OS) like smartphones do – is a window both into the chaos of automotive software and old fashioned approaches to keeping apps updated.

On the surface a Docker based PCM sounds modern – but is in fact a result of not being able to emulate how Apple or Google have deployed complex software to most of the planet.

However – and this is pure speculation on my part – the approach could have worked – but got caught up in traditional penny pinching in the automotive industry.

Docker is relatively CPU friendly – it doesn't add too much pressure on a processor – but where it is resource intensive is in its use of memory.

Running each app as its own "OS" with its own processes is not memory efficient – which is fine in server (or large desktop) situations where memory is usually plentiful.

Every time you tap a "app" or tile in the PCM – its booting a complete Docker container. You can see this in the "loading" screen with the spinning circle. If you hit an app you rarely use such as "smart service" or "info" you'll almost certainly see the loading screen.

For example, if you tap on "Weather" and haven't used it in a while – you'll see the loading spinner – which again is the docker container loading. After it has loaded, you'll be presented with the Weather app – which is auto run through something called Docker Entrypoint on the containers Dockerfile configuration. Anyways, you'll then see "data loading" in the app as it then needs to get its data from the weather service API it uses to show content.

Most apps need data to be useful so you'll see this dual loading which is a characteristic of this One App One Docker container approach they have taken.

One thing Docker will do – automatically – is that it will start to shutdown Docker containers if its running low on memory. I do not know what strategy Porsche uses here – but one reasonable guess is it will shut down the oldest, least used containers that are not currently showing on the screen.

The navigation is just a "app" and will also be shutdown in a memory constrained environment. If you've ever been using an app – such as Navigation – and suddenly it goes blank and you see the "Loading" spinner again – maybe even while driving – that is happening due to a low memory situation across the PCM – otherwise known as the Docker host.

You'll almost immediately see the "loading" spinner again as the PCM realises it has had to shutdown a container that was being used. But still even that – will free up processes and memory – and the PCM wouldn't do it if it had no choice.

I do not know how much physical memory the PCM has – but in my opinion it is not enough.

The one thing I would do if I was at Porsche is get more physical memory into the PCM – and I bet you they'd consider the €50/unit increase in cost as way too much – in what was for me a €200k car.

I'd bet my bottom dollar what @whitex is experiencing is an extreme low memory situation – and because of the "auto restore" to where you were – even on reboot – when this happens rebooting isn't working as its re-loading back to the problem.

You'll notice for example, if you have an active navigation route and you do the "two finger" reset – go into engineering mode and then restart back out of it – the PCM will restart with the current navigation route still "live". This is window into how it wants to get back to where it was pre-reboot.

While sometimes the "two finger" reset is enough of a reboot to fix problems – the lock car and walk away - seems to cause the PCM to do a deeper reset which solves this problem.

The second thing I'd do at Porsche – is just acknowledge that the PCM sometimes gets itself into problems and allow users access the "full/deep reset" function within the "advanced settings" menu option. Stop pretending its not required.

My Macbook that has a battle-hardened macOS with a UNIX linage right back to the original AT&T Unix of the 1970s – has a full restart computer option.

What makes the PCM so special ? Nothing.

I know you are speculating, but based on what is known I think you've made some good guesses as to what is going on.

Insufficient memory is a system architecture problem that only is addressed when the System Architect makes the issue known following an architectural design review. Individual app designers don't have enough knowledge of system requirements as a whole to make this call, and it makes me wonder who the SA for PCM is, or if there even is such a role?

In the medical device space, lack of sufficient real memory has resulted in 483 notices and recalls on devices as varied as infusion pumps and dialysis systems. The cause in these cases was in fact lack of an architectural review of changes prior to implementation and release.

 

[whitex]

I've got absolutely no insight into what causes this – but I'm happy to share my "hunch". We know that PCM 6.0 is based on a docker architecture – Porsche has been reasonably public and proud of this in their Porsche Engineering magazine. Each "app" or tile in the PCM is a docker container.

For Porsche, this solves two big problems: (a) each app can be packaged (and in theory updated) without having any impact on other apps and (b) there doesn't have to be a totally unified approach to each app.

They run in their own protected memory and process space and especially they can run their own dependent processes without impacting other apps. This is a totally different approach to how iOS or Android works on billions of mobile devices. It is in fact closer to how many cloud servers work than consumer devices.

The fact that they feel they can't update apps or have multiple apps running in the same process and address space (or on one single OS) like smartphones do – is a window both into the chaos of automotive software and old fashioned approaches to keeping apps updated.

On the surface a Docker based PCM sounds modern – but is in fact a result of not being able to emulate how Apple or Google have deployed complex software to most of the planet.

However – and this is pure speculation on my part – the approach could have worked – but got caught up in traditional penny pinching in the automotive industry.

Docker is relatively CPU friendly – it doesn't add too much pressure on a processor – but where it is resource intensive is in its use of memory.

Running each app as its own "OS" with its own processes is not memory efficient – which is fine in server (or large desktop) situations where memory is usually plentiful.

Every time you tap a "app" or tile in the PCM – its booting a complete Docker container. You can see this in the "loading" screen with the spinning circle. If you hit an app you rarely use such as "smart service" or "info" you'll almost certainly see the loading screen.

For example, if you tap on "Weather" and haven't used it in a while – you'll see the loading spinner – which again is the docker container loading. After it has loaded, you'll be presented with the Weather app – which is auto run through something called Docker Entrypoint on the containers Dockerfile configuration. Anyways, you'll then see "data loading" in the app as it then needs to get its data from the weather service API it uses to show content.

Most apps need data to be useful so you'll see this dual loading which is a characteristic of this One App One Docker container approach they have taken.

One thing Docker will do – automatically – is that it will start to shutdown Docker containers if its running low on memory. I do not know what strategy Porsche uses here – but one reasonable guess is it will shut down the oldest, least used containers that are not currently showing on the screen.

The navigation is just a "app" and will also be shutdown in a memory constrained environment. If you've ever been using an app – such as Navigation – and suddenly it goes blank and you see the "Loading" spinner again – maybe even while driving – that is happening due to a low memory situation across the PCM – otherwise known as the Docker host.

You'll almost immediately see the "loading" spinner again as the PCM realises it has had to shutdown a container that was being used. But still even that – will free up processes and memory – and the PCM wouldn't do it if it had no choice.

I do not know how much physical memory the PCM has – but in my opinion it is not enough.

The one thing I would do if I was at Porsche is get more physical memory into the PCM – and I bet you they'd consider the €50/unit increase in cost as way too much – in what was for me a €200k car.

I'd bet my bottom dollar what @whitex is experiencing is an extreme low memory situation – and because of the "auto restore" to where you were – even on reboot – when this happens rebooting isn't working as its re-loading back to the problem.

You'll notice for example, if you have an active navigation route and you do the "two finger" reset – go into engineering mode and then restart back out of it – the PCM will restart with the current navigation route still "live". This is window into how it wants to get back to where it was pre-reboot.

While sometimes the "two finger" reset is enough of a reboot to fix problems – the lock car and walk away - seems to cause the PCM to do a deeper reset which solves this problem.

The second thing I'd do at Porsche – is just acknowledge that the PCM sometimes gets itself into problems and allow users access the "full/deep reset" function within the "advanced settings" menu option. Stop pretending its not required.

My Macbook that has a battle-hardened macOS with a UNIX linage right back to the original AT&T Unix of the 1970s – has a full restart computer option.

What makes the PCM so special ? Nothing.

Interesting. Out of curiosity, where did you come across this information? I wonder if there is more where this came from.

Killing unused apps to save on resources is not unique to the docker container approach. Android has been killing background apps for many years (and yes, occasionally it would kill a foreground app too).

€50/unit increase in cost even for a €200k is HUGE for automotive guys. A couple of years ago I spent over a month attempting to convince high level VP's to spend $1.50 extra on a part to make software less complex for a six digit priced car. End result, no go. In the end their math did add up, at least at the surface, spend a couple of million on software development, save more millions over the life of the car production. On the other hand I have seen the opposite too - automotive company paying more for hardware to enable more complex software implementation. At the time I worked for a company which offered instrument cluster software. We pitched two versions, one RTOS based, running on a single core microcontroller and 4MB RAM IIRC, and identically looking/behaving IC running Linux, with 4,096MB, and requiring 2 or 4 core SoC. After evaluation, company made a decision to go with Linux, and more expensive hardware. I asked their VP of engineering why, since I know they are very cost sensitive. The answer surprised me, they had no developers who knew how to write code for RTOS, very few developers who could write C/C++, but plenty of developers who could write Java/Python/etc and splice together open source packages on Linux.

All that said however, it still doesn't explain by the PCM would not bootup all the way, even after engineering mode reboot, but would boot up ok after being parked for an hour. The backup camera worked, so the base OS came up, but somehow none of the docker containers booted up? Why that would change after having the car off for an hour vs. 20 minutes or engineering mode reboot? Either they are keeping some state somewhere in the system which caused it to choke on boot, but that state expires after a while, or the software is just bad quality and randomly fails boot, maybe if LTE connectivity shows as up but it's too weak causing packet loss with which Porsche software does not know how to deal with?

 

[whitex]

Individual app designers don't have enough knowledge of system requirements

I dare you, ask those individual app designers about the size of the memory footprint of their app. Most developers today have no clue how much RAM their app needs to run, not even an order of magnitude.

 

[snstevens]

Interesting. Out of curiosity, where did you come across this information? I wonder if there is more where this came from.

Killing unused apps to save on resources is not unique to the docker container approach. Android has been killing background apps for many years (and yes, occasionally it would kill a foreground app too).

€50/unit increase in cost even for a €200k is HUGE for automotive guys. A couple of years ago I spent over a month attempting to convince high level VP's to spend $1.50 extra on a part to make software less complex for a six digit priced car. End result, no go. In the end their math did add up, at least at the surface, spend a couple of million on software development, save more millions over the life of the car production. On the other hand I have seen the opposite too - automotive company paying more for hardware to enable more complex software implementation. At the time I worked for a company which offered instrument cluster software. We pitched two versions, one RTOS based, running on a single core microcontroller and 4MB RAM IIRC, and identically looking/behaving IC running Linux, with 4,096MB, and requiring 2 or 4 core SoC. After evaluation, company made a decision to go with Linux, and more expensive hardware. I asked their VP of engineering why, since I know they are very cost sensitive. The answer surprised me, they had no developers who knew how to write code for RTOS, very few developers who could write C/C++, but plenty of developers who could write Java/Python/etc and splice together open source packages on Linux.

All that said however, it still doesn't explain by the PCM would not bootup all the way, even after engineering mode reboot, but would boot up ok after being parked for an hour. The backup camera worked, so the base OS came up, but somehow none of the docker containers booted up? Why that would change after having the car off for an hour vs. 20 minutes or engineering mode reboot? Either they are keeping some state somewhere in the system which caused it to choke on boot, but that state expires after a while, or the software is just bad quality and randomly fails boot, maybe if LTE connectivity shows as up but it's too weak causing packet loss with which Porsche software does not know how to deal with?

I’ve noticed the short term persistence of the PCM state in the past.

If I get out of my car, lock it, and walk away, but then come back within a few minutes, the PCM starts up much faster, and Apple CarPlay connects more quickly too.

I have no clue whether this is intentional, or not.

 

[tigerbalm]

teresting. Out of curiosity, where did you come across this information?

Porsche Engineering Magazine (attached).

https://www.porscheengineering.com/peg/en/about/magazine/

 

[tigerbalm]

We're just swimming in third party apps for PCM 6.0 – aren't we?

 

[whitex]

I’ve noticed the short term persistence of the PCM state in the past.

If I get out of my car, lock it, and walk away, but then come back within a few minutes, the PCM starts up much faster, and Apple CarPlay connects more quickly too.

I have no clue whether this is intentional, or not.

Yes, I noticed that too. Not unusual that some systems don't actually shutdown at first, but go into some low power sleep mode, only shutdown completely at night or after some time. HOWEVER, rebooting the PCM via the engineering mode definitely boots it from scratch, not a quick startup either.

Sponsored