Obsessed with VW
So I’m kind of obsessed with the VW story. Specifically, I want to know what happened back in 2009 when they started cheating. What was that conversation like? And how many people were privy to the deception? And how did they think it was going to go undetected?
In case you haven’t read all available articles on this like I have, the VOX article is really informative. Here are the key facts:
- Diesel cars are better at gas mileage, worse at polluting out nitrogen oxide (NOx). We care more about NOx in the US than they do in Europe, which is why there’s so much more diesel in Europe.
- But recently we’ve started caring about gas mileage, so there’s been a spot for diesel cars that can pass the NOx emissions test, which most diesel cars cannot (at least for a given price).
- VW blew everyone away with a diesel car that seemed to have good gas mileage and good emissions test results.
- They were discovered by people hired by an independent group, the International Council on Clean Transportation, who hired people to stick a probe up the tailpipe of some VW cars and drive them from Seattle to San Diego, where it was discovered that the NOx levels were up to 35 times higher than was allowed. And that group wanted to know how VW did it so they could copy them.
- Basically they were discovered because their results were “too good to be true.”
So, back to my question. How did the decision get made, that they’d just cheat? Didn’t they know they’d eventually get discovered? It’s kind of like when teachers and principals change the results on students’ tests so they can get a bonus: short term thinking, and kind of obvious if you track erasure marks.
Or… another way of looking at this was that they really didn’t think they’d get caught. The evidence they’d need for that theory is that they cheated all the time like this and had never gotten caught, or they knew others did.
Another possibility: it was a small group of engineers who did this, looking for a large bonus. This kind of thing happens all the time in finance, where you cook the books in a short term way to get a pay-out. Could this be true? Certainly many of the raw ingredients were already available – surely the software already existed for the engineers to test performance and emissions under all types of conditions, so putting it together with a simple “if” statement wouldn’t be too hard. But that begs the question of how they’d explain it to their boss.
In any case, there’s an internal VW story – or perhaps industry-wide story – here that I’d love to hear.
mathbabe 
“About three quarters of the gap between laboratory test results and real-world driving is explained by vehicle manufacturers exploiting loopholes in the current regulation,” said Peter Mock, managing director of the ICCT in Europe.
Carmakers are known to “game” an outdated laboratory regime using perfectly legal methods such as overinflating tyres to reduce rolling resistance or fully recharging a vehicle’s battery before testing.
The rest of the gap is explained by technologies that have a greater effect in the lab than on the road, such as stop-start systems that turn off the engine in stationary traffic. Equipment that tends to increase fuel consumption, such as air conditioning, is also turned off during lab tests.”
http://www.ft.com/intl/cms/s/0/d812543c-6303-11e5-9846-de406ccb37f2.html#axzz3mkn5AFMH
LikeLike
As Dan Ariely teaches, we all tend to cheat a little. We are also inclined to test the boundaries or limits of our cheating. So a person steals $1. No sweat. $2, ,,, until we are talking millions. Gaming, as indicated in the FT article, is pervasive. So they just gamed it a little more.
LikeLike
FT article has a few quotes of people saying gaming is pervasive but no references and my search so far has not found any actual facts about it. I’m not saying it isn’t true but would like to know if anyone has evidence.
LikeLike
Actually, the FT article I was referring to is not the one you cited. Here is the one I mean (but still, not hard evidence).
http://www.ft.com/intl/cms/s/0/970e54f8-6048-11e5-9846-de406ccb37f2.html#axzz3mOokSHg1
Stuart Pearson, analyst at Exane BNP Paribas, said VW was unlikely to have been the only company to game the system globally. “The artificial gaming of emissions tests threatens to become the car industry’s Libor moment,” he said.
LikeLike
I agree that this is an important question. Wouldn’t it be nice if just once the people who cheat us could actually be punished for it with a jail sentence, so we’re not left with the usual feeling that it’s one rule for the rich and another for the poor.
LikeLike
The peanut butter CEO got a nice long jail sentence, but no one at General Motors was made to pay for actual documented deaths.
LikeLike
Amen.
LikeLike
My favorite thing in the VW scandal so far is this quote from a NYT article:
“Our company was dishonest with the EPA, and the California Air Resources Board and with all of you,” Michael Horn, the head of the VW brand in the U.S., said Monday in Brooklyn, New York, where he was revealing a redesigned version of the Passat. “We have totally screwed up. We must fix the cars to prevent this from ever happening again and we have to make this right. This kind of behavior is totally inconsistent with our qualities.”
Imagine how bad things must have been behind the scenes for a VW senior exec to have a public statement like this one cleared by their lawyers.
LikeLike
I wonder whether we might be seeing something similar to the stories from a couple years back where doctors found that admitting error actually reduced the risks of a major lawsuit or continuing through rather than settling because in many cases teaching someone a lesson was more important than the money involved. There’s zero chance that VW is going to escape lawsuits from this but immediately sacking the CEO, directly admitting error, etc. might avoid both getting a jury riled up with righteous indignation and take the wind out of the sails for any major legislative change. Particularly since they’re a foreign company, it’s really easy to imagine that taking a hard-line legal defense would have people in Congress lining up to propose punitive laws which would cost far more over the long-term.
LikeLike
as far as “Didn’t they know they’d eventually get discovered?” I’ll bet they DIDN’T know… this was largely some bit of software buried within millions(?) of lines of proprietary code… I bet once they went a couple yrs. without being found out they felt home free. But more scary, it makes you wonder what else is buried behind industrial proprietary code that you’d be shocked to learn of (hard to imagine that VW is a lone wolf)!
LikeLike
There’s a self-delusional kind of thing that I saw at a large company when I used to work there. Call it “drinking the Kool-Aid” or “believing your own press releases”.
Also, if upper management is pressuring subordinates on their results and is either unethical or disengaged it can also happen:
“If you bring me bad news or say something is not possible, I’ll get rid of you and bring in someone that tells me what I want to hear.”
LikeLike
I think John has it right. “Engineering get pressured to provide results. Knows that they are fudging the system and most likely will get caught eventually but by the time that they are discovered the guilty parties have moved on.” sort of scenario…
LikeLike
If they were professional engineers, say members of the IEEE, then they ignored the oaths they took and the standards they swore to maintain. They have soiled not just themselves but everyone who was also in that society and took those standards seriously, whatever the cost.
I have paid the price on occasion for adhering to those standards. That they may have been pressured is no excuse. Period.
LikeLike
I have a European engineering degree and no oath was ever involved in it.
LikeLike
I agree with gowers. Until people start going to jail, this kind of thing will not stop. The banking industry is another good example. And the higher the management indicted, the quicker the end of the malfeasance.
LikeLike
You once argued that the models need to be open in addition to the data. This should be the case with the software too, otherwise you dont know if the software combining the data with the models does something tricky.
LikeLike
What I want to know is, “How much does this really matter?” With only 11% of the market diesel and VW with 60% of that market we can determine that about 6-7% of cars are VW diesels. Some portion of those are the cars in question and we should be able to estimate how many miles were driven by these cars and extrapolate the effect of those miles.
Can we determine
a) Just now much additional NO was generated because of the miles driven in these cars?
b) How much additional fuel was consumed because actual milage is less than expected?
c) Are these effects large enough to cause concern?
d) How do these effects compare to the effects of other issues like using air conditioning or using manual vs automatic transmission?
LikeLike
I agree these are very good questions. I also know the EPA is calculating SOME of these numbers now (not all). They have models for doing so, so they can scale the fines accordingly. Thus for example with the heavy-duty truck emissions scandal of about 15 years ago they did calculate an NOx tonnage. I have seen some glancing references to these questions in the press as regards VW, but can’t find any hard facts yet. However, I have heard anecdotally that 500,000 cars at 30-40 times the NOx limit over several years (I know, not all 500,000 were on the road for all the years covered) comes out to a medium-sized unscrubbed coal-fired power plant. (Because remember, it ain’t 500,000 cars, it is more like 20 million “equivalent” cars…. the 6-7% is right in terms of units of cars but not in terms of units of pollution.) Cars are not a huge source of GHG (believe it or not) but if untreated they are a huge source of NOx. I have heard CARB is already calculating if enough of these cars were sold in LA to have materially shoved LA deeper into its NOx non-attainment status. I (personal opinion, freed from the burden of facts!) think the MPG impact is minor (in part because it is not clear that turning off the emissions controls actually REDUCED MPG… it may have actually IMPROVED MPG). It’s the NOx. And while the total NOx impact might be minimal in the grand scheme of things (though I think not, in places like LA), certainly the impact of the VW cheat vastly outstrips the impact of car AC or ATs. Again, the engines were 30-40 times over the limit. Turning on the AC on a car moves fuel economy by a few single percentage points at most, and ATs nowadays get BETTER fuel economy than manuals. So, we don’t know the answers yet, but EPA is working on some of them.
LikeLike
In European markets diesel is much more common. More than half the cars on the road in Spain are diesel.
LikeLike
Punishments for crimes based on accurate measurements of their societal consequences? You’re dreaming man!
LikeLike
No, we cannot determine those answers inexpensively or to the satisfaction of both plaintiff and defendant. Neither am i willing to argue in court with the defendant for twenty years until inflation reduces the maximum fine per incidence ($37,500) to one month’s profits. Punishment is intended as a deterrent, not as a reward or compensation for a victim. In this case VW did not pick up a wallet my Uncle Sam dropped and fail to return it. VW cracked my Uncle Sam in the back of the head when he wasn’t looking, beat him on the ground and stole his wallet. This nephew of Uncle Sam’s is quite a bit angry that individuals who knowingly took actions that harmed my cousins all across the Country are not already sitting in prison waiting for their trials as an example to execs from Monsanto, Citibank and JPM among others.
In answer to the question posed of “how did this conversation go?”, I can speculate, but this is why we need an NSA despite the risks to journalists. I believe it is the best tool we will ever have to convict corporate, especially financial industry, criminals. I think the overwhelming media response to eliminate the NSA programs without any significant discussion of restricting or managing those programs like we do FBI wiretapping is indicative of the whole barrel having become rotten, not just one apple.
I am sure former VW engineers/programmers and executives now work at Cummins or John Deere or Volvo.
LikeLike
My comment could have been expressed better. The type of “measurement” I was thinking of is the one my mind has performed when it makes me laugh out loud at people on TV saying things like “well we don’t know everything about what happened here…things can be tricky” in response to security camera footage of a plainclothes police officer brutally tackling somebody for no reason. It’s that easy kind of measurement I was mainly thinking of.
LikeLike
Internal conversations about this kind of thing are often not about the thing, but about what metric to meet. When the boss wants, say, “x” result on an emission test, and appeals to reason and the laws of physics are repeatedly ignored, employees get the message, and deliver the result. No one asks how the result is being acheived.
LikeLike
Do we really know how the VW testing defeat code works? If it was because it sensed that it recognized that it wasn’t on an all-wheel dynamometer vs. a front-wheel dynamometer, then maybe that WV realized that the EPA’s measurement system is gameable.
LikeLike
Coverage in Bloomberg and the NYT asserts that the defeat code kicked in when it detected the test cycle starting, which it did via the ABS. A car on a test dyno (unless in AWD mode) will have two wheels spinning and two wheels turning, which ABS detects as a skid. So in simplistic terms the car turned on the emissions controls whenever it detected this kind of skid. In the real world this skid would occur once in thousands of miles if at all, so a driver would never notice the shift in power or response. IMHO. And I may very well have mis-read your question (I do that a lot). The testing dynos CARB used I am pretty sure only had two wheels spinning at once (whether the dyno was an all-wheel or two-wheel).
LikeLike
The too-good-to-be-true results that someone attempts to reverse engineer is how Madoff was caught. SEC ignored the proof. Likewise the FT now reports that the EU knew about “defeat devices” back in 2013. But European companies protect so – called national champions to a fault.
There is also the mismanagement from VW’s side: like Citi, for example they decided they wanted to be Numero Uno. Growth and market share objectives overrode other considerations.
LikeLike
This question – how did they decide to just cheat? – was also bothering me. It seemed like a huge amount of effort to go through, with a very high risk of getting caught.
I’ve been poking around on the internet a bit and I think I’ve got a plausible explanation for how the cheating happened. It turns out not to be such a big effort for VW at all, because they already have an existing codebase to do something very similar, but for a legitimate and legal reason.
Some VW cars (for example the Golf, Audi and SEAT) use a form of “all-wheel drive” (AWD) technology, marketed by Haldex (https://en.wikipedia.org/wiki/Haldex_Traction) , which engages the rear-wheel drive only if the front wheels lose traction. This presumably makes the car less prone to skidding (so safer to drive), but more fuel-efficient than a car that uses four-wheel drive all the time.
However, for testing purposes, it’s a problem to have a vehicle that changes drive mode in response to changing road conditions. In a dynamometer test (assuming front-wheel drive), the front wheels of the car are put on rollers, so that they can spin freely, whilst the rear wheels are held against a block to keep the car from moving backwards. You can then run the engine and measure emissions output, fuel consumption, wheel speed etc for a known engine speed. Not only would these measurements be completely messed up if there is a variable amount of power directed to the front wheels, it’s also potentially pretty dangerous – if the rear wheels suddenly engage, the car might move forwards unexpectedly during the test. Hence VW needed to be able to disable the Haldex system and run their AWD cars in front-wheel drive mode (aka “dyno mode”) during testing.
There’s a forum discussion here (http://www.vwroc.com/forums/topic/7631-dyno-mode/) from January 2015 about how to put the VW Golf into this “dyno mode”. It seems that there is a specific sequence of inputs (switching on the hazard lights, pressing the accelerator etc) that will put the car into dyno mode and disengage the rear-wheel drive completely. And a video clip here (https://www.youtube.com/watch?v=ZVmuhVDLt3M) of a VW car in “dyno mode” on a dynamometer, with the rear wheels completely stationary. According to the same forum, “dyno mode” will be switched off once the car is restarted, or if steering is detected.
So now we have a completely legitimate piece of software in the ECM that detects a specific sequence of inputs, switches the output mode of the engine in response, then switches back when normal driving conditions are detected. This is not dissimilar to the description of how the “defeat device” operated (http://www3.epa.gov/otaq/cert/documents/vw-nov-caa-09-18-15.pdf), by switching engine mode in response to testing conditions. And it’s no longer a big step to imagine both changing the required input sequence to match the start of the (standardized) test, and modifiying the behaviour in “dyno mode” to include engaging the emission control systems.
The problem is that it now becomes much more complex to identify those responsible for the cheat. The code was initially written for a legitimate use case. Both the legal and the illegal behaviours have been called “dyno mode”, presumably internally as well as externally. Separating the two may be challenging: Unless there is a massive smoking gun, many people may be able to claim that they had no knowledge of the cheat, only the legitimate usage of the term, and thus leave some poor “rogue” engineer carrying the can.
LikeLike
This seems a key piece of the story. Also, since they already knew that the car’s performance during the test was not the same on the road, it would have provided some moral cover for making additional changes that increased the gap.
LikeLike
This is a MUCH better answer than my half-a**ed answer above. Proving that, as always, I should read all the comments before weighing in! Apologies…
LikeLike
One more factoid (not related to Haldex or dyno’s): an engineer I spoke with who has links to CARB said that whenever CARB asked questions about these TDI beasties (during the multiple meetings over the last year between VW and CARB), the local VW engineers would defer to engineers from Germany, who flew over specifically for these meetings, with instructions to do all the talking (rather than the local girls and boys). Rumor also has it that they always came from Wolfsburg, never Ingolstadt. If you see what I mean.
LikeLike
This question – how did they decide to just cheat? – was also bothering me. It seemed like a huge amount of effort to go through, with a very high risk of getting caught.
I’ve been poking around a bit and I think I’ve got a plausible explanation for how the cheating happened. It turns out not to be such a big effort for VW at all, because they already have an existing codebase to do something very similar, but for a legitimate and legal reason.
Some VW cars (for example the Golf, Audi and SEAT) use a form of “all-wheel drive” (AWD) technology, marketed by Haldex (https://en.wikipedia.org/wiki/Haldex_Traction) , which engages the rear-wheel drive only if the front wheels lose traction. This presumably makes the car less prone to skidding (so safer to drive), but more fuel-efficient than a car that uses four-wheel drive all the time.
However, for testing purposes, it’s a problem to have a vehicle that changes drive mode in response to changing road conditions. In a dynamometer test (assuming front-wheel drive), the front wheels of the car are put on rollers, so that they can spin freely, whilst the rear wheels are held against a block to keep the car from moving backwards. You can then run the engine and measure emissions output, fuel consumption, wheel speed etc for a known engine speed. Not only would these measurements be completely messed up if there is a variable amount of power directed to the front wheels, it’s also potentially pretty dangerous – if the rear wheels suddenly engage, the car might move forwards unexpectedly during the test. Hence VW needed to be able to disable the Haldex system and run their AWD cars in front-wheel drive mode (aka “dyno mode”) during testing.
There’s a forum discussion here (http://www.vwroc.com/forums/topic/7631-dyno-mode/) from January 2015 about how to put the VW Golf into this “dyno mode”. It seems that there is a specific sequence of inputs (switching on the hazard lights, pressing the accelerator etc) that will put the car into dyno mode and disengage the rear-wheel drive completely. And a video clip here (https://www.youtube.com/watch?v=ZVmuhVDLt3M) of a VW car in “dyno mode” on a dynamometer, with the rear wheels completely stationary. According to the same forum, “dyno mode” will be switched off once the car is restarted, or if steering is detected.
So now we have a completely legitimate piece of software in the ECM that detects a specific sequence of inputs, switches the output mode of the engine in response, then switches back when normal driving conditions are detected. This is not dissimilar to the description of how the “defeat device” operated (http://www3.epa.gov/otaq/cert/documents/vw-nov-caa-09-18-15.pdf), by switching engine mode in response to testing conditions. And it’s no longer a big step to imagine both changing the required input sequence to match the start of the (standardized) test, and modifiying the behaviour in “dyno mode” to include engaging the emission control systems.
The problem is that it now becomes much more complex to identify those responsible for the cheat. The code was initially written for a legitimate use case. Both the legal and the illegal behaviours have been called “dyno mode”, presumably internally as well as externally. Separating the two may be challenging: Unless there is a massive smoking gun, many people may be able to claim that they had no knowledge of the cheat, only the legitimate usage of the term, and thus leave some poor “rogue” engineer carrying the can.
LikeLike
There is no way that hundreds of diesel engineers at VW could do their jobs if they didn’t know quite a bit about the emissions properties of the engines and how those might/would be affected with design changes they consider. Emissions are primary design objectives for any diesel engine and the US / CA is one of their most important markets with the strictest standards. The idea that the “cheat” might be known only to a few rogue actors just doesn’t pass the snicker test. Top engineering management for the small diesels would have known from the beginning.
How did they decide to cheat?
Suppose it’s your job to build a small diesel for the US market. Dozens, perhaps hundreds of engineers and staff are hired and under your direction. If you fail, you and your group is out of work (or marginalized in some other way.) To succeed you need the car to seem “peppy” – not like the diesels of the 80s and 90s. It needs to get good mileage – better than gas engines since diesel fuel often costs more than gasoline. It needs to be affordable. And it must meet emissions standards.
Your team works very hard for a long time but cannot meet all main design criteria. If you can’t succeed in time for the 2009 model year, your project will be terminated, your team disbanded and perhaps fired. The world economy is in bad shape and autos aren’t selling well anywhere. Layoffs, even in Germany, might not be out of the question. Failure seems like a really bad option for everyone.
You know that emissions metrics are gamed by virtually everyone in the auto business. Your team invents the emissions “cheat” so the engine can be sold in the US. This allows the team to keep working on the problem. Maybe in a year or two you’ll be able to meet the standards with a marketable car. It’s a lot better than admitting failure and being responsible for your troops losing their jobs. And, it’s only for a short time – you’ll figure it out in short order.
I suspect that most people with leadership responsibility would cheat if the opportunity presented itself (and convince themselves that “it’s just a little cheat” – no worse than what competitors are doing.)
LikeLike
Very consistent with how I picture it: the folks at the top name several objectives, perhaps incompatible objectives, and the metrics for those objectives. No one much questions the apparent success that follows. People, top to bottom, also take cues from what leaders don’t talk about. If ethics are, at most, discussed annually with an accompanying pro forma and rather shallow Powerpoint, folks get what the priorities are.
LikeLike
a great incentive to turn them into Greasecars, IMNSHO.
http://www.caranddriver.com/features/the-vegetable-oil-alternative-feature
LikeLike
I think the fact that the emissions tests were different in the US (non-home) vs German (home) market is also important. This would have blunted some of the moral questioning that should naturally arise with cheating. If people don’t believe in the tests or believe that they are being tested in an arbitrary/unfair way, then tricks to cheat the test no longer feel wrong and can even feel positively right.
One other tidbit worth remembering: the emissions cheats seemed to happen around the time VW and Porsche were involved in various financial shenanigans: http://www.nytimes.com/2008/10/30/business/worldbusiness/30volkswagen.html?_r=0
Taking two data points from very different departments and parts of their corporate structure hints at a culture where sharp interpretation of the rules was encouraged.
LikeLike
It wouldn’t be too hard. Stern demands of engineers to meet spec, a complete lack of willingness by mgmt to listen to problems, and voila: VW problem. It happens in SO many areas.
LikeLike
We are talking about 700.000 direct Jobs and 3.500.000 indirect Jobs. We are talking about Europe future….not about a bunch of guys who wanted to get richer….Be careful since the consequences affect to the spirit of VW owners that are middle class decent people that really hate to cheat….Lets protect VW and isolate the gulty guys
LikeLike
I remember some years ago, when a campaign in the German press and some regional governments stopped all vegetable imports from Spain due to some contamination scare, that afterwards was demonstrated to have originated in The Netherlands. Huge losses for farmers and distributors.
So it wouldn’t be wrong to let Germany feel some pain for once. Maybe the German population wouldn’t be so happy punishing entire countries because that’s always easier than “isolating the guilty guys”…
LikeLike
so you are suggesting that it was a bunch of engineers that did the trick. For what reason? did they get the billions that VW made from this fraud? Big companies get the profits and when something goes wrong we say it is some isolated people who are responsible.
LikeLike
@Peter, this is the exact same dynamic that happens in Investment Banks. Look at the sound and light around the Libor scandal, or any other trading scandal. While profits are flowing, the banks are happy. When they get caught, they send a trader to the sacrificial altar, act shocked, and then carry on as usual.
LikeLike