Harry Shearer interviewed Maria Garzino, the whistle-blower who first pointed out how flawed the M.W.I. pumps were. I listened to the show and took notes and just wanted to share a few random observations. You can listen to the interview for yourself here (Sunday Sept. 13th Show).
Ms. Garzino did give a few conflicting statements over the course of the interview, but not an unreasonable amount, given it was a 30 minute interview on a technical subject.
* An error on the ACOE pump support structure drawings caused the pumps to be installed with 2' less of NPSH(a) than the pump impellers were trimmed to. Ms. Garzino thought it was not an unusual error, given the complexities of the project. It took a while to correct, but it eventually was fixed.
* The "battle test" that the ACOE is so proud of for the M.W.I. wasn't much of test at all. SCADA data shows that the M.W.I. pumps were only run once the heavy lifting was done by the direct drive pumps. For background, all major industrial equipment is controlled by Programmable Logic Controllers (PLC's). These primitive, reliable computers communicate to each other through several different protocols, one of which is SCADA. It's still a little hard to get definitive evidence, but it seems to be backed up by the runtime reports at the back of the Parsons Report (Table 3-10, etc.).
* One of the serious design flaws that kept the M.W.I. pumps from being successfully tested was air getting into the hydraulic pumps. In the M.W.I., diesel engines turn a hydraulic pump. That hydraulic pump pressurizes fluid that runs through lines to a hydraulic motor. The motor turns the impeller blades that moves the water. The reservoir that fed the hydraulic pumps was mounted incorrectly, causing air to carry into the hydraulic pump. Air in a hydraulic pump bad, on a good/bad scale. It tore the guts out of the hydraulic pumps. The reservoirs were mounted higher, to avoid air undercarry. That solved that particular problem.
* Maria Garzino then described the various stages of whistle-blowing she went through. She brought it up with various folks within the Corps. After she exhausted internal appeals, there was an Internal Technical Investigation by the Corps. I believe that was done by the Vicksburg office. She said the report was nice and all, but it didn't set its own scope and didn't really address her concerns. The Parsons Report followed on with more or less the same issues. Finally, the Office of Special Council hired their own technical advisers, went through all the reports and basically agreed with her over the Corps. Here's a letter [PDF] that describes a little more about the OSC report.
* Another issue she went through is how her actions changed some of the Corps plans. The Corps was, at one point, planning on re-using the M.W.I. pumps in the permanent pump stations. Later, the Corps decided that the M.W.I. pumps were only temporary. I don't know if there's any way of knowing whether the Corps had the original intent of using the Corps for the permanent stations, but I can say that they got lots of pushback from all sorts of sources when they discussed re-using the M.W.I. pumps. Mrs. Garzino's actions may or may not have contributed in the change of plans.
* The Corps has a developed a "How To" manual for reworking the Interim Closure Structure (ICS) into a permanent facility. The price tag on ripping out the M.W.I. pumps and replacing them with direct drives is approximately $275 million dollars. According to Ms. Garzino, the original estimate that the Corps submitted to Congress included a 50-year design life for the ICS. Spending $275 million after you told Congress that what you built had a 50-year design life is a waste of money. It's not completely an apples-to-apples comparison, but I think it's still useful to mention that large offshore oil structures have a design life of 35 years.
I listened to the radio interview to make some sort of judgment on Ms. Garzino's credibility. I listened like a hawk and took notes and while there were some contradictory statements, her interview taken as a whole leaves me to conclude her story is quite credible. All of her major assertions check out.
UPDATE- More things are coming to me in bits and pieces:
* The Corps really moved the chains for M.W.I. to pass the tests. Some of the moves were no big deal, but others were not legitimate. When competitively bidding contracts, you have to hold the winning bidders to what they said they'd do, especially with regards to testing. It violates ethical engineering practices and its unfair to the losing bidder. I understand the Corps' complaint (that they were under the gun for time), but the Corps is not the first entity in the world to face a looming deadline for a major, complicated project. If the Corps asked for testing the pumps, M.W.I. should have had a line item of a few thousand per pump for testing. If they didn't and they won the bid because of that, they should be forced to eat the difference. Remember, there's hardly ever only one vendor that can provide a piece of equipment or service.
* Verifying the pumps against their curve (making sure they work as the salesmen claim) is an important step. Hydraulic Institute standards state no worse than +/- 5% of the curve. Hydraulic Institute standards are MINIMAL; I can tell you that most clients I've dealt with have much more stringent standards on the pumps they test. If it's just a $50,000 pump or so, you're probably not going to bother testing it. Millions of dollars worth of pumps? You're going to get them tested. For critical systems, like de-ballast, firewater, etc. you're going to test the pumps to the Nth degree with a good chance of additional factory inspections.
* M.W.I.'s salesmen may have made excessive claims, but its the duty of the engineers not to just take them at their word and verify what they say. I worked on a job about a year ago where an engineering company bought some pipe for a pump discharge that wasn't pressure rated. I talked with the engineer and he said the salesmen said it was "an ideal application." I talked with the salesmen and he he told me it meets certain AASHTO, etc. specifications. I looked up those specs and the first thing they say is that these standards are meant for non-pressure rated pipe. The ratings the specs give were minimal 20 foot of head (~10 psig) ratings that should never have led an engineer to use them as force-mains. It's the engineer's job to run these things down.
R'fuah shleimah to you, man. Take care of that NDF and take it easy.
ReplyDeleteClay,
ReplyDeleteI don't know a lot about pumps so I don't have an opinion on those. What I do know is that the Interim Closure Structures (ICS) always were intended to be temporary. From day one the goal was to build something quickly and then to take the time required to build state-of-the-art permanent structures and pumps. Here's evidence to support this assertion:
Location: Each of the three outfall canal ICS are located away from the lake. That's because engineers wanted to be sure that the ICS were not in the way of the Permanent Pump Stations (PPS). The ICS were not built at the best locations knowing they would only be temporary.
Loss of pump capacity: Construction of the PPS will have to be staged so it does not reduce the ability of the ICS to block storm surge and pump out the canals. It would be very difficult to remove pumps from the ICS and install them in the PPS without causing impacts to pumping capacity.
ICS layout: A key design feature of the PPS will be that all key equipment is elevated to protect from major flood events--not just the 100-year but something larger. That includes all equipment, power generation, communication and controls. The ICS do not currently fulfill this requirement.
In short, I think somebody got an idea one day that the Corps could save a few hundred million in taxpayer money by reworking the ICS to become the PPS, but that is not feasible. From the beginning the official plan has been to build interim protection and then follow with even better permanant pumps and gates.
Peace,
Tim
If the ICS's were always intended to be temporary, why use a 50 year lifespan in the January 2006 Project Information Report (and subsequent follow-on revisions throughout 2006) to justify the project at a 7.3:1 benefit to cost ratio?
ReplyDeleteI think it is possible there was some evolution of thinking going on as problems emerged.
By the way, the "someone" that got the idea to make the ICS's work for 50 years was the Corps, ina report started sometime in 2007 and isued in April of this year:
ReplyDeletehttp://www.nolaenvironmental.gov/nola_public_data/projects/usace_levee/docs/original/IER5ICSPermEnhancementRptAppA-E27Apr09.pdf
That report calls for the replacement of the hydraulic pumps with direct drive pumps in order to extend the life of the system. It also recognizes serious corrosion protection problems, problems which are apparently manifesting themselves right now:
http://www.wwltv.com/topstories/stories/wwl080509cb17canal.b06b692b.html
Note there may be more pumps getting pulled than the ones listed in that report. Publicly available contract information indicates that some of the 42" pumps mounted on the 17th Street cosure structure are also "under repair."
Tim, do you have documentary evidence to back up your assertions about site selection, structure heights, etc?
In short, I think somebody got an idea one day that the Corps could save a few hundred million in taxpayer money by reworking the ICS to become the PPS, but that is not feasible.
ReplyDeleteI more or less agree with you, however it was foolish to think you could save anywhere near that much money. I worked on a project for an overseas client where they wanted to reuse a bunch of old equipment. Turns out, between checking out the sizes and condition of the old parts and doing all the extra work to integrate the parts, we were probably 20% over the cost to do it from scratch. Then, of course, none of the reused items will have quite the same lifespan. You've got to do incredibly intricate inspections if you want to have any confidence with reused industrial equipment.
Another thing I'll give the Corps: this looks like a situation where intense political pressure affected engineering decisions. Had their not been such a big push to get SOMETHING in place by June 1st 2006, I have my doubts we would have ever heard of M.W.I. It would have been unreasonable to expect an engineer to buck that sort of pressure, even if he thought it would result in a better, cheaper system. And, if he was unlucky and something big hit in 2006, he would have been ruined.
ReplyDeleteYes, pity the Corps for setting their own deadline and not meeting it.
ReplyDeleteThe Corps is a political organization that answers to (at least) 435 separate constituencies. It has been feeling "political pressure" for over 100 years. That doesn't exempt it from doing stuff right or doing what it will says it will.
Clay said: "if he was unlucky and something big hit in 2006, he would have been ruined."
ReplyDeleteThis is the perspective I find most disturbing. The engineer's failure and embarrassment is not what needs protection. His job and career are not as important as public safety or even as important as honesty and the truth. Geez, it's not like any engineers even lost their suckbutt jobs as a result of their negligent homicide of many hundreds of people and the taking of many tens of billions of dollars worth of property from US Citizens.
I bet Garzino is about the only one who had to change jobs because of the bad work of co-workers.
"doesn't exempt it from doing stuff right or doing what it will says it will."
Agreed. I have not yet heard an excuse from the Corps that wasn't a self serving blame deflecting red herring.
I really never liked ignorant over confident engineers, especially if they make efforts to cover-up their mistakes. They should man up, on the spot, admit their faults - that is the only way for their community to regain respect.
crescentCityRay
"never liked ignorant over confident engineers"
ReplyDeleteI didn't mean to imply I don't like engineers. After all, I arz one.
I just don't like the selfish stupid unethical ones who care more about their careers than their customers (those that depend on their products - i.e. the people). Does the Corps train employees to lie? Are there seminars or something? Are they given orders to lie by the brown shirts? Do their mothers know what they do? Talk about a bottom of the barrel group of GS-12 wannabes - NOLA district, apparently the end of the line, smart/honest/ethical people need not apply. what a disappointment.
CrescentCityRay
Since Tim didn't step up with any documentary evidence to support his contentions, I'm going to give some that refute them.
ReplyDeleteThis one: "ICS layout: A key design feature of the PPS will be that all key equipment is elevated to protect from major flood events--not just the 100-year but something larger. That includes all equipment, power generation, communication and controls. The ICS do not currently fulfill this requirement."
appears wrong.
The floor level for the engine houses for the hydraulic pumps at all three ICS sites is 12.5'. I pulled this from the report on extending the life of the structures here:
http://www.nolaenvironmental.gov/nola_public_data/projects/usace_levee/docs/original/IER5ICSPermEnhancementRptAppA-E27Apr09.pdf
(see Adobe pages 72, 83, and 92)
In a presentation to the East Bank Levee Authority (the SLFPA-E) last week, which was made public today here:
http://www.mvn.usace.army.mil/news/view.asp?ID=238
Page 13 shows the operating floor elevations for both Option 1 and Option 2.
It is exactly the same as the current engine house floor elevations: 12.5 feet.
In addition, Tim's undocumented contention about the loss of capacity kind of makes sense on its face:
"Loss of pump capacity: Construction of the PPS will have to be staged so it does not reduce the ability of the ICS to block storm surge and pump out the canals. It would be very difficult to remove pumps from the ICS and install them in the PPS without causing impacts to pumping capacity."
Until you consider that the pumps should really only be needed during hurricane season (London Avenue revised policy notwithstanding). That is, the Corps would have half a year to transfer pumps from the old facilities to the new ones.
Tim offers no documentaryy backup for his contention on site selection of the ICS's either.
Matt
My comment about the 2006 flooding was directed at the subjective parts of engineering that really depends on judgment. How important was it to have the pumps in place 4 months (approximate guess) earlier than a better & cheaper option (direct drives)? If nothing happened, then it doesn't seem very important. If something had happened, though, there would have been hell to pay, even if it looked like the best decision at the time.
ReplyDeleteIt wasn't important at all, because the pumps they spent these millions of dollars on hardly had any more capacity than the rental they lined up along the bridges at 17th St and London.
ReplyDeleteMaybe if they were going to be an order of magnitude greater in capacity it would have made a difference. But they were futzing over (nominally) 2400 cfs on the 60" hydraulics at 17th Street while they already had 1000 cfs installed along the Hammond Hwy bridge with a sheet pile closure. 1000 cfs or 2400 cfs - neither was close to the requirement of 10,000 cfs, so what was the hurry?
Guys,
ReplyDeleteI've been around hydraulic driven pumps before during failures and it wasn't pretty. Aside from equipment damage, and production downtime, the most exciting thing about it was the complete coating with hydraulic oil of everything anywhere near the pump. Fortunately, these pumps were in closed spaces. It was usually hose and fitting failures at or near full power. Has this been part of the problem? Have Corps' hydraulic pumps been spilling their guts in Lake Pontchartrain if the Corps tries to use them? I bet if they have been doing that, the Corps has been illegally trying to keep it a secret from the Coast Guard and public. Shame, shame. Most outfits get big fines for that kind of behavior.
Please tell me this is not the case. And, really, in what world do mech engineers specify unproven hydraulic systems over waterways?
So Beedy eyes told us at St. Paul's as recently as a year and a half ago that those pumps are good for fifty years and this past summer his bigger brass boss Welshes and says they ain't good for even seven years and that has always been just seven. And then Tim grasps at straws trying to justify his co-workers ethics problems. The lying seems to run from stem to stern from the keel all the way up to the wheel house. The Leake ship is loaded with nothing but bullship. What's up with that salami?
crescentCityRay
The Corps set the deadline? I thought it was Congress and the White House? Setting a deadline before calling around for long-lead items is not good engineering practice.
ReplyDelete