Austin Water Oversight Committee Meeting Transcript (AWOC) – 3/31/2021 Title: ATXN-1 (24hr) Channel: 6 - ATXN-1 Recorded On: 3/31/2022 6:00:00 AM Original Air Date: 3/31/2022 Transcript Generated by SnapStream ================================== Please note that the following transcript is for reference purposes and does not constitute the official record of actions taken during the meeting. For the official record of actions of the meeting, please refer to the Approved Minutes. [2:05:04 PM] >> We'll call the meeting to order, the Austin water oversight committee. We are coming to order at 2:05 P.M. And there is a quorum present. We have councilmembers kitchen, alter, pool, as well as councilmember Ellis in attendance. And with that, we will get started. First item is public communication. Have we had anyone sign up? No one is signed up. Okay. The next item on our agenda is to approve the minutes from our February 23rd meeting. Is there a motion for approval? Motion by councilmember pool, seconded by alter. Hello. [ Laughing ] All in favor? Okay. That passes unanimously. All right. So, item number 2 is our discussion and possible action on recommendations to council for issues related to the [2:06:05 PM] February 4th boil water event. Colleagues, I wanted to recap the various interactions council has had with staff regarding the recent boil water notice since our last Austin water oversight committee meeting, which was held on February 23rd. Since our last committee meeting, Austin water has sent a memo in response to the committee recommendation that we adopted that was adopted by council regarding the customer impacts to our Austin water customers and consideration of a goodwill credit. Additionally, Austin water has sent an investigation report detailing what went wrong on March 29th, and the audit and finance committee met yesterday to detail the execution of the external audit. So today we will first discuss the findings from the investigation report. And with us we have director [2:07:05 PM] Meszaros to lead us. >> Thank you. Our primary presenter today is Stephanie, our water treatment operations manager. And I'd call Stephanie to the dais to walk through some of the material. >> Testing. Can you hear me? Good afternoon, councilmembers. My name is Stephanie, operations manager for water treatment here at Austin water. Oh, yeah, and I guess I do have some slides. There you go. All right. And you can move to the next slide, which is the agenda slide. My name Stephanie sioux, Austin water, water treatment operations manager. And today I will be providing a briefing of the process analysis [2:08:06 PM] and findings. And this is mostly comprised of the report that was -- the reports that were issued to you earlier this week on Tuesday. We'll be talking about the after-action review we've been conducting since the February event and the status of implementation of those action items. And then finally we'll be talking about the response to the council resolution which we submitted last Tuesday. Next slide. So, the process analysis and findings -- I'm going to try to cover the high points of those reports. There's a lot of detailed information that was included in it, but I'll try to cover the high points. The process analysis and the narrative provides a lot of the technical details, the timeline of the events that was happening. >> Ms. Soo, I wanted to make sure that ctm knew that we have councilmember tovo on the line who needs to be moved over. Councilmember tovo has joined us [2:09:07 PM] virtually. Thank you. >> Stand by, I'll have to take us out of presentation mode for one second. >> Tovo: Thanks very much. I appreciate it, chair. >> All right. So the process analysis and findings consist of a process narrative that we provided. It's a pretty detailed timeline and detail of the events and the chronology that led up to the boil water notice. There's also three investigation reports that were provided. That's a summary from our interviews, or Austin water's interviews with the staff members that were working that night and leading into the incident that led to the boil water notice. [2:10:08 PM] Next slide, please. Before I get too far, I did want to provide a quick overview and background of some of the terms and some of the things that I'm going to mention during this presentation. So, first thing is shift activities. We talk about the day and the night shift a lot. And I wanted to give a description of what our staff does on a day-to-day basis. The first part of it -- each team at our water treatment plants consists of both an operations and a maintenance side of staff. The operations staff are the ones that are monitoring and adjusting the treatment processes. They're watching parameters and making adjustments as needed. Their goal is to meet tceq regulatory and Austin water standards. The other side of our team is the maintenance side. They're performing corrective actions. They're being -- they respond to issues with our equipment. [2:11:09 PM] They troubleshoot it when breakdowns occur. And they are also providing preventive maintenance as well on our equipment -- doing things like changing oil, greasing bearings, making sure our equipment is ready when we need it. So, across the board both on the operations and maintenance sides, we do adjust our tasks based on -- and we re-prioritize our tasks based on what's happening. Things like weather. If we have to increase our flow. If demand goes up. We have to re-prioritize how we conduct our business. But that is a summary of the activities that we deal with on a day-to-day basis. The next item is operations shift roles. The first role that we'll talk about is the lead role. That means, that is the main point of contact for that team. That person is responsible for making decisions to make sure the plant is running efficiently. They can do things like look at [2:12:10 PM] scada trends, pass down notes and help the team decide to make decisions, with the sole responsibility for making sure things go smoothly over the course of their shift. The driver is really the owner of the plant. So that's the person that's typically sitting in the control room watching the screens, making adjustments as needed. But they're normally sitting in the control room trying to assess how things are going at the plant and making adjustments accordingly. The labs role is typically -- that's the person who is running the tests in our lab and each of our plants has a lab in-house and we have to run these tests on a prescribed schedule to meet regulatory requirements. And the purpose of this is to monitor the performance and what's going on at the plant, and to some extent back check the instruments online and [2:13:10 PM] continuously doing samples. We're also manually grabbing them and measuring it in the lab. Finally, there's the station checks role, the person that's out across the plant checking equipment, making sure things are going smoothly. If something doesn't sound right or seems to break, they'll get maintenance involved and help to get that equipment back up and running. They're also collecting samples. One thing to clarify is that each team has a lead. And typically that person is either the most senior or has the highest license on the team. That lead isn't always driving what's going on at the plant. They can take any of the three roles listed there, either the driver, the labs, or the outs person. But at the end of the day, no matter what role that lead is filling, that person is the main point of contact and is responsible for making the decisions for the day, especially if things aren't going right. In the case of the night shift on that Friday night that we'll [2:14:10 PM] talk more about, the lead on that shift was actually filling the role of the labs person. So, just wanted to make sure to point that out. The last thing I wanted to just provide some background on is the staff licensing requirements. So, tceq, the state has prescribed procedures for people that operate and make process control decisions on treating water. And so there is a very rigid and structure to how you progress in your licensure. And the first license that you would get is a class D license and then you kind of progress to C, B, and a. And there is, as I mentioned, very clear requirements as to the coursework, the number of years experience, the fact that you have to pass an exam, you have to submit an application for each one of those levels. [2:15:11 PM] And I will add that Austin water actually is one of the very few utilities in the state of Texas that actually has tceq license trainers on staff that teach these tceq classes to allow our staff to progress in their careers. And then the other thing I wanted to point out is that of the staff that we're going to talk about on that night shift, the lead and the lab person that was on duty that evening has a B license with an a license pending. The driver has a C license and the outs person has a D license. Next slide. So, I'll get into kind of the events leading up to the incident. I'll handle this as a timeline standpoint. The week of January 31st, which was that Monday, we got some pretty heavy rains that night with flood potential. No issues to report at our plants through that weather [2:16:12 PM] event. Tuesday morning, forecast for an arctic cold front were being broadcast. And our first thing that we started doing was to winterize the plants. We've talked about the sops and how we step that up. We instituted our hard freeze sops at all of our plants. That was completed and checked by the middle of that week. Throughout that week, there were normal winter flow rates coming through as you probably know. Wintertime we have lower flows. And we're at a lower flow rate than compared to summer, but nothing crazy was happening. In previous council meetings, it's been referenced that we were having some centrifuge issues that week. And the centrifuge equipment had ongoing issues. That's pretty common for us, especially at Ullrich, which is [2:17:15 PM] a pretty big water treatment plant with lots of moving parts and lots of equipment. But we were dealing with a a centrifuge. The staff was troubleshooting it for most of the week. That was something that had been on-going, and not due to the winter weather that we eventually got later in the week. The incident management team and the departmental operations center were activated on midnight that Wednesday night. And that was due to the freezing temperatures that were forecasted. And then I'll add that we were fully staffed that week. And so I'll clarify by saying that that Monday through Wednesday were normal operation days. We had our full staff of both operations and maintenance staff. We had our supervisors, superintendents, support were all at the plant that day, Monday through Wednesday of that [2:18:15 PM] week. Starting that Thursday, normal operations were suspended because of the icy road conditions that were projected for that Thursday morning. So we were fully staffed from the standpoint of the operations staff on that Thursday. And because normal operations had been suspended, what we typically do is we ground our maintenance staff and they are on call in case something comes up through the event. So our maintenance staff and our supervision was sort of on call through the non- normal operation days, which was that Thursday and Friday of that week. And then finally, as the it temperatures dropped that Wednesday night, we got into Thursday. No issues were reported. Normal operations as far as what we typically see on a normal winter day. No issues were reported. No equipment issues occurred. In fact, we were pumping -- producing water pretty consistently throughout that Thursday and filled up the [2:19:16 PM] distribution system such that that Thursday night we actually were able to come down in flow and took two basins offline going into the night. Next slide. So, Friday morning -- one more. Friday morning arrives. 7:00 A.M. The day shift arrives. And the first thing they're asked to do was to increase the amount of flow that we're treating through the plant. That's just a response to what was going on in the distribution system. They tried to bring up those two basins they had taken offline the night before and realized that the line feed systems had frozen. I'll point out that right now there is a current capital improvements project ongoing at Ullrich. And the contractor had not insulated portions of the lines yet. [2:20:16 PM] And the feed systems had frozen. We were not able to come up in flow immediately and had to make arrangements to give us time to let those lines thaw out. We had some equipment issues with our centrifuges. Between the centrifuges and the line feed systems being frozen, we made the call to bring out maintenance staff and supervision on-site that day to help make sure that those issues were taken care of. Later in that afternoon, we did bring up basin eight followinged followed by basin six once the systems were thawed out and we were able to give the distribution system what it needed. Before they left on that Friday, at about 5:00 P.M., that day shift started the seeding process on basin six. This is a standard operating procedure for when we bring a basin back online. [2:21:17 PM] They started seeding it two hours before they left. And it's noted in the investigation report that they noted it both verbally and in their pass-down notes. I did want to explain the pass-down procedure that we have. That is really the transition time when you go from one shift to the other. It's an important communication period, because you have guys that were there for 12 hours seeing how the plant was doing. And it's important to communicate to the next shift what the state of the plant is so over the course of their shift they know what they're taking on and what kind of adjustments may need to be made and history, what has most recently happened. It's important to note that is a process we have in place, a very important process. As the new shift comes onboard you want to know what you're getting yourself into. It was noted in the invest report that that seeding process had started, both verbally when it was communicated from the day [2:22:17 PM] shift to the night shift, but also written down and attached to a clipboard located in the control room where the guys were working. Next slide. So the Friday night shift begins at 7:00 P.M. After the pass- down. The guys start working. There was a message sent from our superintendent of Ullrich to the staff members asking them to keep their eye on monitoring the sledge disposal, making sure that sledge was able to be disposed of and hauled off-site. The second big thing was to keep the basins online. We knew that we were having freezing temperatures coming and going into the weekend and we wanted to be able to keep our production up to keep up with the demand -- anticipated demand in the distribution system. We wanted to make sure that we were able to continue to produce at the rate we had been on Friday and also to avoid the [2:23:19 PM] lines from freezing again. So that message was sent to the staff that was working over the weekend. One of the things the investigation showed was that the shift driver for the first four hours of his shift was focused on sludge disposal, from 7:00 P.M. Until 11:00 P.M. The driver, who I mentioned is supposed to be in the control room watching the screens, making adjustments, was distracted. He was at the centrifuges helping to get sludge disposed and into a truck and off the site. So that was one thing the investigation report showed. Another thing that was noted is that the lead operator, the most senior operator on that team, did not step in to fill in to drive for those first four hours while the driver was focused on other things, nor did he delegate any other roles and redirect the group. [2:24:19 PM] Those are things that are expected of that lead operator, being the most senior on that team. Meanwhile, we had the basin six turbidities starting to climb at 9:30 P.M. And that was attributed to the fact that the seeding process had started by the day shift and had not yet been stopped. The basin has solids being sent to it, on a continuous basis. And 4 1/2 hours later, turbidities start to climb. By 2:30 A.M., the turbidity exceeded 100 ntu. The sop states the turbidity should normally be two ntu. Sometimes it can spike up to [2:25:20 PM] 10-15. But the fact that we had the basin at 100 at 2:30 in the morning is not an indication of anything normal. And so that's one thing I wanted to point out. The team noted in the investigation report that they acknowledge that they were seeing the high turbidities at 10:00 P.M., shortly after they started climbing. They recognized there was something going on. But unfortunately, they misdiagnosed the gauze of that turbidity and spent six hours looking for a sulfate leak that didn't exist. And so in summary, they really misdiagnosed the cause of what was happening. So, as I mentioned, you've got basin six, hi turbidities because you've been sending solids to it. By 2:30, the turbidities were super high. The next step in the treatment process is the filters. [2:26:22 PM] Next slide, please. The filters are typically used to Polish off what was not are removed in the clarification process, but an hour and a half later after that 2:30 A.M., 100 ntu measurement, at 4:00 in the morning, the filters start to see -- to exhibit that turbidity. What's coming out of the filters now is high and does not meet regulatory requirements. What that means is that for an hour and a half hour filters were able to keep up with that high turbidity. But by 90 minutes, it was just overwhelmed and there was too much. And it was coming through in the filters. And so I think the process analysis says in a matter of minutes some of the filters actually went from .05 to .1 to .3 to 1 to 5. And the tceq requirements say [2:27:23 PM] your turbidity should never be greater than 1 coming out of your filters. Between the hours of 4:00 to 5:00 A.M., we had our series of filters that were just not meeting regulatory requirements. At that time, only two of those eleven filters were taken offline for washing. Our sops require that when our turbidities coming out are at 0.1 is when some action needs to be taken on the filters. It wasn't until 4:00 A.M. When we were hitting the 5 ntu mark, much higher than the regulatory requirements, that some action was taken. But only two of the eleven filters were taken offline. At around 4:00 in the morning, that team says that they looked at the scada screens, looked at the pass-down notes, and they realized that the seeding process has not stopped and they need to act to at least turn [2:28:24 PM] that off to prevent the solids from overwhelming the basin. At that time, for whatever reason, nine hours into their shift, they look at the scada screens, the pass-down notes, and realize what's going on. Despite the fact that we had such high turbidities in that basin six and that we were overwhelming the filters, there was also no action taken to take that basin offline. That means you can stop the flow from moving into the filters and isolate it so you're not sending bad water to the filters. But through that night, the decision was made to only turn off the seeding process and waste the solids from that basin. The final important thing to not note by the end of their shift at 7:00 A.M., is that no supervision was contacted or notified by that night shift, [2:29:25 PM] despite the fact that there were numerous calls from the chain of command. I mention that first four hours they were dealing with sludge disposal, talking to supervisors, to our superintendent at that time. We had the division manager call that evening to kind of wish them good luck and good support and that we had their back, and if they needed anything, to let us know. Nothing was mentioned. We also had our operations manager for process engineering. He happened to be on the incident management team at that time and needed to get progress reports to report out on how things were going at the plants. He called in at midnight and 5:00 A.M. And at no time, on any of those calls, including to him, was it ever mentioned that there were elevated turbidities or that we had a regulatory exceedens on the filters. Next slide. So, Saturday morning comes. They end their shift at 7:00 A.M. It's reported in the investigation report that the [2:30:26 PM] night shift lead, the most senior staff on that team, left the plant without fully participating in the pass- down to the day shift. And I believe it says that two of the three staff he was present for their briefing, but the third one wasn't there. And so the driver actually stayed behind to brief all three staff members, but that lead did not stick around for that pass-down to the day shift. Additionally, that pass-down from the night to day shift did not indicate that there were turbidity exceedences in the filters. It was noted there was a seeding -- the seeding pump was still on. It was noted they came up in flow when they were asked to and they had issues with sludge disposal over the night. But nowhere in the pass-down notes was it noted that they had a turbidity exceedence in the filters. It took 15 minutes for the day shift to look at the scada screens and realize there was [2:31:27 PM] something very wrong. So at 7:15, 15 minutes after they started their shift they notified supervision. That is the first time that anybody in supervision or the chain of command knew what was going on from that night. It took supervision, with that team, to look through the screens, make sure they knew what they were looking at and they could trust what they were looking at, 20 minutes later they made the call to shut down basin six. I pointed out earlier, we go from the basins to the filters. After the filters, the next step is the clear wells, where we collect all of our clean water before we pump it into the distribution system. So at 4:00 A.M. Is when the filters started sending turbid water to the clear wells. At 8:00 A.M. Is when the clear wells were saturated with that highly turbid water and we were starting to send that water into the distribution system. It took four hours for that to happen, 4:00 A.M. To 8:00 A.M. [2:32:29 PM] As soon as management was made aware of what was going on, it was not a hard decision to shut down the water treatment plant and that did occur at 9:30 A.M. I wanted to point out -- I've talked a lot about regulatory requirements and what the tceq requires at various stages in our process. Austin water has our own criteria that we communicate through our sops. And it's actually more stringent than what the state requires. And the reason for that is to allow our operators to be able to make adjustments so that we never -- or we don't exceed regulatory requirements, or that we don't violate regulatory requirements. So, what's hard to swallow about this is that there was no notification made. And there was essentially not enough time for us to make any course correction or to make any adjustments to be able to do anything else except for shutting down Ullrich that [2:33:30 PM] morning. Next slide. So this is kind of a summary slide with some of the high points that I want to make sure that you guys take away as you review the process analysis and investigation findings. The first item is that the night shift was informed that basin six was seeding through the pass-down process from the - - by the day shift. And I know it's noted, too, that the day shift turned on that pump but didn't put a timer on it. And so it ran continuously. But at the very least, they noted it in their pass-down notes. They communicated it verbally and in written format to the night shift so that they were aware that that had been started. As basin six exceeded what our sop standards dictate, the night shift misdiagnosed what was going on. They were searching for a sulfate leak that did not exist and they did not determine quickly enough what was going [2:34:31 PM] on, what was causing that elevated turbidity. The filter sop was not followed when the filters were overloaded. They only took two offline of the eleven that had been running. And several more had exceeded the regulatory requirements for filtration turbidity. Scada and the pass-down notes were not reviewed until 4:00 A.M. So, nine hours into their shift, they're looking at scada and looking at the pass- down, which clearly states that seeding is ongoing. And only at that time is when that is stopped. No action was taken to turn basin six off by the night shift. And so, as I mentioned, basin 6 is the source of that highly turbid water and by 4:00 A.M. It's overwhelming the filters, yet when they determine what's [2:35:32 PM] happened, nothing was done to shut off that basin. And the final important note to make is that, again, supervisors were not notified. Nobody was called that night during that shift by the night shift, even though there were several touchpoints of, you know, people calling in, checking in, checking to make sure things were okay. There was nothing mentioned about high turbidities, the sulfate leak that wasn't able to be found, the fact that we had regulatory exceedences. And none of that was ever communicated out throughout the night. I wanted to point out a few things here. One is that all of our staff at all of our plants have performance goals that we revisit on a six-month basis. And in those goals it is clearly stated what the ntu requirement is at the important steps in the [2:36:32 PM] treatment process. So there is no miscommunication of the fact that we are looking for a .1 ntu, Austin water standards. Additionally, it is clearly stated that supervision needs to be communicated with if there's a regulatory exceedence. I want to be clear that those expectations are clearly stated and are revisited on a six-month basis. The reports that were submitted this week document a lot of the missteps that happened that night. And I think it boils down to two major things that happened that evening. Number 1 is poor judgment. And that's evidenced by the fact that there was a misdiagnosis of the turbidity in the first place, the fact that sops were not being followed, the fact that they waited nine hours to look at scada to figure out what was going on, the fact that basin 6 was not turned off when [2:37:33 PM] it was sending highly turbid water to the filters. And the second big thing was the breakdown in communication. We had the day shift notate they were seeding that basin. And on top of that, supervision was never notified of the events of what happened that night. I want to point out that the poor judgment and that breakdown of communication is not a reflection of the staff that I have working at my water treatment plants. I, as the operations manager, have set that expectation to my staff that they need to follow their sops, they need to follow regulatory requirements, and above anything, they need to notify their supervision if there is an issue, and especially if there's a regulatory exceedence. It is sad and it's frustrating that the inaction by this team that night was the direct cause [2:38:34 PM] of the boil water notice and impacted the lives of the citizens of the city of Austin. And above that, it's unfair that the staff that is doing great quality work and are hard-working at my plants are now having to suffer this intense scrutiny that's being brought by this boil water notice. I talked earlier about the staff members that went from day to night, they put in the notes that they were seeding the basin. I talked about the Saturday morning crew that came in on that day and it took them 15 minutes to reach out to supervision and tell them what happened. We had a team that came in that Saturday night to bring the plant back online and they did so in a methodic and very smooth way. Those are the staff that are -- you know, the great workers and the great people that I have working on my team that are making great water, and that we can rely on. And it's hard to swallow the [2:39:36 PM] fact that the inaction by, you know, all of these things that happened that night, Friday night that caused the boil water notice, you know, is a reflection of the staff, not only at Ullrich, but all of our water treatment plants. So, I'll move on to our after-action review if that's okay. And I guess I want to point out a couple things. And that is that since the February event we have had a lot of discussions internally. We've talked to the executive staff, superintendents, supervisors, our o&m staff to really identify what we need to do from here. But I also want to point out -- I think it's been mentioned previously -- that the business of operating a water treatment plant is not something that we can fully automate. It's not something that robots can do. It's something that we have to rely heavily on our people and invest in our people to be [2:40:42 PM] successful. And this -- that's sort of the crux of what our after-action items are about. But I also want to point out that especially on the operations side, we are very challenged right now with our workforce in that we have numerous vacancies. And we have -- I think it was mentioned in council q&a we've got a market study that's ongoing right now that has shown that especially on the operations side that we are not competitive with other utilities, not to mention other businesses in Austin that pay more for less responsibility. So I want to make sure to get the point across that we are asking our staff to do some really important things, making clean water for the city of Austin. But they are working harder than ever now. We have a lot of vacancies. They're covering for people that don't exist on my team. They're working harder than ever and not getting paid as much as their peers are. I do want to point out that [2:41:43 PM] looking forward into how we invest in our workforce, it has to include having compassion for them and making sure that they are paid enough to do this very important job. So, next slide, please. >> Ms. Sue, I wanted to do a time check. Mayor pro tem alter will have to leave here in about 20 minutes and I want to make sure we have time to get to her questions. How many more minutes do you think the rest of the presentation will be? >> Maybe about five minutes, is that okay? >> Perfect. >> I'll go fast. [ Laughing ] As far as the after-action review goes, the items that we came up with that have specifically to do with what happened in February have to do with making sure that we provide the tools to our staff to make good decisions and also that there's clear communication happening. Some of the completed actions that we've done have to do with increased access to scada for our staff, our management staff, [2:42:43 PM] such that when we do have bad weather events or they are unable to get to the plant, they are able to monitor from afar and they can pick up the phone and say what's going on here, and check in on the staff. We talked about the manual seeding process and how the day shift had turned on the seeding process but there was no timer set to it. We do have a policy now and an infrastructure improvement in place to allow only timer-based seeding. But I want to point out the process of seeding a basin is not something that you can just automate. It is something that you have to monitor, you have to understand where that basin has been, what's going on, and only based on that is when you know when you can turn the seeding process off. But that is why both manual and a timer-based seeding exists. But today, the policy is in place that we are only seeding with the timer. We've got the enhanced shift pass-down communication procedures. [2:43:43 PM] A policy is in place that all staff members from both shifts have to be present for that. And we've also improved the form that is filled out by each shift so that direct information is provided -- helpful information is provided when you're transitioning from one to the other. And we've trained our staff on our plant parameters, things that tell them where you should be operating, what kind of action levels you should be taking, and clearly states what our regulatory requirements are. Next slide, please. Actions under way have to do with looking at our filter control logic improvement. So, this takes a little bit out of the hands of the operators in that we are looking to have an automatic shutdown once the turbidity is hit. The operators will have to know what to do when that happens, but it will prevent turbid water from getting out into the distribution system. There's the external [2:44:43 PM] notification for alarms. This is in progress at the moment where we're going to allow for text messages and/or emails to be sent to our management staff once we hit some thresholds. As I mentioned, operations training is going to be a very important part of this after-action. We do have, like I said, in- house classes that we offer for career progression related to the tceq licensureship, but otherwise, training does occur on the job. It's dependent on your team and how they teach you how to respond to different things. So we are -- we have a formalized training plan you under way to make sure there's consistent operations happening across all teams. And then finally, reviewing and update sops and guidelines. They exist, but this is a good time to review them, make sure that they're up to date and use them and incorporate them into our training plan. [2:45:45 PM] Next slide, please. The last part of this presentation is the response to council resolution, which was a memo that we submitted last week, basically responding to the request to provide alternative recommendations to reduce the risk. And so the memo -- next slide, please. The memo does kind of quantify the bill impacts, but I just wanted to point out the highlights of our recommended enhancements, or reinvestment in our staff. And that is first adding additional staff to the operations core staffing. And that will allow us to provide supervision from the day to the night shift, pass down, making sure there's oversight when that transition happens, and really bolstering our operations team. We're thinking about maybe a four-man operations team versus [2:46:46 PM] three-man, if we can get the staffing. We've also got enhanced support staffing, so additional engineering on-site and regulatory support, as well as trainers to help us institute some of these formalized training plans that I mentioned. Next slide. Finally, there's the modernization software for sops. So, this will allow us to map our standard processes and make sure that we've kind of accounting for everything at all levels of treatment and also is a way for us to use our sops for training materials and to audit how we conduct our training as well. And it will be very useful for enhancing how we transfer knowledge. With that, I think that's all I wanted to present. Happy to answer any questions. >> Fuentes: Thank you. Very helpful. Mayor pro tem, would you like to start us off? >> Alter: Thank you. And I appreciate that. And I'll try and come back down as soon as I can. [2:47:47 PM] Ms. Sue, thank you for the presentation. I think the details are really helpful on what happened. I did want to request if we can get a more detailed analysis of the after-action portion. So, I understood this was an investigation report, which is not the same thing as an after-action report on how everything is changing. You cited many things that need to happen, but I would like to see some timelines and know when those are happening, and be able to know when I would know when you have addressed them. So, the thing that I keep struggling with is this investigation report really squarely -- it affirms most of what was told to us in February about what happened, and ascribes blame to the operators in this case, and not following procedures. The piece that confounds me is [2:48:50 PM] with the supervisory side. And in particular, it's 2022. There are systems. Your own system allows if you enable it to have alarm sent tos sent to supervisors, but yet we didn't have, clearly autonomous notifications going out to supervisors. So, why didn't we have that set up, and where else don't we have that set up where we need that within our plant? >> I guess I'll say that within Austin water, it is not standard to have an external notification. There are lots of alarms that hit throughout the course of a shift. Some are more important than others. But at the moment -- obviously during the boil water -- during the incidents that led to the boil water notice there were not alarms. That is why we did -- we quickly instituted the ability to allow [2:49:51 PM] our supervisors and our staff to remote in and to be able to watch the scada system. But like I said, we have to rely on our people to make the right decisions and to be able to respond to those alarms. And it's not something that -- I don't think we do -- I know we don't do it at our water treatment plants, but I'm fairly sure we don't do that at any of our facilities. >> Alter: So it would be really helpful to have a sense of what is the standard procedure within the industry. I hear what you're saying that you don't do that at Austin water. In the course of reviewing material for this and looking at what happened, several people with a lot more expertise than I have in overseeing water plants have raised this issue about these autonomous alarms and the kinds of information that supervisors are getting. And raising questions about why we don't have them and [2:50:52 PM] suggesting that they are standard in the field. They may not be standard at Austin water, but that's what we're trying to remedy here. >> That was a quick after-action that we came up with, was that external notification. And we didn't have it at the time, but it is something we are actively working on right now to call out to the supervisors when a certain threshold has happened, and taking that out of the operator's responsibility, or out of their hands to make contact. >> Alter: From what I've been told -- I am not an expert on overseeing water infrastructure -- that there are things, these are standard things. I would think a turbidity hitting 100 ntus, you would want an alarm to be sent. Certainly we hit thresholds where you would want those. And so it would be very helpful. [2:51:52 PM] I see Mr. Goode in the audience for us to have some of that perspective relative to other utilities as to what they're doing with respect to that, because it's all fair and well to say our operators need to follow these procedures, etc., but we want to set up the redundancies and have the systems in place so that they can succeed. There will be human error, but there are things that are in our control. And I don't understand at this point whether or not we had state-of-the-art systems in place for something like that. >> Mayor pro tem, vice chair kitchen had a quick related question. >> Kitchen: Yes. And thank you. So it does sound like -- if I'm hearing you all correctly, and thank you, mayor pro tem -- this is not something that was done in the past, but you do have plans to change that and do those alarms in the future. Did I hear that correct? [2:52:53 PM] >> Yes, ma'am. We are -- >> Kitchen: So you are acting to correct. >> Correct. >> Kitchen: What was a problem in the past. >> Yes. >> Kitchen: And I guess I would also be interested in what the mayor pro tem asked about, which is just the timeline. So you can let us know more specifically when those changes will be made. >> Mhmm. >> Kitchen: And while I appreciate -- you know, I think this is a learning experience for us all, that this was something that was not done in the past that needs to be done. And I am happy to hear that it is going to be done right now and that you all have recognized that and you are moving forward to make that change. So then the only other aspect of the question is, are you making that change in all aspects, in all places that you can? So, some more specificity about that would be helpful, but thank you for recognizing the need to make that change and moving forward with it. >> Fuentes: Mayor pro tem. [2:53:53 PM] >> Alter: Did you want to answer the question? >> I was going to say that yes, definitely. Our focus right now -- I've been having conversations like this with my staff about -- turbidity was the issue now, but there are other things we have to be aware of. It is the full gamut of water quality that we need to be aware of. That is on our agenda as far as what we're talking about from the after-action. It's not just Ullrich. We have three water treatment plants and they're set up fairly the same way. They have different ages and nuances to them, but it is important that what we do here as a response to this at Ullrich needs to be instituted at all of our water treatment plants. >> Alter: Thank you. So, the reason I started by asking for the timeline was you have identified this as one of the after-actions, so I recognize that. But it still raises questions in my view about whether we have the state-of-the-art systems that we need to be having. [2:54:54 PM] And we have to support our operators with all the tools possible to help them succeed in a difficult environment. So, we have set in place a new polymer system that was not used in this instance. Why was that not used here? >> So the system is in place when our raw water quality exceeds a certain turbidity. And that is a different kind of turbidity than what we were seeing here. That is organics and sediment material that's coming in from the source water versus in this case it was turbidity caused by, you know, our solids process, the Lyme, the fact that our solids were being recycled back into the process. So it was a different kind of solids and polymer would not have been helpful. [2:55:54 PM] >> Alter: Thank you. So the emergency operations center was still operating at the time this happened, is that correct, from the freeze? >> Ana might know more, but it was on the demobilization phase by Saturday morning. But as soon as we realized what had happened that morning, we activated -- I think 12:00 P.M. On Saturday. Is that right? >> That's correct, Ana, Austin water. The incident management team was on duty overnight but was unaware that these things were taking place. We demobilized at 6:00 A.M. And then as these events unfolded, and the chain of command was informed, then the incident management team reactivated at 12:00 P.M. >> Alter: So we had information here about correspondence that came between the plant supervisor and the [2:56:57 PM] team. Was the incident management team, the emergency operations team, were they in touch with the team overnight? >> Yes, I believe that Brian Haas, one of our operations managers, was serving in an incident management team role overnight, and he was checking in with the plant in those overnight hours. >> Alter: So we've talked a lot about the standard operating procedures for the operators. Do those exist for the supervisors so you have a clear sense of the questions they asked, or how does that work? >> When Brian had called in to get a progress report? I think it's really -- I don't think an sop exists for that. Ana might correct me, but I would say it was just a progress report, like how are things going, are we producing what we need to, any issues at the plant kind. [2:57:58 PM] >> Yes, that's very much in line with the check-in conversations that we had as the incident management team. We had check-in conversations leading up to the evening hours and then again early in the morning. It was the role of Brian, who was serving as the operations section chief on the incident management team, to be in touch with operations personnel, and ensure that operations were -- were proceeding and that there were not specific concerns. The way that our incident management team is operated, we have to be adaptable to a wide variety of incidents. So we don't prescribe the questions that the operation section chief would be asking, because the questions that might be applicable in a freeze situation, which is why we were activated at that time, would be quite different from, let's say, a flood situation or a wildfire situation. But the operations section chief in their expertise is [2:58:59 PM] knowledgeable about what needs to be, I guess, monitored with operations. >> So the four hour it's this is my last question -- the four hours that were spent by one of the operators towards the beginning dealing with something else, did that action -- I think that it was that they were focused on sludge disposal. Did that action relate to the freeze? >> I'd say not directly. I mean, I think that it is acknowledged that the driver of the team was focused on that and -- and was out of the control room, not watching the scada screens and watching the trends and what was happening while he was focused on this sludge disposal. But, at the same time, as I mentioned the lead operator who happened to be working in the lab during that time acknowledged that he didn't step in to start driving and to start looking at the screens and to [3:00:00 PM] kind of redirect the team. >> I guess what I'm trying to understand is if the distraction that moved him away had to do with the freeze, the monitoring -- >> Oh, sorry. >> The sludge disposal. >> I'm sorry, I didn't understand. It had to do with the centrifuge issues that had been ongoing all week. We had a centrifuge down and it was basically just an ask to keep watching what was going on with sludge disposal. There is an indirect -- an indirect -- indirect -- what is the word -- indirect connection to the freeze? In that when normal operations had been suspended on that Thursday and Friday we did have to stop hauling our sludge. I think that it was just that Thursday that we actually stopped. So by that Friday the trucks were able to get on the road but we had kind of built up our sludge on-site so we were trying to make sure that kept moving off site. So there's a bit of an indirect connection. But as far as what caused boiled [3:01:00 PM] water notice, the fact, you know, that inaction -- to take the basin offline and to do anything with the turbidities, that was not related to the winter -- to the winter storm. >> And no one knew when you went off-site that when you switched that that someone needed to monitor the sludge and that you might have needed more staff to do that? >> I mean, there was no request for additional assistance by that night team. And like I said, there was a superintendent that was pretty well in touch with them during those hours and I think that in the investigation report he notes that nobody mentioned, I need some help, can I get some more assistance here. >> Thank you, thank you, chair, for letting me jump ahead. >> Fuentes: Thank you, colleagues, are there questions? Councilmember pool. >> Pool: I just want to thank the staff for a really comprehensive and a clearly articulated walk through for the plan. I can tell it is really hit [3:02:01 PM] everybody pretty hard. And we -- I certainly appreciate the diligence that y'all are bringing to the concerns that we've been raising, because I know that you share them. And we all want to ensure that the water utility is able to provide a really predictable, reliable, clean source of the necessary water for our community. So I just wanted to let you know my real sincere appreciation. >> Thank you. >> Pool: For your presentation today. Thank you. >> Thanks. >> Fuentes: Thank you. I do have a couple of questions to pick up the thread with mayor pro tem alter, I wanted to better to understand, you know, it is mentioned here that staff misdiagnosed the turbidities and what was happening and they spent considerable time looking for a leak, a sulphate leak which is often associated with older iron pipes. And so can you just explain why [3:03:02 PM] they would be so persistent in thinking that it was a leak for such a long time? >> Yeah. Well, I can at least state what I think that was noted in the investigation report. And that is -- so I'll start by saying that sulphate is used as a coagulant at our plants and added to the raw water as it's entering into the basins and what that does is that it helps to kind of destabilize the particles that are in the raw water, and then it also allows for the lime that we add to adjust the ph to allow that -- it allows it to coagulate, to come together and to kind of become heavier and settle out. So that is a chemical that we use on a normal basis. I think what the investigation report said -- and it is correct -- that during the freeze, during winter storm I are uri we did have a sulphate leak at the plant as a result of cold water. So I think their first thought, based on what they said, there must be a leak because it's [3:04:04 PM] freezing. And the thought to look at sulphate is not a wrong one in that the absence of a coagulant, like, they thought that there was a leak and it wasn't getting into the basin and, therefore, the turbidities were climbing for that reason because there's no coagulation happening. It wasn't a wrong thing to look at, it's just that, you know, after so many hours of looking for something and not finding it, that, you know, they didn't determine -- it took them a long time to realize that we were actually sending solids into the basin and that was the cause for the high turbidity. >> Chairman Fuentes, you mentioned that sulphate is associated with iron pipes, that's not correct. It is sometimes referred to as iron because it's an iron derivative chemical but it's not an iron pipe associated chemical. It is something that we do on purpose as Stephanie described to add in coagulation. >> Fuentes: Got Ya, thank you [3:05:06 PM] for that clarification. Vice-chair kitchen. >> Kitchen: Just a question that you may not have an answer to. I am curious, as part of the conversations with the operators involved, did you get an understanding of why they didn't report it to the supervisors earlier? >> I can only report what the investigation -- >> Kitchen: Sure. >> I know that I have not talked to them. But I think what it said is that they -- they saw that the turbidity coming out of the clear well was -- I think that it was .08 as they were leaving the plant. And I think that, you know, they directly said, well, that was nowhere near the 1, and, you know, to trigger for a boiled water notice. When, really, one is really bad. You know, our standard for Austin water is actually a .1, but I think that in their minds at .08 and I think that they said it was a .08 coming out of the plant, no big deal, but what they had not stateside that our [3:06:07 PM] filters had already topped out as far as turbidity goes. >> Kitchen: So it sounds like -- well, I don't want to -- I don't want to make assumptions because that's not appropriate because I wasn't there and I haven't talked to them. But I was trying to get an idea of what we're talking about is a lack of knowledge or just a lack of recognition of what the issue was as opposed to, you know, not understanding the importance of bringing in supervisory assistance. So -- it sounds to me like -- >> The investigation -- I will just convey what is in the investigation. The lead operator during the investigation had been asked why this shift that came on afterwards immediately communicated to the supervisor and he commented that a weak crews talked to their supervisors. >> Kitchen: All right. The reason that I ask that question is because of that -- I would hope that -- that, um, and [3:07:10 PM] it sounds like you have done this -- I would hope that drilling down into the reason behind the reporting that may be an indication of an attitude or a thinking or a culture perhaps to be addressed that -- that is in addition to and perhaps beyond just knowledge base. So I think that that was important -- that was important that was identified as part of this process. So, thank you. >> Fuentes: Councilmember Ellis. >> Ellis: Thank you, chair. You had mentioned the pass down notes from one team change to another team change. How readily available are previous passed down notes to the crews that might be taking over? Is it more that you get to read the notes and discuss verbally with the crew -- only the crew before you? Or is there an accessibility to those notes so people can look at what happened before they came on shift? >> Absolutely. Yeah, that's a great question. So we have a clipboard in the [3:08:11 PM] control room where, you know, you fill it out electronically and you print it out and you put it in there so that it covers the kind of people that want to talk about it, versus wants to see it on paper. And the third option is to go click on -- we have an excel, you know, notebook for tabs of each day of the month, so you can open that up electronically and click through the tabs to kind of look at what is -- what's happened, you know, days before. And then there's always that open communication. You can text or call the operator that was on duty, you know, during the day or night before to say, hey, what did you mean by this. You know, if there's not that -- if it's not the day or the night or the night to the day transition. >> Ellis: Thank you for that. Are there any procedural walk-thrus that happen and I am asking that I know in jobs that I previously held, that when you take over that shift you kind of do your own visual or physical inspection of what you are supposed to be monitoring, so if [3:09:12 PM] something is left off of the passed down notes that you know that you have checked everything. So letting one team know that is happening and then the next team shows up and isn't -- either isn't aware or is aware but didn't quite understand how long it had been seeding and there was not a step that happened to take it back offline -- how did those pre-shift inspections work? >> It's not a physical inspection because the plant is so big and there's so many things to do. I think that first thing that all of our shifts learn to do when they first get there is to look at the scada screens and that's the easiest and the most kind of comprehensive way to see what pump is on, what are your flow rates, what are your dosages set at. And there is a screen on the scada -- on one of the scada screens where you can tell if the seeding operation is happening and that is just indicated by, you know, the color of the pump that it's on essentially. [3:10:12 PM] So as I mentioned, you know, that day shift that came in on Saturday and that reached out at 7:15, they did what they were supposed to do. They looked at their scada screens and, you know, there's sort of a routine that you go through. What is our flow rate and you kind of step through that and very quickly they were able to see what was going on. So I'd say it's that level of walk-thru that they do when they start a shift. >> Ellis: I appreciate that and thank you for the comprehensive information. I know that there's a lot of detailed questions that we and the public are certainly hearing and wanting to make sure that we can provide that clarification for people, but, you know, it does seem for how many days of the year you are running an operation, that, you know, the blessing and the curse of providing clean, reliable drinking water is that when everything is going right by a large number of individuals, that nobody is paying attention because things are happening as expected. >> Right. >> Ellis: So that's why the focus right now is just on what specifically happened in this incident, because it does seem [3:11:13 PM] either doubling up on a supervisor for that shift if one was distracted, may have been able to go a long way in this particular incident. >> Thanks. >> Fuentes: I have a few more questions and we'll wrap up this portion of our agenda. You mentioned earlier that during the winterization process preparing for the winter advisory that was issued, there was insulation of pipes, and you identified that the contractor had not -- I guess adequately or sufficiently insulated key pipes. So can you talk a little bit more about what happened and any action items that were taken as a result? >> Yeah, and I might not have pointed that out. So, at Ullrich we are on the tail end of a project where we are replacing our line feed system, and it is a capital improvements project that we have a contractor that's working actively on that right now. [3:12:14 PM] And the contractor -- like I said, we found out the hard way that the contractor had not yet insulated a portion of the line that the line feed system needs, you know, to be insulated in order to work. And so since then, the work has been completed and they just had not yet gotten to it by the time of the freeze and none of us were really aware that that was, you know, still something that needed to be done. And so as I mentioned it was a capital improvements project. It was quickly identified to the contractor after we noticed what was going on and I think that within two to three weeks. I mean, they were planning to do it, it's just they hadn't yet gotten around to it. >> Have we reviewed the rest of our cep projects to determine if there's any others that would fall under winterization that need to be done? >> That's a good question. I think that at Ullrich, between that and we've got the sub-station project, I think that we have taken a look at the -- you know, to make sure [3:13:19 PM] that recently completed and ongoing projects have the right amount of winterization. We are actively doing that for projects that are about to go out to bid and that we're working on the design, because it's very at the forefront of our minds as far as, you know, is there a way to drain this if we need to, and does this have right amount of insulation? But, yes, this is something that we have on our minds. >> Okay. And the other question that I had is around the chain of command. Notifying the chain of command. You said that the supervisor did call in a couple of times during that night but I guess that the information was intentionally withheld from him? >> I'm not sure what the mode of thinking was. I know that we had a superintendent and I think two -- two to three supervisors that were calling in kind of early in the night. They were kind of providing some guidance on the sludge disposal side of things. And we had the division manager that called in that night and to [3:14:19 PM] kind of just say, hey, good job, guys and I hope that you guys get through the night okay but let us know if anything is going on. I do think that the operations manager that called in on half of that mt from the time frame standpoint, that would have been the time that things were kind of happening turbidity- wise. And, like I said, there was no report back of, you know, even the progress report requests, hey how's it going and what can I report back as far as production -- um, there was nothing mentioned about elevated turbidities. But I'm not sure, you know, as to the reason of that being withheld. >> So I think you mentioned as part of the recommendations that would be to keep a supervisor on site during -- during the shift. Is that right? >> Correct. That's -- part of our ask for the 12 full-time employees would be to allow for a supervisor to be present and at the site during the pass down communication, and that would be [3:15:20 PM] just to kind of provide that oversight to make sure that as the transition happens that it's clearly communicated, any direction that is needed, that they are there for that. >> Yes, and I know that we added tire chains to our vehicles, so I think that it would be more manageable to have someone stay as part of the shift, especially, if we are in a, you know, a high hazard situation. And the last note, you mentioned that the alert system has been changed for remote -- I guess check-ins. So a supervisor could check in remotely, but would they receive some type of notification or an alarm that would ping them to notify them that we have exceeded -- we have violated the levels? >> So that is something that is part of the work in progress and that is something that we have identified and we're actively working on. But that type of notification is not currently in place but it will eventually allow, you know, a text message and an email to [3:16:22 PM] be sent to management when that happens. The scada access allows us as management and as supervision when we are home we can see on a read only basis what is going on with scada. And we can see if things are blinking and that's another way to have us to be prompted to call in to Sao say, what is going on. >> Fuentes: Thank you, colleagues, any further questions on the investigation report? Okay, thank you. >> Thanks. >> Fuentes: The other item on -- or as part of this item on our agenda is one of the items that I wanted to draw your attention to is regarding the previous resolution that our Austin water oversight committee adopted and went to full council for consideration and adoption. We have received a memo from Austin water that was sent out on March 22nd and that was to look at the feasibility and the impacts of one mitigating [3:17:23 PM] adverse utility bill impacted by customers and, two, to provide a one-time allocation into Austin water's operating budget or its five-year capital improvement plan in lieu of direct utility bill relief. And so from this memo, Austin water recommended the second option to invest in staff being and software modernization and investments. So I'd like to really turn it over to Austin water to share with us about the funding impacts and answer any question that the committee members may have about this allocation. >> Thank you, chair. So, I think that you outlined it really well. We provided a response back to the resolution, and engaged some alternatives and Stephanie covered the specifics of staffing and sop software enhancements. In terms of those enhancements, we had -- we had included [3:18:25 PM] estimates of dollars in our -- in our memo. And the estimates were both for current year, this year, as well as for next year, because these costs would be ongoing, particularly for staff. That really the next -- this year, next year, the staffing enhancements and software would be less than the bill credit, it would be in the same ballpark, but slightly less. And it would be, you know, approximately a little over $3 million in total between now and the end of the next fiscal year to implement those recommendations. >> Fuentes:colleagues, any questions? Okay, is there a motion on the table? Kitchen. >> Kitchen: I move that we adopt the motion to accept the staff recommendations, and I believe that it says the Austin water [3:19:27 PM] oversight committee supports Austin water staff recommendations for Austin water to seek full council approvals to add ftes and modernize the software in lieu of utility credits, and recommends the council approve this proposal when Austin water brings this as an item for council approval. >> And I'd be happy to second. >> Fuentes: Thank you, councilmember pool seconds. Any discussion? All right, all in favor? Passes unanimously from the committee. >> I'll include that in my pass down notes to director good. >> Fuentes: Thank you. And speaking of -- oh, yes, councilmember -- mayor pro tem Alison alter was off the dais for that vote. And I want to just take a quick moment since we have the opportunity to introduce Robert good who will be the interim director of Austin water starting April 11th. Mr. Good was recently served as vice president, senior program [3:20:27 PM] manager for the civil engineering firm Lockwood, Andrews and Newman and previously directed the transportation and infrastructure planning departments at such large entities as central Texas regional mobility authority and the city of Austin and the city of Fort Worth and Travis county, so, welcome, Mr. Good. Thank you. >> Chair? I just wanted to say welcome back to our good friend, Robert good. Thanks. >> Kitchen: And I would like to add a welcome back also. I appreciate the opportunity to work with you again. >> Fuentes: Thank you. All right, we are moving on to item number 3 which is for future agenda items. Colleagues, last time we spoke we talked about scheduling a briefing on the city's ser process. And a briefing on the purple pipe program for our may 25th meeting which would be our next committee meeting. Mayor pro tem has also requested a briefing on wildfire [3:21:28 PM] mitigation strategies. Is there anything else that we would like to add to our list for consideration for our may meeting? Or for our August meeting. Okay. Councilmember pool. >> Pool: I didn't have anything to add but I did want to just take a moment of personal privilege. >> Fuentes: Sure. >> Pool: And just to give a fond farewell to director Lazares, I appreciate the work that you have done with the water urkts tilt with your long tenure with the city of Austin. I sometimes say that this gig here is one of the hardest in the city. Because we are so exposed to the community for good purpose, we strive for accountability and transparency and I think that you have -- have stood firm on that foundation. And your expertise and professional knowledge has really carried us a great long distance. And so I -- I bid you a fond farewell and thank you for the [3:22:30 PM] really tremendous services that you have given to this city for many, many years. >> Thank you. >> Fuentes: Wonderful. Thank you, and thank you, director, for year tenure and leadership here at the city of Austin. Colleagues, our next time will convene will be April 15th and we'll be doing a committee tour of the Ullrich plant. And councilmember Ellis, you are invited to attend as well if you'd like. >> Ellis: I'll see if I can make it and it's in my district so I'd love to have a behind-the-scenes tour. >> Fuentes: Wonderful. If there's no other business, the meeting of the water oversight committee is adjourned at 3:23 P.M. Thank you.