Loading...
HomeMy WebLinkAbout1999-11-02; City Council; 15479; Report on City Preparedness Efforts Year 2000, . Y . . p 2 6 5 8 CITY OF CARLSBAD -AGENDA BILL AB# 15.179 DEPT. HD. MTG. 1 O/l 9/99 TITLE: Report on the City’s Year 2000 DEPT. FIR Preparedness Efforts RECOMMENDED ACTION: Adopt Resolution number 99-483 approving the Year 2000 contingency plan and associated funding recommendations as submitted by the Carlsbad Emergency Management Administrative Team. ITEM EXPLANATION: This item is a report from the Carlsbad Emergency Management Administrative Team (CEMAT) on the progress of the City’s Year 2,000 (Y2K) mitigation and contingency planning efforts. It includes an explanation of the actions taken by the Information Systems Team to identify and correct vulnerable mission critical data and communications systems. The report also includes an overview of the contingency plan developed by Police, Fire and Public Works to ensure continuity of basic services should Y2K Technology or service problems actually materialize during the critical period of December 3 1,1999 through January 1,200O. FISCAL IMPACT: The proposed contingency plan requires funding support in the amount of $32,500. This includes personnel costs as well as anticipated equipment, materials and support expenses. Since these expenses were not contemplated during the development of the 1999/2000 budget, CEMAT recommends that this amount be appropriated by City Council to a Y2K Contingency account to be administered by CEMAT. EXHIBITS; 1. Resolution 99-483 2. Year 2000 Business Continuity and Contingency Plan RESOLUTION NO. 99-483 A RESOLUTION OF THE CITY COUNCIL OF THE CITY OF CARLSBAD, CALIFORNIA, APPROVING THE YEAR 2000 BUSINESS CONTINUITY AND CONTINGENCY PLAN, AND ASSOCIATED COSTS. WHEREAS, computer technology problems may occur world wide on January 1,2000, and WHEREAS, although the City of,Carlsbad has investigated its technological vulnerabilities and corrected suspect systems, the potential for failures of critical data, communication and operational systems remains, and Whereas, the potential exists for interruption of basic services such as water and electrical power which are provided by other agencies, and Whereas, such interruptions could result in serious health, safety and service concerns for the citizens of Carlsbad, and Whereas, the potential exists for an extraordinary demand for public safety services during the period December 31,1999 through January 1,2000, and Whereas, the Public Works, Information Systems, Police and Fire Departments have developed a comprehensive plan to preserve order and continuity of services should such failures and demands occur, and Whereas, the current budget of the City of Carlsbad has no provisions for fiscal support required by the plan, NOW, THEREFORE, BE RESOLVED by the City Council of the City of Carlsbad, California, as follows: 1. That the above recitations are true and correct. 2. The Year 2000 Business Continuity and Contingency Ptan submitted by the Carlsbad Emergency Management Administrative Team is approved. 3. Funds in the amount of $32,500 shall be appropriated from the City Council Contingency Fund in support of the Year 2000 contingency plan. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 - PASSED, APPROVED AND ADOPTED at a regular meeting of the City Council of the City of Carlsbad, California, held the 2nd day of November, 1999, by the following vote, to wit: AYES: Council Members Lewis, Hall, Finnila, Nygaard, Kulchin NOES: None ABSENT: None CLAUDE A LEWIS, Mayor ATTEST: KAREN R. KUNDTZ, Assistant City Clerk (SEAL) -2- - - October 4, 1999 TO: FROM: LEADERSHIP TEAM CEMAT YEAR 2000 BUSINESS CONTINUITY AND CONTINGENCY PLAN This report is submitted by the Carlsbad Emergency Management Administrative Team (CEMAT), on behalf of the City’s “Year 2000” (Y2K) planning committees. It describes the city’s remedial and mitigation efforts to counter the unusual technological problems and extraordinary service demands that could occur on January 1,200O. Summary Carlsbad’s Y2K mitigation program responsibilities have been divided to focus separately on internal and external concerns. Internal concerns include the viability of the city’s data and communication systems, namely computer hardware, telephones, emergency dispatch and 911 systems, and various automated control systems used by our Public Works Department. External concerns include services delivered by other agencies, which if interrupted, could result in widespread impact on the community. They include, water, electrical power, communications, public transportation, etc. The potential for Y2K failure within the City’s internal operating systems has been minimized through investigative and corrective action by the Y2K City Systems Team, under the guidance of the Information Systems Department. To date, the City’s mission critical systems have been identified and evaluated through testing or certification, as to their ability to perform after January 1, 2000. Suspect systems have either been repaired or replaced and confidence in their readiness is high. Planning for external technology failures such as interruptions in utility services, is the responsibility of the Y2K Contingency Planning Team and is facilitated by CEMAT. The Y2K Contingency Planning Team identified the most critical external services and evaluated the impact that their interruption could have on the community. Contingency plans were then developed for each. Strategies included preparing special department operating plans, developing liaison relationships with outside agencies such as utilities and public transportation providers, and conducting community education forums to assist private citizens. Although most of the remedial work is complete and the contingency planning is in its final stage, both teams will continue to refine plans, test systems and collaborate with vendors and other agencies on Y2K issues over the next three months. 4 The operational plan requires some over staffing of several departments over a critical 24 hour period beginning December 31, 1999. Personnel, equipment, material and support costs will total $33,500. CEMAT recommends this amount be appropriated in a temporary Y2K contingency account administered by CEMAT. YZK REMEDIATION AND CONTINGENCY PLANS Scope of the problem The challenges associated with the 2000 New Year settle into three categories: 1. Internal technology failures that impact the City’s ability to serve the community; i.e. financial data systems, and our communications networks. 2. External technology failures that interrupt basic services, i.e. utilities, consumer supply, and banking. 3. Extraordinary demands on safety services due to the social import associated with the perceived millennium change. Background of internal remediation effort In late 1992, the City of Carlsbad began a systematic upgrade of hardware and software as part of the normal business plan. At that time, many of the systems had been in place for ten years and were in need of replacement or upgrade. In the Fall of 1998, a committee was established comprised of employees from each City department. The committee has been heavily involved in conducting an inventory of all systems which may be affected, and identifying those that are critical to our operation. The systems survey was completed last June and included equipment containing embedded chips. In addition, the team has been communicating with vendors and suppliers of products and services in an effort to determine their levels of readiness, Internal systems upgrades to date include: Accounts Payable Accounts Receivable General Ledger Purchasing Payroll Human Resources Permits and Business License programs 2 Water Utility Billing and Work Order systems (Fleet & CMWD) will complete their upgrades later this Fall. While the failure of those systems identified as mission critical or high priority is highly unlikely, contingency plans have been and continue to be developed for each. The goal is to be prepared and ensure that key systems will work in the Year 2000. Should Y2K problems occur, the Information Systems Department is confident that the mutual cooperation and coordination to date with the Y2K external contingency planning team will serve to minimize any disruptions and expedite recovery. Background of external mitigation effort Early in 1999, with the Information Systems Department’s remediation plan well under way, the City Manager instructed CEMAT to develop a Y2K contingency plan to deal with potential external failures that could impact City services. CEMAT convened a Y2K Contingency Planning team comprised of representatives from all the City’s major service areas (MSA), to evaluate Carlsbad’s vulnerabilities, and to develop special operating plans to mitigate them. The contingency planning process has followed the Standardized Emergency Management methods outlined in Carlsbad’s Emergency Operating Plan adopted by Council in 1998. The team also employed the Federal Emergency Management Agency (FEMA) Y2K planning guide in evaluating potential problems. Possible scenarios such as power outages, breakdowns in telephone communications, and civil disturbances were prioritized with respect to the likelihood that they would actually materialize, and the impact they would have on the community. Special operating and staffing plans were then developed by each department to counter those problems. Contingency planning concepts Most of the public sentiment regarding Y2K focuses on the potential for interruption of basic public and private services such as power, water, public safety, and banking. The approach of the Y2K contingency plan is to first, develop measures that promote continuity of the services over which Carlsbad has control. For example, in anticipation of power interruption, the Y2K plan relies upon auxiliary power generators at key public facilities to maintain continuity of normal operations and city services. The second function of the plan is to consult with external providers to determine what if any problems they expect. While Carlsbad has little control over day to day operations of agencies such as the North County Transit District or the Metropolitan Water District, knowledge of their potential vulnerabilities allows our community to anticipate and plan for problems or unusual service demands. 3 6 Finally, the contingency plan addressed possible problems in the delivery of commercial goods and services by providing citizens with basic preparedness information that promotes self sufficiency without inciting panic. The information has been distributed by means of pamphlets, the news media, and public meetings. Special telephone numbers will also be published to allow the citizens to report infrastructure problems directly to the appropriate city department. Department specific Y2K issues Following is a description of specific concerns identified and evaluated by the Y2K internal and external planning teams. These descriptions are not all inclusive, but are representative of potential problems affecting certain City departments and the methods developed to counter them. Information systems Vulnerability 1. Hardware and software systems Mitiqation Systematic replacement of vulnerable components, vendor certification, testing, development of alternative methods in case failures occur. Public Works Facilities Vulnerabilities 1. Y2K compliance of support systems of city buildings 2. Power interruptions Mitigation All Heating and air conditioning systems have been or will be evaluated by the Y2K Systems Team. Facilities staff will inspect and test systems in all buildings. Stand by generators will be provided to supply key buildings should power interruptions occur. Water Vulnerabiiities 1. Potential for failure in telemetric controls serving system control valves 2. Possible interruption of water supply. Mitigation Data systems and controllers have been evaluated for Y2K conformance, Staff will be on duty to begin inspection and monitoring of system controls beginning at 12:Ol a.m. January 1, and will continue until confirmation that ail systems are operating properly. Sufficient staff will be available to operate the system manually if necessary. System reservoirs will be filled by December 31, providing a ten day water supply. Long term water conservation measures have been prepared should interruptions in supply occur. Sanitation Vulnerabiiities 1. Potential loss of sewer pump stations due to power interruption or computer technology failures. Mitigation Systems have been evaluated for Y2K conformance. Staff will be on duty to inspect lift stations for proper operation. Should failures occur, staff will use portable generators to power pumps, or portable pumping and transport equipment to maintain sewer holding facilities at proper levels. Streets Vulnerabilities 1. Loss of street lighting and traffic signals due to computer technology failure or power interruption. Mitigation Equipment and systems have been evaluated for Y2K conformance. Staff will be on duty and prepared to install traffic control signs at ail critical intersections. Public Safety General Vulnerabilitv 1. Potential for technology failures in dispatch system, radio communication, and 911 system 2. Extraordinary volume of injuries and other incidents related to millennium celebrations 3. Requests for assistance from other agencies due to numerous and/or large civil disturbances in metropolitan areas. Mitiqation All systems have been evaluated for Y2K conformance. Redundant systems in place should failures occur. The Police and Fire Departments will over staff to accommodate expected additional service and mutual aid requests. Police specific Vulnerability 1. Potential for mutual aid requests from other agencies due to civil disturbance. Mitiqation Additional staffing will be employed to permit participation in special mutual aid detail to assist the County Sheriff and other agencies as needed. Fire specific Vulnerabiiitv 1. Potential extraordinary requests for assistance from small nursing homes due to medical equipment failures 2. Possibility of hazardous materials incidents due to technology failures in industrial process controls 3. potential for fire or medical mutual aid requests for assistance from other agencies. Mitisation Additional staffing will be employed in anticipation of extraordinary incident volume and the potential for mutual aid requests. Staff will conduct cursory inspection of high risk industrial complexes and report conditions. Police Senior Volunteers will assist Fire with evaluation of nursing homes and other needy citizens. Concept of operations First Operational Period As a precaution, the City’s Emergency Operating Center will be open and minimally staffed (one to three persons) during the first operational period beginning at 6 a.m. December 31, 1999. The purpose of the early activation is to track international Y2K related events as the date changes globally. EOC Command Staff will initiate and maintain communication with the County Emergency Operating Center which will also activate at 6a.m. The Situation Unit of the EOC will prepare appropriate reports and will be prepared to brief managers during the day, as well as oncoming staff of the second operational period which begins at 6 p.m. Second Operational Period EOC staff will be expanded during the second operational period to fill positions in the Operations, Planning, and Logistics Sections. Per the City’s Emergency Plan, staff from Public Works, Law Enforcement, and Fire will coordinate responses to ail events and incidents from the EOC. The second operational period will end at 6 a.m. the following day. All affected service departments will employ additional staff as needed during the second operational period. Each of these departments will operate according to their specific plans from the following Department Operating Centers (DOC): City Hall Community Development 405 Oak Street Carlsbad Municipal Water District Offices 6 9 Safety Center Fire Stations Incidents and events will be reported to the City Emergency Operating Center (EOC), and EOC staff will maintain communications with ail DOC and with the County EOC throughout the operational period. Internal svstems The critical test for City systems commences at 12:Ol a.m. January 1, 2000, therefore inspection and testing activities for our internal systems will begin at that time. These operations will be conducted by Facilities and Information Systems staff with assistance from staff occupying DOC positions. Testing will continue until staff is satisfied that all systems are operating properly. Funding Y2K operational costs The cost of the Y2K operational plan totals $32,500. Personnel costs are projected to be $23,500 over both operational periods. An additional $9,000 will be needed to meet equipment and personnel support expenses. It should be noted that this total represents the amount needed to maintain the minimum staff necessary to inspect and evaluate the status of City operating systems, and to respond quickly and efficiently to emergency incidents. Additional personnel and operating costs may be incurred should problems escalate. Because the cost of the plan was not contemplated during the formulation of the 1999/2000 budget, its imposition upon existing MSA accounts is likely to result in some year end shortages, particularly in Public Works. For that reason, CEMAT recommends creation of a Y2K contingency fund account, administered by CEMAT, from which personnel and operating expenses could be met. This will provide the control necessary to ensure an accurate accounting of Y2K expenditures.