1	Incident Response Teams Marty Gillespie
2	Agenda What is Incident Response? Incident Response Goals Incident Response Methodology Emergency Action Plan Questions
3	What Is Incident Response? Incident response is a methodology for reacting to and resolving unexpected events Acts of nature Floods, earthquakes Accidental Fires, electrical outages, excessive heat, system crashes Intentional Web page defacement, Denial of Service attacks, unauthorized access
4	Incident Response Goals The goal of the incident handler is to limit damage, and maximize the capability to recover Restrict the scope of the incident Collect information in a manner suitable for use in returning system(s) to full use Proceed in a manner unlikely to cause further damage
5	Incident Handling Methodology Incident Response Methodology consists of 6 distinct phases Preparation Identification Containment Eradication Recovery Lessons learned Note: Not all steps within each of the 6 phases are pertinent to all types of incidents/events
6	Phase 1: Preparation The best method of dealing with any incident is to be well prepared before one occurs.
7	Preparation I Establish organizational policies for: Dealing with computer incidents Dealing with disasters Privacy Peer notification Home users Extranets Be proactive – test yourself
8	Preparation II Convince upper management of the importance of an incident response capability Collect related news items Make sure to keep track of all past incidents Establish a universal time source for the organization Dealing with issues of time variations between systems can cause real issues during an incident
9	Preparation III Form an incident response team Identify team members Technical Public Affairs Legal Management Other Choose organizational strategy (Centralized? Distributed?) Create checklists of incident response procedures and provide to team members
10	Preparation IV Determine methods of communications Call Lists IVR status system Offsite storage for communications information in case of disaster Have access to passwords Establish command center
11	Preparation V Create Resource Acquisition Plan Alternate Sites Hardware/Software Food Lodging Transportation Media Incidental
12	Preparation VI Create a means for incidents to be reported Educate users early and often Create an Incident website/web page Encourage reporting Telephone numbers and alternate means of communication of key personnel
13	Preparation VII Train the Incident Handling team Meet regularly to discuss Incidents in the news Current organizational events as related to Incident Response New systems New technologies in use Conduct periodic training sessions
14	Preparation VIII Create “Jump Kits” for team members Jump kits are bags containing everything a handler might need during an incident Operating Systems and Applications CDs “Known Good Binary” CDs Backup Software Tape Drive & Tapes Laptop(s) with appropriate tools installed Incident Handling Forms Tape Recorder Network Hub & cables Cell phones/radios
15	Phase 2: Identification The goal of the Identification phase is to confirm the criticality of the incident that has occurred and determine next steps Acts of nature Accidental Intentional
16	Identification I Don’t be afraid to alert early! The sooner an event addressed, the quicker it can be resolved. Remember, time is of the essence! Properly trained users can be a significant resource for reporting incidents Ensure the Incident Response Team is being fed timely and accurate information Check multiple sources for additional information
17	Identification II Use the help desk if available The help desk can be a great method of collecting information Ensure that ALL calls to the help desk are properly recorded and tracked Assign a person to act as the primary handler of an incident Must be aware of what is expected of them Must be empowered to escalate the incident to the appropriate level
18	Identification III Determine if the event is an incident Not all events are incidents, but all incidents start as events Look for the obvious first: Configuration problems Service outages ( bad cables, down Telco circuits, power, etc.) Hardware problems Continually reassess evidence as it presents itself
19	Phase 3: Containment The goal of the Containment phase is to limit the scope of an incident Keep the problem from getting worse Limit the number of systems the incident effects During this phase, modification of the system begins
20	Containment I Assign a team to handle the incident Limit access to those handling the incident Bring all team members up to speed on all aspects of the incident. This includes things that may be considered irrelevant
21	Containment II Do not contaminate the system! Backup the system as soon as possible Make multiple backups, in multiple formats Image backups should be made to preserve deleted files, data hidden in “bad” sectors, etc Filesystem backups should be made to provide easy access to the individual files on the system TEST ALL BACKUPS In the case of systems with hot swappable mirrored disks, the most effective method of backup is to remove half of the mirror set Do not trust the binaries installed on the system. Use the binaries in your Jump Kit CD in all cases
22	Containment III Store the backups in a safe place, under lock and key One copy of the backup should be kept for future reference All other copies should be stored in a safe place Consider placing one copy in a safe deposit box
23	Containment IV Consider removing the system from the subnet Consult the system owner Take into account scope of the incident Has the intruder taken over the system, or just a user account? Will removing the system from the network cause more harm than good? Consider level of expertise of attacker, and duration of time system has been compromised
24	Containment V Analyze the backups Attempt to determine how the system was compromised Search for signs the attacker has moved to other systems If possible, the attacker should not be made aware that you have noticed them Keep management and the Incident Response command center informed
25	Phase 4: Eradication The goal of the Eradication phase is to remove the means by which the incident occurred, as well as to shore up defenses to prevent further attacks Root cause resolution Protect against attacks utilizing reconnaissance information
26	Eradication I Remove the initial attack vector Perform vulnerability assessment Check machine that has been attacked, as well as related systems (same subnet, shared trust relationships, etc.) Obtain software patches to remove attack vector, if available Resolve configuration issues
27	Eradication II Locate most recent clean backup Check integrity of backup Ensure backup has not been contaminated Verify data is not corrupt
28	Phase 5: Recovery During the Recovery phase, systems and networks are restored to functioning order Restore the system(s) to a functioning state Validate the system(s) Restore operation Monitor the system
29	Recovery I Restore the system to a functioning state Restore files and/or system from backups Rebuild system from scratch Ensure backups are not compromised Utilize hot/warm/cold backups, as applicable
30	Recovery II Validate the system Compare system versus system “fingerprint” items such as: MD5 hashes of key files Baseline Performance Metrics Baseline Network activity Perform test plans developed when application was initially deployed End user verification
31	Recovery III Restore Operation Allow system/application owner to have final say in restoring production service Help owner weigh risk of resuming operation versus benefits Monitor the system Ensure system is not involved in further incidents Verify correct operation of system over time (from both a security and application standpoint)
32	Phase 6: Lessons Learned The purpose of the lessons learned phase to take the key factors in the incident and develop countermeasures to ensure the incident is not repeated. Follow up report Follow up meeting
33	Lessons Learned I Follow up report Develop a report to capture as much information pertinent to the incident as possible Start immediately! Circulate drafts to all involved parties Capture input from involved parties, attempt to present consensus opinion Provide Executive Summary Send report to management to review suggested changes Implement changes
34	Lessons Learned II Follow up meeting Should be near conclusion of the incident, but after a sufficient cooling off/rest period Resist the urge to lay blame Concentrate on means to improve
35	The Emergency Action Plan I Remain calm Stress levels climb during incidents Act as an example to others by remaining calm and level-headed Think of repercussions of your actions before you act Keep things friendly Collect information Take notes (Who, what, when, where and how!) Save logs to read-only media (printer, CD-ROM, etc.) Keep in mind this information will be used to evaluate effectiveness of your Incident Response Plan!
36	The Emergency Action Plan II Notify management and get a helper Let management know what is going on Use the helper to take notes on everything you do Do not use the helper for anything but documenting your actions and communications with management Enforce a need to know policy Limit the number of people who know about the incident to a minimum. This includes management. Stick to your Incident Response Plan!
37	The Emergency Action Plan III Use out-of-band communications Use a method of communications that is different from the one involved in the incident In-band communications may alert attacker that you are aware of them In-band communications may not be possible in all circumstances
38	The Emergency Action Plan IV Learn from what happened Set up a meeting with all involved parties as soon as possible following the conclusion of the incident Discuss and document ways the incident could have been handled better or differently Share information with others, if allowed
39	Questions