Incident Report X040702  Blogger FTP Corruption Incident Response Setup

orcmid> sostegno> X040702C> 0.20 2004-08-14 -16:12 -0700

  Approach

1. Announcement
2. Set Expectations
3. Practice Incident Response

The site blogs are currently locked down, and the latest notice is in effect [2004-07-29]

The goal is to revive the blogs one at a time, verifying that I can post successfully without further corruption.  Before I do that, I will establish an incident-response procedure that allows for rapid response, roll-back, and reporting in the event of a new incident.

This is being accomplished in small steps:

1. Announce this process in case anybody is watching.

2. Let people know what they will see as recovery take place.

3. Make the incident-response setup for Spanner Wingnut first.

4. Practice the incident response on Spanner Wingnut, tuning the process and the materials.

5. Stabilize Spanner Wingnut as the the model for incident response and reporting on all of the blogs.

6. For each production blog, clone the appropriate incident-response materials and then reactivate the blog:

   • Post an entry announcing the reactivation.

   • Make other adjustments for being in full operation, including template changes

   • Restore operation with attention to regular backups.

   • Post an entry when full operation is restored.

7. Continue operation with appropriate backup procedures so that new incidents are captured rapidly and the blog is restored to operation quickly.

1. Announcement

The following notice is added in the Spanner Wingnut site feed, followed by replication in the other site feeds:

BLOG RECOVERY STATUS: 2004-07-29-16:45 -0700
My three main blogs were locked down on Friday, July 2.  I am now preparing to restore operation.  Before I do that, I am putting in place a rapid-response structure for site lockdown, incident capture, and site roll-back.  I will have that ready as I restore each blog to operation.  Recovery procedures will first be practiced and verified at Spanner Wingnut's Muddleware Lab.  Then the non-experimental blogs will be restored one-by-one: Orcmid's Lair, Numbering Peano, and Professor von Clueless in the Blunder Dome.  The development of the incident-response procedure can be tracked here. -- Dennis E. Hamilton

This announcement is formed here with the following conditions:

• The   entities that FrontPage inserts for extra spacing must be replaced by the direct character entity for a non-breaking (mandatory) space:   .
• <img ... > must be replaced by <img ... /> to satisfy XHTML formatting.
• <br> must be done as <br /> and the same for all other unbalanced HTML constructs.
• Make sure that all borders show up in preview, adding the px-suffix to attributes that need it.
• All URLs, including in <img>-elements, must be absolute URLs.

The notice is added at the beginning of a site-feed entry is as the content material of an entry having the following customized form:

    <entry>
        <title mode="escaped" type="text/html">Incident Response Setup</title>
        <link href="http://orcmid.com/sostegno/X040702C.htm" 
              rel="alternate" title="X040702C: Incidence Response Setup" 
              type="text/html"/>
        <id>http://orcmid.com/sostegno/X040702C.htm</id>
        <modified>2004-07-30T18:20:00Z</modified>
        <issued>2004-07-29T16:45:00-07:00</issued>
        <content type="application/xhtml+xml" 
                 xml:base="http://orcmid.com/BlunderDome/wingnut/" 
                 xml:lang="en-US" >
            <div xmlns="http://www.w3.org/1999/xhtml">
            <!-- The content material goes in this space. -->
            </div>
        </content>
    </entry>

The variable information is shown in blue text.

2.    Set Expectations

This is the notice designed for announcing what can be expected to each of the currently locked-down site feeds:

BLOG RECOVERY STATUS: 2004-08-14-16:12 -0700

My three main blogs were locked down on Friday, July 2.  An incident analysis and recovery operation is in progress.  When the Incident Response Setup is completed and recovery accomplished, this is what you can expect:

   1. When the lockdown is ended and posting resumes, the default blog page (the one with current postings) and the Atom feed will be restored to earlier, correct versions.  The lockdown notice and recovery-status announcements such as this one will disappear.
   2. New blog entries will provide an account of the incident and the recovery.  There may also be a brief chattering of changes as template adjustments and other alterations are made.  Information about any previous or ongoing incidents and their recovery is found on the Web Log Status page.
   3. In the event of a future incident, there may be sudden replacement of blog pages and the atom feeds.  This incident slam-down step is designed to confine the incident and prevent further access to damaged material.  The blog will be recovered to an earlier state, and then recovered from that point.  There will be incident announcements as needed.   -- Dennis E. Hamilton

3.    Practice Incident Response

[to be continued]

0.20 2004-08-01-00:00 Prepare for Incident Response Practice

The announcement of what to expect is drafted and will be posted as soon as there is enough on future-incident slam-down to link to.

0.01 2004-07-29-16:45 Create Announcement

Create an overview and setup the announcement that for advising readers of the coming changes.

0.00 2004-07-24-19:11 Create boilerplate to fill in with next steps

Capture enough details to include in the next feed announcement..

You are navigating Orcmid's Lair

created 2004-07-24-19:11 -0700 (pdt) by orcmid $$Author: Orcmid $ $$Date: 05-02-11 16:48 $ $$Revision: 10 $