Incident Report X040801 Bad Atom Provokes Gator

orcmid> sostegno> X040801> 0.15 2004-08-26 -21:38 -0700

Category: Data Corruption - User Exposure	Incident ID: X040801
Priority: 9 - Critical	Status: Resolved 2004-08-25 with workaround to be retested
Subject: Atom Site Feeds, NewsGator 2.0.3.3, Blog Incident Response Procedure	Repaired in: 2004-08-24 page updates
Assigned To: Dennis Hamilton	Reported By:  Dennis Hamilton (2004-08-24)
Date Opened: 2004-08-24	Date Closed: none
1. Summary 2. Remedies 3. Actions 4. Analysis 5. Lessons Learned	see also: X040802: Guessing Atom Rules

1. Summary (2004-08-25):

Incident	Spanner Wingnut Site Feed Error (X040801)
2004-08-24	2004-08-23	At 2004-08-23-22:36 pdt, there was an invalid site feed for Spanner Wingnut.  The incident was corrected in 10 minutes.  Any access to the feed in that interval may have resulted in feed errors or even removal of the feed from syndication.  A workaround against recurrences has been confirmed.  See Incident Report X040801: Bad Atom Provokes Gator.  NOTE: Because of web-page caching, the invalid feed may have continued to be served until as recently as 2004-08-24-16:00 pdt.

1. Aggregators for site Atom feeds can depend on identification information in the feed to confirm that the file being accessed is consistent with the feed that is subscribed to.

2. In the case of NewsGator, if a feed does not appear to be the one subscribed to or the feed is considered invalid for some other reason, the feed will be dropped and the subscription will be silently deleted. may be diverted into a new folder.  This is an example of behavior that can occur when a feed is unexpected or unacceptable in some way, even though the XML is well-formed and appears to be Atom-compliant.

3. Experimental introduction of test feeds can lead to the feed being recognized as improper with entries discarded and subscriptions deleted unless key identifying information is preserved.

2. Remedies (2004-08-24):  

1. Substitutions for site feeds must match the descriptive information for the original feed so that a feed reader/aggregator will not have any basis for concluding that the feed is damaged and/or should be unsubscribed.

2. Site feeds that are used for testing should not be the same feeds used for production entries of the blog.  For example, the feed linked to the Spanner Wingnut page should not be used for experimental purposes because this is also where valid feeds are stored by Blogger. 

3. The experiments should be done using a different feed, perhaps one that is not linked from any web log page.  That way, only private, experimental subscriptions are impacted.  The official feeds are not altered in a way that is likely to damage subscriptions placed by public readers of a web log.

4. Whenever a feed is being replaced as part of a fire drill or live test, there should be a warning message.  When the test is concluded, an all-clear message should also be transmitted.   followed-by an all-clear entry.  This provides public readers with warning that the feed might be disrupted.  Re-subscription may be required to restore the feed following a disruption, intended or not.

3. Actions (2004-08-26):

• 2004-08-23-22:46: The feed that was unacceptable to NewsGator was restored to its latest acceptable file.

• 2004-08-24-09:10: Initiate Incident Report for analysis and resolution.

• 2004-08-24-15:07: Workaround: Make a version of the "stand by" entry feed that looks completely like the Spanner Wingnut site feed that Blogger creates, except that it has just the one special entry for the announcement.  This is kept at wingnut/wing-atom.standby.xml.

• 2004-08-24-16:22 Add a warning entry to the current feed indicating that a potentially-disruptive feed experiment is being conducted.  This is identified with X040801B.

• 2004-08-24-16:53 Conducted a timed fire-drill swapping in the "standby" feed, confirming its retrieval, and swapping the "standby" out again.  Repair the "stand by" feed until this works.  [It worked the first time.]

• 2004-08-24-17:10 Added an all-clear entry to the restored current feed once the fire-drill is successful.

• 2004-08-26-21:22 Retest a revised procedure just to be sure it works the way we think.  (No it didn't and I learned some important things about Atom feeds: X040802)

• Review the experience up to this point and document updated procedures, timing results, and other information.

• Document related incidents including the moved feed (X040802) and the the problems with pages temporarily missing.

• Develop Testing Feed not Used by Blogger and Not Announced for Syndication
• Develop Minimal Announcement Feed.  Complete this after higher-priority tasks are completed, finding a way to reduce or eliminate dependence on Blogger in making a "stand by" announcement.

4. Analysis (2004-08-25):

1. On August 23, I created a custom Atom XML file as a form of "Please Stand By" announcement for site feeds. I wanted to make a simple file that could be used in place of any feed to protect subscribers from accessing a possibly-corrupted feed.  This file would be used as part of a site incident response.

2. My file was too simple.  After storing it atop the wing-atom.xml file of the Spanner Wingnut Blog, I ran NewsGator to see if it would fetch the generic announcement. The NewsGator status display showed there being one new entry being retrieved.  But when I looked in the folder that receives Wingnut feed items, the announcement wasn't there.  When I looked at my subscription list to see if there was anything to be discovered there, I saw that the subscription to Spanner Wingnut had been deleted. [dh:2004-08-25: I overlooked the appearance of a new feed with a different folder for its one message, the one that seemed to be lost.]

3. I quickly restored the original file in place of the custom version.  Then I resubscribed to the feed in NewsGator.  The resubscription was successful, with entries re-retrieved into the folder holding the previously-read copies.  Any further information will depend on the success of workarounds and additional experience resolving this incident..