crl3 [2] finger ticket-2678@enoc.barrnet.net Retrieving BARRNET Trouble Ticket 2678 Ticket Number: 2678 Ticket Status: closed Ticket Type: unplanned Ticket Source: other Ticket Scope: 1site Site/Line: barrnet Ticket Owner: cjw Problem Fixer: everyone Telco Number: cscdi35420 Ticket Opened: 06/05/95 17:08 Problem Started: 05/17/95 00:00 Closed: 06/13/95 14:53 Ended: 06/13/95 14:55 Close Code: other Problem Description: It has been reported that BBN Planet Western Region is out of Smarties. Staff moral is at an all time low. Most local sources seem to be out of stock at this time. The situation could get critical if a new supply is not found. Note: 1 BARRNET Systems Group indicates that this issue needs to be 06/05/95 17:36 escalated to BARRNET Field service. tdias The current BARRNET Field Service on-call person has been paged with the following message, "BARRNET ticket 2678 - Smarties Situation Critical Please contact CJW@BARRNET.NET" Note: 2 Cathy is obviously experiencing severe problems due to a 06/05/95 17:39 glucose shortage as her spelling on the ticket open is jhickey notably atrocious. It is clear this Smarties shortage needs to be addressed at the highest levels as quickly as possible. Note: 3 This issue started on the 17th of May and was only reported 06/05/95 18:19 on the 5th of June? tdias CJW, should this issue be escalated to IE or to jrugo? As it is a priority 4 and the the problem was only reported on the 5th of June the NOC can only assume that this issue isn't too urgent. CJW, how's about we ship out a case of Smarties the next time iserver.tc.barrnet.net needs to be replaced? Regards - Tony Dias - BARRNET ops Note: 4 Tony, the problem was only just brought to my attention this 06/05/95 18:29 afternoon. No one realized the consequences until it was cjw too late. Being new to the ticket system, I chose priority 4 because this is an internal (non customer) problem. Perhaps I should upgrade it to a higher priority. Note that the Ismaint group is shipping me a new iserver with a due date of Wednesday at 10:30am. Could relief be in sight? Thanks for your support in this most trying of times ---Cathy Note: 5 Jeff Osborne of BARRNET Field service reponds that although 06/05/95 20:19 Field Service provides many services, trying to "score" or tdias i.e. dealing is not one of them. BBN Planet's Ismaint group responds that the new Iserver is already hermetically sealed and can not be touched by anyone here until the customer receives it (so that we can say, "Hey it worked when it left here." So we won't be able to include it with the new iserver. Note: 6 WR Folks, 06/06/95 07:11 Have you tried Skittles as a substitute to Smarties as a tschlatter workaround until the Smarties supply problem can be addressed? Note: 7 Skittles just won't do. We have brought in chocolate dipped 06/06/95 10:07 strawberries as a substitute and that seems to be a good cjw interim solution, but they're going fast. We are hoping that one of the many staff members searching the Bay Area will come up with a new supplier. It is disappointing that FS will not assist since it is clear that some of them have been seen with their hands in the Smarties orb getting their fix. Some folks just won't admit their addiction publicly. Thanks for your continuing support ---Cathy Note: 8 The lack of Smarties in the WR, has brought Planet HQ to 06/06/95 15:03 the very painful realization that there are no chocolate jbehrle dipped strawberries in the fridge, on the east coast. Like a domino effect morale at HQ has started to crumble, if it continues at this feverish pace the situation could become critical! Rumor has it that Bill Gates is closely eyeing the situation, in hopes that Planet executive mgmt will approach him and turn Planet's WR into the MicroSloth Network running only NetBEUI, and DOS based apps. Note: 9 I think the all the regions can agree that if a supplier is 06/06/95 15:49 found, equal portions should be guaranteed for each and kjc every region for symmetric routing. The integrity of the chocolate supply here has degraded to the point where the overall sucrose input has fallen drastically. Sugar packets are being dropped left and right. This definitely calls for escalation. Note: 10 Due to the crisis forming at HQ, I did some investigation. 06/06/95 16:01 It appears that it is not possible for Mackenzies to ship cjw chocolate covered strawberries due to their perishable nature. The only option would be for the next WR person to venture to HQ to act as a courier. This is not a a guarantee, however, because 5 hours on a plane could make them not to tasty. Mackenzies has other chocolate taste delights that might serve as a substitute... ---Cathy Note: 11 Called and Spoke to Donna at the BJ's Wholesale Club in 06/06/95 17:30 Medford, MA. Asked Donna if they have any Smarties in stock jbehrle and she confirmed that they carry them in "5 Pound Value Packs". Asked Donna how many Smarties come in the "5 Pound Value Pack", she reports that she doesn't know. Requested that she open the bag up and count them, as the future of "the PlaNET" may depend on it, Donna hung up. Note: 12 Have secured 3 - "5 Pound Value Packs" of Smarties that will 06/06/95 18:30 be packaged up and sent to the WR in hopes of improving jbehrle staff morale. Current ETA is Thursday, 8 June at the latest. 06/07 09:47 PST: --- ETA is currently Friday, 9 June 1995 --- Note: 13 In repeated attempts to transmit Smartie packets via a Cisco 06/06/95 22:28 router, the packets are (quite literality) dropped on the jbehrle floor without even a hint of an error. Suspect there is a bug in Cisco's IOS 10.2(6.1), preventing Smartie packets from be passed through the router. I believe a ticket needs to be opened with Cisco on this issue. Note: 14 Regis Donovan of the Systems & Network Services Dept at 06/07/95 07:31 FTP Software, Inc. in N. Andover, MA reports that she has jbehrle duplicated the Cisco bug on 10.2(1) and several versions of 10.0, she suspects this problem may be common to all versions of IOS. After locating a 2.15 lb "orb of Smarties" in the FTP Development Dept, Regis estimates that 3 "5 pound value packs" would contain somewhere on the order of 1256 standard Smartie packets, and ~18,837 individual Smarties which is approx the # of Internet nets in use in late 1993. Note: 15 Barbara Bellissimo from BBN Planet WR reports that it 06/07/95 09:47 appears that they are out of immediate danager as a large jbehrle quantity of Smarties appeared on her chair this morning. 3 - "5 Pound Value Packs" will be shipped out of HQ this afternoon via FedEx Economy Service (air bill number will following for tracking purposes). It was decided that due to the appearant Cisco bug, and multiple failed attempts to use carrier pigeons (per RFC 1149). FedEx would be the next logical step. Note: 16 FedEx Air Bill 272-9533-414 applies, this can be tracked via 06/07/95 14:40 the Web: http://www.fedex.com/cgi-bin/track_it jbehrle -or- By sending email to "track@fedex.com" with "airbill 272-9533-414" in the body of the message. It was picked up from HQ at 17:20 EST 06/07. Note: 17 Have begun to use a variety of fresh cannolis (riccota, 06/07/95 14:46 vanilla and chocolate) as a substitute for chocolate covered tschlatter strawberries in the Planet HQ NOC. We are considering using these as a permanent replacement for the strawberries as we have found a reliable year-round supplier and have not been able to do so for the strawberries, and they appear to be having the desired effect on corporate morale, but are still waiting to evaluate the use of Mackenzies chocolate covered strawberries before making a decision. Note: 18 Tony Li of CISCO thanks us for the notification of the 06/07/95 14:58 problem with their software and has requested our assistance kjc in isolating this problem. He has alerted their staff to watch for supplies and are looking to their telecommuter staff for hope, moving on to field support if that fails. He has filed bugid CSCdi35420 on this problem and requests that we enable "debug ip s" and send them the output. He would also like a copy of the appropriate packet generator software for further help. Note attached to ticket. Note: 19 Three 5 pound bags (one Godzillagram) of 15-bite packets 06/07/95 15:24 have arrived at their first hop enroute to the Western kjc Region. They are in a wait state, holding at Medford, MA. Further updates as they arrive. Once the traceroute is complete, we'll have a better idea of whether this is a routing issue or a saturation/starvation (i.e. feast or famine) problem. The next step will be a determination of how to be pro-active here, and prevent this issue from happening again. Note: 20 Anke Dosedal of cisco systems believes that the bug is a 06/07/95 15:57 serious problem. He is submitting a purchase req. for cjw at least one "orb" of Smarties for their lab in order to try to reproduce this bug. I replied offering to send them a value pack in an effort to facilitate the debugging effort ---Cathy Note: 21 I was just reminded that Anke is not a HIM, but a HER. 06/07/95 17:00 Terribly sorry Anke! --- Cathy cjw Note: 22 Anke (cisco) replies that one orb probably won't be 06/08/95 06:31 sufficient for the testing purposes and believes the testing kjc lab might need two. Jim Helvie heartily welcomes input from us to help reproduce the problem and further details that the quantity of Smarties orbs required will of course depend upon the difficulty of debugging & then testing the fix. For a bug that may affect the future of "the PlaNET" cisco will want to do a lot of testing. Note: 23 The Godzillagram of Smarties was picked up from Medford, MA 06/08/95 08:32 at 17:20 on 06/07/95. kjc Route thus far: o Left Origin Location MEDFORD MA 06/07 20:26 o Pickup Exception MEDFORD MA 06/07 17:20 Note: 24 Bud Osterberg formarly of Quorum, now at Oracle sends this 06/08/95 12:06 note, "Well, I just wanted you to know that I can sympathize cjw with your current situation. I recall the agonizing days when our coffee machine was in the ICU. Just keep your spirits up, sugar is on the way! -- bud Note: 25 Jim Lewinson of ISI suggests that the problem may be some 06/08/95 12:59 sort of hardware problem, perhaps involving the cable or cjw transciever manufacturer. Has this been investigated as a possibility? ---Cathy Note: 26 The cables were in use before attempting to pass Smarties, 06/08/95 13:07 working normally. Post-testing period, cables continue to kjc work flawlessly (after they were wiped off). I'd be inclined to believe this is a protocol issue rather than a problem with the hardware. Note: 27 The 3 - "5 Pound Value Packs" of Smarties have "Left Hub 06/08/95 16:03 INDIANAPOLIS IN" 3 times now: jbehrle o Package Left Hub INDIANAPOLIS IN 06/08 14:52 o Package Left Hub INDIANAPOLIS IN 06/08 15:10 o Package Left Hub INDIANAPOLIS IN 06/08 14:36 Suspect we've hit a routing loop, or were seeing packet fragmentation... Note: 28 Concerned about a routing loop, or packet fragmentation, I 06/08/95 16:35 called "1-800-Go-FedEx" and spoke to Connie. Connie assured jbehrle me that the packet "is still in one piece", and that it was probably handled by three different people in Indianapolis and scanned by each of them. Am still a bit suspicious of a possible routing-loop, but the NOC will continue to monitor. Note: 29 Lisa DeBruine a College of Literature, Science, and the Arts 06/08/95 20:01 Student at the University of Michigan reports that in her jbehrle experiences "chocolate covered strawberries are a better substitute for Smarties than Smarties themselves", but she agrees that they are "not as cost-effective". I suspect she has never attempted to use carrier pigeons to transport 15lb of Smartie packets, which I should point out is not only expensive, but real messy too. X.500 at Umich points out tha Lisa enjoys Kool-Aid "finger Lisa.M.De.Bruine@umich.edu". Note: 30 Nora Lundin from BBN WR admits that during the heat of the 06/08/95 23:28 crisis she stumbled upon and purchased mini Sweet Tarts, jbehrle disguised as Smarties (Same twisty wrapper, same shape and size as Smarties). She reports that "To my surprise, while they only fooled people for a moment, they seemed to be an adequate substitute, and a few have even commented that the flavor was better than Smarties'". Nora points out the it is not clear if mini Sweet Tarts will ever permanently replace Smarties, but they might do in a pinch... Note: 31 Elizabeth Kaufman of BBN Planet SP offers to help in any way 06/09/95 08:08 she can. She has extended the support of the entire SP dricci development team. She warns that SP is not currently "Smarties-capable;" and reports that some have even claimed it has no smarts at all. They are working on a custom Smarties-proxy, to be released Q1, Elizabeth expect to announce "which fiscal year sometime in the next 10 weeks". The proxy is expected to be fully TIS-compatible (so it melts in your mouth, not in your Gauntlet). Note: 32 The 3 - "5 Pound Value Packs" of Smarties appear to have 06/09/95 09:47 cleared the routing-loop in INDIANAPOLIS, and have reached jbehrle FedEx's MENLO PARK Hub. We have high confidence that successful deliverey is imminent! Note: 33 It appears that we may have hit another routing loop: 06/09/95 10:06 o Package on Van MENLO PARK CA 06/09 08:19 jbehrle o Package on Van MENLO PARK CA 06/09 06:48 Note: 34 The 3 - "5 Pound Value Packs" of Smarties have successfully 06/09/95 11:04 arrived in the WR. Witnessed the rejoicing in the WR NOC jbehrle via a BBN PicWin IP video conference!!! The "FedEx Airbill Tracking Information" is still reporting: "Package has not been Delivered!"??? Note: 35 This issue has demonstrated that the NOC at HQ does not have 06/09/95 11:18 adequate tools in place to monitor the "orb of Smarties" jbehrle in BBN Planet WR. If we had we could have prevented this problem before it became critical! Working with Len Rose of BBN Planet WR to see if we can get a "SmartieCam" setup on the Web for proactive monitor of the WR's "orb of Smarties". Note: 36 Len Rose reports that "We will have the Smarties Real Time 06/09/95 12:05 Monitor installed sometime later today. The url will be jbehrle http://www.netsys.com:/smarties.html (jpeg) In order to monitor the orb, they will have to be relocated to my office since we do not have a coax cable that will reach their normal location." Note: 37 At this point we have seen over 1200 finger requests for 06/09/95 12:21 this ticket. Requests are coming in at more that 100/hour. smiller Just a note to our viewers, this is only a test, if this had been an actual emergency you would have been directed to let BBN Planet manage your Internet Services. We promise to work just as hard to get your IP packets to their destination :^) BBN Planet,"How Business does Business on the Internet!" Note: 38 Kobi from the BBN Planet SER NOC reports that they have 06/09/95 13:44 uncovered a similar problem in College Park, MD. Per Kobi: jbehrle "In fact, it has been suggested that the problem is much more serious; there is no record of the SER ever having a cache of smarties". Kobi hopes to avert a similar crisis in SER, she plans to obtain a "Sam's" card today so Smarties should be available in bulk in the SER NOC shortly. Kobi, please keep us posted, as we can always try to fire up the carrier pigeons for one more run... Note: 39 Mark Moraes from D. E. Shaw & Co, has offered to assist in 06/09/95 13:56 Smarties network testing on the Wellfleet router platform. jbehrle Mark reports "We invariably use Gummi Bears to test our New York-London link", as their systems person in London "has this deep-seated addiction for candy". Mark, We welcome your assistance, as we suspect this is an internet-wide problem. It appears that MCI BIPP opened TT# 3001 on this issue, but as it has been closed as this prob is beleived to be at the protocol-layer. Note: 40 Len Rose of the WR is pleased to announce that there is now 06/09/95 14:10 a real time SmartieCam(sm) installed at: dricci http://www.netsys.com/smarties.html This will enable real time monitoring of the Smarties situation in the BBN Planet WR. Note: 41 The "FedEx Airbill Tracking Information" finally confirms 06/09/95 15:16 that the "Package has been Delivered!". It reports the jbehrle delivery time was 10:44 PST, and that it was signed for by "K.NAYLOR" Kristin Naylor, our temp WR receptionist. Some of us saw the rejoicing live over the Internet back in note 34. Suspect that FedEx will be out of business if Cisco ever adds a 'Heisenberg Compensator' to their 7000 line ;-) Note: 42 In ref to note 31, Bob Stratton from UUNET Security Services 06/09/95 17:14 extends the following cautionary note "Experimental jbehrle transmissions of Skittles (the most similar traffic to Smarties available at this site) through a TIS Gauntlet with the optional h/w encryption card appears to produce output (on the red-net BNC connector) roughly similar to 'chocolate Magic Shell'." It is unclear if this substance produces serious throughput bottlenecks (on hardening). Bob urges the use of software crypto until this effect can be studied. Note: 43 Cathy Wittbrodt called in to report the SmartiesCam(tm) is 06/09/95 21:55 non-functional. I have been able to confirm that we can kjc reach http://www.netsys.com/cgibin/smarties from noc.bbn.com without a problem. Recommend that Cathy check into her local connectivity (she is at home). Note: 44 It appears that the web browser on my laptop has some 06/09/95 22:38 problems. Sorry for the false alarm. --- Cathy cjw Note: 45 To catch up on some important information that did not make 06/10/95 00:00 it into the ticket, Peiter Zatko of the Internet Server kjc group reports that "Front Door for Windows should have a point-and-click interface for Smarties capabilities. The BBN IServer server side software already has this capability [per secret code implemented by Dan Franklin]. Unfortunately the route for porting the smarties code implemented on the Macintosh FrontDoor to the Windows frontdoor requires a 3rd party library package which prevents the Windows programers Note: 46 ...from doing any Windows programming." 06/10/95 00:20 kjc Note: 47 John Rugo responded to Elizabeth Kaufman's Site Patrol issue 06/10/95 00:28 asking if the custom Smarties-proxy will include kjc authentication with non-repudiation. He further details, "There are unconfirmed reports that a company in the east bay has learned of the Smarties crisis and has been shipping product with the Smarties label but which are not the real thing!" Note: 48 Elizabeth Kaufman responds to John Rugo that he does in fact 06/10/95 00:31 raise a very complex set of security issues. "The proposed kjc proxy...relies on the accuracy and integrity of the data label (i.e., if it looks like a Smartie and comes from the Smartie-place, it's a Smartie). I observe from Note 30 that this policy has already resulted in the mistaken consumption of "mini Sweet Tarts." Verifying content if we mistrust the labels is technically complex- we could either go to a third party CA (Candy Authority) infrastructure, or we could... Note: 49 ...implement the sensory equivalent of checksumming, and 06/10/95 00:33 implement a "taste signature" on each proxy server. Do you kjc have a preference? (BTW, do you believe the current crisis to be a natural shortage, or a deliberate Denial-of-Smarties attack?)" * Len Rose asserts that "WE MUST ASSURE that we are safe from Smartie-Spoofing." Note: 50 I believe I can report from personal experience that using 06/10/95 01:08 the sensory equivalent of checksumming in conjunction with a kjc taste signature will lead to a stomach ache unless the implementation of Taste Buds works on a personnel rotation basis (i.e. limiting individual consumption to a reasonable number of packets where "reasonable" prevents nausea). However, here we run into variability in accordance with different people's ability to distinguish between Smarties, Mini Sweet Tarts and other similar products. Note: 51 It has been suggested that CERT might want a heads-up on an 06/10/95 08:01 issue of possibly large scale importance - namely reports of kjc Denial-of-Smarties attacks occuring throughout various regions of the Internet from unknown sources. Additionally, there have been reports of Smartie-Spoofing by cleverly disguised "mini Sweet Tarts." If true this issue could make the Morris Worm look like a benign geek prank. Note: 52 I think it would be in BBN PlaNET's best interest NOT to 06/10/95 12:03 involve the CERT in this issue, as it is a contained tdias problem and a CERT advisory concerning this issue would only cause an Internet-wide panic, comparable to the SATAN and IP-Spoofing advisories, i.e. the NOC would have a heck of a time with all the Smarties Spoofing filters that would have to be installed. Also the CERT has no sense of humor, from past experiences. Note: 53 I received a call from Moira, from the CERT. Someone, had 06/10/95 12:40 mailed the CERT concerning this issue. (KJC, naughty, tdias naughty) She wished to know, the impact of this issue, and how we'd like the CERT to be involved, especially with the Smarties Spoofing incidents that have occured. She indicate d that the CERT will possibly create an notice concerning this issue, on 06/12/95, or at least offer their services at that time. Futher she indicated, that... ************* CONTINUED *********** Note: 54 ******** continued from previous note *************** 06/10/95 12:44 ... that the CERT does have M&M's technology and is aware tdias of several issues involving them, but that their Smarties experience is rather limited, in fact, Moira was the only person, within the CERT NOC who was aware of Smarties, because she's originaly from the UK, where Smarties are plentiful and she had to inform other CERT NOC'ers of what Smarties are. Finally she indicated, through her patience & understanding that note 52 is wrong; the CERT has humor. Note: 55 Oh ya, I finally tasted a Smartie today. The suckers are 06/10/95 12:55 vile; it was like licking the inside of a Kool-Aid packet tdias or emptying out a whole Pixie stick on your tongue at once. I would think that only a surplus of those toxic little tablets would be something to be worried about and a shortage of them would be something to rejoice. Gummi Bears,btw, are much better. (Tony attempts a flamewar) Note: 56 A minor flame war has ensued between BBN Planet's 06/10/95 15:12 New England region (formerly NEARNET) and BBN Planet's tdias Southeastern region (formerly SURANET) (Hey it was the only element missing from this saga.) Note: 57 The latest count of acessing the soon-to-be legendary 06/10/95 17:44 "Smarties" ticket, stands at 1777 as of tdias Sat Jun 10 17:27:04 1995. Here are some of the domains that have fingered us: (get your mind out of the gutter) ucl.ac.uk, cisco.com, ludd.luth.se, dec.com, mit.edu, netcom.com, world.std.com, ftp.com, cnri.reston.va.us, convex.com, evtech.com, rutgers.edu, umd.edu, uunet.ca, uiuc.edu, marble.com, ora.com, toronto.edu, qualcomm.com ************ CONTINUED **************** Note: 58 Think.COM, bellcore.com, buffalo.edu, nasa.gov, 06/10/95 17:52 kochi-u.ac.jp, liero.cc.lut.fi, swissbank.com, tdias nyu.edu, anu.edu.au, apple.com, cern.ch, umich.edu. interpath.net, realtime.net, columbia.edu, wais.com, and osf.org Note: 59 Try not to pay too much attention to the rants and raves 06/12/95 09:33 of Tony Dias (tdias) in notes 52-58 as it has come to our jbehrle attention that Tony was suffering from a sugar rush overload, at the time... Note: 60 Nora Lundin from the WR will be send a supply of the 06/12/95 09:35 cleverly disguised "mini Sweet Tarts" to the HQ and SER jbehrle NOC's for additional Smartie-Spoofing testing. Note: 61 On Wed, 07 Jun 1995 at 14:53:22 Cisco released the "Cisco 06/12/95 09:37 Smarties Advisory" (CSCdi35420) warning of a vulnerability jbehrle in Cisco's IOS software when the 'smarties' keyword is used in extended foodstuff access control lists. It warns that: "This bug can, under very specific circumstances and only with certain consumption units, allow unauthorized sugar products that resemble Smarties such as 'Skittles' and 'M&Ms' to circumvent a filtering router". Note: 62 Massachusetts local Security Contractor Hobbit reports that 06/13/95 07:22 a temporary solution to Smarties leakage may be in the form kjc of a new package from Italy called SPREE, currently in beta testing. It can wrap traffic inside DES, 3DES or IDEA sugar coating, using color-coded Drakes-Hershey key exchange at the beginning of the session. He notes that while the resulting SPREE-encapsulated packets are theoretically 2^63 times harder to crack open than UUNet's Skittles, they are big & hard & difficult to push through the ST connectors... Note: 63 ...on some network hubs. One of the beta testers has 06/13/95 07:25 suggested possible lozenge deformation in rare cases where kjc transmission media are run close to hot steam pips as on larger college campuses. This problem has not yet been thoroughly reviewed by the candographic community at large, but alert users will quickly become aware of the problem if their arriving SPREEs start appearing gummi. Note: 64 All of us in Western Region have gotten back to our normal 06/13/95 14:49 sugar high. It is good to know that there is such a spirit cjw of cooperation out there. Thanks to everyone for your ideas and support. We will continue to monitor the Smarties bugs filed with various vendors and will keep the community up to date on the status. ---Cathy Problem Solution: We will continue to monitor the Smarties bugs filed with various vendors and will keep the community up to date on the status.
(From the "Rest" of RHF)