| Where |
What |
Start Date/Time |
End Date/Time |
Description |
| local |
SE packetloss |
2007-08-01 02:00 |
2007-08-01 14:00 |
Both lines are suffering 20-35% packetloss. If I try to move data on the line, packet loss goes to 45%. Ping times are 15ms to DSLAM, which is fine -- except for the fact that almost half the packets don't make it back at all! 11:17: Speakeasy posted a system status message indicating a Chicago-area outage. 14:00 0% packetloss, cause of outage unknown. |
| local |
SE packetloss |
2007-07-09 08:00 |
2007-07-11 00:00 |
Experiencing high latency and low throughput on SE-A, SE-B equally. 13:58 opened web ticket. Called Speakeasy at 16:30 and SE-A is better now, but not perfect. Oddly, had a similar -- but not precisely the same -- issue almost exactly one year ago. 18:30: Problem is gone for now, will monitor for 72 hours or so. 2007-07-10, same problem again today. Symptoms: max download speed is about 150K/s, but sometimes it goes down to 25K/s. Speakeasy speedtest shows 1800Kbps down, 580 up. Even when only a single download is occuring, ping times to the DSLAM skyrocket to 1800ms+. Normal ping should be less than 50ms. If no file transfers are active, there is no packet loss that I can see. However, while transfers are in progress, there is roughly 10% packet loss. At 17:00 exactly, speed improved dramatically! 2007-07-11: hopefully we're on the way to resolution for SE-A. Turns out the backhaul/PVC that I was on was way oversubscribed (95th percentile utilization). Speakeasy switched both SE-A, SE-B to a better PVC. Speeds are back to normal; will monitor for 48 hours or so. |
| local |
SE-B sync |
2007-07-09 08:00 |
2007-07-09 18:30 |
This line drops sync at the slightest hint of bad weather. Opening ticket with Speakeasy. Expect difficult resolution process. 18:30: possible inside wiring issues; removed splitter from the line and it's been behaving ever since. |
| EV1 |
unreachable |
2006-09-11 16:55 |
2006-09-11 17:53 |
Cannot get to EV1's website or forums, nor to my server. Trying from Speakeasy (via Above.net) and Savvis, no go. EV1 says: "The source of this issue was a misconfiguration of the Fireslayer Dos Mitigation system." |
| local |
Liger destroyed |
2006-08-23 09:00 |
2006-08-23 09:00 |
Machine turned itself off overnight. Wouldn't power on but instead emitted a horrible burning smell and made a "pop" sound. It's fried. Further investigation reveals the liquid cooling system has leaked onto the power supply. Drives, cards, and RAM seem to be unaffected. |
| local |
Liger crash |
2006-07-18 13:05 |
2006-07-18 13:10 |
A garbled, medium-high pitched sound about 0.5 seconds in duration preceded this lockup. The machine froze but I didn't see an OS X KP screen. Running 10.4.6 but will now update to 10.4.7. |
| SM |
desdemona crash |
2006-07-18 06:50 |
2006-07-18 08:00 |
Woke up to this machine being down... issued a reboot ticket and it's back. Cause of crash unknown. |
| local |
SE packetloss |
2006-07-08 08:00 |
2006-07-17 08:00 |
Speakeasy is having problems in the Chicago area with packet loss. Connections die for ~30 seconds every so often. They are aware. The problem improved after two rounds of emergency maintenance, but as on 2006-07-17 14:34, they said: "After hardware maintainance in Chicago we are still unfortunately seeing sporadic issues. We are thus continuing to work with our vendor to find the source of the problem." Another "Emergency Network Maintenance Event" is scheduled for 2006-07-20 01:00. That got it, the problem is gone. |
| local |
Liger wedged |
2006-07-08 16:00 |
2006-07-08 16:05 |
Lockup while doing cmd-tab to switch apps. OS X 10.4.6 |
| local |
SE-B down |
2006-05-19 14:30 |
2006-05-19 17:00 |
Secondary outage but just on this one line. Turns out it was the router (wallb). Damn. |
| local |
SE-A, SE-B no surf |
2006-05-19 12:40 |
2006-05-19 13:30 |
Speakeasy DSL lines A, B synched but no net access, or, as they say, "sync no surf". Can ping routers from the outside. This has happened before. Resetting routers. Failed. Called Speakeasy: known problem in the Chicago area. @13:00 Waiting 10 more minutes before opening a ticket. Ignored it for a while and it started working again.... |
| SM |
desdemona crash |
2006-02-12 12:10 |
2006-02-12 12:17 |
Machine was probably powered off accidentally. Either that or there's a faulty bit of hardware. Will keep an eye on it. |
| local |
Liger wedged |
2006-02-01 12:00 |
2006-02-01 12:00 |
Had three lockups while doing cmd-tab to switch apps. Happened on three separate days, but only recording once since I lost track of *which* days. |
| local |
SE routing |
2005-10-31 13:00 |
2005-10-31 13:50 |
Speakeasy DSL apparently having a routing issue with Level 3 - major problems past ge-8-0-340.ipcolo1.chicago1.level3.net. Going through Above.net (444.ge-4-0-0.mpr1.ord7.us.above.net) works fine. This took some time to isolate but meanwhile routes haven't been updated for some reason. Am unable to demonstrate any packet loss -- oddly, pings and *emails* are getting through just fine. But ssh and web are not. I hope this isn't a Level 3 dispute with Speakeasy or something. Will call at 14:00 if not resolved by then. At 13:50 the problem suddenly vaporized and everything returned to normal in an instant. |
| local |
Liger crash |
2005-05-22 15:45 |
2005-05-22 15:45 |
Tiger 10.4.1 bomb. Was FFing a dvd in DVD Player.app. Left screen went all gray except for a small white box near the upper left corner. Right screen was filled with vertical lines of differnt shades of gray, as well as black and white. Killer! |
| local |
Liger crash |
2005-05-16 09:23 |
2005-05-16 09:25 |
Tiger 10.4.0 KP. panic(cpu 0 caller 0xFFFF0003): 0x300 - Data access |
| local |
SE-B intermittent |
2005-05-13 12:45 |
2005-05-13 13:30 |
I guess they're working on this line now. |
| local |
Liger crash |
2005-05-13 11:15 |
2005-05-13 11:17 |
Tiger 10.4.0 KP. unaligned frame address: 0xC870FD18 -- doesn't sound good |
| local |
SE intermittent |
2005-05-12 23:10 |
2005-05-12 23:59 |
Speakeasy DSL intermittently available - they are doing emergency maintenance at the POP |
| local |
SE-B down |
2005-03-07 07:00 |
2005-03-07 10:00 |
SE-B line good connectivity but can't see their router; this has happened before; SE-B DSLAM is pingable from the outside; 120 reset on the bridge did not clear problem; opened ticket; SE reset DSLAM, ticket closed |
| local |
quietriot crash |
2005-03-04 16:00 |
2005-03-04 20:25 |
Kernel panic - turns out to be a faulty RAM stick. Replaced stick and now it's working. |
| local |
Dual-G4 crash |
2005-02-12 09:30 |
2005-02-12 09:45 |
Panther 10.3.7 wedged (GUI unresponsive, no mouse cursor). Keyboard worked enough to open/close DVD tray but that's about all. |
| local |
Dual-G4 crash |
2005-01-30 09:53 |
2005-01-30 11:15 |
Panther 10.3.7 wedged under load of incoming faxes (!) "kernel: InternalUSBModem::setPowerState(1)" 28 times in system.log over a period of one hour, then nothing.... I was doing fax testing during this time. Monitor was in energy saver mode and wouldn't wake up, so I don't know what might have been displayed on the screen. |
| local |
SE intermittent |
2004-09-12 14:00 |
2004-09-12 16:00 |
DDoS attack - 80% packet loss - packet loss at the DSLAM ~15%; called SpeakEasy and they confirmed this - apparently InterNAP sourced DDoS. Went away by itself. |
| local |
Dual-G4 crash |
2004-06-20 14:05 |
2004-06-20 14:15 |
wedged while trying to copy from a CD |
| local |
Dual-G4 crash |
2004-06-07 17:15 |
2004-06-07 17:25 |
Six (6) Panther 10.3.4 kernel panics, at 12:15, 15:40, 17:15, 20:10, 21:45, 21:55 (minutes after boot!); going to attribute the panics today to heavy BT activity (still not sure though - will run all night w/o BT to make sure) |
| local |
Dual-G4 crash |
2004-06-07 09:19 |
2004-06-07 09:22 |
Two (2) Panther 10.3.3 kernel panics, at 08:52 and 09:19 |
| local |
Dual-G4 crash |
2004-06-04 01:12 |
2004-06-06 19:30 |
Panther 10.3.3 kernel panic |
| local |
Dual-G4 crash |
2004-06-03 15:35 |
2004-06-03 15:45 |
Panther 10.3.3 just wedged; if it was going to panic, it didn't have time, as I just restarted after about 5 mins |
| Netmar |
unreachable |
2004-06-01 21:45 |
2004-06-02 00:00 |
Unreachable; traceroutes from three locations fail; came back sometime in the night |
| Netmar |
unreachable |
2004-05-28 19:25 |
2004-05-28 19:55 |
Unreachable; traceroutes from four locations reveal a routing problem (looping?); 19:54: she's breathing |
| Netmar |
unreachable |
2004-05-26 09:00 |
2004-05-26 17:00 |
Intermittently unreachable; eventually called Netmar and they said there is a routing problem of some sort and the problem eventually went away |
| local |
SE intermittent |
2004-05-25 09:00 |
2004-05-25 13:00 |
Speakeasy DSL intermittently available (CPE retraining over and over) perhaps due to noise on the line; eventually changed filters and it got better |
| local |
Dual-G4 crash |
2004-05-13 09:57 |
2004-05-13 10:05 |
Panther 10.3.3 kernel panic after the machine slowly wedged itself |
| local |
Dual-G4 crash |
2004-05-09 04:33 |
2004-05-09 06:55 |
Panther 10.3.3 kernel panic |
| local |
SE intermittent |
2004-05-06 14:00 |
2004-05-06 14:15 |
Speakeasy DSL intermittently available (CPE retraining over and over) perhaps due to noise on the line |
| local |
Dual-G4 crash |
2004-05-02 18:50 |
2004-05-02 19:05 |
Panther 10.3.3 kernel panic |
| local |
Dual-G4 crash |
2004-04-16 20:33 |
2004-04-16 20:57 |
Panther 10.3.3 kernel panic |
| local |
Dual-G4 crash |
2004-04-14 05:32 |
2004-04-14 07:15 |
Panther 10.3.3 kernel panic |
| local |
Dual-G4 anomaly |
2004-03-28 09:45 |
2004-03-28 09:51 |
Panther 10.3.3 GUI locked up, except for the mouse cursor; iTunes was playing via the network, but it paused for a good 15-30 seconds and then continued. After about two minutes, iTunes started just playing normally as if nothing was wrong, although the GUI was still completely frozen. At the five minute point, I held down the power button on the front of the machine, but apparently due to the button's flakiness (by design?) the machine thought I wanted to sleep it (or something). Anyway, it came back to life and I was able to continue on working. Interestingly, I had been exporting records from FileMaker and the export had not progressed during the five minute lockup period, so I guess the lockup went further than just the GUI. In the end, I was able to restart cleanly. |
| local |
power outage |
2004-03-05 10:30 |
2004-03-05 12:30 |
nothing survived this outage; nothing was shut down properly before losing batteries altogether, but all's well |
| local |
Dual-G4 crash |
2004-03-03 10:10 |
2004-03-03 10:20 |
Machine slowly wedged itself, cause unknown. |
| local |
XO down |
2004-01-15 14:15 |
2004-01-15 15:00 |
XO DSL line good connectivity but can't see XO's router; this has happened before; XO DSLAM is pingable from the outside; 120 reset did not help; 14:25 calling XO and a recording says that there is a Chicago area DSL outage for XO so waiting until 15:00 to try again; did another 120 reset at 14:55 and -- it's back |
| local |
Dual-G4 crash |
2004-01-09 15:26 |
2004-01-09 15:30 |
Panther 10.3.1 kernel panic |
| local |
Dual-G4 crash |
2003-12-16 17:45 |
2003-12-16 17:40 |
Panther 10.3.1 kernel panic |
| JohnCompanies |
server unreachable |
2003-11-30 18:15 |
2003-11-30 18:35 |
cannot connect to my virtual server; had crashed, or was rebooted (apparently) |
| local |
XO down |
2003-11-25 09:45 |
2003-11-25 09:50 |
XO DSL line good connectivity but can't see XO's router; this has happened before; XO DSLAM is pingable from the outside; 120 reset on the bridge cleared the problem (again) |
| local |
power outage |
2003-11-12 17:50 |
2003-11-12 19:40 |
nothing survived this outage; nothing was shut down properly before losing batteries altogether, but all's well |
| local |
XO down |
2003-10-13 13:00 |
2003-10-13 13:15 |
XO DSL line good connectivity but can't see XO's router; this has happened before; XO DSLAM is pingable from the outside; 120 reset on the bridge cleared the problem (again) |
| local |
XO down |
2003-10-13 11:30 |
2003-10-13 11:45 |
XO DSL line good connectivity but can't see XO's router; this has happened before; XO DSLAM is pingable from the outside; 120 reset on the bridge cleared the problem |
| local |
XO down |
2003-09-27 18:10 |
2003-09-22 18:40 |
XO DSL line good connectivity but can't see XO's router; this has happened before; XO DSLAM is pingable from the outside; called XO; ticket opened; suspended due to a credit card problem - again! - fixed. |
| ProHosters |
unreachable |
2003-09-27 13:30 |
2003-09-27 16:00 |
seems to be a network problem |
| Netmar |
server lockup |
2003-09-27 10:15 |
2003-09-27 12:30 |
Something odd happened to the server, but I was eventually able to reboot it. |
| local |
XO down |
2003-09-22 14:10 |
2003-09-22 14:55 |
XO DSL line good connectivity but can't see XO's router; this has happened before; XO DSLAM is pingable from the outside; called XO; ticket opened; decide to go for a walk - it works! When I get back from the walk, the line is back up |
| local |
XO down |
2003-08-21 22:35 |
2003-08-22 06:45 |
A 120 reset on the CPE fixed this one! Possibly related to a brief, area-wide power spike during the night. |
| Netmar |
connection timeouts |
2003-08-20 09:00 |
2003-08-27 09:00 |
Odd problem; seems like during busy times, new connections cannot be established, whilst existing connections work fine. Netmar notified, awaiting response. PageSentry now monitoring the problem at 2 minute intervals. Problem seems to affect all IPs except the machine's main IP. 2003-08-25: got response from Netmar, and as usual, they rock! (Apparently they *were* running out of connection slots on their firewall.) Haven't seen any problems in the last 12 hours, continuing to monitor. Zero problems in 48 hours -- considering this to be closed now. |
| local |
power outage |
2003-07-17 15:45 |
2003-07-17 17:00 |
nothing survived this outage; nothing was shut down properly before losing batteries altogether, but all's well |
| local |
Dual-G4 crash |
2003-06-26 12:35 |
2003-06-26 12:40 |
Jaguar 10.2.6 kernel panic! |
| local |
Dual-G4 crash |
2003-06-08 08:30 |
2003-06-08 08:35 |
Jaguar 10.2.6 kernel panic! |
| local |
OpenBSD crash |
2003-05-31 07:00 |
2003-05-31 07:35 |
Out of disk space! ... and it kernel panicked. ::sheepish grin:: |
| Netmar |
unreachable |
2003-05-08 17:30 |
2003-05-09 02:30 |
Cannot reach my server. Page sent at 18:00. Linux crash apparently. |
| Netmar |
unreachable |
2003-04-07 19:20 |
2003-04-07 21:45 |
Cannot reach their network. DoS attack! |
| Netmar |
unreachable |
2003-04-05 11:15 |
2003-04-05 11:28 |
Cannot reach their network |
| QWK.net |
intrusion |
2003-03-27 16:00 |
2003-03-28 09:00 |
OK, switching to another webhost because of this message "Until we know more, we cannot make any guarantees about resolution time. If it is critical that your site be available, it would be best at this time to seek another web host." QWK.net has restored service, but it's too late for me. |
| Netmar |
crash? |
2003-03-28 02:15 |
2003-03-28 02:15 |
Server rebooted for some reason, waiting to hear possibilities as to why.... Turns out it was UPS-related. |
| JohnCompanies |
power outage |
2003-03-10 11:45 |
2003-03-10 14:00 |
transfer switch/UPS cascade failure and subsequent router destruction; repaired quickly considering... |
| local |
XO down |
2003-02-27 18:05 |
2003-03-02 14:15 |
XO DSL line good connectivity but can't see XO's router; this has happened before; XO DSLAM is pingable from the outside; called XO; ticket opened; settling in for long down-time; called on 2003-03-02 and found out that the line was suspended due to a credit card problem - grr, LOL - fixed. |
| local |
G4 crash |
2003-02-17 18:45 |
2003-02-17 18:50 |
Jaguar kernel panic! |
| local |
power outage |
2003-02-12 18:45 |
2003-02-12 20:30 |
nothing survived this outage; server "Beige" ran for about 30 minutes though, and the OpenBSD firewall box made it about 45 minutes; everything was shut down properly before losing batteries altogether |
| local |
XO down |
2003-01-30 11:55 |
2003-01-30 13:28 |
XO DSL line good connectivity but can't see XO's router; this has happened before; XO DSLAM is pingable from the outside; called XO; recorded message says there are known problems in the Illinois area |
| JohnCompanies |
"maintenance" |
2003-01-26 03:45 |
2003-01-25 11:00 |
planned outage was supposed to end by 05:00. another server configuration problem |
| easyCGI |
network slammed |
2003-01-24 23:45 |
2003-01-12 00:30 |
same problem again; unsure of the cause, but possibly related to SQL.Slammer worm loose on the Net; might as well blame that |
| MPA |
network down |
2003-01-22 10:30 |
2003-01-27 15:15 |
normally I do not pay much attention to the frequent outages on this network; as this one extends past 48 hours I wanted to make a note of it; PSI is having network-wide problems and many outages; 2003-12-26 18:25 down for over 100 hours, and no ETR is listed by PSI |
| local |
G4 crash |
2003-01-21 13:56 |
2003-01-21 13:59 |
Freeze upon quitting VPC - that's old skool |
| local |
G4 crash |
2003-01-19 15:19 |
2003-01-19 15:21 |
Jaguar kernel panic! |
| QWK.net |
email server |
2003-01-18 08:00 |
2003-01-18 13:30 |
apparent email system failure caused all subsequent checks to hang, but the system was actually up, it just "needed a break"; update: apparently there WAS a delivery problem as no messages got delivered for 36 hours |
| JohnCompanies |
server unreachable |
2003-01-17 12:20 |
2003-01-18 17:45 |
cannot connect to my virtual server; sent email to support at 12:31 - no response; sent another email at 14:29 - no response; sent a third email (hmm, lack of a phone number is hurting here) at 15:41 saying that four hours had passed without a response and offering other means for them to contact me; that last email got an email response and a phone call from John himself updating me on the situation, which is: disk failure on the server; uptime still unknown at (16:21); received update at 22:07 saying the server would by up tonight (01-16); now it's 01-18, 9:47, sent email asking for an update - no response; 11:48 sent another email asking for an update - no response; 12:15 called and was told ETR of two hours "this time for sure!" which would mean up at about 14:30; their RAID systems must be level 0, which seems to be a major weakness; 3 hours elapse, and it's 15:30 - I call, am told 30-60 minutes to system up; called at 17:26, told "about a minute" - LOL!; 17:45 it's back |
| local |
G4 crash |
2003-01-16 01:38 |
2003-01-16 08:00 |
Jaguar kernel panic! This time I caught Path Finder with 2057 threads open, according to "top", which I had employed for surveillance. I now theorize that under a high CPU load from other apps, Path Finder goes horribly awry after midnight. The machine can run fine all day with no problems under a high load; to test this theory, I will quit Path Finder every night for a up to a month to see if this fixes the problem. |
| Netmar |
server unreachable |
2003-01-15 15:38 |
2003-01-15 16:04 |
Assumed power failure due to UPS switchover problem (again) |
| Netmar |
server unreachable |
2003-01-14 12:45 |
2003-01-14 12:55 |
Power failure due to UPS switchover problem (again) |
| Netmar |
server unreachable |
2003-01-14 11:50 |
2003-01-14 12:20 |
Power failure due to UPS switchover problem (as told by Netmar) |
| easyCGI |
network slammed |
2003-01-12 18:45 |
2003-01-12 19:00 |
again... net is so sluggish that I can't make an FTP connection, although HTTP works _slowly_; I see 10% packet loss with ping |
| local |
XO down |
2003-01-11 11:40 |
2003-01-11 13:10 |
XO DSL line good connectivity but can't see XO's router; this has happened before; XO DSLAM is pingable from the outside; called XO; settling in for long down-time; oh good, it's back! |
| easyCGI |
network slammed |
2003-01-07 13:20 |
2003-01-07 13:41 |
net is so sluggish that I can't make an FTP connection, although HTTP works _slowly_ |
| Netmar |
intermittently unreachable |
2003-01-06 13:30 |
2003-01-06 13:45 |
network problems, I'll wager they're fiddling with their firewall/router |
| local |
G4 crash |
2003-01-03 04:41 |
2003-01-03 08:00 |
Jaguar kernel panic! |
| JohnCompanies |
crashing |
2002-12-19 00:24 |
2003-01-01 18:00 |
work in progress to resolve hardware issues; system is now back up |
| Netmar |
unreachable |
2002-12-08 10:25 |
2002-12-08 12:10 |
called and left emerg. page at 11:05; verified from three different networks that Netmar was totally unreachable; theorized about a router/gateway/firewall failure on their end requiring someone to be dispatched to the NOC to fix the problem; with that in mind, est. completion by 12:30. Started breathing at 12:00 on the nose. |
| local |
G4 crash |
2002-12-05 12:44 |
2002-12-05 09:30 |
Jaguar kernel panic! |
| local |
G4 crash |
2002-11-19 night |
2002-11-19 09:30 |
Jaguar kernel panic! |
| Netmar |
slowness |
2002-11-07 13:00 |
2002-11-07 17:00 |
complaint of slowness (images barely loading); I cannot reproduce it; possibly an issue elsewhere on the Net |
| local |
G4 crash |
2002-11-02 00:57 |
2002-11-02 08:30 |
Jaguar kernel panic! Uh oh. |
| Netmar |
partial connectivity |
2002-10-28 09:20 |
2002-10-28 09:35 |
once again can't connect to my server on any IPs except the main one; ongoing firewall rebuild; "we tend not to notice problems with some IPs as much as with others" |
| local |
G4 crash |
2002-10-25 00:01 |
2002-10-25 08:45 |
Jaguar kernel panic! |
| Netmar |
FTP problem |
2002-10-23 15:30 |
2002-10-23 17:30 |
inbound FTP to non-interface-direct IPs; ongoing firewall rebuild |
| Netmar |
outbound email |
2002-10-23 09:00 |
2002-10-23 11:00 |
can't connect to any hosts on port 25 (or 110); outgoing mail is piling up; ongoing firewall rebuild |
| Netmar |
server unreachable |
2002-10-17 13:00 |
2002-10-17 13:10 |
Linux crash! |
| Netmar |
unreachable |
2002-10-11 16:30 |
2002-10-11 19:00 |
later attributed to Linux.Slapper worm |
