*ABOUT OUR WEEKLY MAINTENANCE WINDOW*

Our weekly maintenance window (WMW) has been changed from Fridays,
from 2am to 6am to Saturdays, from 2am to 6am.  By analyzing usage
statistics and bandwidth consumption, we have determined these hours
to be the least likely hours for customers to be on line.

We will use this period to do upgrades and repairs to our network that
otherwise would be disruptive to service.  Please check the current
Network Status Report for details of actions during the WMW.  

We may need to repair/upgrade/replace hardware or software at times
outside of the WMW.  We understand the inconvenience to you, our
customer, should we need to do extensive, unplanned maintenance.  We
will therefore only disrupt service for maintenance if we determine
the situation to be an emergency.  We will always note emergency
maintenance on our Network Status Report.

If you have questions about reports on the Network Status Report,
please contact our Support Team.  We are available 24 hours a day, 7
days week, excluding major holidays.

Phone: 1.877.688.3200
Email: support@telerama.com
Chat Room:  http://www.telerama.com/irc/


-----------------------------------------------------------------------------


*Office Move*

Impact: High
Systems Affected: Telephone Support
Status: Telephone support unavailable at Sat Jan 31 07:31:48 EST 2004

Telerama's office is moving from Suite 630 to Suite 750, up one floor
from our old office.  During the move, our customer support staff will
be unavailable while we migrate our phone system.

We are very sorry for an inconvenience this may have caused and thank
you very much for your patience while we move.


-----------------------------------------------------------------------------


*DNS Resolution Problems*

Impact: High
Systems Affected: DNS for all Telerama users
Status: Workaround installed on Mon Sep  1 20:04:58 EDT 2003

Due to problems with our primary DNS server, 205.201.1.10, several DNS
outages have occurred today and over the weekend.  The problem appears
to be caused by an astonishing increase in lame DNS delegations.  (For
more information, see http://www.cymru.com/DNS/lame.html and
http://www.apnic.net/mailing-lists/sig-dns/archive/2003/07/msg00000.html.

We have installed a temporary workaround to prevent this problem from
recurring and are now working on a permanent solution which should be
in place by tomorrow morning.

We are very sorry for an inconvenience this may have caused and thank
you very much for your patience in resolving this issue.



-----------------------------------------------------------------------------

*Border Gateway Protocol*

Inpact: High
Systems Affected: Routers
Status: Finished at Sun Apr 25 00:25:05 EDT 2004

MD5 TCP hash was added to our upstream BGP peerings.
Code refresh will be performed on network equipment.
                                                                                
BGP peerings was cleared and hardware was
reloaded. 

If you have any questions, please contact us via phone at (877) 688-3200 or email at support@telerama.com during our normal business hours.

-----------------------------------------------------------------------------

*ADSL Performance Problems: Telerama ADSL Users*

Impact: Medium
Systems Affected: Performance/Throughput for Telerama ADSL users
Status: Problem resolved at Sun Aug 24 17:56:12 EDT 2003

Our monitoring systems are indicating that the ADSL performance
problems subsided on Thursday.  The latency and slow throughput issues
do not seem to have recurred.

Verizon representatives had initially claimed that the problem was due
to congestion caused by the recent Internet worms, but we do not
believe this was the case based on the types of patterns we are
seeing.  Verizon has never given us what we feel is an accurate answer
to what happened.

We apologize for any inconvenience this may have caused and thank you
very much for your patience in resolving this issue.


-----------------------------------------------------------------------------


*ADSL Performance Problems: Telerama ADSL Users*

Impact: Medium
Systems Affected: Performance/Throughput for Telerama ADSL users
Status: Ticket is still open at Wed Aug 20 19:30:19 EDT 2003

Some Telerama ADSL customers are still experiencing high latency and
slow throughput.  We have determined that approximately 100 Telerama
ADSL lines are affected by this problem.

We have finally received confirmation of a problem in Verizon's ATM
network:

"The DNOC is going 'router by router' to address the slow thru-put
issues.  Some routers are back up and functioning normally.  Those who
are not, are being worked on.  You will see customers experiencing
problems as they go thru each router.  The estimate is now at 60 hrs
to finish going thru the routers.  The DNOC has asked that no tickets
be called in for slow thru-put until Saturday AM."

According to this notice, high latency and slow throughput for
Telerama ADSL users may continue until Saturday morning.

Further details will be posted here as soon as an update is available.
We apologize for any inconvenience this may cause.


-----------------------------------------------------------------------------


*ADSL Performance Problems: Telerama ADSL Users*

Impact: Medium
Systems Affected: Performance/Throughput for Telerama ADSL users
Status: Ticket opened at Mon Aug 18 20:15:17 EDT 2003

Some of Telerama's ADSL customers are experiencing high latency and
slow throughput at this time, ostensibly due to a congestion problem
with Verizon.  We are currently working to resolve this problem.

Current tests show that there are problems on two specific PVCs and
not all PVCs, which strongly suggests a congestion problem with
Verizon.

Ping times of 1000ms and higher from our DSL network are an indication
of this problem.  Not all ADSL customers are affected, so we are
working to determine which areas have been affected by what looks to
be a change in Verizon's network.  Please contact us if you are having
slow throughput on an unused DSL connection, as it is difficult to
determine whether high ping times indicate a busy DSL line or a line
that is affected by this problem.

Further details will be posted here as soon as an update is available.
We apologize for any inconvenience this may cause.


-----------------------------------------------------------------------------


*Unscheduled Router Reset*

Impact: High for all Telerama users
Systems Affected: All Connectivity (No Route)
Status: Finished at Mon Apr 14 15:34:27 EDT 2003

Our core router needed to be rebooted while debugging our ADSL
assembly this afternoon.  This reset was not scheduled, and we
apologize for any inconvenience this may have caused.  Further tests
will be performed during off-peak times to minimize the chances of
disrupting the network.


-----------------------------------------------------------------------------


*Planned Verizon ADSL Outage*

Impact: High, for ADSL users
Systems Affected: All Connectivity (No Route)
Status: Finished at Tue Mar 25 01:55:50 EST 2003

In order to troubleshoot the ADSL performance problem described below,
Verizon will momentarily disconnect our DSL uplink to perform some
tests.  Two brief outages are expected, each lasting about a minute or
two at the most, while the technician connects and disconnects our DSL
uplink to their analyzer.


-----------------------------------------------------------------------------


*ADSL Performance Problems: Telerama ADSL Users*

Impact: Medium for Telerama ADSL users
Systems Affected: Performance/Throughput
Status: Workaround instated at Fri Mar 21 14:25:00 EST 2003
        Still Debugging at Tue Mar 25 00:57:07 EST 2003


Telerama ADSL users are currently experiencing high latency and slow
throughput.  We are currently working with Verizon to resolve this
issue.  We have not made any recent changes to the DSL configuration
on our router and we are not seeing any errors on our Verizon ATM DSL
uplink.  This leads us to believe that the problem is a
misconfiguration in Verizon's network.  Some of our DSL users have
reported seeing traffic destined for other users, which also leads us
to believe that the problem lies with Verizon.

Further details will be posted were as we continue to work on this
problem.  We apologize for any inconvenience this may cause.


-----------------------------------------------------------------------------


*Emergency Router Maintenance*

Wed Feb  5 01:53:32 EST 2003

Impact: High
Systems Affected: All Internet connectivity.
Status: Finished at Wed Feb  5 05:30:00 EST 2003


We will be performing some emergency maintenance on our core router
this evening to alleviate high ping times and moderate packet loss.
Due to the nature of this problem, we have determined that this an
emergency and should not be scheduled for the Weekly Maintenance
Window on Saturday, from 2am to 6am.  Short losses of connectivity are
to be expected while the work is being done.



------------------------------------------------------------------------------

*Weekly maintenance: E-Mail Server Upgrade, Routing Modifications
Sat Jan 18 01:30:12 EST 2003

Impact: High
Systems Affected: Primary mail server
Status: Finished at Sat Jan 18 04:00:00 EST 2003

During our weekly maintenance window this week, we will be upgrading
the operating system of our mail server.  The mail server will be
completely off-line during the upgrade.  This upgrade is expected to
take approximately 2 hours.


----------------------------------------------------------------------------


*Weekly maintenance: Shell server maintenance.
Sat Nov 8 02:00:00 EDT 2002

Impact: High
Systems Affected: Frogger (shell server)
Status: Planned for Sat Nov 8 02:00:00 EDT 2002

We've encountered inconsistencies on the hard drive of our shell server,
that might be signs of a future hardware failure. Therefore, we've
decided to replace it during our maintenance hours to avoid any risk of
'crash'. This server will be unreachable only few seconds, we'll dump and
restore it while running and then, reset the SCSI bus.

----------------------------------------------------------------------------


*Weekly maintenance: E-Mail Server Upgrade, Routing Modifications
Sat Oct 26 02:11:04 EDT 2002

Impact: High
Systems Affected: All Internet connectivity
Status: Finished at Sat Oct 26 07:10:00 EDT 2002

During our weekly maintenance window this week, we will be upgrading
the storage system of our mail server to increase capacity.  We will
also be making some modifications to our routing to alleviate
congestion on one of our uplinks.

The mail server will be off-line during the upgrade, and the upgrade
is expected to take approximately 2 hours.

UPDATE: It took nearly 4 hours for the mail server's files to copy, we
are sorry for any inconvenience this delay may have caused.

Several brief losses of connectivity are expected during the routing
modiciations, lasting no more than 5 minutes each.


----------------------------------------------------------------------------


*Problems with Dial-Up Numbers
Mon Oct 21 16:29:23 EDT 2002

Impact: Low
Systems Affected: All Dial-up Access Numbers, including 800 number
Status: Repaired by Adelphia, Confirmed Normal (Tue Oct 22 08:34:30 EDT 2002)

Less than 10% of calls into our dial-up lines are failing due to heavy
static on the lines.  If you experience this, simply re-dial.

Adelphia (the carrier for all of dial-up access lines) is currently
investigating the source of the failure.  We expect resolution to
occur quickly.

If you experience REPEATED failure when trying to dial into Telerama's
access numbers, please notify our support staff immediately, by
calling 412 688 3200 (or toll-free 1 877 688 3200).

Adelphia has determined that the source of the problem was a
malfuntioning circuit between their equiptment and the Verizon
network.  It was repaired as of Mon Oct 21 22:43:55 EDT 2002.

----------------------------------------------------------------------------


*Weekly maintenance: Router and Switch OS Upgrades
Sat Sep 14 01:12:58 EDT 2002

Impact: High
Systems Affected: All Internet connectivity
Status: Completed at Sat Sep 14 04:09:51 EDT 2002

During our weekly maintenance window this week, we will be upgrading
IOS on our core and edge routers.  We will also be upgrading our
switch to the latest version of CatOS.

Three brief losses of connectivity are expected, which should last no
more than 5 minutes each.



----------------------------------------------------------------------------


*Weekly maintenance: System-wide upgrades
Sat Sep  7 01:52:03 EDT 2002

Impact: High
Systems Affected: All Internet connectivity
Status: Completed at Sat Sep  7 05:27:22 EDT 2002

During our weekly maintenance window this week, we will be performing
several system-wide upgrades.  This includes an upgrade to one of our
routers and also a switch.

Three or four brief losses of connectivity are expected, lasting no
more than 5 minutes each.



============================================================================


*Second Widespread Power Outage*

Mon Aug 26 21:05:52 EDT 2002

Impact: High
Systems Affected: All Internet Services
Status: Mon Aug 26 23:13:22 EDT 2002

At approximately 9pm on Monday, August 26, 2002, an underground
electrical fire at Baum Blvd. and Millvale Ave. caused a power outage
to the building that Telerama's offices and main data center are
located in.  Most Internet services and phone services were disrupted.

As of 10pm, power was restored to Telerama's building, and services
were restarted.

------------------------------------------------------------------------------


*Widespread Power Outage*

Sat Aug 24 7:14:58 EDT 2002

Impact: High
Systems Affected: All Internet Services
Status: Resolved Sat Aug 24 10:27:07 EDT 2002

At 7:15 this morning, a power outage occurred in the east end of
Pittsburgh, where Telerama is located.  All services operating on
emergency power had to be shut down.  After 9:30am, dial-up service
was restored via our facilities colocated in at another location.

As of 10am, power was restored to Telerama's building, and services
were restarted.

There has been no explanation from Duquesne Light regarding the cause
of the power outage.  

------------------------------------------------------------------------------

*Emergency Email Server Maintaince*

Tue Jul 23 11:55:03 EDT 2002

Impact: High
Systems Affected: All email related services.
Status: Completed

In the past 24 hours, our email server has rebooted spontanously 3
times.  This morning will we be adding a new hard drive, which will
resolve this problem.  We determined that this maintenance should not
be scheduled for the Weekly Maintenance Window on Saturday, from 2am
to 6am.



------------------------------------------------------------------------------

*Widespread Routing Difficulties*

Mon Jul  1 05:19:41 2002

Impact: High
System Affected: Routing to various sites across the Internet.
Status: Resolved

Telerama users are having difficulties reaching some websites and
sending email to various addresses.  Some notable examples of this are
difficulties reaching www.cnn.com and www.ebay.com.

The problem seems to be with some routes that traverse the AT&T
network, but this is still being investigated.

We are currently working with our upstream provider to get this
corrected.  Thank you for your patience.



------------------------------------------------------------------------------

*Mail Server Emergency Maintenance, Pt. II*

Began: Wed May 15 13:23:41 EDT 2002
Ended: Wed May 15 15:16:26 EDT 2002

Impact: High
System affected: Primary email server
Status: Resolved
        

The primary mail server requires additional emergency maintenance
after replacing the failed hard drive last night.  

Incoming email delivery (originating outside of Telerama) will only be
delayed.  There will not be any lost incoming email as a result of
this outage.  Outgoing email delivery (originating inside of Telerama)
will be unavailable while the server is being repaired.  Any mail
reading services, such as IMAP or POP, will also be unavailable during
this time.

We are sorry for any inconvenience this may cause.



------------------------------------------------------------------------------

*Mail Server Emergency Maintenance*

Impact: High
System affected: Primary email server
Status: Started at Wed May 15 03:00:00 EDT 2002
        Ended at Wed May 15 06:00:00 EDT 2002

The primary mail server requires emergency maintenance to replace a
failed hard drive.  Incoming email delivery (originating outside of
Telerama) will only have been delayed.  There will not be any lost
incoming email as a result of this outage.  Outgoing email delivery
(originating inside of Telerama) will be unavailable while the server
is being repaired.  Any mail reading services, such as IMAP or POP,
will also be unavailable during this time.  

We are sorry for any inconvenience this may cause.



-----------------------------------------------------------------------------

*Mail Server Down*

Impact: High
System affected: Primary email server
Status: Started at Mon Apr  8 19:37:59 EDT 2002
        Ended at Mon Apr  8 20:10:27 EDT 2002

The master email server was down during this interval.  This outage
prevented anyone from reading email, and would prevent most people
from sending email as well.  Since the downtime interval was
relatively short, incoming email delivery should only have been
delayed.

There will not be any lost email as a result of this outage.

-----------------------------------------------------------------------------

*Emergency Switch Maintenance*

Impact: High
Systems Affected: All Production Servers
Status: Started at Tue Feb 26 02:52:50 EST 2002
        Finished at Tue Feb 26 05:07:59 EST 2002

Due to minor problems with a switch, we are upgrading the firmware for
the switch connecting our production servers to the Internet.

The switch started experiencing some packet loss this week and
therefore we do not want to delay this upgrade until the weekly
maintenance period.  A brief interruption of all production services
(commercial web hosting, e-mail, login authentication) will be
experienced while the switch is upgraded.

We apologize for any inconvenience this may cause.


-----------------------------------------------------------------------------

*Verizon ADSL Outage: East Liberty CO*

Impact: High for ADSL users wired to the East Liberty CO
Systems Affected: All Connectivity (No Route)
Status: Resoved Tue Feb 12 14:20:33 EST 2002


From: verizon.com
Subject: DSL Network Outage Notification - PITBPAEL - East Liberty,
     PA - 539132NET - Initial

The Verizon Data Network Management Center Communications Team is reporting
a/an
ADSL Network Outage

Location    East Liberty, PA (PITBPAEL)

Customers may be experiencing or reporting no route

The outage started at 10:50:00 AM on 2/12/2002

The potential number of customers affected is 420

The trouble has been initially isolated to the DSLAM

Initial
ASAM IN ISOLATION

The Verizon Network Management ticket number is 539132NET

Comments:
DNOC TECHNICIANS ARE WORKING TO RESOLVETHIS ISSUE.

If you should have any questions, please call the Verizon Data Network
Operations Center at:

1-877-389-7400, option 2



-----------------------------------------------------------------------------

*Emergency Mail Server Maintenance*

Impact: High
Systems Affected: All Email Access
Status: Scheduled for 3am-4am Thu Jan 31 2002
        Started at Thu Jan 31 03:09:22 EST 2002
	Finished at Thu Jan 31 04:26:25 EST 2002

An extra drive is being installed on Telerama's mail server this
evening, in order to process email more efficiently.  The server was
experiencing occasional poor performance in the past week, the new
drive is being installed as soon as it is possible to do so, rather
than wait until the weekly maintenance period.

We apologize for any inconvenience.  


-----------------------------------------------------------------------------

* NEW BETA SHELL BOX NOW ONLINE! *

Mon Jan 14 00:14:48 EST 2002

Telerama is pleased to announce the availability of our new user shell
box, pong.telerama.com.  This machine is now available for open beta
testing by all Telerama users.  Just ssh or telnet to
pong.telerama.com and log in with your regular Telerama username and
password.  This system is still being configured so please send your
questions and comments to support@telerama.com



----------------------------------------------------------------------------

*Maintenance on all 412 Dial Up Lines*
Fri Jan 18 04:04:28 EST 2002

Impact: High
Systems Affected: All Dial up Users in the 412 area code
Status: Completed, Fri Jan 18 08:30:00 EST 2002 

Due to restrictions from our vendors, the upgrades to our 412 dial up
lines must take place outside of our regularly scheduled maintenance
period.  Users may experience frequent disconnects during this
period. We apologize for any inconvience that this may cause you.


----------------------------------------------------------------------------


*Weekly maintenance: Dialup server and router maintenance 
Sun Dec 30 01:30:00 EST 2001

Impact: High
Systems Affected: All Internet connectivity
Status: Finished, Sun Dec 30 05:30:58 EST 2001

During our weekly maintenance window this week, we will be moving our
dialup servers to a new managed power strip.  A single disconnection
will be required while the power cords are moved to their new outlets.

We will also be replacing an Ethernet cable used by our primary
gateway.  Three or four brief losses of connectivity are expected,
lasting no more than 1-2 minutes each.

Finally, we will be moving our SDSL network to another router.  A
brief SDSL outage is expected, approximately 2-5 minutes.  This change
will not affect our ADSL customers.

	
-----------------------------------------------------------------------------

*Instability of Unix Shell Server (frogger.telerama.com)*
Mon Dec 17 12:03:51 EST 2001

Impact: Medium
Systems Affected: All Unix Shell Users, All Support Channel Users
Status: Pending replacement by Telerama technical staff

Our Unix Shell Server is experiencing hardware instability, which
began this weekend.  The server experiencing spontaneous rebooting,
caused by errors on the hard drive.  The hard drive will be replaced
as soon as a new drive can be prepared from backup files.  The ETA for
this repair is late afternoon.


-----------------------------------------------------------------------------

*Weekly maintenance: Router and server memory upgrades
Sat Dec  1 02:00:00 EST 2001

Impact: High
Systems Affected: All Internet connectivity
Status: Maintenance completed at Sat Dec  1 05:11:07 EST 2001

During our weekly maintenance window this week, we will be upgrading
the software on two routers.  A short interruption of all Internet
connectivity is expected while new software revisions are installed.
Each outage should last no longer than 5 to 10 minutes while the
router reboots.

We will also be performing memory upgrades on our nameserver and our
commercial web server.  A short interruption of web and DNS serving
services (less than 5 minutes) is expected while additional memory is
installed in the servers.