incubatec system status

Due to the increase in traffic and to provide you with optimal performance, we will install an additional linecard on our core-router and then reboot the router. The network will be unreachable for a few minutes, but it should be the last upgrade required for the next couple of months.

A causa del volume di traffico aumentato e per garantirVi ottime prestazioni, questa notte a partire da mezzanotte provvederemo ad installare una linecard aggiuntiva sul core-router e poi a riavviare questo. Durante il riavvio la rete sara’ irraggiungibile per un paio di minuti, ma e’ l’ultimo aggiornamento pianificato per il sistema di routing nei prossimi mesi.

Wegen des gestiegenen Traffic-Aufkommens und um optimale Leistung sicherzustellen, werden wir diese Nacht eine zusätzliche Linecard in unseren Routing-Systemen einbauen; anschließend muss der Core-Router neu gestartet werden, was zu einem Downtime von einigen Minuten führen wird. Anschließend sind aber für einige Monate keine weiteren Upgrades geplant oder notwendig.

UPDATE: The upgrade was performed without problems, but the router crashed at 12:03 CET today due to Cisco bug CSCsc53393. We have to switch to another IOS-Version this night with another downtime of about 170 seconds. This is an emergency measure and cannot be delayed.

UPDATE2: Another IOS-Version has been installed and the router rebooted two times after 1:00 o’clock during the night.

UPDATE3: The router crashed again today, the 25th of November 2006 at 13:49 CET and the total outage was an one hour and ten minutes, but we were finally able to find the root-cause of the problem: a memory-stick in the Cisco Catalyst 12016 GSR was defect. We have removed the defect module and will receive a replacement within a few days (no common RAM modules can be mounted in the router). In the meantime the router is working with half the memory as normal, but seems to be stable. Again, we apologize for the inconveniences.

Abbiamo appena attivato un nuovo server di monitoraggio esterno, in questo modo la distribuzione geografica dei sistemi di monitoraggio e’ migliorata.

We will perform maintenance and upgrade operations on our core-router at 0:00 CET and the core-router will be rebootet. The network will be unreachable for a few minutes.

UPDATE: The upgrade was completed successfully, the router took 171 seconds to get online again and it took about 8 minutes to get the full BGP-sessions ready. Now all systems are working perfectly again and should be stable.

This night we have updated our domain-database so that all managed domain-names hosted on our nameservers have a valid SPF record to prevent abuse of our customer’s domain-names by spammers and unauthorized third-parties. Should you encounter any problems or want changes applied to your specific domain-name, do not hesitate to contact us.

Dear customer,

to solve the load-issues caused by the high amount of spam and viruses we have to upgrade the server where your account is hosted.

Therefore from 10 pm CET to 4 am CET the services will not be avaible or avaible only in part.

Thank your for the patience,
sincerely,
Your webhosting24-team
—

Gentile cliente,

per far fronte al numero elevato di virus ed e-mails contenenti spam dobbiamo potenziare drasticamente il server sul quale risiede il Vs account.

Per questo motivo a partire dalle ore 22 fino alle 4 del mattino i servizi non saranno raggiungibili o raggiungibili soltanto in parte.

Grazie per la pazienza,
cordiali saluti,
il Vs team webhosting24
—

Sehr geehrte Kundin, sehr geehrter Kunde,

heute ab 22 Uhr bis 4 Uhr morgens werden wir dringend notwendige Wartungsarbeiten durchführen, um der gestiegenen Last durch SPAM- und Viren-Mails entgegenzuwirken.

Während dieser Zeit werden die Dienste nur sehr eingeschränkt oder gar nicht nutzbar sein.

Vielen Dank für Ihre Verständnis,
mit freundlichen Grüßen,
Ihr webhosting24-Team

This night we will perform various meinteinance works on the shared hosting servers for the webhosting24-service to adapt for the strongely increased spam- and virus-levels. All shared-hosting customers are affected.

Questa notte effettueremo vari lavori di manutenzioni ai servers shared hosting per il servizio webhosting24 necessari per adattare i sistemi al livello in forte rialzo di e-mails contenenti spam e virus. I downtimes e lavori di manutenzione riguardano tutti i clienti shared hosting.

Diese Nacht werden Wartungsarbeiten an allen Shared-Hosting-Servern für den webhosting24-Service durchgeführt, um diese dem stark angestiegenem Aufkommen von SPAM- und Viren-E-Mails anzupassen. Alle Shared-Hosting-Kunden sind davon betroffen.

Due to link flaps the Cogent-uplink was switched to another linecard, Cogent connectivity should be up again in a few minutes. Datacenter connectivity was never affected, as lines are redundant.

Yesterday we experienced a problem on the core-routing system at 14:05 CET in Munich, after controlling all cables, electricity and running diags on the linecards, we had to perform a cold restart of the router to get back online. We’re still investigating the cause of this problem, most probably a software-bug with the Cisco 12016 GSR IOS. After one hour and 40 minutes all systems were back online and fully operative.

Ieri alle ore 14:05 CET i sistemi di core-routing hanno cessato di funzionare a Monaco. Dopo aver controllato cavi, cablaggio, elettricita’ e le varie linecards, abbiamo dovuto togliere e rimettere la corrente al router per ritornare online. Stiamo tuttora indagando la causa esatta del problema, probabilmente si tratta di un software-bug del sistema IOS del router Cisco 12016 GSR. Dopo un ora e 40 minuti tutti i sistemi erano di nuovo online e ripristinati.

Gestern um 14.05 Uhr CET kam es zu einem Totalausfall des Core-Routing-Systems in München. Nachdem wir Kabel, Elektrizität und die versch. Linecards (Hardware) kontrolliert haben, war ein kalter Neustart des Hauptrouters notwendig, um wieder online zu kommen. Wir prüfen immer noch, was die genaue Fehlerursache war, höchstwahrscheinlich ist ein Software-Bug im IOS-Betriebssystem des Cisco 12016 GSR Routers verantwortlich, da keine Hardware-Fehler gefunden wurden. Nach einer Stunde und 40 Minuten waren alle Systeme wieder online und operativ.

[UPDATE]: The problem was due to a heavy increase in the BGP-table, the memory was upgraded. Additional processor memory has been ordered and will installed within 2 weeks.

Gentili clienti,

da ieri a causa di problemi tecnici a Milano di Telecom Italia, la raggiungibilita’ dalla rete ADSL non e’ stabile, inoltre ci sono sporadici problemi con i DNS. Il problema e’ noto ed il NOC di TI ci sta’ gia’ lavorando, ma riteniamo nostro dovere informarVi lo stesso.

Cordiali saluti,
Thomas Moroder
Presidente – incubatec S.r.l.

UPDATE: I problemi di Telecom Italia sono stati segnalati anche su vari altri siti, tra i quali:

http://gioxx.wordpress.com/2006/11/13/oh-mamma-saura-hanno-sifonato-la-telecom/
http://punto-informatico.it/p.aspx?id=1732855&r=PI
http://forum.html.it/forum/showthread.php?s=&threadid=1035963