There are no Open Network Issues Currently
DC-01 Power Maintenance (Resolved) Low

Affecting Other - MTH Data Centre 1

  • 06/03/2023 08:00 - 14/03/2023 12:29
  • Last Updated 06/03/2023 09:54

Starting on the 9th March 2023 and continuing for 4 days, electrical engineers at the MTH DC-01 datacentre will be performing critical upgrade works to the power monitoring systems.

We do not anticipate any interruption to service, and although this work has been categorised as a "LOW" risk, following best practice, engineers will be working on only one power feed at a time.

Indicident Affecting Broadband Services: E16 (Resolved) High

Affecting Other - OFNL Fibre Service: E16 Area

  • 02/03/2022 13:13 - 02/03/2022 19:43
  • Last Updated 08/03/2022 17:45

209491: INCIDENT AFFECTING BROADBAND   Last updated:(02 Mar 2022 17:43hrs)
Status: Resolved.
Location: E16 Heartwell London.
We are currently experiencing a network incident in the Heartwell, E16 area, affecting data services. Engineers are currently investigating the issue and will provide an update as soon as possible or within 2 hours.

Please do not reset your equipment.

We apologise for any inconvenience this may cause.
 
Update (02 Mar 2022 17:43hrs)
This incident was closed on Wednesday, March 2nd, 2022 at 17:43hrs.
 
Update (02 Mar 2022 17:43hrs)
Work to restore service was completed at 17:25 on 02/03/2022. Customers should now find services to be working as expected. We will continue to monitor services to ensure they remain stable.

Any issues being experienced after this time should be treated as individual cases and reported in the usual manner.

We apologise again for the inconvenience caused and thank you for your patience throughout this issue.
 
Update (02 Mar 2022 16:02hrs)
It has been identified that this incident has been caused by an issue with a third party carrier, who provide service to us in this area. OFNL and third party engineers are currently working on fault restoration.

We are working with the provider to resolve this issue and a further update will be provided in 2 hours or as soon as possible.

We apologise again for the inconvenience caused and thank you for your patience.

Indicident Affecting Broadband Services: LN2 (Resolved) High

Affecting Other - OFNL Fibre Service: LN2 Area

  • 02/03/2022 11:28 - 02/03/2022 17:34
  • Last Updated 03/03/2022 09:57

209472: INCIDENT AFFECTING BROADBAND   Last updated:(02 Mar 2022 17:34hrs)
Status: Currently open.
Location: Lincoln, LN2 3.
 
We are currently experiencing a network incident in the Lincoln, LN2 3 area, affecting data services. Engineers are currently investigating the issue and will provide an update as soon as possible or within 2 hours.

Please do not reset your equipment.
 
Update (02 Mar 2022 17:34hrs)
This incident was closed on Wednesday, March 2nd, 2022 at 17:34hrs.
 
Update (02 Mar 2022 17:34hrs)
Work to restore service by our Third Party Carrier was completed at 16:34 on 02/03/2022. Customers should now find services to be working as expected. We will continue to monitor services to ensure they remain stable.

Any issues being experienced after this time should be treated as individual cases and reported in the usual manner.

We apologise again for the inconvenience caused and thank you for your patience throughout this issue.
 
Update (02 Mar 2022 16:13hrs)
Engineers from the third party carrier have found a fault on their network and are continuing to work to resolve this issue. A further update will be provided as soon as possible.

We apologise again for the inconvenience caused and thank you for your patience.

Update (02 Mar 2022 13:38hrs)
It has been identified that this incident has been caused by an issue with a third party carrier, who provide service to us in this area.

We are working with the provider to resolve this issue and a further update will be provided as soon as possible.

We apologise again for the inconvenience caused and thank you for your patience.

 

Indicident Affecting Broadband Services (Resolved) Critical

Affecting Other - OFNL Fibre Service: Multiple Areas

  • 24/11/2021 15:09 - 24/11/2021 20:00
  • Last Updated 02/03/2022 16:21

200693:INCIDENT AFFECTING BROADBAND SERVICES   Last updated:(24 Nov 2021 15:47hrs)
Status: Currently open.
Location: TN39,SP3,CT21,BN17,RG1,RG30,SO22.
We are currently experiencing a network incident in all areas stated above affecting data services. Engineers are currently investigating the issue and will provide an update as soon as possible or within 2 hours.

Please do not reset your equipment.

We apologise for any inconvenience this may cause.
 
Update (24 Nov 2021 20:00hrs)
This incident was closed on Wednesday, November 24th, 2021 at 20:00hrs.
 
Update (24 Nov 2021 20:00hrs)
Work to restore service by our third party carrier was completed at 17:11 on 24/11/21. Customers should now find services to be working as expected. We will continue to monitor services to ensure they remain stable.

Any issues being experienced after this time should be treated as individual cases and reported in the usual manner.

We apologise again for the inconvenience caused and thank you for your patience throughout this issue.
 
Update (24 Nov 2021 17:48hrs)
Our provider is continuing to work on their network equipment, towards a resolution for this issue, and a further update will be provided as soon as possible.

We apologise again for the inconvenience caused and thank you for your patience.
 
Update (24 Nov 2021 15:53hrs)
It has been identified that this incident has been caused by an issue with a third party carrier, who provide service to us in these areas.

We are working with the provider to resolve this issue and a further update will be provided in 2 hours or as soon as possible.

We apologise again for the inconvenience caused and thank you for your patience.

Indicident Affecting Broadband Services (Resolved) Critical

Affecting Other - OFNL Fibre Service: Multiple Areas

  • 03/11/2021 08:19
  • Last Updated 02/03/2022 16:20

Hello,

Please see below the recent information regarding the internet for Colchester CO3, Sudbury C010, Northamptonshire NN18, Bedfordshire SG17, SG16, MK54.

 

Update (03 Nov 2021 09:12hrs)
This incident was closed on Wednesday, November 3rd, 2021 at 09:12hrs.

 

Update (03 Nov 2021 03:40hrs)

Work to restore service by our supplier Vodafone in the Bedfordshire SG17, SG16, MK54 areas was completed at 02:35 on 03/11/21. Customers should now find services to be working as expected. We will continue to monitor services to ensure they remain stable.

Work to restore service by our supplier Sky in the Sudbury C010 and Northamptonshire NN18 areas is continuing.

We apologise again for the inconvenience caused and thank you for your patience throughout this issue.

Update (03 Nov 2021 00:29hrs)

It has been identified that this incident has been caused by an issue with multiple 3rd party carriers, who provide service to us in these areas.

We are working with the providers to resolve this issue and a further update will be provided as soon as possible.

We apologise again for the inconvenience caused and thank you for your patience.

Update (02 Nov 2021 23:19hrs)

Engineers are continuing to work to resolve this issue and a further update will be provided within 2 hours or as soon as possible.

We apologise again for the inconvenience caused and thank you for your patience.

Indicident Affecting Broadband Services: BN24 (Resolved) High

Affecting Other - OFNL Fibre Service: BN24 Area

  • 23/02/2022 09:32 - 24/02/2022 13:34
  • Last Updated 02/03/2022 16:20

208380:INCIDENT AFFECTING BROADBAND AND VOICE SERVICES   Last updated:(24 Feb 2022 13:34hrs)
Status: Closed.
Location: Pevensey, BN24.
 
We are currently experiencing a network incident in BN24 affecting all services.

Engineers are aware that after a recent power outage in this area UKPN restored power to all properties at 03:30am this morning. however power has not returned to our equipment which provides fibre connection to this site.

We have engaged with UKPN and we hope to have this resolved shortly.

Please do not reset your equipment.

We apologise for any inconvenience this may cause.
 
Update (24 Feb 2022 12:35hrs)
Third party engineers have restored temporary power to our equipment. This is currently being powered by a generator. Services are starting to restore. However services are still deemed at risk of further power outages when we switch back to mains supply.

Please do not reset your equipment.

We apologise for any inconvenience this may cause.
 
Update (24 Feb 2022 08:22hrs)
UK-Power Networks have attended the area carrying out a further temporary fix to return supply.
This has unfortunately been unsuccessful.

Multiple escalations have been made, however at this time they aim to be back working on this fault tonight at 23:30 tonight.

Every effort possible is being made to source a temporary power supply to support our equipment to bring services back online.

Further updates will be provided when they become available.

Indicident Affecting Broadband Services: TQ (Resolved) Critical

Affecting Other - OFNL Fibre Service: TQ Area

  • 29/10/2021 09:23
  • Last Updated 02/03/2022 16:20

OFNL are currently having issues with broadband in the TQ6, TQ7, TQ10, TQ12 area.

Any updates will be posted here.


Update (29 Oct 2021 12:04hrs)
The issue with the 3rd party carrier has been resolved and work to restore service was completed at 11:03 on 29/10/2021. Customers should now find services to be working as expected. We will continue to monitor services to ensure they remain stable.

Any issues being experienced after this time should be treated as individual cases and reported in the usual manner.

We apologise again for the inconvenience caused and thank you for your patience throughout this issue.


Update (29 Oct 2021 11:00hrs)
It has been identified that this incident has been caused by an issue with a 3rd party carrier, who provide service to us in this area.

We are working with the provider to resolve this issue and a further update will be provided within 2 hours.

We apologise again for the inconvenience caused and thank you for your patience.


198211: INCIDENT AFFECTING BROADBAND SERVICES   Last updated:(29 Oct 2021 09:15hrs)
Status: Currently open.
Location: Kingsbridge(TQ7), Newton Abbot(TQ12), Dartmouth(TQ6) and South Brent(TQ10).
We are currently experiencing a network incident in Kingsbridge(TQ7), Newton Abbot(TQ12), Dartmouth(TQ6) and South Brent(TQ10) affecting data services. Engineers are currently investigating the issue and will provide an update as soon as possible or within 2 hours.

Please do not reset your equipment.

We apologise for any inconvenience this may cause.

Indicident Affecting Broadband Services (Resolved) High

Affecting Other - OFNL Fibre Service: HG3

  • 21/02/2022 21:21 - 23/02/2022 10:04
  • Last Updated 23/02/2022 10:04

208692: INCIDENT AFFECTING BROADBAND SERVICES   Last updated:(23 Feb 2022 09:43hrs)
Status: Closed.
Location: HG3 2 HARROGATE.
 
We are currently experiencing a network incident in the HG3 2 Harrogate area affecting data services. Engineers are currently investigating the issue and will provide an update as soon as possible or within 2 hours.

Please do not reset your equipment.

We apologise for any inconvenience this may cause.
 
Update (23 Feb 2022 09:43hrs)
This incident was closed on Wednesday, February 23rd, 2022 at 09:43hrs.
 
Update (23 Feb 2022 09:43hrs)
Work to restore service was completed at 22:34 on 21/02/2022. Customers should now find services to be working as expected. We will continue to monitor services to ensure they remain stable.

Any issues being experienced after this time should be treated as individual cases and reported in the usual manner.

We apologise again for the inconvenience caused and thank you for your patience throughout this issue.

Indicident Affecting Broadband Services (Resolved) High

Affecting Other - OFNL Fibre Service: BN24

  • 21/02/2022 08:32 - 21/02/2022 15:57
  • Last Updated 22/02/2022 09:24

208380: INCIDENT AFFECTING BROADBAND AND VOICE SERVICES   Last updated:(21 Feb 2022 15:57hrs)
Status: Currently open.
Location: Stone Cross (BN24).
We are currently experiencing a network incident in Pevensey (BN24) affecting voice and data services. Engineers are currently investigating the issue and will provide an update as soon as possible or within 2 hours.

Please do not reset your equipment.

We apologise for any inconvenience this may cause.
 
Update (21 Feb 2022 15:57hrs)
This incident was closed on Monday, February 21st, 2022 at 15:57hrs.
 
Update (21 Feb 2022 12:19hrs)
Engineers have visited the site and found no defects with our equipment. The issue is due to an ongoing power outage in the area, that's scheduled to be resolved by UKPN between 15:30 and 16:30 21/02/2022.
 
Update (21 Feb 2022 10:20hrs)
Engineers have been investigating the problem and found that the issue has been caused by a power outage in the area. OFNL engineers are en route to site to investigate further.

We are continuing to work to resolve this issue and a further update will be provided as soon as possible.

We apologise again for the inconvenience caused and thank you for your patience.

Indicident Affecting Broadband Services (Resolved) High

Affecting Other - OFNL Fibre Service: IP30

  • 21/02/2022 08:25 - 21/02/2022 14:09
  • Last Updated 21/02/2022 15:25

208545: INCIDENT AFFECTING BROADBAND SERVICES   Last updated:(21 Feb 2022 14:09hrs)
Status: Closed.
Location: Woolpit (IP30).
 
We are currently experiencing a network incident in Woolpit (IP30) affecting data services. Engineers are currently investigating the issue and will provide an update as soon as possible or within 2 hours.

Please do not reset your equipment.

We apologise for any inconvenience this may cause.
 
Update (21 Feb 2022 14:09hrs)
This incident was closed on Monday, February 21st, 2022 at 14:09hrs.
 
Update (21 Feb 2022 10:45hrs)
The local power issue has been resolved and work to restore service was completed at 09:56 on 21/02/2022. Customers should now find services to be working as expected. We will continue to monitor services to ensure they remain stable.

Any issues being experienced after this time should be treated as individual cases and reported in the usual manner.

We apologise again for the inconvenience caused and thank you for your patience throughout this issue.

Indicident Affecting Broadband Services (Resolved) High

Affecting Other - OFNL Fibre Service: RH13

  • 18/02/2022 15:28 - 20/02/2022 11:39
  • Last Updated 21/02/2022 11:39

208413 : INCIDENT AFFECTING BROADBAND AND VOICE SERVICES   Last updated:(18 Feb 2022 15:28hrs)
Status: Currently open.
Location: Horsham RH13.
 
We are currently experiencing a network incident in Horsham affecting data services. Engineers are currently investigating the issue and will provide an update as soon as possible or within 2 hours.

Please do not reset your equipment.

We apologise for any inconvenience this may cause.
 
- No updates for this incident yet.
 
Update (18 Feb 2022 16:09hrs)
It has been identified that this incident has been caused by an issue with a third party carrier, who provide service to us in this area. They have advised this issue has been caused by loss of power to their equipment, due to the storm.

We are working with the provider to resolve this issue and a further update will be provided in 2 hours or as soon as possible.

We apologise again for the inconvenience caused and thank you for your patience.

Indicident Affecting Broadband Services (Resolved) High

Affecting Other - OFNL Fibre Service: CO7, CO13, CO14, CO16

  • 09/01/2022 15:19 - 09/01/2022 18:12
  • Last Updated 18/02/2022 15:50

204649:INCIDENT AFFECTING BROADBAND SERVICES   Last updated:(09 Jan 2022 18:12hrs)
Status: Closed.
Location: Essex, CO7. Walton on the Naze CO14. Essex, CO13. Weeley Heath, CO16. Great Bentley, CO7..
We are currently experiencing a network incident in Essex, CO7. Walton on the Naze CO14. Essex, CO13. Weeley Heath, CO16. Great Bentley, CO7 affecting data services. Engineers are currently investigating the issue and will provide an update as soon as possible or within 2 hours.

Please do not reset your equipment.

We apologise for any inconvenience this may cause.
 
Update (09 Jan 2022 18:12hrs)
This incident was closed on Sunday, January 9th, 2022 at 18:12hrs.
 
Update (09 Jan 2022 18:11hrs)
Work to restore service was completed at 17:16 on 09/01/2022. Customers should now find services to be working as expected. We will continue to monitor services to ensure they remain stable.

Any issues being experienced after this time should be treated as individual cases and reported in the usual manner.

We apologise again for the inconvenience caused and thank you for your patience throughout this issue.
 
Update (09 Jan 2022 16:03hrs)
It has been identified that this incident has been caused by an issue with a third party carrier, who provide service to us in this area.

We are working with the provider to resolve this issue and a further update will be provided in 4 hours or as soon as possible.

We apologise again for the inconvenience caused and thank you for your patience.
 
 

Indicident Affecting Broadband Services (Resolved) High

Affecting Other - OFNL Fibre Service: TS21

  • 12/01/2022 00:32 - 12/01/2022 04:08
  • Last Updated 18/02/2022 15:50

204936:INCIDENT AFFECTING BROADBAND SERVICES   Last updated:(12 Jan 2022 04:08hrs)
Status: Closed.
Location: Cleveland, TS21.
We are currently experiencing a network incident in Cleveland, TS21 affecting data services. Engineers are currently investigating the issue and will provide an update as soon as possible or within 2 hours.

Please do not reset your equipment.

We apologise for any inconvenience this may cause.
 
Update (12 Jan 2022 04:08hrs)
This incident was closed on Wednesday, January 12th, 2022 at 04:08hrs.
 
Update (12 Jan 2022 04:08hrs)
Work to restore service was completed at 02:42 on 12/01/2022. Customers should now find services to be working as expected. We will continue to monitor services to ensure they remain stable.

Any issues being experienced after this time should be treated as individual cases and reported in the usual manner.

We apologise again for the inconvenience caused and thank you for your patience throughout this issue.
 
Update (12 Jan 2022 01:59hrs)
It has been identified that this incident has been caused by an issue with a third party carrier, who provide service to us in this area.

We are working with the provider to resolve this issue and a further update will be provided as soon as possible.

We apologise again for the inconvenience caused and thank you for your patience.
 

 

Indicident Affecting Broadband Services (Resolved) High

Affecting System - OFNL Fibre Service: DY3

  • 15/01/2022 23:55 - 20/01/2022 13:33
  • Last Updated 18/02/2022 15:49

205303: INCIDENT AFFECTING BROADBAND AND VOICE SERVICES   (Last updated: 18 Jan 2022 12:13hrs)
Status: Resolved.
Location: Wolverhampton (DY3).
We are currently experiencing a network incident in Wolverhampton (WV5) affecting voice and data services. Engineers are currently investigating the issue and will provide an update as soon as possible.

Please do not reset your equipment.

We apologise for any inconvenience this may cause.
 
Update (18 Jan 2022 12:13hrs)
Engineers on site are continuing to work on restoring service to the New cabinet.

We are continuing to work to resolve this issue and a further update will be provided as soon as possible.

We apologise again for the inconvenience caused and thank you for your patience.
 
Update (17 Jan 2022 23:44hrs)
Engineers will be attending the site again tomorrow morning 18/01/2022 to finish setting up the new cabinet.

We are continuing to work to resolve this issue and a further update will be provided as soon as possible.

We apologise again for the inconvenience caused and thank you for your patience.
 
Update (17 Jan 2022 18:00hrs)
Engineers have attended the site and a new cabinet has been installed. Work is now underway to restore service to the remaining components.

We are continuing to work to resolve this issue and a further update will be provided as soon as possible.

We apologise again for the inconvenience caused and thank you for your patience.
 
Update (17 Jan 2022 14:54hrs)
Contractors have recently arrived on site, working alongside Electrical Engineers to install the new cabinet.

We are continuing to work to resolve this issue and a further update will be provided as soon as possible.

We apologise again for the inconvenience caused and thank you for your patience.
 
Update (17 Jan 2022 10:39hrs)
Engineers are due on site at 14:00 to begin install of New Fibre Cabinet.

We are continuing to work to resolve this issue and a further update will be provided as soon as possible.

We apologise again for the inconvenience caused and thank you for your patience.
 
Update (16 Jan 2022 10:32hrs)
Our suppliers have been made aware of the incident and work is underway to ensure that a replacement cabinet is ready as soon as possible. Further updates will be provided as soon as we have new information.

We apologise again for the inconvenience caused and thank you for your patience.
 
Update (16 Jan 2022 03:17hrs)
Field engineers have attended the site and found that a vehicle has collided with the OFNL street cabinet and all equipment has been severely damaged. Due to the serious nature of the incident, we do not have a service restoration estimate at this time.
 
Update (15 Jan 2022 23:56hrs)
Field engineers have been dispatched to the site to investigate further, ETA 01:45.

We are continuing to work to resolve this issue and a further update will be provided as soon as possible.

We apologise again for the inconvenience caused and thank you for your patience.

Indicident Affecting Broadband Services (Resolved) High

Affecting System - OFNL Fibre Service: E20

  • 31/01/2022 10:37 - 31/01/2022 14:48
  • Last Updated 18/02/2022 15:49

206490: INCIDENT AFFECTING BROADBAND AND VOICE SERVICES   Last updated:(31 Jan 2022 14:48hrs)
Status: Resolved.
Location: Stratford (E20).
We are currently experiencing a network incident in Stratford (E20) affecting voice and data services. Engineers are currently investigating the issue and will provide an update as soon as possible or within 2 hours.

Please do not reset your equipment.

We apologise for any inconvenience this may cause.
 
Update (31 Jan 2022 14:48hrs)
This incident was closed on Monday, January 31st, 2022 at 14:48hrs.
 
Update (31 Jan 2022 14:48hrs)

It has been identified that this incident had been caused by an issue with a third party carrier, who provide service to us in this area.

Work to restore service was completed at 12:22 on 31/01/22. Customers should now find services to be working as expected. We will continue to monitor services to ensure they remain stable.

Any issues being experienced after this time should be treated as individual cases and reported in the usual manner.

We apologise again for the inconvenience caused and thank you for your patience throughout this issue.
 
Update (31 Jan 2022 14:17hrs)
Engineers have arrived at the site to continue investigating the brief loss of services earlier. A further update will be provided within 2 hours or as soon as possible.

We apologise again for the inconvenience caused and thank you for your patience.
 
Update (31 Jan 2022 12:20hrs)
Engineers are continuing investigations into the problem. We are continuing to work to resolve this issue and a further update will be provided within 2 hours or as soon as possible.

We apologise again for the inconvenience caused and thank you for your patience.

Indicident Affecting Broadband Services (Resolved) High

Affecting Other - OFNL Fibre Service: SG8

  • 04/01/2022 10:06 - 05/01/2022 16:48
  • Last Updated 18/02/2022 15:49

204196: INCIDENT AFFECTING BROADBAND SERVICES   Last updated:(04 Jan 2022 09:36hrs)
Status: Resolved.
Location: Royston (SG8).
 
We are currently experiencing a network incident in Royston (SG8) affecting data services. Engineers are currently investigating the issue and will provide an update as soon as possible or within 2 hours.

Please do not reset your equipment.

We apologise for any inconvenience this may cause.
 
 
Update (04 Jan 2022 16:48hrs)
This incident was closed on Tuesday, January 4th, 2022 at 16:48hrs.
 
Update (04 Jan 2022 16:48hrs)
Work to restore service was completed at 12:57 on 04/01/2022. Customers should now find services to be working as expected. We will continue to monitor services to ensure they remain stable.

Any issues being experienced after this time should be treated as individual cases and reported in the usual manner.

We apologise again for the inconvenience caused and thank you for your patience throughout this issue.
 
Update (04 Jan 2022 15:20hrs)
We are seeing services have been stable since 12:57 and we are requesting a further update from our third party carrier.

We are working with the provider to resolve this issue and a further update will be provided within 2 hours.

We apologise again for the inconvenience caused and thank you for your patience.
 
Update (04 Jan 2022 13:20hrs)
Engineers have been investigating the problem and found there is an intermittent power issue in the third party carrier's network feeding this site. Third party engineers have been tasked to investigate.

We are continuing to work to resolve this issue and a further update will be provided within 2 hours or as soon as possible.

We apologise for any inconvenience this may cause.
 
Update (04 Jan 2022 11:28hrs)
It has been identified that this incident has been caused by an issue with a third party carrier, who provide service to us in this area.

We are working with the provider to resolve this issue and a further update will be provided in 2 hours or as soon as possible.

We apologise again for the inconvenience caused and thank you for your patience.

Indicident Affecting Broadband Services (Resolved) Critical

Affecting Other - OFNL Fibre Service: SN3

  • 09/12/2021 11:58 - 15/12/2021 15:23
  • Last Updated 18/02/2022 15:48

MTH Networks are aware of an issue on the OFNL Network, please see below from OFNL site acknowledging the issue.
 
202222: INCIDENT AFFECTING BROADBAND SERVICES   Last updated:(09 Dec 2021 11:58hrs)
Status: Resolved.
Location: Swindon (SN3).
We are currently experiencing a network incident in Swindon (SN3) affecting data services. Engineers are currently investigating the issue and will provide an update as soon as possible or within 2 hours.

Please do not reset your equipment.

We apologise for any inconvenience this may cause.
 
Update (09 Dec 2021 16:03hrs)
This incident was closed on Thursday, December 9th, 2021 at 16:03hrs.
 
Update (09 Dec 2021 15:52hrs)
Our third party carrier has resolved the issue and work to restore service was completed at 15:32 on 09/12/2021. Customers should now find services to be working as expected. We will continue to monitor services to ensure they remain stable.

Any issues being experienced after this time should be treated as individual cases and reported in the usual manner.

We apologise again for the inconvenience caused and thank you for your patience throughout this issue.
 
Update (09 Dec 2021 13:51hrs)
It has been identified that this incident has been caused by an issue with a third party carrier, who provide service to us in this area.

We are working with the provider to resolve this issue and a further update will be provided within 2 hours.

We apologise again for the inconvenience caused and thank you for your patience.

Indicident Affecting Broadband Services (Resolved) High

Affecting Other - OFNL Fibre Service: TN12

  • 18/02/2022 13:10 - 18/02/2022 15:29
  • Last Updated 18/02/2022 15:47

208394: INCIDENT AFFECTING BROADBAND AND VOICE SERVICES   Last updated:(18 Feb 2022 15:29hrs)
Status: Closed.
Location: Tonbridge (TN12).
 
We are currently experiencing a network incident in Tonbridge (TN12) affecting voice and data services. Engineers are currently investigating the issue and will provide an update as soon as possible or within 2 hours.

Please do not reset your equipment.

We apologise for any inconvenience this may cause.
 
Update (18 Feb 2022 15:29hrs)
This incident was closed on Friday, February 18th, 2022 at 15:29hrs.
 
Update (18 Feb 2022 15:09hrs)
Work to restore service was completed at 14:49 on 18/02/2022. Customers should now find services to be working as expected. We will continue to monitor services to ensure they remain stable.

Any issues being experienced after this time should be treated as individual cases and reported in the usual manner.

We apologise again for the inconvenience caused and thank you for your patience throughout this issue.

Indicident Affecting Broadband Services (Resolved) High

Affecting Other - OFNL Fibre Service: GL50

  • 17/02/2022 01:17 - 17/02/2022 04:59
  • Last Updated 17/02/2022 16:59

208211: INCIDENT AFFECTING BROADBAND SERVICES   Last updated:(17 Feb 2022 05:15hrs)
Status: Closed.
Location: Cheltenham (GL50).
We are currently experiencing a network incident in Cheltenham (GL50) affecting data services. Engineers are currently investigating the issue and will provide an update as soon as possible.

Please do not reset your equipment.
We apologise for any inconvenience this may cause.
 
Update (17 Feb 2022 05:15hrs)
This incident was closed on Thursday, February 17th, 2022 at 05:15hrs.
 
Update (17 Feb 2022 05:15hrs)
Work to restore service was completed at 04:59 on 17/02/2022. Customers should now find services to be working as expected. We will continue to monitor services to ensure they remain stable.

Any issues being experienced after this time should be treated as individual cases and reported in the usual manner.

We apologise again for the inconvenience caused and thank you for your patience throughout this issue.
 
Update (17 Feb 2022 02:56hrs)
Engineers have been investigating the problem and found that the issue has been caused by electrical maintenance work in the area, resulting in loss of power to our equipment. We have engaged with the supplier and work is underway to restore
power. We expect this to be completed by 08:00 17/02/2022.

We apologise again for the inconvenience caused and thank you for your patience.

Emergency Maintenance Affecting Broadband Services (Resolved) High

Affecting Other - OFNL Fibre Service: NN5

  • 14/02/2022 08:00 - 15/02/2022 16:00
  • Last Updated 15/02/2022 17:09

205775: EMERGENCY MAINTENANCE AFFECTING BROADBAND AND VOICE SERVICES   Last updated:(14 Feb 2022 16:00hrs)
Status: Closed.

Location: Upton Park, NN5.
There is emergency maintenance taking place between the hours of 08:00 on 14/02/22 and 16:00 on 15/02/22. Customers may experience an interruption to all services for up to 2 hours during this time.

Please do not reset your equipment during this period.

We apologise for the inconvenience this may cause.
No updates for this incident yet.

Indicident Affecting Broadband Services (Resolved) Critical

Affecting System - OFNL Fibre Service: NR14

  • 11/02/2022 07:17 - 11/02/2022 08:17
  • Last Updated 11/02/2022 09:17

207651: INCIDENT AFFECTING BROADBAND SERVICES   Last updated:(11 Feb 2022 08:19hrs)
Status: Closed.
Location: Loddon NR14.
We are currently experiencing a network incident in Loddon NR14 affecting data services. Engineers are currently investigating the issue and will provide an update as soon as possible or within 2 hours.

Please do not reset your equipment.

We apologise for any inconvenience this may cause.
 
Update (11 Feb 2022 08:19hrs)
This incident was closed on Friday, February 11th, 2022 at 08:19hrs.
 
Update (11 Feb 2022 08:15hrs)
Work to restore service was completed at 07:40 on 11/02/2021. Customers should now find services to be working as expected. We will continue to monitor services to ensure they remain stable.

Any issues being experienced after this time should be treated as individual cases and reported in the usual manner.

We apologise again for the inconvenience caused and thank you for your patience throughout this issue.
 
Update (11 Feb 2022 08:13hrs)
It has been identified that this incident has been caused by an issue with a third party carrier, who provide service to us in this area.

We are working with the provider to resolve this issue and a further update will be provided in 2 hours or as soon as possible.

We apologise again for the inconvenience caused and thank you for your patience.

Indicident Affecting Broadband Services (Resolved) High

Affecting System - OFNL Fibre Service: BS4

  • 07/02/2022 08:00 - 07/02/2022 21:50
  • Last Updated 08/02/2022 09:10

207130 : INCIDENT AFFECTING BROADBAND SERVICES   Last updated:(07 Feb 2022 21:50hrs)
Status: Resolved.
Location: Bedminster BS4.
We are currently experiencing a network incident in Bedminster affecting data services. Engineers are currently investigating the issue and will provide an update as soon as possible or within 2 hours.

Please do not reset your equipment.

We apologise for any inconvenience this may cause.
 
Update (07 Feb 2022 21:50hrs)
This incident was closed on Monday, February 7th, 2022 at 21:50hrs.
 
Update (07 Feb 2022 21:50hrs)
Work to restore service was completed at 21:23 on 07/02/2022. Customers should now find services to be working as expected. We will continue to monitor services to ensure they remain stable.

Any issues being experienced after this time should be treated as individual cases and reported in the usual manner.

We apologise again for the inconvenience caused and thank you for your patience throughout this issue.
 
Update (07 Feb 2022 20:55hrs)
Third party engineers are expecting a replacement part on site shortly and are continuing to work to restore service.

We are continuing to work to resolve this issue and a further update will be provided within 2 hours or as soon as possible.
 
Update (07 Feb 2022 18:27hrs)
Third party engineers are continuing to work to restore service.

We are continuing to work to resolve this issue and a further update will be provided within 2 hours or as soon as possible.
 
Update (07 Feb 2022 16:28hrs)
Third party engineers are on-site and continuing to work to restore service.

We are continuing to work to resolve this issue and a further update will be provided within 2 hours or as soon as possible.
 
Update (07 Feb 2022 14:44hrs)
Third party engineers are continuing to work to restore service.

We are continuing to work to resolve this issue and a further update will be provided within 2 hours or as soon as possible.
 
Update (07 Feb 2022 11:54hrs)
Engineers have been investigating the problem and found that a Power outage to third party equipment has caused the outage. We are continuing to work to resolve this issue and a further update will be provided within 2 hours or as soon as possible.

We apologise again for the inconvenience caused and thank you for your patience.
 
Update (07 Feb 2022 09:46hrs)
It has been identified that this incident has been caused by an issue with a third party carrier, who provide service to us in this area.

We are working with the provider to resolve this issue and a further update will be provided in 2 hours or as soon as possible.

We apologise again for the inconvenience caused and thank you for your patience.

Indicident Affecting Broadband Services (Resolved) Critical

Affecting Other - OFNL Fibre Service: B15

  • 15/12/2021 12:15
  • Last Updated 07/02/2022 14:37

MTH Networks are aware of an issue on the OFNL Network, please see below from OFNL site acknowledging the issue.
 
202842: INCIDENT AFFECTING BROADBAND SERVICES   Last updated:(15 Dec 2021 12:15hrs)
Status: Resolved.
Location: Birmingham, B15.
We are currently experiencing a network incident in Broadway, Birmingham affecting data services. Engineers are currently investigating the issue and will provide an update as soon as possible or within 2 hours.

Please do not reset your equipment.

We apologise for any inconvenience this may cause.
No updates for this incident yet.

OFNL Network Issue (Resolved) Critical

Affecting System - OFNL Network

  • 06/12/2021 12:42 - 15/12/2021 15:24
  • Last Updated 07/12/2021 18:46

MTH Networks are aware of an issue on the OFNL Network, please see below from OFNL site acknowledging the issue.

201324: INCIDENT AFFECTING BROADBAND SERVICES   Last updated:(06 Dec 2021 13:52hrs)
Status: Currently open.
Location: Swindon (SN1), Stratford (E20), Littlehampton (BN17), Upper Heyford (OX25), Greenwich M.V. (SE10).
OFNL are aware of extremely high demands on our network during peak times and some customers are experiencing a degradation of their internet service. We have verified the performance within the OFNL network and we are working closely with our upstream service providers to improve the issue.
Update (07 Dec 2021 17:07hrs)
Work will be taking place tonight to make further improvements as part of our ongoing investigation. Some customers may experience a short disruption to their service as part of this process.

We apologise for the inconvenience caused and thank you for your patience.
Update (06 Dec 2021 16:07hrs)
We are continuing to work with our carrier to investigate and improve upstream performance affecting some customers.

We apologise for the inconvenience caused and thank you for your patience.

Datacentre Connectivity Issues (Resolved) Critical

Affecting Other - MTH-Datacentre 1

  • 17/11/2021 14:38 - 24/11/2021 16:09
  • Last Updated 24/11/2021 16:08

To our Valued MTH Clients

We are currently experiencing DataCentre connectivity issues at our main site: MTH-Datacentre 1

A team is on site investigating currently.

An update will be provided as soon as possible.

Kind Regards,
Your MTH Team

-- update as of 15:10
Our on-site team has restored connectivity to our services.

-- update as of 15:13
A full RFO can be available on request, a faulty Fibre Module was to blame here - which was replaced as soon as possible.

Shared Hosting Server Issue (Resolved) High

Affecting Server - MTH SSD Host

  • 17/08/2021 08:53 - 17/08/2021 16:00
  • Last Updated 12/10/2021 12:59

Good Morning,


We are experiencing high loads on our Shared Hosting server. This is being investigated.

-- updated as of 09:30 - we are narrowing down rogue processes causing CPU Spikes

Good Afternoon,

The issues surrounding our Shared Hosting Server look to be ironed out now and we're seeing flattened resource usage.

Please accept our apologies for any inconvenience caused.

Datacentre Planned Maintenance & Updates (Resolved) High

Affecting Other - MTH-Datacentre 1

  • 30/07/2021 22:00 - 30/07/2021 23:00
  • Last Updated 10/08/2021 12:26

To all our Valued Clients,

To continually offer our high level of system uptime, and server/network performance, common maintenance and updates are required.

Our next Update/Maintenance Window is on the 30/07/2021 between 22:00 and 23:00.

There will be a period of 30 minutes downtime during this window.

Should you have any questions or concerns ahead of this window, please raise a ticket from your Client Area or email us at [email protected]

We will be available during this Window to respond to any queries if necessary!

Kind Regards,
Your MTH Team

 

-- update as of 31/07/2021
The scheduled maintenance and upgrades were completed with no issues!

Planned Engineering Works (Resolved) High

Affecting System - MTH VoIP

  • 18/05/2020 00:00 - 29/06/2020 00:00
  • Last Updated 30/07/2021 15:58

Dear Clients,

As part of our ongoing program of MTH VoIP platform expansion and improvement, we will be expanding and upgrading the edge cluster of access servers to the MTH VoIP network. This will allow us to accommodate for further platform growth along with our clients and increasing platform demand, as well as introduce improved resilience and giving greater flexibility in how we handle calls and handset registration traffic.

The schedule of upgrade works will commence on the 18th May 2020 and is expected to take six weeks.

There are no actions required for users; handsets will automatically receive updates overnight and refresh their configuration to take advantage of the platform.

Most importantly, we don't expect any impact to service due to these works.

If you have any questions, please do raise a support ticket from your MTH Client Area.

Kind Regards,

Your MTH Networks Team

Datacentre Connectivity Issues (Resolved) High

Affecting Other - MTH-DC-3

  • 11/06/2021 11:37
  • Last Updated 21/07/2021 12:39

To our Valued Customers,

We are experiencing connectivity issues out of our Coventry DataCentre.

This is affecting inbound and outbound traffic of VPS' and other services based here.

This is being investigated by the onsite team, an update will be provided as soon as possible.

Kind Regards,
Your MTH Team

 

-- update as of 12:20
Services look to be restoring, we are monitoring for stability and investigating the root cause.

Scheduled Shared Host Maintenance & Downtime (Resolved) High

Affecting Server - MTH SSD Host

  • 26/06/2021 22:00 - 26/06/2021 22:30
  • Last Updated 26/06/2021 22:37

To Our Valued Shared Hosting/Reseller customers,

 

To ensure we continue to offer our high level of service, we consistently maintain and upgrade our servers.

 

Some maintenance does require more extensive work and downtime.

 

Our next maintenance window is tomorrow evening (Saturday 26th of June), where a reboot will take place at 22:00. We only envisage up to 15 minutes downtime, however the 'at risk' window will be up to 30 minutes.

 

If you have any queries regarding this, please do not hesitate to raise a support ticket with us by emailing [email protected].

 

Please may we also point you to our Network Status page, where you can keep up to date with all current and upcoming updates across our Infrastructure. This can be reached by navigating to: https://secure.monkeytreehosting.com/serverstatus.php

 

Kind Regards,

 

The Monkey Tree Hosting Team

 

--UPDATE--

 

The scheduled update went ahead as planned, and we have seen services return to normal.

 

Should you encounter any issues at all please do not hesitate to get in contact with us by emailing [email protected].

 

Have a wonderful evening, and enjoy the rest of your weekend!

 

Kind Regards,

 

The Monkey Tree Hosting Team

 

 

Network Connectivity Issues (Resolved) Medium

Affecting Other - UK Networks

  • 23/03/2021 15:35
  • Last Updated 12/04/2021 16:40

To our Valued Clients,

We are aware of an issue at London Internet Exchange (LINX) in which internet traffic to various Internet Service Proviers (ISPs) is affected.

Our Datacentres reported under a 5 minute drop in connection at 15:35

Our Datacentre Networks remain stable. However UK Internet providers relying on LINX will observe instability until engineers are able to troubleshoot the issue.

Further updates will follow.

Kind Regards,
Your MTH Team

Inbound Mail Issue (Resolved) Critical

Affecting Other - Exchange Inbound

  • 07/04/2021 08:11 - 07/04/2021 15:00
  • Last Updated 12/04/2021 16:39

To our Valued Customers,

We are aware of an issue affecting Inbound Mail across our Internal Exchange Services.

This is being investigated as a matter of urgency, we will provide an update as soon as possible.

 

-- update as of 10:00
Confirmed no MTH Services are under fault, this is being raised with our Mail Delivery Partners

-- update as of 13:30
We are noticing new mail delivering through, and historic mail being delivered on a retry rate. We are continuing to monitor this closely.

-- update as of 15:00
We consider this issue fully resolved. Historic Mail is still being delivered on a retry rate.

MTH-3 Server Outages (Resolved) Critical

Affecting Other - MTH-3 Datacentre

  • 05/03/2021 11:20 - 05/03/2021 11:48
  • Last Updated 05/03/2021 12:04

To our Valued Clients.

We are aware of an outage in our MTH-3 DataCentre.

All connectivity is currently affected.

The Network Team on site are investigating.

-- update as of 11:36
connection is restored, service is still considered at risk

-- update as of 11:48
We are confident this issue has been fully resolved. We will continue to monitor services as usual

Major Outage (Resolved) Critical

Affecting Other - MTH-1

  • 12/12/2020 10:58
  • Last Updated 05/03/2021 12:03

Good Morning,

We are aware of ongoing connectivity issues affecting all MTH Services.

-- update as of 08:45

Our DataCentre team is investigating this outage

-- update as of 09:04

The on-site team are honing in on a root cause of the issue

-- update as of 09:06

Our datacentre partners have admitted to an Internet Traffic Issue

-- update as of 09:30

All top-level network kit has been investigated and found to be fully functional

-- update as of 09:48

Information has been provided by the Uplink Partners that Emergency Fibre works are taking place due to physical Breakages in the Fibre Lines

-- update as of 10:18

The on-site Engineers have identified the fibre breakages and are working to resolve. Current ETA 1-2 hours.

-- update as of 11:50

We are seeing a restoration of connectivity across our networks, however, our service is still 'at risk' until we've monitored the situation closely and concluded the issue is fully resolved.

-- update as of 12:03

Our datacentre partners have confirmed that all services should now be back online and we are not at further risk of disruption. We will continue to monitor closely.

Managed Signature Service (Resolved) High

Affecting Other - Managed Signature Service

  • 16/10/2020 10:59 - 16/10/2020 13:05
  • Last Updated 05/03/2021 12:03

Good Morning,

 

We are aware of an issue affecting our Managed Signature Services.

This is being investigated.

An update will be provided ASAP.

Kind Regards,
Your MTH Team

 

--update 13:05

After locating the issue on our side, resolving and testing I can confirm this issue has been resolved.

Managed Signatures are now applying.

Apologies for any inconvenience caused.

MTH Networks Residential Outages - OFNL (Resolved) Critical

Affecting Other - MTH Networks - OFNL Infrastucture

  • 04/03/2021 16:00 - 04/03/2021 16:30
  • Last Updated 05/03/2021 12:03

To our Valued MTH Networks Customers,

One of our Network Partners are experiencing an Outage affecting multiple sites.

This is being investigated by OFNL's network engineers. An update will be provided as soon as possible.

-- update as of 16:30

Work to restore service was completed. Customers should now find services to be working as expected. We will continue to monitor services to ensure they remain stable.

Gateway Certificate Issue (Resolved) Critical

Affecting System - MTH Hosted Desktop

  • 11/11/2020 08:35 - 12/12/2020 11:04
  • Last Updated 11/11/2020 09:56

To our Valued MTH Customers,

 

We have an ongoing issue affecting logins to our Hosted Desktop services from Windows Computers. MTH OS and Mac Computers are unaffected.

This is being worked on as a matter of urgency - we will have an update for you shortly.

Kind Regards,
Your MTH Team

-- update as of 09:56
This issue has been fully resolved.
Please accept our apologies for any inconvenience caused.

Network Connectivity Issues (Resolved) Critical

Affecting Other - SDC, KAO & Leased Line Services

  • 13/10/2020 09:13 - 13/10/2020 11:00
  • Last Updated 14/10/2020 11:22

Good Morning,

We are aware of Network Connectivity Issues affecting our Hosted Desktop Services, some Leased Lines and other services.

Our MTH HQ Office is also experiencing Network Issues so our Phone Lines are intermittent.

We will provide an update as soon as possible.

Kind Regards,
Your MTH Team

-- update as of 10:11

The issue has been identified and is related to routing across varoius networks. Access is intermittent and various teams are working towards a resolution.

-- update as of 10:45

We are seeing stability return across our networks. We will closely monitor. Should you experience any further issues, please do contact us on 0203 358 0210 opt.1

Inbound Mail - Service Issue (Resolved) Critical

Affecting Other - Exchange Email Hosting & Filtered IMAP Hosting - Inbound

  • 15/09/2020 14:30 - 15/09/2020 22:29
  • Last Updated 15/09/2020 22:42

To our Valued Customers,

We are aware of an issue affecting Inbound Mail to both our Exchange and Filtered IMAP Customers.

This is being investigated internally, and by our Mail Delivery Partners.

We will provide an update as soon as possible.

Kind Regards,
Your MTH Team

--update: 15:40 UK BST
An issue has been identified and is being worked on as a matter of urgency.

--update: 17:50 UK BST
We are in constant contact with our inbound mail filtering partners and they are working tirelessly to bring this back into service.
Unfortunately, at this moment they have no further updates for us, but have assured us that we should see restoration to inbound mail soon.

--update: 18:50 UK BST
MailChannels have deployed a configuration change as well as increasing their system capacity extensively. Inbound email is now flowing again and we will continue to monitor the flow with internal testing and monitoring as well as liaising directly with MailChannels until they are certain that their system has stable, consistant operation for an extended period of time

--update: 20:45 UK BST
MailChannels have identified further issues are at play here and delays continue as they work on rolling out a fix. Customer logs will show that we are taking in email to our queues; however, messages are not being delayed quickly to customer mail servers. MailChannels are working diligently to get that email flowing so that queues can drain and customers can receive email.

--update: 21:02 UK BST
Messages are now draining from MailChannels internal queues and customers should see messages that were delayed hitting their inboxes now.

--update: 21:47 UK BST
We are continuing to monitor for any further issues.

--update: 22:12 UK BST
All the mails have now been drained out from MailChannels internal queues

--update: 22:29 UK BST
Email delays are back within normal parameters. The issue is now considered fully resolved by MailChannels. MTH will be monitoring mailflow throughout the night to ensure this is 100% correct.

Inbound Mail - Service Issue (Resolved) Critical

Affecting Other - Exchange Email Hosting & Filtered IMAP Hosting - Inbound

  • 09/09/2020 14:42 - 09/09/2020 15:35
  • Last Updated 09/09/2020 15:42

To our Valued Customers,

We are aware of an ongoing issue affecting Inbound Mail to our Exchange Email Hosting customers and Filtered IMAP Hosting.

This is being investigated as a matter of urgency by relevant departments.

An update will be posted as soon as possible.

Kind Regards,
Your MTH Team

--update 15:07
Our Mail Delivery partners who offer our Mail Filtering and Delivery services have acknowledged this issue.
A further update will be posted ASAP.

--update 15:20
An Internal Service within our Mail Partners systems has caused a delay in inbound mail routing.
While email may be delayed, senders will queue and retry later. We appreciate how frustrating any delay in email delivery is.

-- update 15:35
Mail Delivery has restored to it's full capacity.
As ususal, we will monitor delivery closely to ensure there are no further mail delays.

MBORC declaration for all Openreach products (Resolved) Critical

Affecting Other - OpenReach

  • 25/03/2020 14:50
  • Last Updated 08/09/2020 14:17

MBORC: Matters Beyond Our Resonable Control

https://www.openreach.co.uk/orpg/home/mborc.do

This MBORC declaration notification is issued as a special message to all MTH customers utilising the OpenReach network due to the impact of COVID-19 in the UK.

Due to the effects of the extensive Government restrictions across the UK, OpenReach's day-to-day operational activities are seriously impacted.

OpenReach is therefore declaring MBORC for provision, repaid, missed appointments, on a national basis, for all SD SAM patches and FND operational areas.

This declaration will take effect from 23:59hrs on the 24th March 2020.

As always, we will continue to monitor the situation, and will keep you updated on further developments.

MTH Hosted Desktop - Service issues (Resolved) High

Affecting System - MTH Hosted Desktop

  • 07/09/2020 05:00 - 07/09/2020 11:30
  • Last Updated 08/09/2020 14:17

To our Valued Customers,

We are unfortunately experiencing a few routing issues between our MTH-1 and MTH-2 Datacentres.

This is causing a side affect of some Hosted Desktop issues. Including, Print Services and Hosted Application services.

Standard Usage of the Hosted Desktop service remains unaffected.

This issue is being investigated by all relevant departments, and will be resolved ASAP.

Please do give us a call on 0203 358 0210 opt.1 or raise a ticket for further information.

Kind Regards,
Your MTH Team

-- update as of 11:30, our Datacentre Team have resolved the routing issues.
Following a complete review of our system, all Hosted Desktop services have been restored.

Emergency Maintenance Notification (Resolved) High

Affecting Other - Exchange Email Hosting & Foldr File Sharing

  • 10/06/2020 20:00 - 10/06/2020 22:00
  • Last Updated 11/06/2020 09:17

To our Valued Clients,

As per the current issues with the File Sharing Service, Emergency Maintenance Work has been scheduled for this evening.

A downtime window of 5-10 minutes is expected during the scheduled Maintenance of 20:00 and 22:00. This will affect some mail routing, and file access services.

Should you have any queries, or issues durirng this time; please raise a ticket from within your Client Area, or send an email to '[email protected]'

Kind Regards,
Your MTH Team

--update as of 22:30, these works have been successful. Service has been fully restored.

Exchange & Foldr Maintenance (Resolved) Critical

Affecting System - Exchange Email Hosting & Foldr File Sharing

  • 21/05/2020 23:00 - 22/05/2020 02:00
  • Last Updated 11/06/2020 09:17

To our Valued Clients,

To ensure a high level of service to our Clients, some maintenance work is often required to various systems.

The next Maintenance Window is Thursday 12st May 2020, at 23:00. This work will approximately take 3-4 hours. Finishing at around 03:00 Friday 22nd May 2020.

The total downtime during this time will be limited to 1 or 2 15 minute windows.

The services effected will be Exchange Email Hosting, along with access to Foldr fire shares.

Should you have any concerns or queries, please do let me know.

Kind Regards,
Your MTH Team

File Sharing Service issue (Resolved) High

Affecting Other - File Sharing Service issue

  • 10/06/2020 09:15 - 10/06/2020 22:30
  • Last Updated 11/06/2020 09:17

To Our Valued Clients,

We are experincing a Certificate Issue with our File Sharing system.

Unfortunately we cannot guarantee full resolution until out of hours, this evening, due to the other knock-on effects this may have with other services.

Please contact our Support Team on 0203 358 0210 opt.1 should you experience any issues, and we can guide you through a temporary resolution for today.

Thank you in advance for your patience with this matter.

Kind Regards,
Your MTH Team

--update 22:30, as per the scheduled emergency maintenance, this issue has been resolved.

Coventry DataCentre Connectivity (Resolved) High

Affecting Other - DC-3

  • 01/06/2020 13:55 - 01/06/2020 15:00
  • Last Updated 10/06/2020 10:19

To our Valued Clients,

We are experiencing connectivity issues in our Conventry DataCentre. This is affecting all traffic.

Our on-site team is investigating this issue as a matter of urgency.

An update will be posted ASAP.

Kind Regards,
Your MTH Team

--update as of 14:08
This issue has been resolved, network traffic is fully restored.

--update as of 14:52
This issue has recurred, we are investigating.

--update as of 14:58
Connectivity has once again been restored. We do not expect this issue to recur.

LINX Network Maintenance (Resolved) Low

Affecting Other - Datacentre MTH-1

  • 06/05/2020 00:00 - 06/05/2020 03:00
  • Last Updated 12/05/2020 09:06

Description of works:

We have received notification from LINX (London Internet Exchange) that they are going to be carrying out maintenance during the above period that involves upgrading to a new software release which will fix longstanding bugs and implement support for new hardware. Our 2x 100Gb/s network ports to LINX during this time are expected to flap however due to the redundant nature of our network no service interruption is expected during this window however the network should be deemed at risk for the duration of this maintenance window.

Impacted Services:

  • Some VPS services
  • Some Dedicated Servers

 

While we expect zero impact to customers services, please do let us know if you have any concerns.

Your MTH Team

London Network Issue - continued (Resolved) High

Affecting Other - Datacentres MTH-1, MTH-2

  • 06/05/2020 11:38 - 06/05/2020 12:00
  • Last Updated 12/05/2020 09:06

To our Valued MTH Clients

We are aware of continued issues affecting connectivity of the London Network. These issues are subsequently affecting connectivity to 2 of our datacentres.

Network Engineers are on-site and investigating the root cause of these issues for resolution.

Connectivity to our services may be affected during this time.

Noticed outages:
- 11:38 to 11:46

Further updates will be posted.

Kind Regards,
Your MTH Team

-- update as of 12:00
Impacts of connectivity issues have been reduced and stability of the London Network has been restored.

Close monitoring will continue.

Emergency Maintenance Notification (Resolved) High

Affecting Other - Datacentres MTH-1, MTH-2

  • 06/05/2020 00:15 - 06/05/2020 00:30
  • Last Updated 06/05/2020 10:24

To our Valued Clients,

Following today's Network Connectivity issues affecting the London Network, Further Emergency Maintenance is required.

Network Maintence is requied on the affected routers which were the root-cause of today's issues. The affected routers require a reboot and further work.

An outage is expected overnight. Specicically in the early hours of tomorrow morning (06/05/2020: 00:15 - 00:30)

An update will be provided at the completion of these works.

Please raise any issues via a ticket from your client area, or by an email to [email protected]

Kind Regards,
Your MTH Team

London Network Issue (Resolved) High

Affecting Other - Datacentres MTH-1, MTH-2

  • 05/05/2020 11:03 - 06/05/2020 09:18
  • Last Updated 06/05/2020 10:24

To our Valued MTH Clients

We are aware of London Network issues affecting connectivity to 2 of our datacentres.

Network Engineers are on-site and investigating the root cause of these issues for resolution.

Connectivity to our services may be affected during this time.

Noticed outages:
- 11:03 to 11:10
- 11:28 to 11:32

Further updates will be posted.

Kind Regards,
Your MTH Team

-- update as of 11:39
Stability is noticed across the London Network. No further drops have occurred.
Our Network Engineers are still investigating the root-cause of the issues and ensuring stability of the network.

--- update as of 12:27
Our Engineers have confirmed Network Stability and have moved to a 'monitoring' status.

-- update as of 16:10
Our on-site Network Engineers have scheduled in Emergency Maintenance over night to fully resolve these issues.

-- update as of 16:23
A further drop in connectivity has occurred: 16:17 to 16:22

Continued: 06/05/2020

-- update as of 09:18
The works as part of the Emergency Maintenance Window last night were successful and services have been stable throughout the night.

This issue is now considered fully resolved. We will closely monitor as usual and provide any updates as they develop.

MTH Networks - VoIP Issues (Resolved) High

Affecting Other - MTH Networks - VoIP

  • 29/04/2020 09:15 - 05/05/2020 00:00
  • Last Updated 05/05/2020 10:46

To our Valued MTH Networks Customers

We are aware of intermittent connectivity issues to our MTH VoIP Service as of 09:15 this morning.

This may affect incoming/outgoing calls and device registrations. Also, this may be affecting the OfficeUC mobike application.

This issue is being investigated as a matter of urgency by our VoIP System Technicians.

An update will be posted shortly.

Kind Regards,
Your MTH Team

-- update as of 10:10
The affected server has been recognised and further work is taking place.

-- update as of 11:55
An improval of service is being noticed across the Network.
Our Technical Team are closely monitoring the service to ensure full resolution.

-- update as of 12:30
This issue is considered resolved, however, to ensure stability a decision has been made for the BLF functionaity to be disabled temporarily.
We will monitor this closely, and enable BLF functionaity upon full system stability.

Continued: 30/04/2020

-- update as of 09:20
We have continued to hold back on re-enabling the BLF functionaity to ensure stability across the system.
Our monitoring team will make the decision when the BLF functionaity will be restored.

Continued: 01/05/2020

--update as of 12:00
Further issues are seen within the infrastructure relating to BLF functionality. However, the VoIP System remains stable for calls and device registrations.

Continued: 04/05/2020

--update as of 12:00
Following further investigational work and testing, BLF Functionality was restored late Friday evening (01/05/2020). This was closely monitored along with stability of the entire Infrastrcuture.

This issue is now considered fully resolved.
May we apologise for any inconvenience caused.

MTH VoIP - Number Porting (Resolved) Critical

Affecting Other - OpenReach

  • 24/03/2020 13:00 - 30/03/2020 13:00
  • Last Updated 29/04/2020 10:36

Today MTH have been advised that OpenReach have closed all porting activities. This means that no number porting can be completed from today onwards.

We have also been advised that all ports which were scheduled for today will now fail; further, all number ports scheduled up to 6th April will be cancelled.

As soon as OpenReach re-open this process, we will ensure all ports are re-submitted; if this affects you, we will be directly in touch.

We can only apologise for the disruption this will no doubt cause to some of your businesses, however due to the current climate, this is out of our control.

 

As always, we will provide further updates as and when they come available.

 

UPDATE: 26/03/2020 12:45hrs

Please see the below statement on our Number Porting services function from OpenReach:

  • The Indian Government has introduced expansive restrictions and stringent controls on the movement of people in its attempt to protect the Indian population from the Coronavirus pandemic. As a result this is having a significant impact on our ability to serve you
  • We have been working to understand what porting capability can be offered to our customers in light of the rapid lockdown experienced in India.
  • Our current service is significantly impacted, during this time there will be delays in responses however we are standing up additional agent resource with the aim to return to as near a business as usual service as possible on data actions by early next week. We will address existing backlogs and focussing on activation delays with potential loss of service as a priority.

 

As soon as we have updated details on what level of service will be restored, we will update this post as a priority.

 

UPDATE:30/03/2020 13:00hrs

Number porting service is restored to normal, however with potential for slightly longer than normal response times due to smaller and remote working teams.

Scheduled Exchange Server Maintenance (Resolved) High

Affecting System - Exchange Mail Hosting

  • 06/04/2020 22:00 - 06/04/2020 23:30
  • Last Updated 28/04/2020 09:07

To Our Valued Customers,

In order to maintan our High Level of service, constant maintenance is required to our Infrastructure.

Our next scheduled maintenance is Monday 6th April, 22:00-23:30.

A 10 minutes outage during the maintenance window may be experienced during this time.

Should you experience any issues at all, please send an email to [email protected] or raise a ticket from your Client Area.

Kind Regards,
Your MTH Team

Scheduled Exchange Server Maintenance (Resolved) Medium

Affecting Other - Exchange Mail Hosting

  • 03/04/2020 20:00 - 04/04/2020 23:00
  • Last Updated 06/04/2020 10:04

To Our Valued Customers,

In order to maintan our High Level of service, constant maintenance is required to our Infrastructure.

Our next scheduled maintenance is Friday 3rd April between 20:00-23:00 and Saturday 4th April between 20:00 and 23:00

No Service Impact is expected during these maintenance windows.

Should you experience any issues at all, please send an email to [email protected] or raise a ticket from your Client Area.

Kind Regards,
Your MTH Team

Coventry - Emergency Datacentre Maintenance (Resolved) High

Affecting Other - SC-1

  • 04/04/2020 01:00 - 04/04/2020 03:00
  • Last Updated 06/04/2020 10:03

To Our Valued Customers,

In order to maintan our High Level of service, constant maintenance is required to our Infrastructure.

Our upcoming Maintenance involves high-prioirty works to our Coventry Datacentre.

All Virtual and Dedicated Servers will require a reboot at 01:00 (Saturday 04/04/2020). Please allow for up to 15 minutes downtime after this reboot.

Please be aware this is only affecting servers that are hosted in our Coventry Data Centre. If you are unsure if your server is at risk of this downtime, please contact us.

Should you have any queries at all, please don't hestitate to contact our team via a Support Ticket that can be raised from within your Client Area, or via email to: [email protected]

Kind Regards,
Your MTH Team

MTH VoIP - Device Registration Issue (Resolved) High

Affecting System - MTH VoIP

  • 12/03/2020 12:22 - 12/03/2020 13:50
  • Last Updated 13/03/2020 16:12

To our Valued MTH VoIP Customers

We are currently aware of a MTH VoIP Platform issue.

This is being investigated as a matter of urgency.

An update will be posted when possible.

Kind Regards.
Your MTH Team

--update: 12:53hrs
Service is beginning to restore, we will be closely monitoring the service. Further updates to follow

--update: 13:50hrs
We are happy that this issue is fully resolved. As usual, this service will be fully monitored to ensure stability.
May we apologise for any inconvenience caused

--update: 13/03/2020 09:50hrs
Service has remained stable with no further issues having been seen overnight or during this morning's peak call period.
Our technical team will continue to monitor the platform closely throughout the day.

Routing Abnormalities (Resolved) Medium

Affecting System - Hosted Desktop, VPS and Dedicated Servers

  • 12/03/2020 09:19 - 13/03/2020 01:37
  • Last Updated 13/03/2020 10:42

Update - We are pleased to announce that the residual disruption affecting specific services has now been resolved and we are seeing stability. As of such, we are able to close the emergency maintenance window.
We will continue to monitor services throughout a stability period passing into the office hours of Friday, 13th March.

If you continue to experience any substantial disruption, please contact us directly via [email protected] or by calling the office on 020 3358 0210.

We apologise for any inconvenience caused throughout these vital works.
Mar 13, 01:37 GMT
 
Update - As a result of the continuing residual disruption seen after the emergency maintenance window, we are re-opening this window for a further 2 hours until 03:00am.
While we are hopeful that the disruption will not continue until this time, we are re-opening the window to make the required changes in order to stabilise affected services.
 
A further update will be provided at 01:30am
Mar 13, 00:59 GMT
 
Update - Please be aware that we are still experiencing residual disruption as a result of the final changes of the emergency maintenance window.
We are still enacting remedial action to resolve these and will provide a further update at 01:00 where a decision will be made to re-open the emergency maintenance window for further works if necessary
Mar 13, 00:45 GMT
 
Update - Please be aware that we are still experiencing residual disruption as a result of the final changes of the emergency maintenance window.
We are now taking remedial action to resolve these and will provide a further update at 00:45.
Mar 13, 00:29 GMT
 
Update - Please be aware that we are still experiencing residual disruption as a result of the final changes of the emergency maintenance window.
 
A further update will follow at 00:30 before we give the all-clear.
Mar 13, 00:14 GMT
 
Monitoring - Please be aware that services affected by this emergency maintenance window are now stabilising. While works have ended, we are continuing to monitor the situation closely.
In the event that any corrective actions are required, we will provide an update here immediately.
 
Further update to follow at 00:15.
Mar 12, 23:54 GMT
 
Update - Please be aware that we are nearing the end of the emergency maintenance works, while we are seeing affected services stabilise, there is still the possibility of further disruption for a short while.
We expect that this will be resolved and affected services fully stable before the end of the 23:59 maintenance window.
Mar 12, 23:34 GMT
 
Update - Please be aware that these emergency maintenance works are still progressing. While we are still witnessing disruption to specific services, we are not currently expecting this to continue past the end of the emergency maintenance window at 23:59.

Further updates to follow.
Mar 12, 23:01 GMT
 
Update - As expected, we are seeing further disruptions throughout the emergency works for certain services. The works are still progressing and we are not currently expecting disruption past the end of this emergency maintenance window.

We will continue to provide further updates as the works progress.
Mar 12, 22:27 GMT
 
Update - Please be aware that while these works are underway, we are currently experiencing a related issue that is causing the disruption to select services to be higher than anticipated.
We have made the decision that the works should still finish, but we are expecting further disruption to a sub set of services until the end of the maintenance window.
Mar 12, 21:28 GMT
 
Update - Please be aware that we will be undertaking emergency maintenance this evening, between 20:00 and 23:59 to resolve this issue.
We expect minor disruption during this window for a subset of affected layer 3 services.

We will provide a further update here once these works are completed.
Mar 12, 15:29 GMT
 
Update - Please be aware that we have mitigated the impact of this issue for the majority of affected clients. There may still be a small number of services that become affected. Please contact us directly at [email protected] if there are any queries.

We are currently designing a package of out of hours works to fully resolve this issue, further details of the timings for these works are still to follow in a subsequent update.
Mar 12, 11:37 GMT
 
Identified - We have identified the issue and are currently working on a resolution.
 
Further updates will follow shortly.
Mar 12, 09:34 GMT
 
Investigating - Please be aware that we are currently investigating routing abnormalities, affecting a subset of leased lines and other services.

Updates will be provided here as we resolve the issue.
Mar 12, 09:19 GMT

Connectivity Issues - Virgin Media (Resolved) Critical

Affecting Other - Virgin Media Connectivity

  • 05/03/2020 09:09 - 05/03/2020 16:54
  • Last Updated 12/03/2020 22:43

Resolved - Virgin media successfully repaired the fibre break in the Southampton area. The works required were substantial due to the extent of the break.

If you are still encountering issues please raise a support case to [email protected].
Mar 516:54 GMT
Identified - We are aware of an issue that is affecting several of our leased line customers that utilize Virgin Media tails. Virgin Media are experiencing a Major incident, currently affecting a large number of customers. We are speaking with them to ascertain the time to fix and we will update this page with any news.
Mar 509:09 GMT

Vodafone Circuit Instability (Resolved) High

Affecting Other - Vodafone

  • 31/01/2020 02:16 - 31/01/2020 10:47
  • Last Updated 12/03/2020 22:41

Resolved - This incident has been resolved.
Mar 509:04 GMT
Monitoring - This issue has been resolved by Vodafone and we are currently observing a period of stability before closing this incident.

We are still working with Vodafone to understand the root cause.

Please contact MTH Networks directly if you have any queries.
Jan 3110:47 GMT
Update - Please be aware that we have been in contact with Vodafone directly and are now awaiting their response.

We have identified the issue as our 1G NNI port with Vodafone and we are expecting a reply within the next 3 hours.

We will provide updates here as we receive them.
Jan 3102:34 GMT
Investigating - Please be aware that we are currently investigating an issue raised by our automatic alerting, affecting a subset of customers with Vodafone provided connectivity.

We currently have this raised with the provider and are attempting to escalate now to get a resolution.

Updates will be provided here as we receive them.
Jan 3102:16 GMT

Scheduled Datacentre Network Maintenance (Resolved) High

Affecting Other - SC1

  • 18/02/2020 21:00 - 19/02/2020 10:34
  • Last Updated 18/02/2020 11:33

To our Valued Customers,

Our Datacentre Partners are undertaking required Maintenance Works this evening at 21:00hrs.

An at risk window will affect internet traffic between 21:00 and 23:00hrs.

Should you have any issues during this time, or have any concerns please email [email protected] or raise a ticket from within your Client Area.

Kind Regards
Your MTH Team

VoIP Platform - Call Handling, BLF & Availability Issues (Resolved) Medium

Affecting Other - Hosted VoIP Service

  • 27/01/2020 14:23 - 18/02/2020 11:02
  • Last Updated 30/01/2020 17:22

To our VoIP customers,

We are currently (14:23hrs) receiving reports of issues with various VoIP platform feautes, including BLF, Call Waiting, and Call Transfers.

These are currently under investigation by our technical team, as well as our provider BT, alongside the VoIP platform vendor support.

This is being investigated as a matter of urgency and we will post updates  as soon as we have any further information.

Kind Regards,
Your MTH Team

 

UPDATE: 14:45hrs

We are now getting reports that this issue is also affecting phone registrations intermittently; this may interrupt live calls, stop a phone receiving a call, and show "No Service" on the handset periodically.

Please be assured that this is now being increased to a High priority issue and escallated with various teams involved.

Further updates will follow as soon as we have information for you.

 

UPDATE: 15:38hrs

BT have identified the issue at play here; traffic is moving from Primary to Secondary Servers and during the sync it is lost. BT are now on a Tech Bridge with high level Cisco engineers to establish the root cause of this and work on a resolution.

For clarification, BT's Cloud voice platform is build on a multi-million pound Broadworkssoft infrastructure; Cisco aquired Broadworkssoft back in 2018 and this is why their world expert infrastructure and networking teams have been brought in on this.

 

UPDATE: 16:38hrs

We have now escalated this to a Critical Outage, due to the extreme length of service outage; MTH can only apologise for the impact this is having to all of your business'.

BT have updated us, however unfortunately nothing too much to report. Traffic is still bouncing between Application Servers (the entry point to the VoIP network) and remains a major problem and the cause of the issue all clients are facing. 

All involved are actively working on a Technical Bridge with the platform provider (Cisco Broadworkssoft) however no work around has been identified as yet.

As soon as we have any hint of an update, we'll be in touch.

 

UPDATE: 17:08hrs

We appreciate everyone's patience here; it's not easy when you're experiencing this extent of downtime.

The update here is mainly the same as 16:38hrs; the Primary Application Server is in overload control, causing traffic to move to the Secondary Application Server. This secondary device is then trying to process this and dropping into overload itself and passing traffic back to the Primary Server.

Currently Cisco are working on a potential solution to resolve the routing issue; the alternative is to roll the platform back to a previous itterative version which should bring stability back, however this option would take a few hours to complete.

When we hear more, we will update everyone.

 

UPDATE: 19:20hrs

We've just received an update from BT:

  • They're still working to fix the issue on their VoIP platform. The platform is currently experiencing a total loss of service; Cisco and Broadworkssoft senior engineers are working across the platform, restarting applications in an attempt to resore service.
  • This is continued to be treated as a Major Critical issue, with the utmost urgency and all key personel, and relevant resources are engaged  on this.

We fully expect to receive further updated throughout the evening until resolution has been reached; as always, as soon as we have anything, we'll update you here.

 

UPDATE: 20:20hrs

Since the last update at 19:20hrs, we’ve seen stable VoIP service from c. 19:30hrs; this has been carefully monitored and tested, and will continue to be throughout the evening.

Additionally, we’ve received the following from BT, Cisco and Broadworkssoft:

We have reinstated service on our VoIP platform. Testing and close monitoring will continue through the night.

We continue to treat this issue with the upmost urgency and all relevant resources are engaged. We sincerely apologise for the inconvenience this has caused you and your customers

We are continuing to monitor and test the platform throughout this evening; we have been given a confirmed issue resolved status, however will complete our own testing before signing it off ourselves later this evening.

 

UPDATE: 22:30hrs

Following a further 2hrs of MTH testing and checking, and a successful 3hrs of stable service, we’re finally signing this VoIP platform issue off as Resolved.

We can only apologise for the massive inconvenience and impact this has had on all your business, and assure you the whole situation is being investigated at the highest level.

If anyone experiences any further issues following this resolution, please first of all reboot your phone handsets; if this doesn’t resolve the issue, please do contact our support team directly and we will assist and resolve as promptly as possible.

 

UPDATE: 09:00hrs 28/01/2020

Unfortunately we have been advised this morning that issues are still being seen across the platform, despite the overnight works lastnight; urgent investigations are still underway.

We have checked with various clients and service is seemingly OK for our clientbase, the platform is still in an "at risk" state.

Further updates will follow as soon as we have any information.

 

UPDATE: 09:35hrs

BT have advised that a large number of their customers are experiencing further issues already this morning, specifically making and receiving calls and phones de-registering from the platform. Senior BT engineers are continuing to work with Cisco and Broadworkssoft to isolate and resolve these issues.

Further update to be provided c. 10:30am. Please accept our sincere apologies for the impact this is causing.

 

UPDATE: 10:30hrs

BT Update:

We continue to investigate issues on the platform. While we have received reports of issues, not all customers are impacted. All relevant technical resource is present on a technical bridge. 

We'll provide further updates by 12:00hrs.

 

UPDATE: 11:00hrs

Further information from BT is that they're looking into applying a new 'tactical fix' due to the overnight fix failing. They are also looking into the possibility of rolling back the Broadworks R22 upgrade.

As soon as we have any update regarding the intended course of action, this will be posted below.

 

UPDATE: 12:30hrs

BT Update:

We’re sorry for the continued disruption to service.  We estimate this impact to be around 40%, with many of these customers able to make calls, but not use some of the key features. Over 1.5 million calls have been successfully made so far today.  We of course recognise the criticality of these features to our customers and are working through a plan to resolve.

We believe the issue is due to the planned R22 upgrade which was deployed over the weekend and as a result we are building parallel plans to roll back to the previous version. This is likely to take around 8 hours, so have taken the decision to defer this until 8pm.  At this point in time if we don’t have certainly on a resolution, which will resolve the issue in a loaded environment, we’ll execute our tried and tested roll back plans to ensure service is fully restored before 7:30 am tomorrow.

 

UPDATE: 15:30hrs

Early indications are that the issue appears to be resolved, however we are giving it more time before confirming. If anyone is having issues, please contact the MTH Support team.

 

UPDATE: 16:30hrs

While MTH clients haven't seen any issues through today, BT, Cisco and Broadworks have repoted a further influx of client issue reports, thus MTH are keeping this service notification open.

 

UPDATE: 17:00hrs

BT Update:

Due to the further impact experienced 16:30 decision has been made that a full rollback of the R22 upgrade will be completed this evening to resolve all outstanding issues. 

Platform will revert platform back to as it was evening of Friday 24th 

Rollback will commence at 17:30, no customer impact is expected during the rollback process. Rollback is estimated to take 8 hours and BTW engineers will be monitoring throughout the evening / early hours. 

MTH have a scheduled call with senior engineers and directors at 07:30hrs to confirm platform status prior to the working day. Updates will be posted following this call.

 

UPDATE: 07:50hrs 29/01/2020

Work to roll back to the previous version of software has continued through the night, but at 2am the team encountered a number of issues.  In order to ensure customers have service this morning the decision was taken to re-instate the newer version of operating firmware.

While the rollback continues, calls and services will be working normally although we are likely to see similar issues as Monday and yesterday.  In parallel to the roll back, we have a team assembled to resolve these issues and implement a fix.  

Business Zone is available but any orders will be queued and not progressed.   Business Portal is unavailable during the roll back procedure. This is not a fault and is part of the process, and we currently anticipate this will be completed in the next 3 hours.

All necessary resource is available on the technical bridge as is support from vendors at the most senior level.

 

UPDATE: 09:35hrs

We can finally confirm some good progress is being made to revert the platform rollback to R21. Unfortunately this has resulted in the MTH VoIP portal currently being unavailable, however we expect this to be back online this afternoon.

Please be aware "call gapping" is in place now to alleviate any further customer impact; this could result in a small number of calls returning busy to control peak volumes and allow for complete resolution to carry on.

Updates to follow.

 

UPDATE: 10:10hrs

BT Update:

Work to re-instate the newer version of operating firmware continues and progress is being made. In order to ensure that service is maintained throughout the process it is taking longer than initially expected. As a result provision activities remain suspended and MTH VoIP Portal is unavailable. This is expected to remain for some hours yet and may not be resolved until later this afternoon. 

In order to mitigate the issues we have seen over the last two days we have implemented some minimal call gapping. This will mean that at during our busiest periods some customers may get busy tone. Initial feedback this morning suggests that this approach is working.  

In parallel to the roll back, we have a team assembled to resolve these issues and implement a fix.

All necessary resource is available on the technical bridge as is support from vendors at the most senior level. 

Please accept our sincere apologies for the impact this is causing.

 

UPDATE: 11:40hrs

Latest from BT:

Work to re-instate the newer version of operating firmware continues. Provision activities remain suspended and MTH VoIP Portal is unavailable. We are making progress and we expect to restore both provision activities and MTH VoIP Portal by 14:00hrs.

We have implemented some minimal call gapping in order to mitigate the issues we have seen over the last two days. This will mean that during our busiest periods some customers may get busy tone. This approach continues to be effective.

In parallel to the roll back, we have a team assembled to resolve the issues with the firmware and implement a fix.  This remains our top priority.

All necessary resource is available on the technical bridge as is support from vendors at the most senior level.  We will update you by 3:00 p.m. or sooner if the situation changes.

 

UPDATE: 12:15hrs

We've been monitoring the situation today and everything has been continued stable for MTH and all it's clients; this has been the case since 19:30hrs on 27/01/2020.

Despite the continued stability for MTH and clients, we have been keeping ontop of the updates, and keeping everyone posted here due to the extensive issues being reported elsewhere. Work is still ongoing to completely resolve the platform issues; the MTH VoIP portal will be available later this afternoon.

Next update expected c. 15:00hrs

 

UPDATE: 14:10hrs

Latest from BT:

The newer version of operating firmware has now been re-instated. Provision activities have restarted and the MTH VoIP Portal is now available. 

Minimal call gapping will continue to be applied in order to mitigate the issues we have seen over the last two days. This will mean that during our busiest periods some of your customers may get busy tone. This approach continues to be effective. 

A team from across BT and our vendor continue working on resolving the issues with the firmware in order to implement a fix. This remains our top priority.

Close monitoring of the platform will continue. We will update you by 5 p.m. or sooner if the situation changes. 

Please accept our sincere apologies for the impact this is causing.

 

UPDATE: 17:09hrs

Latest update from BT:

The platform has remained stable since the new firmware was re-instated. Provision orders have all progressed and Business Portal is fully restored.  Close monitoring of the platform continues and none of the call drop and function issues experienced on Monday and Tuesday have been observed today. 

The BT and our vendor team are now fully focussed on working to resolving the issues with the firmware; progress has been made but its too early to confirm a resolution timescale at this time.

Minimal call gapping will continue to be applied in order to mitigate the issues we have seen over the last two days. This will mean that during our busiest periods some of your customers may get busy tone. This approach continues to be effective. 

We do not foresee any further issues with performance through the night period. We will provide a further update at 8 a.m. tomorrow. However if there is any change to this we will update you accordingly.  

Please accept our sincere apologies for the impact this has caused you.

MTH will update further here as soon as we receive information; additionally, we have had zero clients report issues since 19:30hrs on 27/01/2020. These updates are for completeness.

 

UPDATE: 08:20hrs 30/01/2020

We have seen the platform remain stable overnight, without any issues seen or reported. The new version of the firmware is successfully loaded across all clusters with all the latest patches. Parameter changes were made lastnight on all clusters and these will be monitored closely throughout the day to ensure that they have had the expected impact.

Update from BT:

The BT and Cisco team are now fully focussed monitoring the platform following yesterday’s changes. While we are confident of the work carried out yesterday it is too early to state that the issue is resolved and we will not be making that statement today as we need to monitor stability over a longer period. 

Minimal call gapping will continue to be applied. As we gain confidence this will be relaxed and we will keep you updated on this.  This will mean that during our busiest periods some of your customers may get busy tone. 

Clearly your customers experienced a lot of issues on Monday and Tuesday most of which should now be resolved. If they experience issues today can we please ask that you report these on a new ticket with all the necessary detail. This will allow us to prioritise these issues and work closely with our technical team to resolve. 

We do not foresee any further issues but will provide a further update at 12 noon today. However if there is any change to this we will update you accordingly.  

Please accept our sincere apologies for the impact this has caused you.

Updates will be posted below midday.

 

UPDATE: 12:07hrs

The platform has remained stable, with no issues processing calls. During the morning busy period (08:15 to 08:45hrs) the platform handled twice the call volume of yesterday without any issue.

We appreciate over the last few days there have been a number of sporadic issues, and service interruptions; here's the latest progress update:

•	Voicemail – is not working correctly if the call is delivered via auto-attendant or a hunt group. We have  identified the issue and a fix is being worked on now. 
•	SIM Ring – is not working correctly and only ringing on one phone. Our technical team are working on this. 
•	Transferring Calls – if delivered via auto attendant the call was failing on transfer. This is now resolved.
•	BLF (Busy Lamp Field) – this was not working correctly. This is now resolved. 
•	Call Waiting – Despite being disabled it was still alerting to a second call. This is now resolved. 
•	Phones de-registering – Phones were de-registering and calls were dropped. This is now resolved.

Both the outstanding issues are being worked on with utmost priority.

Further updates will be provided at 13:00hrs today.

 

UPDATE: 13:09hrs

The platform has remained stable without issues; call processing has been flowing continually all day.

Following the outstanding issue update earlier, the two outstanding issues are as below:

•	Voicemail – is not working correctly if the call is delivered via auto-attendant or a hunt group.  We have a proposed fix which is currently in test. We expect to deploy this today. 
•	SIM Ring – is not working correctly and only ringing on one phone. This is currently with our vendor who are working on a fix. 

Further updates will be posted at the end of today, c. 17:00hrs.

 

UPDATE: 14:30hrs

In addition to the issues advised above, an issue has also been identified with provisioning of new devices to the platform. Initial indications are that this is only impacting companies built before September 2019

Details from BT:

The platform is receiving the requests to build device configurations but is currently very slow to respond. It will appear to customers that the device is there in the portal however the background configuration is not  and handsets will not register. Some are getting "no config" error on the display.

We have been advised that the platform should eventually catchup, however BT and Cisco are working on the cause of the build up of requests and slow response. We do not have an ETA on this fix, but will notify as soon as posisble.

 

UPDATE PENDING

MTH Services Maintenance (Resolved) High

Affecting Other - Datacentre MTH-2 & MTH-3

  • 15/11/2019 22:00 - 16/11/2019 06:00
  • Last Updated 29/01/2020 18:32

To Our Valued Customers.

To follow our efforts on providing an excellant, reliable service, we are scheduling some upcoming works to the Core of our networks.

The maintenance is to be carried out on the core of our Infrastructre Services.

This maintenance will temporarily affect access to Website Hosting, Hosted Desktop, VPS and dedicated server access.

Work will begin at 22:00 today, 15th November 2019 and continue through to around 06:00 tomorrow morning, 16th November 2019.

We will reduce outages as much as possible. However, please expect outage windows of up to 2 hours, whilst the maintenance work takes place.

Our team of Engineers will work tirelessly throughout this window to complete the upgrade work, and keep outages to a minimum.

Should you have any queries regarding the upcoming works, or whilst they are taking place, please raise a ticket through your client area, or send an email to [email protected]

 

Thank you.

Kind Regards,
Y
our MTH Team

 

**UPDATE: 22:00hrs**

Hosted Desktop access is currently disabled for the network maintenance, expected ETA for access is 00:30hrs

**UPDATE: 22:55hrs**

VPS Services in our MTH-3 datacentre are currently going down, expected ETA to come back up is 23:15hrs

**UPDATE: 23:15hrs**

VPS Services in our MTH-3 datacentre unfortunately are delayed in coming up; expected ETA 5-10mins

**UPDATE: 23:20hrs**

VPS Services in our MTH-3 datacentre are now fully restored; apologies for the inconvenience. Total service interruption 20-25mins.

**UPDATE: 00:15hrs**

Hosted Desktop access expected ETA is now 01:30hrs due to unforeseen delays; apologies for the extension.

**UPDATE: 01:23hrs**

Hosted Desktop access has now been re-activated; this has been tested from 6 separate office and network connections and all is performing as expected.

Main MTH Number (Resolved) Critical

Affecting Other - Hosted VoIP System

  • 27/01/2020 15:00 - 27/01/2020 20:21
  • Last Updated 27/01/2020 22:38

Hi,

We are currently experiencing a phone system outage ourselves, due to the on-going VoIP outage.

During this time, please raise support tickets via your MTH Client Area, or email [email protected] from a registered, authorised contact.

Kind Regards,

MTH Support

 

UPDATE: 20:21hrs

Following VoIP issue resolution, our office phone lines are now working again.

VoIP Platform Outage (Resolved) High

Affecting Other - Hosted VoIP Service

  • 22/01/2020 10:40 - 27/01/2020 14:32
  • Last Updated 27/01/2020 14:31

Good Morning

10:40 - An outage is currently being experienced which will affect all Hosted VoIP customers.

Incidently, our platform is affected too, our contact number 0203 358 0210 will be in limited service currently.

This is being investigated as a matter of urgency.

An update will be posted as soon as possible.

Apologies for any inconvenience caused.

Please raise a ticket from your client area, or email [email protected] if your issue is of an urgent nature.

Kind Regards,
Your MTH Team

 

--update 10:52

We are noticing service resuming across the platform, with most customers reporting VoIP service restored.

The MTH Office is back online, should you wish to reach out to us on 0203 358 0210.

We will monitor closely and provide any further updates.

Google Chrome on Hosted Desktop (Resolved) High

Affecting System - Hosted Desktop Services

  • 12/12/2019 10:48
  • Last Updated 22/01/2020 11:13

To Our Valued Hosted Desktop Customers,

Google Chrome have enforced an update over the last 24 hours which has broken compatibility with our Terminal Servers.

Unfortunately this means no user can access Google Chrome on our Hosted Desktop environment. An error: 'aw snap' will be experienced on all Web Pages.

A workaround is being investigated and tested.

May we apologies for any inconvenience in the meantime.

Thank you.

Your MTH Team

host.mthservers.com: SQL Service (Resolved) High

Affecting Server - MTH SSD Host

  • 06/11/2019 06:00 - 06/11/2019 09:21
  • Last Updated 06/11/2019 09:29

Early this morning we were alerted to the crash of the SQL Service on one of our cPanel hosting servers.

This is currently being investigated and work is being carried out to bring the service back up.

As soon as we have an update, we will update this notice.

Kind Regards,

MTH Support

===UPDATE: 09:20hrs===

MariaDB published an update on November 5th, 2019 that can cause MariaDB to fail upon startup on cPanel & WHM servers. We've identified this as an upstream issue with MariaDB (MDEV-19073)

cPanel themselved have also realised this overnight and a large number of servers have been impacted on this.

https://forums.cpanel.net/threads/known-issues-status-page.644133/#post-2709865

A rollback resolution has been implimented on our server and we believe this issue to be resolved; please do let us know if you have any further issues.

DataCentre Issue (Resolved) Critical

Affecting System - Datacentre MTH-1

  • 30/10/2019 10:15 - 30/10/2019 11:00
  • Last Updated 06/11/2019 08:20

Good Morning,

Our Datacentre: MTH-1 is currently experiencing  some routing issues. Most outbound connections are currently down.

Unfortunately this will be affecting some of our VPS and Dedicated Server customers.

Our Datacentre Technicians are on hand resolving the issue, with constant communication to ourselves.

A positive update will be posted as soon as possible.

Kind Regards,
Your MTH Team

-------
UPDATE: 10:30

This issue is still being investigated by the on-hand Team.

A resolution will be reached shortly.

 

UPDATE: 11:00

This issue has been resolved by our Datacentre partners. If anyone is still experiencing outage, please raise a support ticket from your MTH Client Area.

Scheduled Shared Host downtime (Resolved) High

Affecting Server - MTH SSD Host

  • 18/10/2019 23:00 - 18/10/2019 23:30
  • Last Updated 30/10/2019 10:15

To Our Valued Shared Hosting/Reseller customers

To ensure we continue to offer our high level of service, we consistently maintain and upgrade our servers.

Some maintenance does require more extensive work and downtime.

Our next maintenance window is tomorrow evening (Friday 18th October), A reboot will take place at 23:00. We only envisage up to 15 minutes downtime, however the 'at risk' window will be up to 30 minutes.

If you have any queries regarding this, please don’t hesitate to raise a support ticket.
Support will also be available during this migration period.

Please may we also point you to our Network Status page, where you can keep up to date with all current and upcoming updates across our Infrastructure. This can be reached by navigating to: https://secure.monkeytreehosting.com/serverstatus.php

Kind Regards,

The Monkey Tree Hosting Team

MTH Hosted Desktop Maintainance (Resolved) High

Affecting System - Underlying Hypervisor Infrastructure

  • 22/10/2019 22:30 - 22/10/2019 23:30
  • Last Updated 23/10/2019 00:26

Good Evening,

We need to perform some emergency maintenance this evening to complete some core infrastructure updates; this will require a number of reboots to some key infrastructure.

During this work, the Hosted Desktop service may experience some short periods of interruption, however these will be limited to a couple of 5min windows; this will be mitigated as much as possible, however these works are necessary.

If you experience a temporary profile, this is expected while the work is underway; please proceed to work as normal.

Works Timeframe: 22:30hrs to 23:30hrs

- We would usually give longer notice for works of this nature, however feel it is necessarily high importance to complete these updates.

We will update below as and when works start and complete.

 

==Update 00:25hrs==Complete==

This evening's works have now been complete; all works have been successful.

 

==Update 23:30hrs==

Progress is coming along nicely with updates and system actions all completing successfully; unfortunately due to mitigating service impact, the actions are taking longer than expected.

The works timeframe is being extended by 1hr to 00:30hrs

 

==Update 22:30hrs==

Works starting...

 

==Update 18:51hrs==

Works scheduled on-time; confirmed 

Scheduled Datacentre Emergency Network Maintenance (Resolved) High

Affecting System - Datacentre MTH-2

  • 10/10/2019 23:00 - 11/10/2019 01:00
  • Last Updated 22/10/2019 19:04

Good Afternoon,

Our Datacentre team are required to undertake some emergency maintenance this evening to resolve some minor routing issues.

An at risk window has been posted from 23:00 this evening (Thursday 10th October) to 01:00 tomorrow morning (Friday 11th October)

The expected impact is limited to a single 20 minute period of network instability during this time.

Please do raise a ticket from within your Client Area, should this impact any service with ourselves.

Thank you for your understanding.

Kind Regards,
Your MTH Team

Scheduled Shared Host downtime (Resolved) High

Affecting Server - MTH SSD Host

  • 11/10/2019 23:00 - 11/10/2019 23:30
  • Last Updated 18/10/2019 15:49

To Our Valued Shared Hosting/Reseller customers

To ensure we continue to offer our high level of service, we consistently maintain and upgrade our servers.

Some maintenance does require more extensive work and downtime.

Our next maintenance window is tomorrow evening (Friday 11th October), A reboot will take place at 23:00. We only envisage up to 15 minutes downtime, however the 'at risk' window will be up to 30 minutes.

If you have any queries regarding this, please don’t hesitate to raise a support ticket.
Support will also be available during this migration period.

Please may we also point you to our Network Status page, where you can keep up to date with all current and upcoming updates across our Infrastructure. This can be reached by navigating to: https://secure.monkeytreehosting.com/serverstatus.php

Kind Regards,

The Monkey Tree Hosting Team

Hosted Desktop Maintenance (Resolved) High

Affecting System - Hosted Desktop Services

  • 18/09/2019 20:00 - 18/09/2019 22:00
  • Last Updated 10/10/2019 15:46

To Our Valued Customers.

To follow our efforts on providing an excellant, reliable service, we are scheduling some upcoming works to the Core of our networks.

The maintenance is to be carried out on the core of our Hosted Desktop Infrastructure.

Use of our Hosted Desktop services will be limited during the period of 8pm to 10pm this evening (Wednesday 18th September).

Should you have any queries regarding the upcoming works, please raise a ticket through your client area, or send an email to [email protected]

Kind Regards,
Your MTH Team

Scheduled Datacentre Maintenance (Resolved) High

Affecting Other - Datacentre MTH-2 & MTH-3

  • 07/08/2019 21:00 - 07/08/2019 03:00
  • Last Updated 18/09/2019 16:11

To Our Valued Customers.

To follow our efforts on providing an excellant, reliable service, our partners are scheduling some upcoming works to the Core of our networks.

The maintenance is to be carried out on our core network, in an effort to complete migrations to our new Juniper platform.

Following on from the work carried out on Monday, the Contingency Window has been invoked for further works.

Just as with the original maintenance, this window will encompass all sites and involve our network team physically moving your services onto new Juniper hardware. This move will impact all transit and MPLS connectivity services, and customers should expect to see multiple brief outages throughout the window. Any resilient customers should not see an outage but should consider their services at risk during this window

Should you have any queries regarding the upcoming works, please raise a ticket through your client area, or send an email to [email protected]

Kind Regards,
Your MTH Team

Scheduled Datacentre Network Maintenance (Resolved) High

Affecting Other - Datacentre MTH-2 & MTH-3

  • 05/08/2019 21:00 - 06/08/2019 03:00
  • Last Updated 06/08/2019 16:41

To Our Valued Customers.

To follow our efforts on providing an excellant, reliable service, our partners are scheduling some upcoming works to the Core of our networks.

The maintenance is to be carried out on our core network, in an effort to complete migrations to our new Juniper platform.

The work scheduled for Monday 5th August 2019 at 21:00-03:00 includes actions to the following services:

  • Leased Line services - Customers who have a leased line will be affected by this work. The impact is expected to be up to 15 minutes per leased line at various intervals throughout the maintenance window. Where customers have resilient connections, only 1 connection will be migrated at a time.
  • Layer 3 services - Upstream connectivity to the Internet is being migrated to our new Juniper core. We do not expect any impact to customers during this migration as proactive steps have been taken to avoid downtime, however, services should be considered 'at-risk' for the duration of the window. 
  • Layer 2 services - Site-to-Site connectivity is not directly in scope for these works and we do not expect any impact to customers, however, services should be considered 'at-risk' for the duration of the window due to other work being carried out on the network.

Should you have any queries regarding the upcoming works, please raise a ticket through your client area, or send an email to [email protected]

Kind Regards,
Your MTH Team

Scheduled Datacentre Network Maintenance (Resolved) High

Affecting System - Datacentre MTH-2 & MTH-3

  • 16/05/2019 22:00 - 17/05/2019 02:00
  • Last Updated 29/07/2019 10:54

Good Afternoon, 

 

As part of our commitment to maintain a high standard of service, MTH and our datacentre partners are instigating a major hardware and architecture upgrade to our core network infrastructure.

Over the past year the teams have been working tirelessly to build a brand-new core network based on SDN technology, separating L2 and L3 infrastructure and using new best-of-breed Juniper hardware. This represents a great investment by MTH and our datacentre partners into building a world-class network to support our customers' future growth, and is also in part a response to the issues with longer-than-anticipated re-convergence times following network disturbances and fibre breaks.
Having been through extensive testing, the upgrade is now ready to be installed. The commissioning process involves moving physical fibre links, which means there will be an interruption to services during the following maintenance windows:

 

We will be using the below window to complete the works on the network upgrade. Please be advised that updates of this window will be posted on our status page:

 

https://secure.monkeytreehosting.com/serverstatus.php 

16th May 2019, 22:00-02:00 (contingency)
MTH-2 and MTH-3 Data Centres, all traffic
This is a contingency-only window, scheduled to accommodate any complications during the previous maintenance window.

This upgrade work is the culmination of extensive research, planning and financial investment, and I'm proud to announce that the new network is ready for installation and commissioning. If you have any questions about the nature of the work or would like more information, please get in touch with us.

 

Kind Regards,

 

Your MTH Team

Scheduled Datacentre Maintenance (Resolved) Medium

Affecting Other - Datacentre MTH-2 & MTH-3

  • 03/06/2019 20:30 - 03/06/2019 21:30
  • Last Updated 29/07/2019 10:53

To our Valued Hosted Desktop customers,

We are constantly upgrading our infrastucture to provide a better Hosted Desktop experience for our clients.

Our next upgrade focuses on the Shared FileServer where all your files and folders are stored safely and securely.

The window of these works are between 20:30 and 21:30 this evening.

Any noticable impact will be between 21:00 and 21:15. During this time, access to any 'Shared Network Drives' may be disrupted.

Please raise a ticket on your client area, or email [email protected] should you run into any issues, or have any queries.

Kind Regards,

Your MTH Team

National Network Outage (Resolved) Critical

Affecting Other - Internet Line Connections

  • 12/07/2019 09:16
  • Last Updated 12/07/2019 14:54

Good Morning,

Our Internet Partner are having Major Network Routing issues at the moment, affecting the below services

  • FTTC (Fibre to the Cabinet) Lines
  • Leased Line connectivity
  • DSL Lines

This outage is spanning from Telehouse North, On of the UK's largest Network Infrastrcture's. It is affecting many Internet Providers across the UK and Europe.

This is being worked on by a large team of experienced network engineers. We are constantly being updated on a resolution.

Any further updates will also be posted below.

Thank you for your patience with this matter and may you accept our apologies for any inconvenience caused during this time.

Kind Regards,

Your MTH Team

--update: 10:00

Our engineers are on site resolving the issue.

Two avenues are being taken. 1, to mitigate the current issue in hand with the non-working services. 2, to build a alternative service and route network through there.

--update: 11:00

Our Engineers are still working in parallel to bring a new device online to help resolve the remaining issues.

We apologise that we do not have anything more concrete at this time, but can assure you that our team are treating this as their number one priority. An update will follow within the hour.

--update: 12:00

Our Engineers have pushed a Network-Wide fix for all traffic. We are seeing services slowly come back online.

--update: 12:20

Services are fully online. We are closely monitoring all services to ensure stability.

--update: 15:00

Services are fully stable. Monitoring will continue as normal. This issue is considered resolved.

Scheduled Datacentre Network Maintenance (Resolved) High

Affecting System - Datacentre MTH-2 & MTH-3

  • 04/05/2019 22:00 - 05/05/2019 06:00
  • Last Updated 08/05/2019 12:15

Our DataCentre Partners are undertaking a rolling programme of network upgrades, with the following objectives:

• Building a brand-new core network based on SDN technology

• Seperating L2 and L3 infrastructure, using top-of-the-line Juniper Hardware

•Preventing network disturbances and fibre breaks

•Building a World-Class network to support customer's future growth

•Upgrading all current physical fibre-links

The first maintenance window will be this coming Saturday (Saturday 4th May 2019) from 22:00-06:00. During this time all traffic is considered to be at risk, but with minimal downtime.

If you have any questions, please raise a support ticket from your client area.

Kind regards,

Your MTH Team

Scheduled Shared Host downtime (Resolved) High

Affecting Server - MTH SSD Host

  • 04/02/2019 23:59 - 05/02/2019 03:00
  • Last Updated 30/04/2019 17:40

To Our Valued Shared Hosting/Reseller customers

To ensure we continue to offer our high level of service, we consistently maintain and upgrade our servers.

Some maintenance does require more extensive work and downtime.

Our next maintenance window is next week, from midnight Monday night to the early hours of Tuesday morning. To be exact: (Monday 4th February at 00:00 to 03:00 on Tuesday 5th February). We estimate only up to an hour of downtime during this period.

If you have any queries regarding this, please don’t hesitate to raise a support ticket.
Support will also be available during this migration period.

Please may we also point you to our Network Status page, where you can keep up to date with all current and upcoming updates across our Infrastructure. This can be reached by navigating to: https://secure.monkeytreehosting.com/serverstatus.php

Kind regards,

The Monkey Tree Hosting Team

Hosted Desktop - Desktop Issues (Resolved) Critical

Affecting System - HostedDesktop

  • 14/02/2019 11:23 - 14/02/2019 12:22
  • Last Updated 14/02/2019 12:33

Hosted Desktop - Known Issues
Last Updated: Thursday 14th @ 12:16PM

Good Afternoon.

At 11:23, our monitoring alerted us to an issue within the Hosted Desktop Infrastructure, which caused Microsoft Programs to crash, and subsequently crashing a users profile.
We have narrowed this down to 2 Microsoft Updates for Outlook, which we believe to be corrupting the profile on use, and causing the issues.

Please bear with us whilst we investigate this further. We will be calling round all effected customers once we have a further resolution and fix in place.

UPDATE: 12:22
We are seeing connections resolve, and users log back into the platform. A full log out and log back in is resolving the issue, we are still actively searching for the conflicting update within MS architecture, and we hope to have this removed by close of play today.

Kind Regards

MTH Team

Scheduled Datacentre Network Maintenance (Resolved) Medium

Affecting System - Datacentre MTH-2 & MTH-3

  • 24/01/2019 23:00 - 25/01/2019 03:00
  • Last Updated 28/01/2019 16:28

Our DataCentre Partners are undertaking a rolling programme of network upgrades, with the following objectives:

• Reducing latency

• Increasing reliability

• Enhancing QoS

• Improving our Constant Quality Monitoring (CQM) Platform

• Improving failover/failback response times

• Better oversight of maintenance impact assessments

• General housekeeping improvements

The next maintenance window will be next Thursday (Thursday 24th January 2019) from 23:00-01:00. During this time there will be a small impact to Internet services, expected to be a few minutes of instability. This is estimated to be approximately 4x 10-minute outages for each maintenance window, however for contingency we have allowed the entirety of the two-hour period.

If you have any questions, please raise a support ticket from your client area.

Kind regards,

Your MTH Team

Monkey Tree Hosting - VoIP Outage (Resolved) Critical

Affecting System - VoIP Phone Network

  • 18/01/2019 12:42 - 18/01/2019 13:40
  • Last Updated 18/01/2019 13:45

Hi all,

We are currently  (12:41) experiencing a system-wide VoIP outage,

- This is affecting all VoIP services and VoIP customers

We are actively looking into this and investigating the root cause as a P1 issue.

Please keep an eye on this page for updates.

Kind Regards
MTH Team

UPDATE - 13:10

Our engineers are investigating the root cause of this issue.  ETA of resolution: 20 minutes.

UPDATE - 13:30

This issue has been resolved and we are seeing calls flowing through the VoIP network. 

Please accept our apologies for any inconvenience caused.

Kind Regards, MTH Team

Monkey Tree Hosting - VoIP Outage (Resolved) Critical

Affecting System - VoIP Phone Network

  • 15/01/2019 12:01
  • Last Updated 15/01/2019 12:28

Hi all,

We are currently  (11.40) experiencing a system-wide VoIP outage,

- This is affecting all VoIP services and VoIP customers

We are actively looking into this and investigating the root cause as a P1 issue.

Please keep an eye on this page for updates.

Kind regards
MTH Team

UPDATE 11.55

Service has been restored and we are seeing calls flowing through the network again.
For all affected customers, please accept our apologies for this outage.
Any further issues, please do not hesitate to contact us.

Kind Regards,
MTH Team

Scheduled Datacentre Network Maintenance (Resolved) Medium

Affecting Other - Datacentre MTH-2 & MTH-3

  • 10/01/2019 23:00 - 11/01/2019 01:00
  • Last Updated 15/01/2019 09:18

Our DataCentre Partners are undertaking a rolling programme of network upgrades, with the following objectives:

• Reducing latency

• Increasing reliability

• Enhancing QoS

• Improving our Constant Quality Monitoring (CQM) Platform

• Improving failover/failback response times

• Better oversight of maintenance impact assessments

• General housekeeping improvements

The next maintenance window will be This Evening (Thursday 10th January 2018) from 23:00-01:00. During this time there will be a small impact to Internet services, expected to be a few minutes of instability. This is estimated to be approximately 4x 10-minute outages for each maintenance window, however for contingency we have allowed the entirety of the two-hour period.

If you have any questions, please raise a support ticket from your client area.

Kind regards,

Your MTH Team

Hosted Desktop Certificate Expiry (Resolved) Critical

Affecting System - Hosted Desktop UK

  • 14/11/2018 07:05 - 14/11/2018 08:01
  • Last Updated 14/11/2018 08:10

We have been alerted by our systems that our gateway security certificate is expired and is thus blocking any Windows devices connecting to the hosted desktop; the certificate is incorrectly reporting it's expired status as this is not the case. We are working with the SSL provider to urgently resolve this.

Affecting: New session login

Not affecting: Existing user sessions

 

UPDATE 08:01hrs

We have issued a new certificate and implemented as this was the fastest route to resolution. Apologies for the inconvenience caused by this interruption to service.

Scheduled Datacentre Network Maintenance (Resolved) Medium

Affecting Other - Datacentre MTH-2 & MTH-3

  • 01/11/2018 23:00 - 02/11/2018 01:00
  • Last Updated 02/11/2018 10:04

Our DataCentre Partners are undertaking a rolling programme of network upgrades, with the following objectives:

• Reducing latency

• Increasing reliability

• Enhancing QoS

• Improving our Constant Quality Monitoring (CQM) Platform

• Improving failover/failback response times

• Better oversight of maintenance impact assessments

• General housekeeping improvements

The next maintenance window will be This Evening (Thursday 1st September 2018) from 23:00-01:00. During this time there will be a small impact to Internet services, expected to be a few minutes of instability. This is estimated to be approximately 4x 10-minute outages for each maintenance window, however for contingency we have allowed the entirety of the two-hour period.

If you have any questions, please raise a support ticket from your client area.

Kind regards,

Your MTH Team

 

---

UPDATE: 02/10/2018 08:00hrs

This work was completed successfully without issue; network disruption was limited to a few minutes.

Scheduled Datacentre Network Maintenance (Resolved) Medium

Affecting Other - Datacentre MTH-2 & MTH-3

  • 20/09/2018 23:00 - 21/09/2018 01:00
  • Last Updated 21/09/2018 14:19

Our DataCentre Partners are undertaking a rolling programme of network upgrades, with the following objectives:

• Reducing latency

• Increasing reliability

• Enhancing QoS

• Improving our Constant Quality Monitoring (CQM) Platform

• Improving failover/failback response times

• Better oversight of maintenance impact assessments

• General housekeeping improvements

The next maintenance window will be Thursday 20th September 2018, from 23:00-01:00. During this time there will be a small impact to Internet services, expected to be a few minutes of instability. This is estimated to be approximately 4x 10-minute outages for each maintenance window, however for contingency we have allowed the entirety of the two-hour period.

If you have any questions, please raise a support ticket from your client area.

Kind regards,

Your MTH Team

 

---

UPDATE: 21/09/2018 08:00hrs

This work was completed successfully without issue; network disruption was limited to a few minutes.

Scheduled Datacentre Network Maintenance (Resolved) Medium

Affecting Other - Datacentre MTH-2 & MTH-3

  • 04/10/2018 23:00 - 15/01/2019 12:27
  • Last Updated 20/09/2018 11:13

Our DataCentre Partners are undertaking a rolling programme of network upgrades, with the following objectives:

• Reducing latency

• Increasing reliability

• Enhancing QoS

• Improving our Constant Quality Monitoring (CQM) Platform

• Improving failover/failback response times

• Better oversight of maintenance impact assessments

• General housekeeping improvements

The next maintenance window will be Thursday 4th October 2018, from 23:00-01:00. During this time there will be a small impact to Internet services, expected to be a few minutes of instability. This is estimated to be approximately 4x 10-minute outages for each maintenance window, however for contingency we have allowed the entirety of the two-hour period.

If you have any questions, please raise a support ticket from your client area.

Kind regards,

Your MTH Team

Scheduled Datacentre Network Maintenance (Resolved) Medium

Affecting Other - Datacentre MTH-2 & MTH-3

  • 18/10/2018 23:00 - 15/01/2019 12:27
  • Last Updated 20/09/2018 11:13

Our DataCentre Partners are undertaking a rolling programme of network upgrades, with the following objectives:

• Reducing latency

• Increasing reliability

• Enhancing QoS

• Improving our Constant Quality Monitoring (CQM) Platform

• Improving failover/failback response times

• Better oversight of maintenance impact assessments

• General housekeeping improvements

The next maintenance window will be Thursday 18th October 2018, from 23:00-01:00. During this time there will be a small impact to Internet services, expected to be a few minutes of instability. This is estimated to be approximately 4x 10-minute outages for each maintenance window, however for contingency we have allowed the entirety of the two-hour period.

If you have any questions, please raise a support ticket from your client area.

Kind regards,

Your MTH Team

Scheduled Datacentre Network Maintenance (Resolved) Medium

Affecting Other - Datacentre MTH-2 & MTH-3

  • 01/11/2018 23:00 - 15/01/2019 12:27
  • Last Updated 20/09/2018 11:13

Our DataCentre Partners are undertaking a rolling programme of network upgrades, with the following objectives:

• Reducing latency

• Increasing reliability

• Enhancing QoS

• Improving our Constant Quality Monitoring (CQM) Platform

• Improving failover/failback response times

• Better oversight of maintenance impact assessments

• General housekeeping improvements

The next maintenance window will be Thursday 1st November 2018, from 23:00-01:00. During this time there will be a small impact to Internet services, expected to be a few minutes of instability. This is estimated to be approximately 4x 10-minute outages for each maintenance window, however for contingency we have allowed the entirety of the two-hour period.

If you have any questions, please raise a support ticket from your client area.

Kind regards,

Your MTH Team

Scheduled Datacentre Network Maintenance (Resolved) Medium

Affecting Other - Datacentre MTH-2 & MTH-3

  • 15/11/2018 23:00 - 15/01/2019 12:27
  • Last Updated 20/09/2018 11:13

Our DataCentre Partners are undertaking a rolling programme of network upgrades, with the following objectives:

• Reducing latency

• Increasing reliability

• Enhancing QoS

• Improving our Constant Quality Monitoring (CQM) Platform

• Improving failover/failback response times

• Better oversight of maintenance impact assessments

• General housekeeping improvements

The next maintenance window will be Thursday 15th November 2018, from 23:00-01:00. During this time there will be a small impact to Internet services, expected to be a few minutes of instability. This is estimated to be approximately 4x 10-minute outages for each maintenance window, however for contingency we have allowed the entirety of the two-hour period.

If you have any questions, please raise a support ticket from your client area.

Kind regards,

Your MTH Team

Scheduled Maintenance Notification (DataCentr (Resolved) Medium

Affecting Other - Datacentres: MTH-2 & MTH-3

  • 06/09/2018 00:00 - 15/01/2019 12:27
  • Last Updated 20/09/2018 11:02

As part of our commitment to upholding the best standards of service, our DataCentre Partners have identified an upgrade opportunity for our network infrastructure. This will take the form of a rolling programme of works, starting in September, which will meet the following objectives:

  • Reducing latency
  • Increasing reliability
  • Enhancing QoS
  • Finishing our Constant Quality Monitoring (CQM) Platform
  • Improving failover/failback response times
  • Better oversight of maintenance impact assessments
  • General housekeeping improvements

These will be achieved through various methods, including:

  • Further optimising the OSPF cost model
  • Conducting technical preparations for future international expansions
  • Optimising LSPs
  • Firmware updates to numerous
  • Building out a meshed monitoring platform to enable fine-grain loss monitoring

During the maintenance windows there will be a small impact to Internet services, expected to be a few minutes of instability. This is estimated to be approximately 4x 10-minute outages for each maintenance window, however for contingency we have allowed the entirety of the two-hour period. Due to the nature and complexity of the work, we are unable to give customers granular details on how their services will be affected.

Our DataCentre Partners have extensively reviewed our network use to determine the optimal times for this programme of works to take place. Given our diverse, global client base, this will not be a perfect fit for every customer, however it is the time when the network is least used (amongst other factors) and so will have the least impact.

The works will take place on Thursday nights, on a fortnightly schedule. The schedule has been arranged to provide a balance between making progress with the work and providing service stability. The maintenance windows are:

  • Thursday 6th September 2018
  • Thursday 20th September 2018
  • Thursday 4th October 2018
  • Thursday 18th October 2018
  • Thursday 1st November 2018
  • Thursday 15th November 2018

On each of the dates listed above, the at-risk window for the maintenance work is 23:00-01:00.

We will send reminder notifications before each maintenance window.

Kind Regards,

Your MTH Team

Scheduled Datacentre Network Maintenance (Resolved) Medium

Affecting Other - Datacentre MTH-2 & MTH-3

  • 06/09/2018 23:00 - 07/09/2018 01:00
  • Last Updated 20/09/2018 10:56

Our DataCentre Partners are undertaking a rolling programme of network upgrades, with the following objectives:

• Reducing latency

• Increasing reliability

• Enhancing QoS

• Improving our Constant Quality Monitoring (CQM) Platform

• Improving failover/failback response times

• Better oversight of maintenance impact assessments

• General housekeeping improvements

The next maintenance window will be Thursday 6th September 2018, from 23:00-01:00. During this time there will be a small impact to Internet services, expected to be a few minutes of instability. This is estimated to be approximately 4x 10-minute outages for each maintenance window, however for contingency we have allowed the entirety of the two-hour period.

If you have any questions, please raise a support ticket from your client area.

Kind regards,

Your MTH Team

 

---

UPDATE: 07/09/2018 08:00hrs

This work was completed successfully without issue; network disruption was limited to a few minutes.

Core Network Maintenance Notice (Resolved) High

Affecting System - Datacentres: MTH-2 & MTH-3

  • 19/06/2018 22:30 - 20/06/2018 02:30
  • Last Updated 30/08/2018 18:04

Dear Customer,

Our datacentre provider will be performing scheduled maintenance on 19th-20th June 2018, from 22:30-02:30hrs. During this time they will be upgrading a node on their network to ensure they continue to offer the highest levels of resilience and up-time.

Though we have posted an at-risk window of four hours, we expect the impact to be limited to two connectivity outages, each up to 20 minutes. If you have any questions, please contact your account manager.

Kind Regards,

MTH Team



UPDATE
This was completed successfully with minimal impact

Emergency Maintenance Notification (Resolved) High

Affecting Other - Datacentres: MTH-2 & MTH-3

  • 06/06/2018 01:00 - 06/06/2018 03:00
  • Last Updated 06/06/2018 09:06

Dear Customer,

Following some connectivity issues earlier today, our Datacentre provider will be conducting urgent maintenance on 6th June 2018 between 01:00 and 03:00 to replace a faulty line card on their core router.
 
Though we have posted an at-risk window of 2 hours, we expect impact to be limited to two outages of up to five minutes whilst traffic is directed away from the device, then back to it following the replacement.
 
If you have any questions, please contact support.
 
Kind regards,

MTH Team


UPDATE 03:04

We are pleased to confirm that the maintenance is complete successfully. Should you experience any issues after the advertised window, please raise a support ticket via your usual channels.

Datacentre Routing issue (Resolved) Critical

Affecting Other - Datacentres: MTH-2 & MTH-3

  • 05/06/2018 16:24 - 05/06/2018 19:20
  • Last Updated 05/06/2018 19:19

Hi All,

We're currently (16:24hrs) experiencing core routing issues across two of our Datacentre partners across MTH-2 and MTH-3 facilities.

This is impacting service to the following:

- Hosted Desktop
- Hosted Exchange - NB. No email loss, however intermintent connection drops. Advised to use OWA (https://exchange.mthservers.com) until the issue is resolved.
- Shared Hosting - some clients
- Reseller Hosting - some clients
- VPS - few clients

Our datacentre team are actively looking into this and will resolve ASAP. Please keep posted on this page for updates.

Kind Regards
MTH Team


UPDATE 

Datacentre Team: We have identified a problem with connectivity between all of our sites and the internet at 15:45 this afternoon. We have taken some corrective action at 16:18 which temporarily restored connectivity; however at 16:25 our systems reported more problems.

We are still investigating the problems here.



UPDATE 17:18hrs

Datacentre Team: As a further update, we are now isolating links sequentially in our network throughout Hertfordshire to help us identify the issue.

We are still investigating the problems.



UPDATE 17:45hrs

Datacentre Team: As a further update, we are still working with affected hardware vendors to track down and resolve these ongoing connectivity problems.



UPDATE 18:49hrs

Datacentre Team: We believe we have now identified the cause of these issues and are still working with hardware vendors to resolve them. We have put temporary measures in place to improve the issues. It is likely that we will be arranging emergency maintenance for this evening to put the final fix in place.
More details will follow shortly.


UPDATE 18:53:hrs

We have received confirmation of full resolution to the issues faced this evening. MTH are currently undertaking full systems checks to ensure everything is back to our normal level of service.
Final update to follow, to mark issue as resolved.


UPDATE 19:20hrs

We can now confirm full resolution of the issue and have fully checked all MTH systems successfully; an RFO will be issued in the coming days to outline a summary of the issue and resolution.

Network Notification (Resolved) High

Affecting Other - MTH Hosted and Exchange infrastructure

  • 09/12/2017 12:00 - 09/12/2017 14:00
  • Last Updated 14/05/2018 17:06

To our customers,

This is a notification that there will be scheduled upgrade work to our systems this evening.

Unfortunately this will cause a network outage across all our services for a short amount of time, as we re-route half our network across new, dedicated fibre links to increase capacity and redundancy.

We forsee the outage to only be for 30 minutes at a maximum to allow for minimum disruption, and this can be between the window 23:00 - 04:00.

We thank you in advance for your patience, should you have any problems or questions, please call our offices or email [email protected].

Many thanks,
The MTH team

Network Notification (Resolved) High

Affecting Other - MTH Hosted and Exchange infrastructure

  • 30/11/2017 23:00 - 01/12/2017 04:00
  • Last Updated 01/12/2017 08:11

To our customers,

This is a notification that there will be scheduled upgrade work to our systems this evening.

Unfortunately this will cause a network outage across all our services for a short amount of time, as we re-route half our network across new, dedicated fibre links to increase capacity and redundancy.

We forsee the outage to only be for 30 minutes at a maximum to allow for minimum disruption, and this can be between the window 23:00 - 04:00.

We thank you in advance for your patience, should you have any problems or questions, please call our offices or email [email protected].

Many thanks,
The MTH team

Update 01/17 The upgrade work has been completed succesfully, if you encounter any problems please us at [email protected].

MTH System: Desktop and Exchange Issues (Resolved) Critical

Affecting System - MTH System

  • 25/10/2017 12:01 - 25/10/2017 21:26
  • Last Updated 25/10/2017 21:26

Hi All,

We have noted an issue pertaining to Destop and Exchange sub-systems.
We are actively working to resolve this ASAP and will update when we have a further update.

MTH Team.

UPDATE: 12:32 - SubSystems have been identified, we are working to resolve these now and currently have an ETA of 13:00 for a target resolution.
UPDATE: 13:02 - This issue is now resolved. Please contact [email protected] for an RFO.

Exchange Maintainance (Resolved) High
  • 03/10/2017 21:00 - 04/10/2017 09:38
  • Last Updated 03/10/2017 12:22

On Tuesday 3rd of October at 9PM (BST) we will be performing maintenance on our Exchange system.

Maintenance Start: 21:00 (BST) 3rd of October 2017

Maintenance End: 21:10 (BST) 3rd of October 2017

During the maintenance window users might experience some slowness of the system however the service should remain online; if you do experience any issues please raise a support case in the usual manner.

Exchange Maintanace (Resolved) Critical
  • 14/09/2017 22:00 - 15/09/2017 00:00
  • Last Updated 15/09/2017 08:18

 

On Thursday the 14th of September at 10PM (BST) we will be performing maintenance and upgrades on our Exchange system.

Maintenance Start: 20:00 (BST) 14th September 2017

Maintenance End: 00:00 (BST) 15th September 2017

During the maintenance window users might experience some reduced performance, however the service should remain online; if you do experience any issues please raise a support case in the usual manner

Exchange Delivery Issue (Resolved) Critical
  • 04/08/2017 10:41 - 04/08/2017 12:32
  • Last Updated 04/08/2017 10:46

Dear all,

We are aware of an issue with incoming mail delivery on our Exchange platform and our senior systems engineers are investigating this right now. All emails will be cachced and delivered once full service has been restored.

We apologise for the inconvenience that this has caused and hope to have this resoved as quickly as possible.

Kind regards,

MTH Support Team

OWA Patch (Resolved) High
  • 02/06/2017 23:00 - 28/07/2017 20:44
  • Last Updated 02/06/2017 11:54

Hello,

We have been made aware of a new issue related to OWA (Outlook Web App) and we will be applying a patch to fix this.

There will be no downtime whilst this patch is being applied but you may notice some slowness when using the service.

This will commence tonight at 23:00 - we envisage this to be resolved by 1am.

If you have any queries related to this, please email [email protected].

Kind regards,

MTH Support


West Midlands Data Centre issue (Resolved) Critical

Affecting Server - VPS Control Panel

  • 30/11/2016 10:55 - 08/12/2016 17:08
  • Last Updated 08/12/2016 17:08

All,

We are currently aware of an issue affecting services at our West Mdland data centrre and we hope to have the issue resolved as soon as possible.
Please note this will cause some services to be inaccessible whilst the data centre engineers work on resolving this issue.

An update will follow in due course.

Kind Regards
MTH Team



MTH Desktop Gateway (Resolved) Medium

Affecting System - MTH Desktop

  • 09/12/2016 20:00 - 25/04/2017 23:52
  • Last Updated 07/12/2016 11:49

Hi All,

During the window above we will be performing maintainance to our Remote Desktop Gatways. This is intended to have no effect on the end user but as we will be working on one, and then the other system; it will be classed as "at risk".

Kind Regards

MTH Team

MTH Hosted Desktop Maintainance (Resolved) Medium

Affecting System - File Systems

  • 07/11/2016 20:00 - 25/04/2017 23:52
  • Last Updated 08/11/2016 16:11

All,

We are performing some maintainance on the file structure for our MTH Hosted Desktop product.
Please note this will cause some tempory performance issues, and thus is being performed out of business hours.

Kind Regards
MTH Team

==
Maintainance was conducted. Further restarts are required.
==

Microsoft Exchange (Resolved) High

Affecting System - Microsoft Exchange

  • 19/08/2016 13:00 - 08/11/2016 16:11
  • Last Updated 08/11/2016 16:11

We are currently experiencing client connectivity issues to our Microsoft Exchange platform from one of our data centre locations; we've re-routed this traffic and our users should not be affected by this issue.

We will update this notification once traffic route returns to normal

Network Maintainance (Resolved) High

Affecting Other - MTH Network

  • 06/10/2016 21:00 - 07/11/2016 13:17
  • Last Updated 06/10/2016 10:33

Hi All,

During the period mentioned, we will be taking the links between two of our DCs offline for some maintainance. As all our applications and systems are designed to match across multiple Data Centres this will not be a problem and should have no impacts on the end user.

Kind Regards
MTH Team

Skype for Business Maintainance (Resolved) Critical

Affecting System - MS Skype For Business

  • 27/09/2016 18:30 - 06/10/2016 10:31
  • Last Updated 26/09/2016 13:24

Hi All,

During the window above, Skype for Business will be unavailable.
This is to proceed and deploy new 2015 versions of Skype for Business.

If you have any problems, please email [email protected]

Kind Regards
MTH Team

Power Works (Resolved) Low

Affecting Other - MTH-2 DataCentre

  • 13/08/2016 00:00 - 06/10/2016 10:31
  • Last Updated 02/08/2016 10:56

We are informing you of on-site power works at our SC-2 Data Centre site, on the weekend of 13th-14th August 2016. Though the works will not affect power to the data centre itself, it will impact access to the wider SC-2 data centre site complex.


Thank you for your understanding.

 
Kind regards,

Your MTH Team

BT Network Issue (Resolved) Critical

Affecting Other - BT Network

  • 20/07/2016 07:23 - 19/08/2016 13:14
  • Last Updated 20/07/2016 09:24

All,

We are aware of an on-going BT network issue that is preventing many customers from connecting to our hosted infrastructure.
We will keep you up to date as this progresses. BT are aware of this issue and have publically acknowledged the issue.

Kind Regards
MTH Team

Exchange Reboots (Resolved) Medium

Affecting System - MTH Hosted Exchange

  • 18/07/2016 21:00 - 06/10/2016 10:31
  • Last Updated 18/07/2016 12:34

All,

During the window above the Exchange Servers will be rebooted one by one. This will cause no interuption, and as each server comes online we will establish connection before rebooting the other.
There may be a slight delay in email propagation during this period.

Kind Regards
MTH Team

Exchange Maintenance (Resolved) Critical

Affecting System - MS Hosted Exchange

  • 13/07/2016 23:59 - 14/07/2016 01:00
  • Last Updated 18/07/2016 12:31

Hi All,

During the window above we will be performing some network and exchange routing maintainance.
This will cause a slight delay in mail delivery, and a short outage with Outlook Web Access.

Kind Regards
MTH Team

Hypervisor Networking Issue (Resolved) Critical

Affecting System - Site: WGC

  • 13/07/2016 23:59 - 19/08/2016 13:15
  • Last Updated 13/07/2016 16:24

Hi All,

We have identified a network routing issue with the base Hypervisors in one of our Data Centres. This is not causing any system-problems at the moment, however - has shown on trial and test devices to be causing problems later down the line.

Due to this we are taking emergency action tonight to investigate this.

Kind Regards
MTH Team

Network Maintainance (Resolved) Medium

Affecting System - MTH Hosted Infrastucture

  • 08/07/2016 23:30 - 09/07/2016 06:00
  • Last Updated 13/07/2016 16:22

Hi All,

We are scheduling some critical network maintainance in the time window shown above.
During this time, we do not expect any network outages, however - the network will be running "at risk" due to the works being carried out.

We will be replacing some of our site-to-site links with new hardware routes during this time, and also re-configuring out cross-site replication.
Due to this, those on Hosted applications balanced accross multiple sites may experience slight slowness during this period.

Should anyone have any concerns, please raise a ticket through your client area.

Kind Regards
MTH Team

==
Update:

This is complete.
==

Fibre infrastructure performance improvements (Resolved) Medium

Affecting Other - FTTC

  • 30/06/2016 22:00 - 01/07/2016 00:00
  • Last Updated 24/06/2016 15:26

We will be conducting scheduled maintenance on Thursday 30th June 2016, from 22.00 till 00:00. This is in order to improve the performance of our FTTC network infrastructure.

Your Business FTTC service will lose connectivity for a period of 1-2 minutes several times throughout this window.

All other services including dedicated fibre (lease line) will not be affected.

If you have any questions, please do raise a support ticket or give us a call in the office.

Kind Regards,

The MTH Team

**UPDATE**
This work was carried out with minimal interuption to client FTTC services. All actions complete

Exchange - Upgrade Maintenance (Resolved) Low

Affecting System - Hosted Exchange

  • 20/05/2016 20:00 - 21/05/2016 00:00
  • Last Updated 20/06/2016 16:09

Dear all,

On Friday 20th May between 20:00 - midnight, we will be performing upgrade maintenance on our Highly Available Exchange Cluster.

During this period some Hosted Desktop users may see a certificate error message notifcation, it is ok to click continue if you receive this message.

If you have any questions please feel free to contact us via the support portal. 

Kind regards,

MTH team

**UPDATE 21/05/16 01:00am**
Works have been complete with no problems

Exchange Configuration (Resolved) Low

Affecting System - MS Hosted Exchange

  • 07/05/2016 20:00 - 07/05/2016 23:59
  • Last Updated 17/05/2016 14:05

Hi,

During the hours of 20:00 > 23:59 on Saturday 7th May, we will be performing some Exchange configuration changes.
This could result in temporary slowness in Outlook or Web Access, however - the service will not be at risk.

Should you have any problems please open a support ticket from your client area or by emailing [email protected]

Kind Regards
MTH Team

** Update **

This work has now been completed sucessfully

MS Exchange Upgrade (Resolved) Critical

Affecting System - MS Hosted Desktop

  • 06/05/2016 20:00 - 07/05/2016 08:00
  • Last Updated 07/05/2016 11:11

Hi,

We are performing routine maintainance on the night of 06/05/2016, starting at 8pm.
This maintainance includes the installation of a MS Update patch to our Exchange Infrastructure. 

As we will have to restart one of the servers during this time, the service will be deemed 'at-risk'. 

Users may experience slowness during the period of 8pm > 8am.

Should you have any issues please raise a support ticket through the client area or by emailing [email protected]m

Kind Regards
MTH Team

[update 2000]
This is now in progress

[update 2300]
50% of the work is now complete without issue. Small slowness is noted but nothing effecting the majority.

[update 0100]
We are now 80% through, few more tasks and we will be complete.

[update 0300]
This work is now complete.

Mail Flow Delay (Resolved) Critical
  • 21/04/2016 09:25 - 21/04/2016 21:00
  • Last Updated 06/05/2016 16:25

UPDATE 14:30 UTC 

All infrastructure is now back in service and mail is being processed. Customers should see their mail queues dropping and new mail being delivered immediately. 

UPDATE 14:00 UTC 


Please be advised that Email is being processed but customers might still experience delays. 

Our Engineering team believe that they have identified the networking issue and are currently preparing to attempt to bring the London DC back into service. 

UPDATE 12:08 UTC 

We have seen evidence that the mitigation we have put in place is taking effect and that customers' mail queues are dropping. 


UPDATE 11:49 UTC 

Please be advised that mitigation is being rolled out across the infrastructure to reduce the load and should take effect in 20-30 mins. 

We will continue to update this notice as new information becomes available.

UPDATE 10:37 UTC 

Please be aware that customers using our infrastructure are still experiencing mail delays


UPDATE 09:45 UTC 

Email traffic is flowing without delay through the data centres whilst engineers continue to investigate the underlying issue. 

ORIGINAL MESSAGE 

Please be aware that customers are experiencing delays when sending and receiving email through infrastructure. 

The effected data centre has been taken out of service and mail is flowing through the other two locations. Networking and infrastructure engineering team are investigating the issue. 

We will continue to update this notice as new information becomes available. We sincerely apologize for any inconvenience caused while we work to resolve this issue. 

Exchange Modifications (Resolved) Medium

Affecting System - MS Hosted Exchange

  • 04/05/2016 20:00 - 06/05/2016 16:25
  • Last Updated 06/05/2016 16:25

Hi,

We are scheduling maintainance on our Exchange servers on the evening of 04/05/16. Starting at 8pm.
Each server will be patched in turn and then restarted. This will push users around the hosts, and they will not experience any disruption.

Users may experience slowness in Outlook and Outlook Web Access during this time.

If you have any issues please raise a support ticket through your client area.

Kind Regards
MTH Team

--

[update]
This work is now complete.

Exchange Connectivity (Resolved) Critical

Affecting System - Exchange

  • 24/03/2016 15:29
  • Last Updated 24/03/2016 15:49

Hi All,

We're experiencing connectivity issues on the Exchange platform at current.

As soon as we have an update for you, we will post it here.

Apologies,

MTH Team

**UPDATE** 24/03/16 15:40hrs
All connectivity is now restored; if there's any remaining issues, please do raise a support ticket. Slow searching in OWA or outlook is expected while the system re-indexes, however this will resolve within the hour.

Hosted Desktop - Addition of new site (Resolved) Critical

Affecting Other - MTH Hosted Desktop

  • 29/01/2016 22:00 - 04/05/2016 13:59
  • Last Updated 19/02/2016 12:53

All,

This weekend will see the addition of a new DC site to our Hosted Desktop Platform.
Due to the work of this nature, from Friday 10PM; until Monday 3AM - the system will be deemed "at risk".

We do not envisiage any outages in access to the platform, however - we will be altering system and network configurations outside of working hours (0900 - 1700), which could result in brief disconnection.

Should you have any issues or questions; please raise a support ticket by emailing [email protected]m

Kind Regards
MTH Team

==
Update: Work on the w/e 22/01 went well. We have successfully complete the addition and testing.
We are extending these working hours to this weekend too (29/01 - 31/01). Over this weekend we will be moving desktop access to originate from the secondary site as it has higher redundancy and converting the existing sites into fall-back configurations.
==

==
Update: Work on the w/e 30/01 went well. Addition of the new gateway servers has now been completed and testing will be with the team this week.
All going well; they will be implimented into action on 07/02/16.
==

==
Update: 07/02/16: New Gateway servers are being introduced this evening.
==

Server Move (Resolved) Critical

Affecting System - MTH Hosted Desktop

  • 19/02/2016 21:00 - 04/05/2016 13:59
  • Last Updated 19/02/2016 10:56

On 19/02/2016 we will be commencing work on one of our File Servers.
This file server is one of the first from the system design, and thus still has a few dependancies upon the operation of this.

During the window of time above, Hosted Desktop may be unavailable to some users with old dependancies on UK-MTH-FS01. This server is being migrated from onesite to another in order to combat latency issues that we have seen. Following this migration - we will be working to decommision the server and remove all dependancies from this server.

Should you have any questions; please email [email protected] to raise a support ticket.

Kind Regards
MTH Team

Exchange Server Network Expansion (Resolved) Critical

Affecting System - MS Hosted Exchange

  • 29/01/2016 22:00 - 31/01/2016 03:00
  • Last Updated 07/02/2016 16:00

All,

Futher to adding another DC location to our Hosted Desktop System, we are also taking this time to introduce an extra location for our MS Hosted Exchange system.
Through this weekend we will be bringing online and configuring a few more Exchange Servers in the new location, and configuring the network and cross connects between the locations.

As a result of this active network and system change and configuration - the service will remain "at risk" for the weekend.
You may notice temporary unavailablity of Outlook, and Outlook Web Access as configurations are updates and systems rebooted. These outages should last no longer than 5-10mins, and will not interupt mail flow.

Should you have any issues that you would like to raise please email support on [email protected]

Kind Regards
MTH Team

==
Update: Work on the w/e 22/01 went well. We have successfully complete the addition and testing.
We are extending these working hours to this weekend too (29/01 - 31/01). Over this weekend we will be moving mail flow to originate from the secondary site as it has higher redundancy and converting the existing sites into fall-back configurations.
==

==
Update: This is now complete.
==

MS Lync Server Upgrade (Resolved) Medium

Affecting System - MS Lync

  • 28/01/2016 18:00 - 28/01/2016 23:59
  • Last Updated 07/02/2016 16:00

Hi,

This evening from 1800 - 2359 Microsoft Lync (Skype for Business) will be unavailable.
This is due to upgrades and patches being applied by our system engineers; which will fix issues with Lync for Mobile and Remote Access of the service.

Should you have any issues, please raise a support ticket through your client account.

Kind Regards
MTH Team

==
Update: This is complete.
==

MS Exchange & Mac Mail (Resolved) High

Affecting System - MS Hosted Exchange

  • 27/01/2016 13:00 - 27/01/2016 15:36
  • Last Updated 27/01/2016 15:36

Hi,

We are aware of a current issue effecting Exchange and Mac Mail. This seems to be resolving from the Exchange Web Services however, is only effecting a select number of clients.
We will update this network notification as we have further updates for you.

Kind Regards
MTH Team

==
Update: 15:10
This is now resolved, and was caused by Exchange Web Services as expected.
==

External Interface Configuration (Resolved) Low

Affecting System - MTH Hosted Exchange

  • 10/01/2016 22:00 - 11/01/2016 03:00
  • Last Updated 27/01/2016 14:43

Hi All,

Sunday night, between the hours of 22:00 and Monday 03:00, we will be introducing a new external interface to the exchange platform.
This is in order to facilitate a new feature - Encrypted Mail Delivery.

We will notice a small delivery outage (30secs) when we introduce this to the platform - however; all mail will be stored at Symantec - and further processed once back online.

If you have any concerns please raise a support ticket and we will happily assist or answer any queries.

Kind Regards
MTH Team

==
Update: This work is now complete. There was no interuption to service.
==

MS Exchange Configuration Change (Resolved) High

Affecting System - MS Hosted Exchange

  • 13/01/2016 19:00 - 13/01/2016 20:00
  • Last Updated 21/01/2016 21:53

Hi All,

We are introducing a new aspect of our secure email flow this evening between 1900 // 2000. This may cause disruption to email flow in this period of time; however - everything will be queued and delivered once email flow is restored. We are not expecting this to last longer than 10mins.

Kind Regards
MTH Team

==
Update: This is now complete
==

Connectivity issues affecting VoIP gateway (Resolved) High

Affecting System - VoIP

  • 11/01/2016 15:56 - 11/01/2016 20:54
  • Last Updated 11/01/2016 23:02

We have been made aware of a problem affecting trunks on one of our gateways. This will be causing some trunks to deregister and loss of inbound and outbound service. Our systems engineers are currently looking into this issue as a priority and are working to resolve it as soon as possible.

If you have any questions please do not hesitate to contact us via the support ticket portal.



**UPDATE 16:51hrs**

MTH engineers are still investigating connectivity issues affecting the affected gateway. This appears to be a widespread connectivity issue affecting multiple data centres across the UK. Regular updates will be posted here. Any customers with open tickets will be updated on resolution, however are advised to keep check on this status page.

MTH apologise for the impact to our customers' service today.



**UPDATE 18:10hrs**

Engineers have confirmed that they have found a problematic route to one of our data centres. They are planning to implement a change of routing at 19:00hrs in an attempt to resolve ongoing connectivity issues.

Another update will be sent after this routing change is complete. Engineers are continuing to monitor.



**UPDATE 20:54hrs**

Engineers have successfully re-routed traffic away from problematic routes. All alarms have now cleared, and call traffic is returning to normal. Engineers are continuing to monitor, but a recurrence is not expected.

Once again, we apologise for the inconvenience caused today, and appreciate your patience.

If you have any further questions, or if you see a recurrence of any issues, please do not hesitate to contact us.


**UPDATE 22:50hrs**

Further to this evening's earlier update of issue resolution, we have confirmd that the root cause of this issue was due to a UK wide BT Openreach issue - this also largely affected a number of mobile network providers as well as mobile broadband providers.
A BBC article on EE and O2's issues can be seen at: http://www.bbc.co.uk/news/business-35286448

Exchange Server Introduction (Resolved) Medium

Affecting System - MS Hosted Exchange

  • 29/12/2015 18:00 - 08/01/2016 23:33
  • Last Updated 08/01/2016 23:33

Hi All,

We are introducing a new Exchange Server into the Cluster this evening (29/12/15).
This will not cause any network or server issues, however - the mailbox's are replicating over to the Exchange Server, and this may have an effect on the speed of the system.

Expected to be complete by: 29/12/15 @ 23:59:00

Kind Regards
MTH Team

==
Update 18:00:00
This has now started.
==

==
Update 23:45:00
This is now complete
==

Emergency Network Maintainance (Resolved) Critical

Affecting System - MTH Core

  • 03/01/2016 02:00 - 08/01/2016 23:32
  • Last Updated 29/12/2015 18:44

Hi All,

We will need to perform some emergency network maintainance to a set of our redundant routers in order to configure a redundany Layer 2 link to a new Data Center.
This has the pottential to cause a 5s outage, however - could cause no impact at all.

Target Date: Sunday 3rd January 2016 @ 02:00(am).

Kind Regards
MTH Team

Symantec.Cloud core issue (Resolved) Medium

Affecting System - MS Hosted Exchange

  • 15/12/2015 13:00 - 16/12/2015 00:25
  • Last Updated 16/12/2015 00:53

Afternoon,

Unfortunately we've been informed by Symantec (our spam and anti-virus filter providers for our exchange platform) that they're experiencing slow email throughput on their system. The cause is unknown at current and is only thought to be affecting a small percentage of domains routing through them. Emails are still flowing fine for most clients, however there is unfortunately a delay for a select few.

We will update once we have some more information; unfortunately this issue is out of our hands.

Kind Regards,

MTH Team

**UPDATE** 14:00
We've had confirmation that Symantec have identified the issue.

**UPDATE** 14:34
Symantec are implimenting a temporary work around to improve mail flow.

**UPDATE** 15:45
Temporary work around is in place; some clients may still be experiencing issues, however this will improve over the next few hours.

**UPDATE** 19:04
Symantec have reported to us that the issue was due to a core switch stack failure in their core London datacentre causing slowness of outbound routing; a full RFI (Reason For Issue) will be issued to us by tomorrow morning and this will be updated on this network issue page.

**UPDATE** 20:54
We haven't had any further news back from Symantec yet, however testing carried out by MTH has shown large improvements with minimal if any delay. If you are still experiencing delays, please do let us know by raising a support ticket. We're not stating that this issue is resolved, however initial testing is showing promising results.

**UPDATE** 00:25
We have just finished in discussions with Symantec Engineers and the service is now fully restored. MTH testing shows this to be the case.
Anyone with any further issues should contact [email protected]

MTH UK Shared Server Issues (Resolved) Critical

Affecting Server - UK Shared (N)

  • 23/11/2015 06:00 - 24/11/2015 18:39
  • Last Updated 24/11/2015 18:39

Hi All,

We are seeing high-load and network issues on the MTH UK Shared cPanel server (uk.mthservers.com).
We will update you as soon as we have a further update.

Kind Regards
MTH Team

==
Update: 11:30
This seems to be the result of several wordpress hacks; please take this time as a reminder to secure your wordpress installations.
We have an ETA of 22:00 for resolution.
==

==
Update: 16:24
We are still working on restoring a few accounts and bringing the server back to a viable state. Please bear with us.
==

==
Update 17:39
We are starting to see the load on this server reduce and sites are beginning to load further.
==

==
Update 23:00
This issue is all now resolved, we are seeing sites return to normal speed.
Clients experiencing any further issues should email "[email protected]" in order to open a support ticket.
==

MS Hosted Exchange Issue (Resolved) Critical

Affecting System - MS Hosted Exchange

  • 20/11/2015 10:00 - 20/11/2015 11:51
  • Last Updated 20/11/2015 11:50

Hi All,

We are seeing large Exchange Delays with MS Hosted Exchange.
We are investigating this at the moment, will update as we progress further.

Kind Regards
MTH Team

==
Update: 10:34
This is identified to be Disk I/O on HV3. We are restarting EX13-2 at the moment to kick exchange back to HV1
==

==
Update: 11:04
This is now resolved
==

MS Hosted Exchange - Emergency DR Testing (Resolved) Critical

Affecting System - MS Hosted Exchange

  • 13/11/2015 20:00 - 13/11/2015 23:59
  • Last Updated 13/11/2015 21:30

Hi All,

In light of recent issues that have occured with the MS Hosted Exchange Platform, we are bring forward some emergency disaster recovery testing in order to isolate and confirm no further problems exist.
There is a posiblity for the following services to go offline during testing:

- Outlook Web Access
- Outlook Desktop
- Mobile connectivity
- Inbound / Outbound Mail

All inbound mail will be buffered by out external delivery platform; and thus on the platform issues being resolved - any inbound email will be delivered.

The plan will be as follows;

1/ Suspend uk-mth-ex13-2 so that it appears it’s just fallen offline
Ø  Test OWA for mailbox access
Ø  Test Outlook for mailbox access
Ø  Test Internal messaging
 
Assuming all passes, bring back up uk-mth-ex13-2 and allow to re-sync (15 mins), before moving onto point 2; else resolve issues and re-test before moving forward.
 
2/ Suspend uk-mth-ex13 so that it appears it’s just fallen offline
Ø  Test OWA for mailbox access
Ø  Test Outlook for mailbox access
Ø  Test Internal messaging

Should you have any issues please open a support ticket and raise one of the team,

Kind Regards
MTH Team

==
Update: 20:49: This is now starting..
==

==
Update: 21:09: UK-MTH-EX13 has been shutdown; there was a brief 2min of inactivity where the system was slow; but access was retained. Copy is just being re-sync'd across the two servers.
==

==
Update: 21:24: UK-MTH-EX13-2 has been shutdown; there was a brief 2min of inactivity where the system was slow; but access was retained. Copy is just being re-sync'd across the two servers, further to this the system is back online completely now. DR Testing is complete and has been deemed as a success. A full report will be generated and reased before close of business on 16/11/2015.
==

MS Hosted Exchange - Outbound Route (Resolved) High

Affecting System - MS Hosted Exchange

  • 13/11/2015 21:00 - 13/11/2015 23:59
  • Last Updated 13/11/2015 20:49

Hi all,

Due to recent issues with outbound mail flow we have bought forward our upgrade to our outbound mail flow systems.
This upgrade will allow us greater control and flexibility on the management and tracking of outbound mails on the MS Hosted Exchange infrastructure.

This upgrade means that we will be moving over to pass mail through a new outbound cluster; that is security scanned by Symantec. Due to the change in outbound clusters; we will require a change in SPF records.
During the period mentioned below; MTH will go through all clients with MS Hosted Exchange and change SPF records, however - any clients hosting external DNS will need to make this change themselves. MTH will endevour to find out these domains and contact you before Friday.

The new SPF record is: v=spf1 +a +mx +a:spf.messagelabs.com ~all

Please open a support ticket should you need any assistance with this,

Kind Regards
Steve

==
Update: 21:02: New outbound route is now in place.
SPF records are being updated by MTH; if you control DNS yourself please update your DNS to as above.
==

MS Hosted Exchange Route (Resolved) Low

Affecting System - MS Hosted Exchange

  • 12/11/2015 19:00 - 12/11/2015 19:30
  • Last Updated 13/11/2015 00:38

Hi All,

We are making an energency upgrade to our inbound spam filering platform with regards to recent issues that have occured.

Part of this upgrade will involve the change of MX records used for the MS Hosted Exchange platform, as we route mail through our new Anti-Spam cluster.

New MX records are:

cluster8.eu.messagelabs.com - Priority: 10
cluster8a.eu.messagelabs.com - Priority: 20

MTH will be performing this update manually on all clients with DNS held locally at MTH; further to this - we will be contacting anyone with DNS held externally in order to make this change.
This update will be performed on 12/11/2015 @ 19:00 and is expected to take 30mins to process all clients.

We will be running both Anti-Spam systems in parralel for the following 2 weeks in order to allow everyone time to transisiton over, and to prevent the loss of any mail during change over.

Please raise a support ticket if you have any queries,

Kind Regards
Steve

==
Update: 12/11/2015 19:53
This is now complete. All clients on MS Hosted Exchange have been migrated over to our new inbound cluster. Should we have any issues with your individual accounts we will be contacting you tomorrow AM.
==

Exchange Configuration Changes (Resolved) Critical

Affecting System - MTH Hosted Exchange

  • 10/11/2015 23:59 - 13/11/2015 00:37
  • Last Updated 13/11/2015 00:37

All,

Tonight we will be performing emergency maintainance on our Exchange Cluster.

The below is the work that will be carried out:

Separation of the MAPI Replication Network to an additional adapter
Reschedule of the Shadow Reply
Reschedule of the Exchange Index Route

Kind Regards
MTH Team

==
Update; 11/11/2015 @ 01:35

This work is all now compelete. No downtime was occured.
==

MS Hosted Exchange Outage (Resolved) Critical

Affecting System - MS Hosted Exchange

  • 10/11/2015 13:05 - 10/11/2015 14:00
  • Last Updated 10/11/2015 23:16

All,

During the period above - an outage was experienced on the MS Hosted Exchange Platform.
A further RFO (Reason for Issue) will be issued once investigation work is complete.

Kind Regards
MTH Team

MTH Desktop - Improvements (Resolved) High

Affecting System - MTH Desktop Infrastructure

  • 16/10/2015 21:00 - 10/11/2015 23:08
  • Last Updated 16/10/2015 17:23

All,

On 16/10/2015 there will be a short outage of the internet from within the MTH Desktop System.
Access to the service will remain available, but between the hours of 2100 - 2130; there will be no internet access to external applications.

These improvements are in order to route the internet through a Threat Management Gateway; which will allow finer details and control of internet access through organisations.

Kind Regards
MTH Team

Issue on Hypervisor #3 (Resolved) Critical

Affecting System - HV-WM-03

  • 07/10/2015 12:22 - 16/10/2015 11:23
  • Last Updated 16/10/2015 11:22

All,

We have identified a stability issue with one of our hypervisors. Currently all nodes are stable and connection can be established, however - we have scheduled emergency maintainance for this evening in which will see the effected hypervisors restart with patch updates.

Kind Regards
MTH Team

==
Update:
This is now complete and the stability issues are resolved.
==

Hosted Desktop & Exchange Maintenance (Resolved) Medium

Affecting System - Hosted Desktop & Exchange

  • 08/09/2015 22:00 - 10/11/2015 23:08
  • Last Updated 05/10/2015 11:15

Hi All,

Following the recent issues we've had with the Hosted Desktop and Exchange platforms we've taken decision to implement some improvments to the system; this is also going to include 2 additional measures of failover to ensure future reliability is retained.

The 1st maintenance period will be proceeding Saturday 29th August, 22:00hrs till Sunday 01:00hrs - through this, we expect to see around 20-30mins of intermittent downtime; due to our email buffering system, no incoming emails will be affected or bounced.

The 2nd maintenance period will be proceeding Sunday 30th August 21:00hrs till Monday 03:00hrs - through this, we don't expect to see any downtime, however maintenance will be going on through this timeframe.

Updates will be posted below following each maintenance completion period.

Regards,

MTH Team

**Update 21:59hrs 29/08/15**
Work has started.

**Update 23:35hrs 29/08/15**
Work has now all completed successfully with no further interruption to service.

**Update 23:50 30/08.15**
Work has been suspended for this evening. This is due to a routing issue identified by our pre-work tests. We have re-scheduled this work for next weekend.

The 1st maintenance period will be proceeding Saturday 5th September, 22:00hrs till Sunday 01:00hrs - through this, we expect to see around 20-30mins of intermittent downtime; due to our email buffering system, no incoming emails will be affected or bounced.

The 2nd maintenance period will be proceeding Sunday 6th September, 21:00hrs till Monday 03:00hrs - through this, we don't expect to see any downtime, however maintenance will be going on through this timeframe.

**Update**
The final phase of this maintenance upgrade work will be carried out on Tuesday 8th from 22:00hrs - Wednesday 9th 01:00hrs - during this time, we expect to see around 10-15mins downtime to the hosted desktop platform.

**Update**
This work is confirmed to be proceeding this evening - if anyone has any last minute questions, comments or requests, please do let us know via the support ticket system.

**Update**
This work has been returned to the testing phase due to unforseen circumstances. Issues that did not show up during phase 1 testing have become apparent and we are just returning to testing to check these senarios for any pottential complications.

West Midlands Datacentre Core upgrade (Resolved) High

Affecting System - Core Networking Infrastructure

  • 21/08/2015 14:04
  • Last Updated 05/10/2015 11:13

Dear Customers,

For some time we have been experiencing a large number of network issues and outages at our West Midlands site, this issue has been with the redundant stack of gateway switches that form part of our core networking infrastructure at this site.


We have been working tirelessly with Juniper Technical support to try and resolve the issues that we have been experiencing however we have reached the conclusion that the platform that we use at this site is no longer fit for purpose. 

As a result within the last few weeks we have placed an order for a number of Juniper MX104 routers that will take over the main routing of our network at our West Midlands site. Once implemented all existing customers at the West Midlands site will be terminated on these MPLS enabled devices. We have been utilising the MX series routers at our core Docklands sites for a number of years where they have performed faultlessly and the MX104 is the latest generation in this range of devices. 

We are going to be conducting a number of tests on these devices before migrating customers to the platform however given the recent issues with services at this site we are keen to bring about this migration swiftly. We anticipate a thirty second outage whilst migrating your services to this platform and this will be carried out between the hours of 02:00 and 05:00 GMT, an update will be posted below in due course with more information on when this upgrade will take place.

If you have any questions with regards to the above, please raise a support ticket and we will happily speak with you.

Kind Regards,

MTH Team

Network Maintenance (Resolved) High

Affecting Other - MTH Core Infrastructure

  • 12/09/2015 11:00 - 12/09/2015 13:00
  • Last Updated 05/10/2015 11:13

We have had notification that during the following dates one of our fibre providers providing the fibre links between our West Midlands datacentre and London PoPs will be performing work on their network. This will result in one leg of our ring being unavailable for this time. 

Due to the diverse nature of our network this should not be service impacting for any of our customers but you should be aware that our ring will be operating without it’s second redundant leg during this period and therefore during this time the network to our West Midlands datacentre will be deemed as “at risk”.

Start Time 12/09/2013 – 11:00 GMT
End Time 12/09/2013 – 13:00 GMT

Start Time 20/09/2013 – 01:30 GMT
End Time 20/09/2013 – 02:00 GMT

Hypervisor Updates (Resolved) High

Affecting System - MTH HV1, 2 & 3

  • 02/10/2015 23:00 - 03/10/2015 03:00
  • Last Updated 05/10/2015 11:13

Hi,

We have scheduled in an update patch for 3 of our Hypervisors on Friday 2nd October between the hours of 23:00 and 03:00 (Saturday 3rd October).

During this period, each Hypervisor will be performing updates and patches, and then will restart 2 times.
Each Hypervisor will be restarted in turn, so only one system will ever be offline at any time.

Please let us know if yo have any queries,

Kind Regards
MTH Team.

**Update**
03/10/2015 - 1:05
All updates and restarts have been completed and this Network Notification is now closed and all services are returning to normal.

02/10/2015 - 23:06
Work as described above is now starting - please call the office if you have any issues.

New / One off Direct Debit not completing (Resolved) Medium

Affecting System - Direct Debit System

  • 23/08/2015 13:33 - 24/08/2015 13:00
  • Last Updated 27/08/2015 17:10

Afternoon,

We've identified an issue with the GoCardless module we use for DirectDebit processing - this only affects those setting up new payments and trying to make one-off or upgrade payments.

This issue is being looked into and a fix will be implimented soon, however payments affected will have to use another payment method such as Card or BACS.

Apologies for this inconvenience; we will update this notification once we have any further information.

Kind Regards,

MTH Team

==
Update: 24/08/2015 13:00

This issue is now resolved, and all payments are sucesfully processing.
==

Virtualisation Issue (Resolved) Critical

Affecting Other - HV 1

  • 19/08/2015 11:34 - 19/08/2015 15:54
  • Last Updated 19/08/2015 19:47

Morning All,

Unfortunately, we've just experienced a major issue with one of our hypervisors which has taken some Virtual Machines offline at current.

We apologies for this and we're working on this as we speak to get things back up and running.

Regards,

MTH Support Team

**UPDATE 13:43hrs 19/08/15**

This isssue is showing to be resolved now and was not related to the Hypervisor, however two specific Virtual Machines. This did take offline our Hosted Desktop service and Exchange, however service has been restored and all sessions have been retained.

There will be some remedial works on the Hosted Desktop platform this evening out of hours, however everything is looking stable at current.

This report will be updated once a full system check is complete this evening.

**UPDATE 15:18hrs 19/08/15**

The Hosted Desktop platform is still experiencing issues with user access. Mac users will be able to connect, however Windows users are unable to bypass a certificate warning. This is being worked on as we speak and will update the ticket when resolved.

**UPDATE 15:14hrs 19/08/15**

Hosted Desktop access is restored for Mac users, however Windows users are still experiencing certificate issues.

**UPDATE 15:54hrs 19/08/15**

The Hosted Desktop platform is back online and full service is restored. If anyone is experiencing any further connection issues, please report them ASAP to our support team and we will investigate, however our tests are showing full issue resolution; we can only apologise for the length of disruption to your business critical IT systems today.

UK Shared Crash (Resolved) Critical

Affecting Server - UK Shared (N)

  • 05/08/2015 13:50
  • Last Updated 07/08/2015 13:10

Hi,

Unfortunately we've had the UK cPanel Shared server crash on us this afternoon.

We're currently working to bring it back online - please standby for further updates.

Our apologies for this,

Regards,

MTH Team

**UPDATE**
As of 15:00hrs this issue was resolved. A full RFO will be updated on this shortly.

UK Shared Incoming Email issue (Resolved) Critical

Affecting Server - UK Shared (N)

  • 14/07/2015 12:00 - 14/07/2015 14:00
  • Last Updated 21/07/2015 14:41

Hi,

We have identified an incoming email issue with the UK Shared node with the hostname uk.mthservers.com
This is currently being investigated and we will update this notice when we have a timeframe.

Please open a suppport ticket if you have any queries,

Kind Regards
MTH Team

===
**UPDATE** 14/7/15, 12:22pm

We believe to have resolved this issue, however are running some tests to confirm resolution.
===
**UPDATE** 14/7/15, 14:00hrs
Issue confirmed resolved.

UK Shared hostname DNS (Resolved) Medium

Affecting Server - UK Shared (N)

  • 21/07/2015 08:43 - 21/07/2015 12:40
  • Last Updated 21/07/2015 14:40

Hi all,

Unfortunatley, we've encountered a DNS issue with the hostname uk.mthservers.com; this is currently resolving incorrectly.

This shouldn't have a massive impact, however those of you who are accessing your reseller accounts, or using that URL for webmail/cpanel access will encounter blank screens.

A temporary measure is to use your own domain in place ie domain.com/webmail or domain.com/cpanel and this will get yourself into where you require.

As soon as we have an update, we will post this below.

Regards,

MTH Team

**UPDATE**
This issue has now been resolved and all should be back to normal. If you see any further issues, please open a support ticket.

UK Shared Server (Resolved) Critical

Affecting Server - UK Shared (N)

  • 14/07/2015 09:04 - 14/07/2015 10:05
  • Last Updated 14/07/2015 10:22

Hi,

We have identified a MySQL issue with the UK Shared node with the hostname uk.mthservers.com
This is currently being investigated and we have a timeframe of roughly 10mins for resolution.

Please open a suppport ticket if you have any queries,

Kind Regards
MTH Team

===

**UPDATE** 14/7/15, 09:41hrs
Our apologies, the resolution here is taking a little longer than expected. We're working as hard as we can to get all back up and running; thank you for your patience.

===

===

**UPDATE** 14/7/15, 10:05hrs
This is all resolved and all sites are back up. This was a major MySQL crash that we recovered, caused by a massive number of external connections. In order to combat this, we have limited external connections to known IP address' only.

===

MTH Core Network Issue (Resolved) Critical

Affecting Other - MTH Core Network

  • 27/05/2015 11:05 - 27/05/2015 12:36
  • Last Updated 27/05/2015 14:28

We have a reported DC Network Outtage caused by an upstream router issue. We are investigating this issue at the moment and updates will follow shortly.

MTH Team

==
12:00 Update - Stability Increase
We've seen increased stability for the last 10 mins; however - we are still waiting on final response from the DC on root cause and for them to give the all clear.
==
12:15 Update - DC Cause
We have identified this issue to seem to be a wide spread issue affecting much of the UK's connectivity at the London Internet Exchange (LINX).
We have disabled our peering at LINX for now and all services should now be back online.
==
12:34 Update - Complete 
We have seen around 1 hour of uptime and stability now, and are confident that this issue has been resolves by the DC.
==

Hosted Desktop Maintenance Notification (Resolved) Medium

Affecting Other - Hosted Desktop / Cloud IT

  • 12/05/2015 00:00 - 12/05/2015 00:30
  • Last Updated 26/05/2015 06:57

A small maintenance windows is required to impliment some additional performance and security measures into the file system of the Hosted Desktop Infrastructure.

Normally this wouldn't incur any downtime, however we will need to reboot a cluster of servers for this enhancement to be picked up.

Apologies for the short notification window, however if anyone has any issues with the scheduled time of these works, please do let us know ASAP by raising a ticket from your client area.

Kind Regards,

MTH Team

Planned Maintenance (Resolved) Medium

Affecting System - Network

  • 09/04/2015 03:01 - 09/04/2015 04:01
  • Last Updated 11/05/2015 11:50

Dear Customer,

Notification Type: Planned Maintenance - Coventry

When:
Maintenance Window Start Time: 09/04/2015 – 03:01 GMT +1
Maintenance Window End Time: 09/04/2015 – 04:01 GMT +1

Service Impacting?
Due to the redundant nature of our network this maintenance should not be service affecting however due to the work involved the network is deemed at risk during this period.

Detail of work being carried out:
During this period we are going to be carrying out work on our switching platform at our Coventry site which involves breaking the virtual chassis stack of switches currently providing services to customers at this site and inserting a number of new switches into the stack.

Service(s) Affected:
All dedicated server, co-location and IP transit services at our Coventry site.

Updates throughout this maintenance window will be posted on our status page.

Please feel free to get in contact with our support team with any questions you may have.

Hosted Desktop Maintenance Notification (Resolved) Medium

Affecting System - Hosted Desktop / Cloud IT

  • 27/03/2015 21:00 - 27/03/2015 22:00
  • Last Updated 27/03/2015 22:14

Hi All,

We're performing some routine maintenance and upgrades to the Hosted Desktop platform Friday 27th evening.

The works require us to unfortunately make the service unavailable for up to an hour (9pm to 10pm), however following this works there will also be noticeable performance improvements to the system.

If anyone has any issues with this timing, please do let us know via a support ticket ASAP so we can accommodate.

Kind Regards,

MTH Team

UPDATE 27/3/15, 19:14hrs
Just a brief update to confirm this work is proceeding as planned. To confirm, the hosted desktop service will be unavailable this evening from 9pm for up to 1-hour max.

UPDATE 27/3/15, 22:11hrs
Completion update - all has been completed, tested and signed off as complete. Any issues, please open a support ticket.

Network Issues: Coventry (Resolved) Critical

Affecting System - DC Core Network

  • 03/02/2015 13:23 - 03/02/2015 14:07
  • Last Updated 03/02/2015 14:07

All,

We are experiencing network issues at our Coventry DC and the DC Technicials are rebooting a core piece of network equipment, we will provide updates soon.

MTH Team

==
14:02 UPDATE:

This issue is now resolved. A full RFO will be available within the next 24 hours by request.
==

Backup Licensing issue (Resolved) Medium

Affecting Other - Backup System

  • 30/12/2014 00:02 - 30/12/2014 13:18
  • Last Updated 30/12/2014 14:16

Hi All,

For those of you who have upgraded to our R1Soft backup service, the R1Soft licensing server had some maintenance undertaken on it lastnight - this kicked out our license key and we've had to revalidate this today.

Unfortunately this means backups from midnight until midday today have failed due to no valid licence key, however this is all back online and valid now - we've processed manual backup's to replace those which should have occurred lastnight.

This issue is deemed to be fully resolved, but if anyone has any questions, comments or issues remaining, please do get in touch with out Support Team.

Kind Regards,

MTH Team

Internal Network Maintenance & Upgrade (Resolved) Medium

Affecting System - All Coventry Services

  • 12/12/2014 00:00 - 12/12/2014 05:00
  • Last Updated 27/12/2014 15:20

Hi guys,

We've got some internal network upgrades to impliment which will add a higher data throughput allowance to our network as well as increasing redundancy by adding an additional stacked switch configuration.

Unfortunately, we do expect intermittent network downtime, however this will be kept to a minimum; expected is 5mins max network downtime.

As shown in the Network Details, we've planned this work to be undertaken between midnight and 5am on Friday 12th December as to not interupt anyone in too much of a critical time. If anyone has any issues with this planned time, please do raise these with us and we'll ensure this is re-arranged around your requirements; whilest this cannot be guaranteed, we will do our best to do so. The sooner a request is received the more likely we'll be able to re-schedule.

Kind Regards,

MTH Team

UPDATE: 02:38am, 12/12/14
Switch and network maintenance has started. We don't expect any major outages, however there could be a few 10-15min blips in network connectivity.

UPDATE: 05:00am, 12/12/14
All switch and network maintenance has been completed - network connectivity was limited to less than 2min total.

UPDATE: 03:18pm, 27/12/14
Following the maintenance no issues or problems have been reported - network maintenance closed off as completed successfully.

MTH Mail: Upgrade & Maintenance (Resolved) Low

Affecting System - MTH Mail System - mthmail.com

  • 01/12/2014 08:45 - 02/12/2014 23:59
  • Last Updated 03/12/2014 00:27

Hi All,

We've taken our mthmail.com app offline to do some maintenance on it's server and also perform a few program updates.

This should be back online Tuesday night for you, however email is still accessible via https://ukshared.monkeytreehosting.com/webmail

Kind Regards,

MTH Team

==
Update: 03/12/14 - 00:10
MTH Mail System is now back online. We've added features such as Dropbox Connections, and a place to upload files you send to people.
For more information, please open a support ticket and one of our technical team will be more than happy to publish the changelog.
==

VPS Node 1 Hardware Issue (Resolved) High

Affecting Other - VPS Node 1

  • 30/11/2014 17:00 - 01/12/2014 00:10
  • Last Updated 01/12/2014 00:26

Hi All,

Unfortunately we've been notified by our monitoring systems of a hardware issue with VPS node 1. The implications aren't critical, however we'd prefer that client's servers on this node were not sitting "at risk", so have made the decision to migrate them across to another stable node this evening.

The implications of this will be a minor unavailability of access as they will need to be shutdown, migrated, and re-started; depending on the size of HDD you have on your VPS, this could be between 30mins - 60mins.

We apologise for the inconvenience, however we feel this is in the best interest of your continued service. Those of you that we've been able to contact to discuss this, we've initiated the migration already - those who we've been unable to contact will be done later on this evening.

If anyone has any questions, concerns or general comments, please do open a support ticket and we'll respond to you shortly.

Kind Regards,

MTH Team

========

19:11hrs UPDATE:
5 of 13 VPSes have completed migration

00:10hrs UPDATE:
100% migration complete - all VPSes have been tested and showing live. If anyone has any issues, please raise a support ticket.

Google Service Disruption (Resolved) Medium

Affecting Other - Google Apps for Work

  • 22/10/2014 07:01 - 27/10/2014 00:00
  • Last Updated 06/11/2014 22:33

Evening All,

We've been made aware that Google are reporting some service disruption currently across a number of their services.

Of course we'll keep this page updated with anything we have, but the best port of call is to keep up with the Google Status page:

http://www.google.co.uk/appsstatus#hl=en&v=status

Regards,

MTH Team

========
UPDATE
Apologies for the delay in updating this network issue; Google have confirmed to have resolved this issue last week and this is also reflected on their status page linked above.

Emergency Switch Reboot (Resolved) Critical

Affecting System - Core Network Stack

  • 21/10/2014 14:10 - 23/10/2014 16:13
  • Last Updated 21/10/2014 15:02

Notification Type: Un-Planned Maintenance

Maintenance Window Start Time: 21/10/2014 14:10:00 GMT +1
Maintenance Window End Time: 15/10/2014 14:15:00 GMT +1
Service: Shared, VPS, Dedicated Servers, Co-Location, Rackspace, IP Transit

Location: Coventry

During the period above we had to place an emergency reboot on the Core Network Stack in our Coventry DC due to an update package as a result of attacks on the Network Infrastructure and advise taken from CISCO Engineers. This has patched all issues relating to the network, and we are now in the process of implimenting a dual-stack network Infrastructure, which will allow us to perform such actions to the network core without interuption to services.

We apologise for the minor outage, although this was limited to 5 minutes network downtime.

We have checked all services, and they are up and running - however; should you have any problems - please contact us by opening a support ticket.

Kind Regards
MTH Team

Planned Maintenance - Coventry - 15/10/2014 (Resolved) High

Affecting System - Core Network Stack

  • 15/10/2014 03:00 - 15/10/2014 10:00
  • Last Updated 15/10/2014 10:52

Notification Type: Planned Maintenance

Maintenance Window Start Time: 15/10/2014 03:00 GMT +1
Maintenance Window End Time: 15/10/2014 07:00 GMT +1
Service: Shared, VPS, Dedicated Servers, Co-Location, Rackspace, IP Transit

Location: Coventry

During this maintenance window we are going to be upgrading the JUNOS software on the redundant network stack of devices that form part of customers gateways at our Coventry site. For shared, VPS, dedicated server and co-location customers this should not be service affecting but due to the nature of the works all clients at this site should consider the network at risk during this period. Customers with rackspace/transit services at this site connected with a single uplink may notice downtime of up to ten minutes whilst the particular switch they are connected to reboots once upgraded.


Full service updates will be posted at https://secure.monkeytreehosting.com/serverstatus.php throughout this maintenance window. This post will also be updated below as always, with time and date stamps too.

If you have any questions regarding this maintenance please do not hesitate to submit to submit a support ticket via your client portal.

Regards
MTH Team

UPDATE 15/10/14 04:20 GMT +1
We are currently seeing lost connectivity to a number of racks in relation to the planned maintenance this morning. We hope to have service fully restored shortly.

UPDATE 15/10/14 05:55 GMT +1
Maintenance is now complete at our Coventry site. Full service has been restored.

UPDATE 15/10/14 08:31 GMT +1
All maintenance has been complete and all clients should be back online. Apologies to those affected by the network downtime. As always, if anyone has any issues, please raise a support ticket and we'll gladly assist you to resolve these.

Please note, all servers remained online and live, only network connectivity was lost and subsequently restored.

UPDARE 15/10/14 10:45 GMT + 1
We have experienced a temporary network outage as a result of a fault of an upstream providor. This outage was experienced between 09:42 and 09:59. We are now seeing 100% connectivity through the network and all servers are online.
If you have any questions regarding this maintenance please do not hesitate to submit to submit a support ticket via your client portal.


Planned Maintenance (Resolved) Low

Affecting System - Planned Maintenance

  • 12/09/2014 11:00 - 20/09/2014 02:00
  • Last Updated 08/10/2014 18:20

We have had notification that during the following dates one of our fibre providers providing the fibre links between our Coventry datacentre and London PoPs will be performing work on their network. This will result in one leg of our ring being unavailable for this time. 

Due to the diverse nature of our network this should not be service impacting for any of our customers but you should be aware that our ring will be operating without it’s second redundant leg during this period and therefore during this time the network to our Coventry datacentre will be deemed as “at risk”.

Start Time 12/09/2013 – 11:00 GMT
End Time 12/09/2013 – 13:00 GMT

Start Time 20/09/2013 – 01:30 GMT
End Time 20/09/2013 – 02:00 GMT

UK Node 2 Maintenance (Resolved) Medium

Affecting System - UK VPS Node 2

  • 19/09/2014 01:00 - 19/09/2014 01:30
  • Last Updated 19/09/2014 01:32

Hey guys,

We've got some BIOS and Firmware updates to do to the hardware node your Virtual Servers are situated on. We've already complete this work on UK Node 1 without any issues.

This work requires 2x reboots of the server which we'll plan at 01:10hrs and 01:20hrs accordingly.

Through the process we'll be updating this ticket, however don't expect more than 2x 10mins downtime. If anyone has any concerns, questions, or comments, please let us know by raising a support ticket.

Regards,

MTH Team

13/09/14 UPDATE:
Apologies, we've delayed this until Friday 19th, 1am.

19/09/14 UPDATE:
The BIOS update is currently running. First reboot expected around 12:30am

19/09/14 UPDATE:
BIOS update is complete and successful @ 00:36am
Reboot complete and everything has been checked to be back online @ 00:45am

19/09/14 UPDATE:
RAID Card update complete and successful @ 01:30am
Server is rebooted and back online.

As always, if anyone has any issues, please raise a support ticket and we'll look into it immediately for you.

MTH Team

UK VPS Node 1 - VPS Migration (Resolved) Medium

Affecting Other - UK VPS Node 1

  • 26/07/2014 12:00 - 01/08/2014 00:00
  • Last Updated 19/08/2014 10:22

Following the issues we've had with UK VPS Node 1 recently, we've decided to give it a full re-build to be sure we can completely eliminate any risk factors associated with it.

To enable us to do this, those clients who's VPSes are on this Node will be contacted shortly to arrange their migration across to one of our other VPS Nodes.

If anyone has any questions with regards to this, or is experiencing any issues, please raise a support ticket.

Kind Regards,

MTH Team

UPDATE 30/7/14 07:39hrs
This is confirmed as still planned for this evening. As mentioned, we're expecting around 1-2hours of downtime from midnight till 2am. This ticket will be updated once complete.

UK Node 1 - SQL Issue (Resolved) Critical
  • 25/07/2014 21:03 - 27/07/2014 11:00
  • Last Updated 27/07/2014 11:20

We currently have an issue with UK Node 1 and the SQL service. We are looking into it at the moment and will keep this System updated.

MTH Server Team

UPDATE 27/7/14 11:00hrs
We've isolated this issue and believe to have resolved this. If anyone experiences any further issues, please raise a support ticket.

Regards,

MTH Server Team

UK VPS Node 1 - Downtime (Resolved) Critical

Affecting Other - UK Node 1

  • 25/07/2014 19:45 - 25/07/2014 19:45
  • Last Updated 25/07/2014 19:52

Hi guys,

We just experience about 1.5hrs downtime on our UK Node 1 server - this will have affected any VPS clients on this server as well as all shared hosting clients and email hosting clients.

The reason for this was initially a drive failure in the machine which was in RAID-10 so we replaced it and it was re-building; mid-rebuild a 2nd drive failed which almost corrupted our array.

We managed to quickly resolve this very rare situation and luckily retain everyone's data through a lot of hard work from the team.

If anyone's VPS or website isn't loading, please raise a support ticket and we'll look into it for you.

Cheers,

MTH Team

UK VPS Node 1 - HDD Failure (Resolved) High

Affecting Other - UK Node 1

  • 23/07/2014 12:22 - 25/07/2014 15:04
  • Last Updated 25/07/2014 19:50

Hi all,

We've been alerted by our monitoring systems that we've had a Hard Drive failure in the RAID10 array of UK VPS Node 1.

New hardware is on it's way to the DC and should be installed shortly.

This server is currently running on "at risk" status until the new HDD's have been installed, however all your data and running should be unaffected.

As always, if anyone has any issues they notice, please raise a support ticket and we'll respond for you shortly.

Kind Regards,

MTH Team

UPDATE 24/7/14 16:03hrs
The new HDD has been put in and is currently rebuilding. VPSes on this node may notice a slow HDD I/O speed while this is occuring, however will resume to normal once complete.
This is currently at 6% complete.

UPDATE 24/7/14 16:03hrs
New HDD is still rebuilding nicely and is now at 11%, however a 2nd HDD has failed during this re-build, so we'll have to await the first HDD to finalise re-build and then replace the 2nd failed HDD and await this to rebuild. This is a very rare occurance, however one of the main reasons appart from performance that we use RAID10.
We'll keep you updated.

UPDATE 24/7/14 19:46hrs
The new HDD is still re-building: 46% complete now. Assumining this continues at a linear rate, estimated completion is around midnight.

UPDATE 24/7/14 21:27hrs
The new HDD is still re-building: 65% complete now.

UPDATE 25/7/14 00:04hrs
The new HDD is still re-building: 86% complete now.

UPDATE 25/7/14 03:00hrs
The new HDD has completed re-build.
The 2nd HDD failure is being replaced shortly and re-build will commence immediately upon installation; re-build is expected to take around 9-10hours.

UPDATE 25/7/14 08:38hrs
The 2nd HDD replacement has been done and rebuild is undergoing: 8% complete now.

UPDATE 25/7/14 11:17hrs
The 2nd HDD is still re-building: 49% complete now.

UPDATE 25/7/14 12:41hrs
The 2nd HDD is still re-building: 65% complete now.

UPDATE 25/7/14 15:04hrs
Rebuild complete.

Emergency Core Routing Platform Reboot (Resolved) Critical

Affecting System - Network - Core Routers

  • 17/07/2014 15:07 - 17/07/2014 15:22
  • Last Updated 17/07/2014 13:32

Hi,

We recently experienced a short amount of downtime; around 15mins. This should now be back online, the outage was caused by an emergency reboot of our core routing platform at our Coventry site as recommended by JTAC engineers due to an error we were seeing on these devices. If your service is not back online then please let us know.

As always, any further issues, please raise a support ticket and we'll get this looked into ASAP for you.

Kind Regards,

MTH Support Team

Emergency Works - RAID Card battery failure (Resolved) Medium

Affecting System - UK Node 1

  • 09/06/2014 22:18 - 20/06/2014 06:00
  • Last Updated 25/06/2014 12:47

Hi all,

Our system has notified us of a RAID card battery failure within UK Node 1; this isn't affecting anything directly operational, but is purely RAID write protection upon unplanned shutdown of the Node.

There's a new battery on order, however until this is delivered and installed, we're deeming this Node to be "At Risk" - however small the risk, we always take hardware issues seriously.

Those affected are anyone hosted on UK Node 1:

  • UK Shared Hosting
  • UK Email Hosting
  • Some VPS Clients - to check, please log into your VPS control panel and it will show you which Node your VPS is on.
As always, if you have any concerns, please raise a support ticket and we'll happily answer your questions.

This ticket will be updated below with all progress.

18 June '14 - UPDATE
Apologies for the delay in updating this; we've finally got the new battery delivered and this should be installed shortly.
Provisionally, 5am on Friday 20th June, however this is yet to be confirmed.

19th June '14 - UPDATE
This is a message to confirm that we're replacing the RAID card battery at 05:00hrs on Friday 20th June. (Tomorrow)
Expected downtime should only be around 10-15mins, if that. A message will be added below when this work starts and is completed.

20th June '14 - UPDATE
The RAID card battery was replaced this morning, and the VPS node is back up and running 100% now.

Issue resolved.

MTH: Planned Maintenance Notification (Resolved) Medium

Affecting System - UK Node 1

  • 01/05/2014 02:00 - 01/05/2014 03:00
  • Last Updated 01/05/2014 02:45

Services affected:

  • MTH Website
  • MTH Mail (Email Hosting)
  • MTH Client Area
  • Web Hosting
  • VPS - if on "Coventry Node 1"
Please note, your VPS will not be affected if you're on any other Coventry Node or any of our London Nodes.
Please note, no Dedicated Server clients will be affected either.


This is a planned maintenance notification for those clients who's service is hosted on our Coventry Node 1. We're currently in the process of upgrading our network infrastructure and this server is in the schedule for the beginning of May. We will be fully replacing the RAM in the server, doubling the resources available in the process and adding a new extensive monitoring card to the Server Node for increased monitoring systems.

As our own website and Client Area are also hosted on this Node, unfortunately the Network Status page and Support system will be down for the upgrade process - All support requests for clients on our other nodes should be sent to [email protected]

While we've set aside 1-hour for this maintenance, we're positive it'll actually span only 30mins or less; the hour has been accommodated for if anything else happens during the upgrade.

If you've got any concerns, worries, questions or comments with regards to this maintenance period, please do raise a support ticket or give us a call and we'll be more than happy to discuss this with you.

Kind Regards,

MTH Team

MTH: Planned Maintenance Notification (Resolved) High
  • 06/04/2014 03:00 - 08/04/2014 01:00
  • Last Updated 08/04/2014 01:21

Hi All,

This is a small maintenance notification that will only affect our cPanel hosting clients - we need to increase the virtual and swap resources on the server you're on so your accounts aren't strained to work with what they've got. Seeing as it's a VPS system, we've allocated the new resource settings already and this will be a 4min job.

Unfortunately, to do this required a reboot of the shared server, however due to the equipment and setup we use, this will only be a 3-4min job - I've only put 10mins on the schedule to allow for any over-run, however we don't plan on needing this at all.

99% of you won't even notice the reboot due to the speed of it, plus we'll be running it at 3am on Sunday 6th.

As always, if there's any issues, questions, comments or queries, please raise a support ticket and we're more than happy to discuss this with you.

Kind Regards,

MTH Team

UPDATE:
07/04/14 08:00am
Our apologies, this work was delayed. This will now be happening at midnight tonight.

UPDATE:
08/04/14 01:00am
Following a longer than planned reboot, all services are back up and running.

Shared Hosting IP Blocked on some black-lists (Resolved) Low
  • 27/03/2014 11:30 - 04/04/2014 09:00
  • Last Updated 05/04/2014 10:36

Hi All,

This is a quick message to inform you that we've discovered that our shared IP has been added to some blacklists due to spam emails that some clients have been sending. We're currently in the process of narrowing down the account(s) which are doing this and then this will enable us to delist from the blacklists.

UPDATE:
27/03/14 22:30
We've isolated the account that was doing this, suspended it, and are in the process of delisting from a number of blacklists.

UPDATE:
27/03/14 23:00
We've been able to remove ourselves from 4 of 7 blacklists instantly.

UPDATE:
27/03/14 23:05
Just received notification that 2 further blacklists have accepted our delist requests; this means we're awaiting for 1 more to give the all clear.


UPDATE:
28/03/14 10:00
The final blacklist is unable to be instantly delisted from and unfortunately takes 7-days of "no reported spam" before it automatically delists our IP. The information about this final blacklist is shown below:

IP Address: Listed
Reason for listing: abusive activity (spam) seen within the last 7-days
Number count: 1-hit (means 1 email)
Last Impact: 27.03.2014 22:23hrs
Expire time: 04.04.2014 01:00hrs (Thursday evening)

UPDATE:
04/04/14 09:00
Removed from UCEPROTECTL1 4/4/2014 8:54:19 AM

Issue resolved.

VPS Control Server (Resolved) High

Affecting Server - VPS Control Panel

  • 03/03/2014 06:00 - 05/03/2014 03:00
  • Last Updated 08/03/2014 15:01

Morning guys,

This morning we were informed that our VPS Control server had a HDD failure overnight. Unfortunatly this server needs to be shutdown for the new HDD to be installed and system restored to working condition.

Due to this, the VPS control panel is currently in maintenance mode and unaccessible - in addition, your VPS metric stats in the MTH Client Area will be incorrect.

Please note: Your VPS operation is not affected by this and should be working fine.

The maintenance is undergoing this morning and is expected to be back up and running by early afternoon.

As usual, if anyone has any queries, please do raise a Technical Support ticket from your client area and we'll address them swiftly.


Regards,

MTH Team

140303 17:00 UPDATE: The server hard-drive has been replaced this afternoon and the system is currently in the restoration phase.

140303 22:00 UPDATE: The server is up and running and VPS control panel is back online.

140305 06:00 UPDATE: The VPS backup images have been updated and the backup schedule has been restored. Everything should be back up and running as normal. If anyone has any concerns or notices any problems, please raise a support ticket and we'll get it looked into.

PSU Failure (Resolved) Medium
  • 10/02/2014 13:30 - 13/02/2014 12:00
  • Last Updated 13/02/2014 15:29

Afternoon,

We've just been informed by our Datacentre staff that we've had a PSU failure on our cPanel Shared Hosting Server. This means the server is running "at risk" as we always run servers with Dual-PSU incase of hardware failure like this happens.

A new unit has been ordered and will be installed in due course.

If you have any concerns, please don't hestiate to raise a support ticket and we'd be more than happy to discuss.

Regards,

MTH Team


140210 15:44 UPDATE: Our suppliers have shipped this out and the new PSU should be installed tomorrow morning.

140213 12:00 UPDATE: New PSU unit installed - server is now back up to redundancy level.

Office Phones down (Resolved) Low

Affecting Other - Office Phones

  • 14/01/2014 14:00 - 16/01/2014 19:47
  • Last Updated 16/01/2014 19:47

Hi guys,

Our office phones have gone offline at current - we've got an engineer working on it and should be back up and running this evening.

If anyone needs to get in touch with us, we're monitoring the support ticket system as always.

Regards,

MTH Team

14/01/2014 - 20:00

Update, these are now working now and we're fully contactable should you need us.

Regards
MTH Team

Very brief Network issue (Resolved) Medium

Affecting System - UKNODE1

  • 02/01/2014 10:44 - 02/01/2014 10:47
  • Last Updated 02/01/2014 11:15

Hi guys,

We've just been informed of a very brief network issue at our Midland Datacentre - this should now be resolved and we are looking into the cause. We will update this ticket with the cause information as soon as we receive our DC tech report.

If anyone has been affected by this, please let us know if we can help.

Kind Regards,

MTH Team

Email issue (Resolved) Low
  • 16/12/2013 18:00 - 18/12/2013 06:30
  • Last Updated 18/12/2013 06:39

Hi all,

Since moving our server to the UK, for some reason Microsoft have picked up our new IP and blocked it as it has no previous history. Due to this, any email sent through our server gets junked in hotmail, outlook or office365 at current.

We're currently working on resolving this issue, and as soon as Microsoft answer the phone, we'll have an update posted here.

Regards,

MTH Tech Team

**UPDATE** 20:03hrs
We have got through to Microsoft and submitted all the information they've requested. We're now awaiting on their response to whitelist us to get back to normal. Luckily, I believe it's all USA support, so they'll get the issue in their afternoon due to the time difference.

**UPDATE** 00:40hrs Tuesday 17th Dec '13
Just a quick update to this issue - We've been on the phone intermitently with Microsoft this evening for the last 7 hours. After many escalations, I believe we're getting somewhere.

**UPDATE** 13:42 Tuesday 17th Dec '13
Hi guys, so we've manually gone through all the accounts on the affected server IP and enabled DKIM and SPF Records - This activates email authentication and starts giving our IP a reputation. Following this, recipients should receive ALL emails, however since Microsoft are still filtering emails, they may still go to Junk folders until your domain develops it's reputation. To do this, recipients need to "Mark as safe sender" or select you email as "Not Junk"

We are monitoring the IP reputation and it's been unblocked by Microsoft, however their automated filter is still junking it. - As soon as we see this improve, we'll inform you further. The more emails sent and marked as safe, the quicker this process will repair itself.

Apologies to those affected by this, unfortunately it's out of our control as we've done as much as possible.

**UPDATE** 22:15hrs Tuesday 17th Dec '13

We've found a "work-around" for this issue at current until we find how Microsoft want to validate our IP.
 
Use your domain email, ie "[email protected]" from within a free Gmail account, the email goes through no problem straight to the inbox; I assume this would be the case for a Yahoo, Hotmail or Outlook.com account too as using any of their systems would bypass our server IP. I know this isn't an ideal solution or way to work in your environment, however it maintains your professional email image (using your domain) to the outside world (ie, clients)

**UPDATE** 04:29hrs Wednesday 18th Dec '13
So, finally Microsoft have reviewed our IP and removed it from their "blocklist 1". They've said to wait 1-2hours for this to take effect so we're going to run a few checks and tests around 6-7am and update the ticket following this!

**UPDATE** 06:37hrs Wednesday 18th Dec '13
I can confirm that blocklist 1 removal was successful and all emails successfully route into Microsofts network.

Server Migration (Resolved) Medium
  • 14/12/2013 22:30 - 15/12/2013 00:00
  • Last Updated 15/12/2013 02:17

As part of our ongoing development, we're migrating all our shared hosting packages to our newly installed UK Server - In terms of uptime, unfortunately this means a small amount of downtime following the duplication and DNS changes.

The the planned maintenance is to go ahead tonight (Saturday 14th December), and should take just over 60mins; data migration will start at 22:30hrs (GMT) and everything should be running from the new UK server by midnight.

If you wish to work on your site, or make any alterations, please do so before 22:30hrs tonight - We will be taking a full backup prior to migration, but we expect everything to go smoothly.

If anyone has any issues, questions or worries, please let us know and we'll be more than happy to assist with any personal backup's or answer any questions.

Kind Regards,

MTH Team

-- UPDATE-- 23:28hrs
Ok, so a successful migration has been done and our nameserver DNS changes have been made. We're now just waiting for the DNS to update itself for your sites to be fully live again. Unfortunately this isn't a job in our hands, but we estimate it to be no longer than 30mins as planned in the scheduled post this morning.

--UPDATE-- 00:30hrs
So, following the migration, the DNS has completed now too. All hosting is up and running on the new UK based server in a much faster environment than previously. We apologise for the downtime this evening, but hope you enjoy the increase in speed and response from the new server and location!

If anyone has any problems following the migration, please raise a Technical Support ticket and we'll get onto it ASAP for you.