host.mthservers.com: SQL Service (Resolved)
  • Priority - High
  • Affecting Server - MTH SSD Host
  • Early this morning we were alerted to the crash of the SQL Service on one of our cPanel hosting servers.

    This is currently being investigated and work is being carried out to bring the service back up.

    As soon as we have an update, we will update this notice.

    Kind Regards,

    MTH Support

    ===UPDATE: 09:20hrs===

    MariaDB published an update on November 5th, 2019 that can cause MariaDB to fail upon startup on cPanel & WHM servers. We've identified this as an upstream issue with MariaDB (MDEV-19073)

    cPanel themselved have also realised this overnight and a large number of servers have been impacted on this.

    https://forums.cpanel.net/threads/known-issues-status-page.644133/#post-2709865

    A rollback resolution has been implimented on our server and we believe this issue to be resolved; please do let us know if you have any further issues.

  • Date - 06/11/2019 06:00 - 06/11/2019 09:21
  • Last Updated - 06/11/2019 09:29
DataCentre Issue (Resolved)
  • Priority - Critical
  • Affecting System - Datacentre MTH-1
  • Good Morning,

    Our Datacentre: MTH-1 is currently experiencing  some routing issues. Most outbound connections are currently down.

    Unfortunately this will be affecting some of our VPS and Dedicated Server customers.

    Our Datacentre Technicians are on hand resolving the issue, with constant communication to ourselves.

    A positive update will be posted as soon as possible.

    Kind Regards,
    Your MTH Team

    -------
    UPDATE: 10:30

    This issue is still being investigated by the on-hand Team.

    A resolution will be reached shortly.

     

    UPDATE: 11:00

    This issue has been resolved by our Datacentre partners. If anyone is still experiencing outage, please raise a support ticket from your MTH Client Area.

  • Date - 30/10/2019 10:15 - 30/10/2019 11:00
  • Last Updated - 06/11/2019 08:20
Scheduled Shared Host downtime (Resolved)
  • Priority - High
  • Affecting Server - MTH SSD Host
  • To Our Valued Shared Hosting/Reseller customers

    To ensure we continue to offer our high level of service, we consistently maintain and upgrade our servers.

    Some maintenance does require more extensive work and downtime.

    Our next maintenance window is tomorrow evening (Friday 18th October), A reboot will take place at 23:00. We only envisage up to 15 minutes downtime, however the 'at risk' window will be up to 30 minutes.

    If you have any queries regarding this, please don’t hesitate to raise a support ticket.
    Support will also be available during this migration period.

    Please may we also point you to our Network Status page, where you can keep up to date with all current and upcoming updates across our Infrastructure. This can be reached by navigating to: https://secure.monkeytreehosting.com/serverstatus.php

    Kind Regards,

    The Monkey Tree Hosting Team

  • Date - 18/10/2019 23:00 - 18/10/2019 23:30
  • Last Updated - 30/10/2019 10:15
MTH Hosted Desktop Maintainance (Resolved)
  • Priority - High
  • Affecting System - Underlying Hypervisor Infrastructure
  • Good Evening,

    We need to perform some emergency maintenance this evening to complete some core infrastructure updates; this will require a number of reboots to some key infrastructure.

    During this work, the Hosted Desktop service may experience some short periods of interruption, however these will be limited to a couple of 5min windows; this will be mitigated as much as possible, however these works are necessary.

    If you experience a temporary profile, this is expected while the work is underway; please proceed to work as normal.

    Works Timeframe: 22:30hrs to 23:30hrs

    - We would usually give longer notice for works of this nature, however feel it is necessarily high importance to complete these updates.

    We will update below as and when works start and complete.

     

    ==Update 00:25hrs==Complete==

    This evening's works have now been complete; all works have been successful.

     

    ==Update 23:30hrs==

    Progress is coming along nicely with updates and system actions all completing successfully; unfortunately due to mitigating service impact, the actions are taking longer than expected.

    The works timeframe is being extended by 1hr to 00:30hrs

     

    ==Update 22:30hrs==

    Works starting...

     

    ==Update 18:51hrs==

    Works scheduled on-time; confirmed 

  • Date - 22/10/2019 22:30 - 22/10/2019 23:30
  • Last Updated - 23/10/2019 00:26
Scheduled Datacentre Emergency Network Maintenance (Resolved)
  • Priority - High
  • Affecting System - Datacentre MTH-2
  • Good Afternoon,

    Our Datacentre team are required to undertake some emergency maintenance this evening to resolve some minor routing issues.

    An at risk window has been posted from 23:00 this evening (Thursday 10th October) to 01:00 tomorrow morning (Friday 11th October)

    The expected impact is limited to a single 20 minute period of network instability during this time.

    Please do raise a ticket from within your Client Area, should this impact any service with ourselves.

    Thank you for your understanding.

    Kind Regards,
    Your MTH Team

  • Date - 10/10/2019 23:00 - 11/10/2019 01:00
  • Last Updated - 22/10/2019 19:04
Scheduled Shared Host downtime (Resolved)
  • Priority - High
  • Affecting Server - MTH SSD Host
  • To Our Valued Shared Hosting/Reseller customers

    To ensure we continue to offer our high level of service, we consistently maintain and upgrade our servers.

    Some maintenance does require more extensive work and downtime.

    Our next maintenance window is tomorrow evening (Friday 11th October), A reboot will take place at 23:00. We only envisage up to 15 minutes downtime, however the 'at risk' window will be up to 30 minutes.

    If you have any queries regarding this, please don’t hesitate to raise a support ticket.
    Support will also be available during this migration period.

    Please may we also point you to our Network Status page, where you can keep up to date with all current and upcoming updates across our Infrastructure. This can be reached by navigating to: https://secure.monkeytreehosting.com/serverstatus.php

    Kind Regards,

    The Monkey Tree Hosting Team

  • Date - 11/10/2019 23:00 - 11/10/2019 23:30
  • Last Updated - 18/10/2019 15:49
Hosted Desktop Maintenance (Resolved)
  • Priority - High
  • Affecting System - Hosted Desktop Services
  • To Our Valued Customers.

    To follow our efforts on providing an excellant, reliable service, we are scheduling some upcoming works to the Core of our networks.

    The maintenance is to be carried out on the core of our Hosted Desktop Infrastructure.

    Use of our Hosted Desktop services will be limited during the period of 8pm to 10pm this evening (Wednesday 18th September).

    Should you have any queries regarding the upcoming works, please raise a ticket through your client area, or send an email to support@monkeytreehosting.com

    Kind Regards,
    Your MTH Team

  • Date - 18/09/2019 20:00 - 18/09/2019 22:00
  • Last Updated - 10/10/2019 15:46
Scheduled Datacentre Maintenance (Resolved)
  • Priority - High
  • Affecting Other - Datacentre MTH-2 & MTH-3
  • To Our Valued Customers.

    To follow our efforts on providing an excellant, reliable service, our partners are scheduling some upcoming works to the Core of our networks.

    The maintenance is to be carried out on our core network, in an effort to complete migrations to our new Juniper platform.

    Following on from the work carried out on Monday, the Contingency Window has been invoked for further works.

    Just as with the original maintenance, this window will encompass all sites and involve our network team physically moving your services onto new Juniper hardware. This move will impact all transit and MPLS connectivity services, and customers should expect to see multiple brief outages throughout the window. Any resilient customers should not see an outage but should consider their services at risk during this window

    Should you have any queries regarding the upcoming works, please raise a ticket through your client area, or send an email to support@monkeytreehosting.com

    Kind Regards,
    Your MTH Team

  • Date - 07/08/2019 21:00 - 07/08/2019 03:00
  • Last Updated - 18/09/2019 16:11
Scheduled Datacentre Network Maintenance (Resolved)
  • Priority - High
  • Affecting Other - Datacentre MTH-2 & MTH-3
  • To Our Valued Customers.

    To follow our efforts on providing an excellant, reliable service, our partners are scheduling some upcoming works to the Core of our networks.

    The maintenance is to be carried out on our core network, in an effort to complete migrations to our new Juniper platform.

    The work scheduled for Monday 5th August 2019 at 21:00-03:00 includes actions to the following services:

    • Leased Line services - Customers who have a leased line will be affected by this work. The impact is expected to be up to 15 minutes per leased line at various intervals throughout the maintenance window. Where customers have resilient connections, only 1 connection will be migrated at a time.
    • Layer 3 services - Upstream connectivity to the Internet is being migrated to our new Juniper core. We do not expect any impact to customers during this migration as proactive steps have been taken to avoid downtime, however, services should be considered 'at-risk' for the duration of the window. 
    • Layer 2 services - Site-to-Site connectivity is not directly in scope for these works and we do not expect any impact to customers, however, services should be considered 'at-risk' for the duration of the window due to other work being carried out on the network.

    Should you have any queries regarding the upcoming works, please raise a ticket through your client area, or send an email to support@monkeytreehosting.com

    Kind Regards,
    Your MTH Team

  • Date - 05/08/2019 21:00 - 06/08/2019 03:00
  • Last Updated - 06/08/2019 16:41
Scheduled Datacentre Network Maintenance (Resolved)
  • Priority - High
  • Affecting System - Datacentre MTH-2 & MTH-3
  • Good Afternoon, 

     

    As part of our commitment to maintain a high standard of service, MTH and our datacentre partners are instigating a major hardware and architecture upgrade to our core network infrastructure.

    Over the past year the teams have been working tirelessly to build a brand-new core network based on SDN technology, separating L2 and L3 infrastructure and using new best-of-breed Juniper hardware. This represents a great investment by MTH and our datacentre partners into building a world-class network to support our customers' future growth, and is also in part a response to the issues with longer-than-anticipated re-convergence times following network disturbances and fibre breaks.
    Having been through extensive testing, the upgrade is now ready to be installed. The commissioning process involves moving physical fibre links, which means there will be an interruption to services during the following maintenance windows:

     

    We will be using the below window to complete the works on the network upgrade. Please be advised that updates of this window will be posted on our status page:

     

    https://secure.monkeytreehosting.com/serverstatus.php 

    16th May 2019, 22:00-02:00 (contingency)
    MTH-2 and MTH-3 Data Centres, all traffic
    This is a contingency-only window, scheduled to accommodate any complications during the previous maintenance window.

    This upgrade work is the culmination of extensive research, planning and financial investment, and I'm proud to announce that the new network is ready for installation and commissioning. If you have any questions about the nature of the work or would like more information, please get in touch with us.

     

    Kind Regards,

     

    Your MTH Team

  • Date - 16/05/2019 22:00 - 17/05/2019 02:00
  • Last Updated - 29/07/2019 10:54
Scheduled Datacentre Maintenance (Resolved)
  • Priority - Medium
  • Affecting Other - Datacentre MTH-2 & MTH-3
  • To our Valued Hosted Desktop customers,

    We are constantly upgrading our infrastucture to provide a better Hosted Desktop experience for our clients.

    Our next upgrade focuses on the Shared FileServer where all your files and folders are stored safely and securely.

    The window of these works are between 20:30 and 21:30 this evening.

    Any noticable impact will be between 21:00 and 21:15. During this time, access to any 'Shared Network Drives' may be disrupted.

    Please raise a ticket on your client area, or email support@monkeytreehosting.com should you run into any issues, or have any queries.

    Kind Regards,

    Your MTH Team

  • Date - 03/06/2019 20:30 - 03/06/2019 21:30
  • Last Updated - 29/07/2019 10:53
National Network Outage (Resolved)
  • Priority - Critical
  • Affecting Other - Internet Line Connections
  • Good Morning,

    Our Internet Partner are having Major Network Routing issues at the moment, affecting the below services

    • FTTC (Fibre to the Cabinet) Lines
    • Leased Line connectivity
    • DSL Lines

    This outage is spanning from Telehouse North, On of the UK's largest Network Infrastrcture's. It is affecting many Internet Providers across the UK and Europe.

    This is being worked on by a large team of experienced network engineers. We are constantly being updated on a resolution.

    Any further updates will also be posted below.

    Thank you for your patience with this matter and may you accept our apologies for any inconvenience caused during this time.

    Kind Regards,

    Your MTH Team

    --update: 10:00

    Our engineers are on site resolving the issue.

    Two avenues are being taken. 1, to mitigate the current issue in hand with the non-working services. 2, to build a alternative service and route network through there.

    --update: 11:00

    Our Engineers are still working in parallel to bring a new device online to help resolve the remaining issues.

    We apologise that we do not have anything more concrete at this time, but can assure you that our team are treating this as their number one priority. An update will follow within the hour.

    --update: 12:00

    Our Engineers have pushed a Network-Wide fix for all traffic. We are seeing services slowly come back online.

    --update: 12:20

    Services are fully online. We are closely monitoring all services to ensure stability.

    --update: 15:00

    Services are fully stable. Monitoring will continue as normal. This issue is considered resolved.

  • Date - 12/07/2019 09:16
  • Last Updated - 12/07/2019 14:54
Scheduled Datacentre Network Maintenance (Resolved)
  • Priority - High
  • Affecting System - Datacentre MTH-2 & MTH-3
  • Our DataCentre Partners are undertaking a rolling programme of network upgrades, with the following objectives:

    • Building a brand-new core network based on SDN technology

    • Seperating L2 and L3 infrastructure, using top-of-the-line Juniper Hardware

    •Preventing network disturbances and fibre breaks

    •Building a World-Class network to support customer's future growth

    •Upgrading all current physical fibre-links

    The first maintenance window will be this coming Saturday (Saturday 4th May 2019) from 22:00-06:00. During this time all traffic is considered to be at risk, but with minimal downtime.

    If you have any questions, please raise a support ticket from your client area.

    Kind regards,

    Your MTH Team

  • Date - 04/05/2019 22:00 - 05/05/2019 06:00
  • Last Updated - 08/05/2019 12:15
Scheduled Shared Host downtime (Resolved)
  • Priority - High
  • Affecting Server - MTH SSD Host
  • To Our Valued Shared Hosting/Reseller customers

    To ensure we continue to offer our high level of service, we consistently maintain and upgrade our servers.

    Some maintenance does require more extensive work and downtime.

    Our next maintenance window is next week, from midnight Monday night to the early hours of Tuesday morning. To be exact: (Monday 4th February at 00:00 to 03:00 on Tuesday 5th February). We estimate only up to an hour of downtime during this period.

    If you have any queries regarding this, please don’t hesitate to raise a support ticket.
    Support will also be available during this migration period.

    Please may we also point you to our Network Status page, where you can keep up to date with all current and upcoming updates across our Infrastructure. This can be reached by navigating to: https://secure.monkeytreehosting.com/serverstatus.php

    Kind regards,

    The Monkey Tree Hosting Team

  • Date - 04/02/2019 23:59 - 05/02/2019 03:00
  • Last Updated - 30/04/2019 17:40
Hosted Desktop - Desktop Issues (Resolved)
  • Priority - Critical
  • Affecting System - HostedDesktop
  • Hosted Desktop - Known Issues
    Last Updated: Thursday 14th @ 12:16PM

    Good Afternoon.

    At 11:23, our monitoring alerted us to an issue within the Hosted Desktop Infrastructure, which caused Microsoft Programs to crash, and subsequently crashing a users profile.
    We have narrowed this down to 2 Microsoft Updates for Outlook, which we believe to be corrupting the profile on use, and causing the issues.

    Please bear with us whilst we investigate this further. We will be calling round all effected customers once we have a further resolution and fix in place.

    UPDATE: 12:22
    We are seeing connections resolve, and users log back into the platform. A full log out and log back in is resolving the issue, we are still actively searching for the conflicting update within MS architecture, and we hope to have this removed by close of play today.

    Kind Regards

    MTH Team

  • Date - 14/02/2019 11:23 - 14/02/2019 12:22
  • Last Updated - 14/02/2019 12:33
Scheduled Datacentre Network Maintenance (Resolved)
  • Priority - Medium
  • Affecting System - Datacentre MTH-2 & MTH-3
  • Our DataCentre Partners are undertaking a rolling programme of network upgrades, with the following objectives:

    • Reducing latency

    • Increasing reliability

    • Enhancing QoS

    • Improving our Constant Quality Monitoring (CQM) Platform

    • Improving failover/failback response times

    • Better oversight of maintenance impact assessments

    • General housekeeping improvements

    The next maintenance window will be next Thursday (Thursday 24th January 2019) from 23:00-01:00. During this time there will be a small impact to Internet services, expected to be a few minutes of instability. This is estimated to be approximately 4x 10-minute outages for each maintenance window, however for contingency we have allowed the entirety of the two-hour period.

    If you have any questions, please raise a support ticket from your client area.

    Kind regards,

    Your MTH Team

  • Date - 24/01/2019 23:00 - 25/01/2019 03:00
  • Last Updated - 28/01/2019 16:28
Monkey Tree Hosting - VoIP Outage (Resolved)
  • Priority - Critical
  • Affecting System - VoIP Phone Network
  • Hi all,

    We are currently  (12:41) experiencing a system-wide VoIP outage,

    - This is affecting all VoIP services and VoIP customers

    We are actively looking into this and investigating the root cause as a P1 issue.

    Please keep an eye on this page for updates.

    Kind Regards
    MTH Team

    UPDATE - 13:10

    Our engineers are investigating the root cause of this issue.  ETA of resolution: 20 minutes.

    UPDATE - 13:30

    This issue has been resolved and we are seeing calls flowing through the VoIP network. 

    Please accept our apologies for any inconvenience caused.

    Kind Regards, MTH Team

  • Date - 18/01/2019 12:42 - 18/01/2019 13:40
  • Last Updated - 18/01/2019 13:45
Monkey Tree Hosting - VoIP Outage (Resolved)
  • Priority - Critical
  • Affecting System - VoIP Phone Network
  • Hi all,

    We are currently  (11.40) experiencing a system-wide VoIP outage,

    - This is affecting all VoIP services and VoIP customers

    We are actively looking into this and investigating the root cause as a P1 issue.

    Please keep an eye on this page for updates.

    Kind regards
    MTH Team

    UPDATE 11.55

    Service has been restored and we are seeing calls flowing through the network again.
    For all affected customers, please accept our apologies for this outage.
    Any further issues, please do not hesitate to contact us.

    Kind Regards,
    MTH Team

  • Date - 15/01/2019 12:01
  • Last Updated - 15/01/2019 12:28
Scheduled Datacentre Network Maintenance (Resolved)
  • Priority - Medium
  • Affecting Other - Datacentre MTH-2 & MTH-3
  • Our DataCentre Partners are undertaking a rolling programme of network upgrades, with the following objectives:

    • Reducing latency

    • Increasing reliability

    • Enhancing QoS

    • Improving our Constant Quality Monitoring (CQM) Platform

    • Improving failover/failback response times

    • Better oversight of maintenance impact assessments

    • General housekeeping improvements

    The next maintenance window will be This Evening (Thursday 10th January 2018) from 23:00-01:00. During this time there will be a small impact to Internet services, expected to be a few minutes of instability. This is estimated to be approximately 4x 10-minute outages for each maintenance window, however for contingency we have allowed the entirety of the two-hour period.

    If you have any questions, please raise a support ticket from your client area.

    Kind regards,

    Your MTH Team

  • Date - 10/01/2019 23:00 - 11/01/2019 01:00
  • Last Updated - 15/01/2019 09:18
Hosted Desktop Certificate Expiry (Resolved)
  • Priority - Critical
  • Affecting System - Hosted Desktop UK
  • We have been alerted by our systems that our gateway security certificate is expired and is thus blocking any Windows devices connecting to the hosted desktop; the certificate is incorrectly reporting it's expired status as this is not the case. We are working with the SSL provider to urgently resolve this.

    Affecting: New session login

    Not affecting: Existing user sessions

     

    UPDATE 08:01hrs

    We have issued a new certificate and implemented as this was the fastest route to resolution. Apologies for the inconvenience caused by this interruption to service.

  • Date - 14/11/2018 07:05 - 14/11/2018 08:01
  • Last Updated - 14/11/2018 08:10
Scheduled Datacentre Network Maintenance (Resolved)
  • Priority - Medium
  • Affecting Other - Datacentre MTH-2 & MTH-3
  • Our DataCentre Partners are undertaking a rolling programme of network upgrades, with the following objectives:

    • Reducing latency

    • Increasing reliability

    • Enhancing QoS

    • Improving our Constant Quality Monitoring (CQM) Platform

    • Improving failover/failback response times

    • Better oversight of maintenance impact assessments

    • General housekeeping improvements

    The next maintenance window will be This Evening (Thursday 1st September 2018) from 23:00-01:00. During this time there will be a small impact to Internet services, expected to be a few minutes of instability. This is estimated to be approximately 4x 10-minute outages for each maintenance window, however for contingency we have allowed the entirety of the two-hour period.

    If you have any questions, please raise a support ticket from your client area.

    Kind regards,

    Your MTH Team

     

    ---

    UPDATE: 02/10/2018 08:00hrs

    This work was completed successfully without issue; network disruption was limited to a few minutes.

  • Date - 01/11/2018 23:00 - 02/11/2018 01:00
  • Last Updated - 02/11/2018 10:04
Scheduled Datacentre Network Maintenance (Resolved)
  • Priority - Medium
  • Affecting Other - Datacentre MTH-2 & MTH-3
  • Our DataCentre Partners are undertaking a rolling programme of network upgrades, with the following objectives:

    • Reducing latency

    • Increasing reliability

    • Enhancing QoS

    • Improving our Constant Quality Monitoring (CQM) Platform

    • Improving failover/failback response times

    • Better oversight of maintenance impact assessments

    • General housekeeping improvements

    The next maintenance window will be Thursday 20th September 2018, from 23:00-01:00. During this time there will be a small impact to Internet services, expected to be a few minutes of instability. This is estimated to be approximately 4x 10-minute outages for each maintenance window, however for contingency we have allowed the entirety of the two-hour period.

    If you have any questions, please raise a support ticket from your client area.

    Kind regards,

    Your MTH Team

     

    ---

    UPDATE: 21/09/2018 08:00hrs

    This work was completed successfully without issue; network disruption was limited to a few minutes.

  • Date - 20/09/2018 23:00 - 21/09/2018 01:00
  • Last Updated - 21/09/2018 14:19
Scheduled Datacentre Network Maintenance (Resolved)
  • Priority - Medium
  • Affecting Other - Datacentre MTH-2 & MTH-3
  • Our DataCentre Partners are undertaking a rolling programme of network upgrades, with the following objectives:

    • Reducing latency

    • Increasing reliability

    • Enhancing QoS

    • Improving our Constant Quality Monitoring (CQM) Platform

    • Improving failover/failback response times

    • Better oversight of maintenance impact assessments

    • General housekeeping improvements

    The next maintenance window will be Thursday 4th October 2018, from 23:00-01:00. During this time there will be a small impact to Internet services, expected to be a few minutes of instability. This is estimated to be approximately 4x 10-minute outages for each maintenance window, however for contingency we have allowed the entirety of the two-hour period.

    If you have any questions, please raise a support ticket from your client area.

    Kind regards,

    Your MTH Team

  • Date - 04/10/2018 23:00 - 15/01/2019 12:27
  • Last Updated - 20/09/2018 11:13
Scheduled Datacentre Network Maintenance (Resolved)
  • Priority - Medium
  • Affecting Other - Datacentre MTH-2 & MTH-3
  • Our DataCentre Partners are undertaking a rolling programme of network upgrades, with the following objectives:

    • Reducing latency

    • Increasing reliability

    • Enhancing QoS

    • Improving our Constant Quality Monitoring (CQM) Platform

    • Improving failover/failback response times

    • Better oversight of maintenance impact assessments

    • General housekeeping improvements

    The next maintenance window will be Thursday 18th October 2018, from 23:00-01:00. During this time there will be a small impact to Internet services, expected to be a few minutes of instability. This is estimated to be approximately 4x 10-minute outages for each maintenance window, however for contingency we have allowed the entirety of the two-hour period.

    If you have any questions, please raise a support ticket from your client area.

    Kind regards,

    Your MTH Team

  • Date - 18/10/2018 23:00 - 15/01/2019 12:27
  • Last Updated - 20/09/2018 11:13
Scheduled Datacentre Network Maintenance (Resolved)
  • Priority - Medium
  • Affecting Other - Datacentre MTH-2 & MTH-3
  • Our DataCentre Partners are undertaking a rolling programme of network upgrades, with the following objectives:

    • Reducing latency

    • Increasing reliability

    • Enhancing QoS

    • Improving our Constant Quality Monitoring (CQM) Platform

    • Improving failover/failback response times

    • Better oversight of maintenance impact assessments

    • General housekeeping improvements

    The next maintenance window will be Thursday 1st November 2018, from 23:00-01:00. During this time there will be a small impact to Internet services, expected to be a few minutes of instability. This is estimated to be approximately 4x 10-minute outages for each maintenance window, however for contingency we have allowed the entirety of the two-hour period.

    If you have any questions, please raise a support ticket from your client area.

    Kind regards,

    Your MTH Team

  • Date - 01/11/2018 23:00 - 15/01/2019 12:27
  • Last Updated - 20/09/2018 11:13
Scheduled Datacentre Network Maintenance (Resolved)
  • Priority - Medium
  • Affecting Other - Datacentre MTH-2 & MTH-3
  • Our DataCentre Partners are undertaking a rolling programme of network upgrades, with the following objectives:

    • Reducing latency

    • Increasing reliability

    • Enhancing QoS

    • Improving our Constant Quality Monitoring (CQM) Platform

    • Improving failover/failback response times

    • Better oversight of maintenance impact assessments

    • General housekeeping improvements

    The next maintenance window will be Thursday 15th November 2018, from 23:00-01:00. During this time there will be a small impact to Internet services, expected to be a few minutes of instability. This is estimated to be approximately 4x 10-minute outages for each maintenance window, however for contingency we have allowed the entirety of the two-hour period.

    If you have any questions, please raise a support ticket from your client area.

    Kind regards,

    Your MTH Team

  • Date - 15/11/2018 23:00 - 15/01/2019 12:27
  • Last Updated - 20/09/2018 11:13
Scheduled Maintenance Notification (DataCentr (Resolved)
  • Priority - Medium
  • Affecting Other - Datacentres: MTH-2 & MTH-3
  • As part of our commitment to upholding the best standards of service, our DataCentre Partners have identified an upgrade opportunity for our network infrastructure. This will take the form of a rolling programme of works, starting in September, which will meet the following objectives:

    • Reducing latency
    • Increasing reliability
    • Enhancing QoS
    • Finishing our Constant Quality Monitoring (CQM) Platform
    • Improving failover/failback response times
    • Better oversight of maintenance impact assessments
    • General housekeeping improvements

    These will be achieved through various methods, including:

    • Further optimising the OSPF cost model
    • Conducting technical preparations for future international expansions
    • Optimising LSPs
    • Firmware updates to numerous
    • Building out a meshed monitoring platform to enable fine-grain loss monitoring

    During the maintenance windows there will be a small impact to Internet services, expected to be a few minutes of instability. This is estimated to be approximately 4x 10-minute outages for each maintenance window, however for contingency we have allowed the entirety of the two-hour period. Due to the nature and complexity of the work, we are unable to give customers granular details on how their services will be affected.

    Our DataCentre Partners have extensively reviewed our network use to determine the optimal times for this programme of works to take place. Given our diverse, global client base, this will not be a perfect fit for every customer, however it is the time when the network is least used (amongst other factors) and so will have the least impact.

    The works will take place on Thursday nights, on a fortnightly schedule. The schedule has been arranged to provide a balance between making progress with the work and providing service stability. The maintenance windows are:

    • Thursday 6th September 2018
    • Thursday 20th September 2018
    • Thursday 4th October 2018
    • Thursday 18th October 2018
    • Thursday 1st November 2018
    • Thursday 15th November 2018

    On each of the dates listed above, the at-risk window for the maintenance work is 23:00-01:00.

    We will send reminder notifications before each maintenance window.

    Kind Regards,

    Your MTH Team

  • Date - 06/09/2018 00:00 - 15/01/2019 12:27
  • Last Updated - 20/09/2018 11:02
Scheduled Datacentre Network Maintenance (Resolved)
  • Priority - Medium
  • Affecting Other - Datacentre MTH-2 & MTH-3
  • Our DataCentre Partners are undertaking a rolling programme of network upgrades, with the following objectives:

    • Reducing latency

    • Increasing reliability

    • Enhancing QoS

    • Improving our Constant Quality Monitoring (CQM) Platform

    • Improving failover/failback response times

    • Better oversight of maintenance impact assessments

    • General housekeeping improvements

    The next maintenance window will be Thursday 6th September 2018, from 23:00-01:00. During this time there will be a small impact to Internet services, expected to be a few minutes of instability. This is estimated to be approximately 4x 10-minute outages for each maintenance window, however for contingency we have allowed the entirety of the two-hour period.

    If you have any questions, please raise a support ticket from your client area.

    Kind regards,

    Your MTH Team

     

    ---

    UPDATE: 07/09/2018 08:00hrs

    This work was completed successfully without issue; network disruption was limited to a few minutes.

  • Date - 06/09/2018 23:00 - 07/09/2018 01:00
  • Last Updated - 20/09/2018 10:56
Core Network Maintenance Notice (Resolved)
  • Priority - High
  • Affecting System - Datacentres: MTH-2 & MTH-3
  • Dear Customer,

    Our datacentre provider will be performing scheduled maintenance on 19th-20th June 2018, from 22:30-02:30hrs. During this time they will be upgrading a node on their network to ensure they continue to offer the highest levels of resilience and up-time.

    Though we have posted an at-risk window of four hours, we expect the impact to be limited to two connectivity outages, each up to 20 minutes. If you have any questions, please contact your account manager.

    Kind Regards,

    MTH Team



    UPDATE
    This was completed successfully with minimal impact

  • Date - 19/06/2018 22:30 - 20/06/2018 02:30
  • Last Updated - 30/08/2018 18:04
Emergency Maintenance Notification (Resolved)
  • Priority - High
  • Affecting Other - Datacentres: MTH-2 & MTH-3
  • Dear Customer,

    Following some connectivity issues earlier today, our Datacentre provider will be conducting urgent maintenance on 6th June 2018 between 01:00 and 03:00 to replace a faulty line card on their core router.
     
    Though we have posted an at-risk window of 2 hours, we expect impact to be limited to two outages of up to five minutes whilst traffic is directed away from the device, then back to it following the replacement.
     
    If you have any questions, please contact support.
     
    Kind regards,

    MTH Team


    UPDATE 03:04

    We are pleased to confirm that the maintenance is complete successfully. Should you experience any issues after the advertised window, please raise a support ticket via your usual channels.

  • Date - 06/06/2018 01:00 - 06/06/2018 03:00
  • Last Updated - 06/06/2018 09:06
Datacentre Routing issue (Resolved)
  • Priority - Critical
  • Affecting Other - Datacentres: MTH-2 & MTH-3
  • Hi All,

    We're currently (16:24hrs) experiencing core routing issues across two of our Datacentre partners across MTH-2 and MTH-3 facilities.

    This is impacting service to the following:

    - Hosted Desktop
    - Hosted Exchange - NB. No email loss, however intermintent connection drops. Advised to use OWA (https://exchange.mthservers.com) until the issue is resolved.
    - Shared Hosting - some clients
    - Reseller Hosting - some clients
    - VPS - few clients

    Our datacentre team are actively looking into this and will resolve ASAP. Please keep posted on this page for updates.

    Kind Regards
    MTH Team


    UPDATE 

    Datacentre Team: We have identified a problem with connectivity between all of our sites and the internet at 15:45 this afternoon. We have taken some corrective action at 16:18 which temporarily restored connectivity; however at 16:25 our systems reported more problems.

    We are still investigating the problems here.



    UPDATE 17:18hrs

    Datacentre Team: As a further update, we are now isolating links sequentially in our network throughout Hertfordshire to help us identify the issue.

    We are still investigating the problems.



    UPDATE 17:45hrs

    Datacentre Team: As a further update, we are still working with affected hardware vendors to track down and resolve these ongoing connectivity problems.



    UPDATE 18:49hrs

    Datacentre Team: We believe we have now identified the cause of these issues and are still working with hardware vendors to resolve them. We have put temporary measures in place to improve the issues. It is likely that we will be arranging emergency maintenance for this evening to put the final fix in place.
    More details will follow shortly.


    UPDATE 18:53:hrs

    We have received confirmation of full resolution to the issues faced this evening. MTH are currently undertaking full systems checks to ensure everything is back to our normal level of service.
    Final update to follow, to mark issue as resolved.


    UPDATE 19:20hrs

    We can now confirm full resolution of the issue and have fully checked all MTH systems successfully; an RFO will be issued in the coming days to outline a summary of the issue and resolution.

  • Date - 05/06/2018 16:24 - 05/06/2018 19:20
  • Last Updated - 05/06/2018 19:19
Network Notification (Resolved)
  • Priority - High
  • Affecting Other - MTH Hosted and Exchange infrastructure
  • To our customers,

    This is a notification that there will be scheduled upgrade work to our systems this evening.

    Unfortunately this will cause a network outage across all our services for a short amount of time, as we re-route half our network across new, dedicated fibre links to increase capacity and redundancy.

    We forsee the outage to only be for 30 minutes at a maximum to allow for minimum disruption, and this can be between the window 23:00 - 04:00.

    We thank you in advance for your patience, should you have any problems or questions, please call our offices or email support@monkeytreehosting.com.

    Many thanks,
    The MTH team

  • Date - 09/12/2017 12:00 - 09/12/2017 14:00
  • Last Updated - 14/05/2018 17:06
Network Notification (Resolved)
  • Priority - High
  • Affecting Other - MTH Hosted and Exchange infrastructure
  • To our customers,

    This is a notification that there will be scheduled upgrade work to our systems this evening.

    Unfortunately this will cause a network outage across all our services for a short amount of time, as we re-route half our network across new, dedicated fibre links to increase capacity and redundancy.

    We forsee the outage to only be for 30 minutes at a maximum to allow for minimum disruption, and this can be between the window 23:00 - 04:00.

    We thank you in advance for your patience, should you have any problems or questions, please call our offices or email support@monkeytreehosting.com.

    Many thanks,
    The MTH team

    Update 01/17 The upgrade work has been completed succesfully, if you encounter any problems please us at support@monkeytreehosting.com.

  • Date - 30/11/2017 23:00 - 01/12/2017 04:00
  • Last Updated - 01/12/2017 08:11
MTH System: Desktop and Exchange Issues (Resolved)
  • Priority - Critical
  • Affecting System - MTH System
  • Hi All,

    We have noted an issue pertaining to Destop and Exchange sub-systems.
    We are actively working to resolve this ASAP and will update when we have a further update.

    MTH Team.

    UPDATE: 12:32 - SubSystems have been identified, we are working to resolve these now and currently have an ETA of 13:00 for a target resolution.
    UPDATE: 13:02 - This issue is now resolved. Please contact support@monkeytreehosting.com for an RFO.

  • Date - 25/10/2017 12:01 - 25/10/2017 21:26
  • Last Updated - 25/10/2017 21:26
Exchange Maintainance (Resolved)
  • Priority - High
  • On Tuesday 3rd of October at 9PM (BST) we will be performing maintenance on our Exchange system.

    Maintenance Start: 21:00 (BST) 3rd of October 2017

    Maintenance End: 21:10 (BST) 3rd of October 2017

    During the maintenance window users might experience some slowness of the system however the service should remain online; if you do experience any issues please raise a support case in the usual manner.

  • Date - 03/10/2017 21:00 - 04/10/2017 09:38
  • Last Updated - 03/10/2017 12:22
Exchange Maintanace (Resolved)
  • Priority - Critical
  •  

    On Thursday the 14th of September at 10PM (BST) we will be performing maintenance and upgrades on our Exchange system.

    Maintenance Start: 20:00 (BST) 14th September 2017

    Maintenance End: 00:00 (BST) 15th September 2017

    During the maintenance window users might experience some reduced performance, however the service should remain online; if you do experience any issues please raise a support case in the usual manner

  • Date - 14/09/2017 22:00 - 15/09/2017 00:00
  • Last Updated - 15/09/2017 08:18
Exchange Delivery Issue (Resolved)
  • Priority - Critical
  • Dear all,

    We are aware of an issue with incoming mail delivery on our Exchange platform and our senior systems engineers are investigating this right now. All emails will be cachced and delivered once full service has been restored.

    We apologise for the inconvenience that this has caused and hope to have this resoved as quickly as possible.

    Kind regards,

    MTH Support Team

  • Date - 04/08/2017 10:41 - 04/08/2017 12:32
  • Last Updated - 04/08/2017 10:46
OWA Patch (Resolved)
  • Priority - High
  • Hello,

    We have been made aware of a new issue related to OWA (Outlook Web App) and we will be applying a patch to fix this.

    There will be no downtime whilst this patch is being applied but you may notice some slowness when using the service.

    This will commence tonight at 23:00 - we envisage this to be resolved by 1am.

    If you have any queries related to this, please email support@monkeytreehosting.com.

    Kind regards,

    MTH Support


  • Date - 02/06/2017 23:00 - 28/07/2017 20:44
  • Last Updated - 02/06/2017 11:54
West Midlands Data Centre issue (Resolved)
  • Priority - Critical
  • Affecting Server - VPS Control Panel
  • All,

    We are currently aware of an issue affecting services at our West Mdland data centrre and we hope to have the issue resolved as soon as possible.
    Please note this will cause some services to be inaccessible whilst the data centre engineers work on resolving this issue.

    An update will follow in due course.

    Kind Regards
    MTH Team



  • Date - 30/11/2016 10:55 - 08/12/2016 17:08
  • Last Updated - 08/12/2016 17:08
MTH Desktop Gateway (Resolved)
  • Priority - Medium
  • Affecting System - MTH Desktop
  • Hi All,

    During the window above we will be performing maintainance to our Remote Desktop Gatways. This is intended to have no effect on the end user but as we will be working on one, and then the other system; it will be classed as "at risk".

    Kind Regards

    MTH Team

  • Date - 09/12/2016 20:00 - 25/04/2017 23:52
  • Last Updated - 07/12/2016 11:49
MTH Hosted Desktop Maintainance (Resolved)
  • Priority - Medium
  • Affecting System - File Systems
  • All,

    We are performing some maintainance on the file structure for our MTH Hosted Desktop product.
    Please note this will cause some tempory performance issues, and thus is being performed out of business hours.

    Kind Regards
    MTH Team

    ==
    Maintainance was conducted. Further restarts are required.
    ==

  • Date - 07/11/2016 20:00 - 25/04/2017 23:52
  • Last Updated - 08/11/2016 16:11
Microsoft Exchange (Resolved)
  • Priority - High
  • Affecting System - Microsoft Exchange
  • We are currently experiencing client connectivity issues to our Microsoft Exchange platform from one of our data centre locations; we've re-routed this traffic and our users should not be affected by this issue.

    We will update this notification once traffic route returns to normal

  • Date - 19/08/2016 13:00 - 08/11/2016 16:11
  • Last Updated - 08/11/2016 16:11
Network Maintainance (Resolved)
  • Priority - High
  • Affecting Other - MTH Network
  • Hi All,

    During the period mentioned, we will be taking the links between two of our DCs offline for some maintainance. As all our applications and systems are designed to match across multiple Data Centres this will not be a problem and should have no impacts on the end user.

    Kind Regards
    MTH Team

  • Date - 06/10/2016 21:00 - 07/11/2016 13:17
  • Last Updated - 06/10/2016 10:33
Skype for Business Maintainance (Resolved)
  • Priority - Critical
  • Affecting System - MS Skype For Business
  • Hi All,

    During the window above, Skype for Business will be unavailable.
    This is to proceed and deploy new 2015 versions of Skype for Business.

    If you have any problems, please email support@monkeytreehosting.com

    Kind Regards
    MTH Team

  • Date - 27/09/2016 18:30 - 06/10/2016 10:31
  • Last Updated - 26/09/2016 13:24
Power Works (Resolved)
  • Priority - Low
  • Affecting Other - MTH-2 DataCentre
  • We are informing you of on-site power works at our SC-2 Data Centre site, on the weekend of 13th-14th August 2016. Though the works will not affect power to the data centre itself, it will impact access to the wider SC-2 data centre site complex.


    Thank you for your understanding.

     
    Kind regards,

    Your MTH Team

  • Date - 13/08/2016 00:00 - 06/10/2016 10:31
  • Last Updated - 02/08/2016 10:56
BT Network Issue (Resolved)
  • Priority - Critical
  • Affecting Other - BT Network
  • All,

    We are aware of an on-going BT network issue that is preventing many customers from connecting to our hosted infrastructure.
    We will keep you up to date as this progresses. BT are aware of this issue and have publically acknowledged the issue.

    Kind Regards
    MTH Team

  • Date - 20/07/2016 07:23 - 19/08/2016 13:14
  • Last Updated - 20/07/2016 09:24
Exchange Reboots (Resolved)
  • Priority - Medium
  • Affecting System - MTH Hosted Exchange
  • All,

    During the window above the Exchange Servers will be rebooted one by one. This will cause no interuption, and as each server comes online we will establish connection before rebooting the other.
    There may be a slight delay in email propagation during this period.

    Kind Regards
    MTH Team

  • Date - 18/07/2016 21:00 - 06/10/2016 10:31
  • Last Updated - 18/07/2016 12:34
Exchange Maintenance (Resolved)
  • Priority - Critical
  • Affecting System - MS Hosted Exchange
  • Hi All,

    During the window above we will be performing some network and exchange routing maintainance.
    This will cause a slight delay in mail delivery, and a short outage with Outlook Web Access.

    Kind Regards
    MTH Team

  • Date - 13/07/2016 23:59 - 14/07/2016 01:00
  • Last Updated - 18/07/2016 12:31
Hypervisor Networking Issue (Resolved)
  • Priority - Critical
  • Affecting System - Site: WGC
  • Hi All,

    We have identified a network routing issue with the base Hypervisors in one of our Data Centres. This is not causing any system-problems at the moment, however - has shown on trial and test devices to be causing problems later down the line.

    Due to this we are taking emergency action tonight to investigate this.

    Kind Regards
    MTH Team

  • Date - 13/07/2016 23:59 - 19/08/2016 13:15
  • Last Updated - 13/07/2016 16:24
Network Maintainance (Resolved)
  • Priority - Medium
  • Affecting System - MTH Hosted Infrastucture
  • Hi All,

    We are scheduling some critical network maintainance in the time window shown above.
    During this time, we do not expect any network outages, however - the network will be running "at risk" due to the works being carried out.

    We will be replacing some of our site-to-site links with new hardware routes during this time, and also re-configuring out cross-site replication.
    Due to this, those on Hosted applications balanced accross multiple sites may experience slight slowness during this period.

    Should anyone have any concerns, please raise a ticket through your client area.

    Kind Regards
    MTH Team

    ==
    Update:

    This is complete.
    ==

  • Date - 08/07/2016 23:30 - 09/07/2016 06:00
  • Last Updated - 13/07/2016 16:22
Fibre infrastructure performance improvements (Resolved)
  • Priority - Medium
  • Affecting Other - FTTC
  • We will be conducting scheduled maintenance on Thursday 30th June 2016, from 22.00 till 00:00. This is in order to improve the performance of our FTTC network infrastructure.

    Your Business FTTC service will lose connectivity for a period of 1-2 minutes several times throughout this window.

    All other services including dedicated fibre (lease line) will not be affected.

    If you have any questions, please do raise a support ticket or give us a call in the office.

    Kind Regards,

    The MTH Team

    **UPDATE**
    This work was carried out with minimal interuption to client FTTC services. All actions complete

  • Date - 30/06/2016 22:00 - 01/07/2016 00:00
  • Last Updated - 24/06/2016 15:26
Exchange - Upgrade Maintenance (Resolved)
  • Priority - Low
  • Affecting System - Hosted Exchange
  • Dear all,

    On Friday 20th May between 20:00 - midnight, we will be performing upgrade maintenance on our Highly Available Exchange Cluster.

    During this period some Hosted Desktop users may see a certificate error message notifcation, it is ok to click continue if you receive this message.

    If you have any questions please feel free to contact us via the support portal. 

    Kind regards,

    MTH team

    **UPDATE 21/05/16 01:00am**
    Works have been complete with no problems

  • Date - 20/05/2016 20:00 - 21/05/2016 00:00
  • Last Updated - 20/06/2016 16:09
Exchange Configuration (Resolved)
  • Priority - Low
  • Affecting System - MS Hosted Exchange
  • Hi,

    During the hours of 20:00 > 23:59 on Saturday 7th May, we will be performing some Exchange configuration changes.
    This could result in temporary slowness in Outlook or Web Access, however - the service will not be at risk.

    Should you have any problems please open a support ticket from your client area or by emailing support@monkeytreehosting.com

    Kind Regards
    MTH Team

    ** Update **

    This work has now been completed sucessfully

  • Date - 07/05/2016 20:00 - 07/05/2016 23:59
  • Last Updated - 17/05/2016 14:05
MS Exchange Upgrade (Resolved)
  • Priority - Critical
  • Affecting System - MS Hosted Desktop
  • Hi,

    We are performing routine maintainance on the night of 06/05/2016, starting at 8pm.
    This maintainance includes the installation of a MS Update patch to our Exchange Infrastructure. 

    As we will have to restart one of the servers during this time, the service will be deemed 'at-risk'. 

    Users may experience slowness during the period of 8pm > 8am.

    Should you have any issues please raise a support ticket through the client area or by emailing support@monkeytreehosting.com

    Kind Regards
    MTH Team

    [update 2000]
    This is now in progress

    [update 2300]
    50% of the work is now complete without issue. Small slowness is noted but nothing effecting the majority.

    [update 0100]
    We are now 80% through, few more tasks and we will be complete.

    [update 0300]
    This work is now complete.

  • Date - 06/05/2016 20:00 - 07/05/2016 08:00
  • Last Updated - 07/05/2016 11:11
Mail Flow Delay (Resolved)
  • Priority - Critical
  • UPDATE 14:30 UTC 

    All infrastructure is now back in service and mail is being processed. Customers should see their mail queues dropping and new mail being delivered immediately. 

    UPDATE 14:00 UTC 


    Please be advised that Email is being processed but customers might still experience delays. 

    Our Engineering team believe that they have identified the networking issue and are currently preparing to attempt to bring the London DC back into service. 

    UPDATE 12:08 UTC 

    We have seen evidence that the mitigation we have put in place is taking effect and that customers' mail queues are dropping. 


    UPDATE 11:49 UTC 

    Please be advised that mitigation is being rolled out across the infrastructure to reduce the load and should take effect in 20-30 mins. 

    We will continue to update this notice as new information becomes available.

    UPDATE 10:37 UTC 

    Please be aware that customers using our infrastructure are still experiencing mail delays


    UPDATE 09:45 UTC 

    Email traffic is flowing without delay through the data centres whilst engineers continue to investigate the underlying issue. 

    ORIGINAL MESSAGE 

    Please be aware that customers are experiencing delays when sending and receiving email through infrastructure. 

    The effected data centre has been taken out of service and mail is flowing through the other two locations. Networking and infrastructure engineering team are investigating the issue. 

    We will continue to update this notice as new information becomes available. We sincerely apologize for any inconvenience caused while we work to resolve this issue. 

  • Date - 21/04/2016 09:25 - 21/04/2016 21:00
  • Last Updated - 06/05/2016 16:25
Exchange Modifications (Resolved)
  • Priority - Medium
  • Affecting System - MS Hosted Exchange
  • Hi,

    We are scheduling maintainance on our Exchange servers on the evening of 04/05/16. Starting at 8pm.
    Each server will be patched in turn and then restarted. This will push users around the hosts, and they will not experience any disruption.

    Users may experience slowness in Outlook and Outlook Web Access during this time.

    If you have any issues please raise a support ticket through your client area.

    Kind Regards
    MTH Team

    --

    [update]
    This work is now complete.

  • Date - 04/05/2016 20:00 - 06/05/2016 16:25
  • Last Updated - 06/05/2016 16:25
Exchange Connectivity (Resolved)
  • Priority - Critical
  • Affecting System - Exchange
  • Hi All,

    We're experiencing connectivity issues on the Exchange platform at current.

    As soon as we have an update for you, we will post it here.

    Apologies,

    MTH Team

    **UPDATE** 24/03/16 15:40hrs
    All connectivity is now restored; if there's any remaining issues, please do raise a support ticket. Slow searching in OWA or outlook is expected while the system re-indexes, however this will resolve within the hour.

  • Date - 24/03/2016 15:29
  • Last Updated - 24/03/2016 15:49
Hosted Desktop - Addition of new site (Resolved)
  • Priority - Critical
  • Affecting Other - MTH Hosted Desktop
  • All,

    This weekend will see the addition of a new DC site to our Hosted Desktop Platform.
    Due to the work of this nature, from Friday 10PM; until Monday 3AM - the system will be deemed "at risk".

    We do not envisiage any outages in access to the platform, however - we will be altering system and network configurations outside of working hours (0900 - 1700), which could result in brief disconnection.

    Should you have any issues or questions; please raise a support ticket by emailing support@monkeytreehosting.com

    Kind Regards
    MTH Team

    ==
    Update: Work on the w/e 22/01 went well. We have successfully complete the addition and testing.
    We are extending these working hours to this weekend too (29/01 - 31/01). Over this weekend we will be moving desktop access to originate from the secondary site as it has higher redundancy and converting the existing sites into fall-back configurations.
    ==

    ==
    Update: Work on the w/e 30/01 went well. Addition of the new gateway servers has now been completed and testing will be with the team this week.
    All going well; they will be implimented into action on 07/02/16.
    ==

    ==
    Update: 07/02/16: New Gateway servers are being introduced this evening.
    ==

  • Date - 29/01/2016 22:00 - 04/05/2016 13:59
  • Last Updated - 19/02/2016 12:53
Server Move (Resolved)
  • Priority - Critical
  • Affecting System - MTH Hosted Desktop
  • On 19/02/2016 we will be commencing work on one of our File Servers.
    This file server is one of the first from the system design, and thus still has a few dependancies upon the operation of this.

    During the window of time above, Hosted Desktop may be unavailable to some users with old dependancies on UK-MTH-FS01. This server is being migrated from onesite to another in order to combat latency issues that we have seen. Following this migration - we will be working to decommision the server and remove all dependancies from this server.

    Should you have any questions; please email support@monkeytreehosting.com to raise a support ticket.

    Kind Regards
    MTH Team

  • Date - 19/02/2016 21:00 - 04/05/2016 13:59
  • Last Updated - 19/02/2016 10:56
Exchange Server Network Expansion (Resolved)
  • Priority - Critical
  • Affecting System - MS Hosted Exchange
  • All,

    Futher to adding another DC location to our Hosted Desktop System, we are also taking this time to introduce an extra location for our MS Hosted Exchange system.
    Through this weekend we will be bringing online and configuring a few more Exchange Servers in the new location, and configuring the network and cross connects between the locations.

    As a result of this active network and system change and configuration - the service will remain "at risk" for the weekend.
    You may notice temporary unavailablity of Outlook, and Outlook Web Access as configurations are updates and systems rebooted. These outages should last no longer than 5-10mins, and will not interupt mail flow.

    Should you have any issues that you would like to raise please email support on support@monkeytreehosting.com

    Kind Regards
    MTH Team

    ==
    Update: Work on the w/e 22/01 went well. We have successfully complete the addition and testing.
    We are extending these working hours to this weekend too (29/01 - 31/01). Over this weekend we will be moving mail flow to originate from the secondary site as it has higher redundancy and converting the existing sites into fall-back configurations.
    ==

    ==
    Update: This is now complete.
    ==

  • Date - 29/01/2016 22:00 - 31/01/2016 03:00
  • Last Updated - 07/02/2016 16:00
MS Lync Server Upgrade (Resolved)
  • Priority - Medium
  • Affecting System - MS Lync
  • Hi,

    This evening from 1800 - 2359 Microsoft Lync (Skype for Business) will be unavailable.
    This is due to upgrades and patches being applied by our system engineers; which will fix issues with Lync for Mobile and Remote Access of the service.

    Should you have any issues, please raise a support ticket through your client account.

    Kind Regards
    MTH Team

    ==
    Update: This is complete.
    ==

  • Date - 28/01/2016 18:00 - 28/01/2016 23:59
  • Last Updated - 07/02/2016 16:00
MS Exchange & Mac Mail (Resolved)
  • Priority - High
  • Affecting System - MS Hosted Exchange
  • Hi,

    We are aware of a current issue effecting Exchange and Mac Mail. This seems to be resolving from the Exchange Web Services however, is only effecting a select number of clients.
    We will update this network notification as we have further updates for you.

    Kind Regards
    MTH Team

    ==
    Update: 15:10
    This is now resolved, and was caused by Exchange Web Services as expected.
    ==

  • Date - 27/01/2016 13:00 - 27/01/2016 15:36
  • Last Updated - 27/01/2016 15:36
External Interface Configuration (Resolved)
  • Priority - Low
  • Affecting System - MTH Hosted Exchange
  • Hi All,

    Sunday night, between the hours of 22:00 and Monday 03:00, we will be introducing a new external interface to the exchange platform.
    This is in order to facilitate a new feature - Encrypted Mail Delivery.

    We will notice a small delivery outage (30secs) when we introduce this to the platform - however; all mail will be stored at Symantec - and further processed once back online.

    If you have any concerns please raise a support ticket and we will happily assist or answer any queries.

    Kind Regards
    MTH Team

    ==
    Update: This work is now complete. There was no interuption to service.
    ==

  • Date - 10/01/2016 22:00 - 11/01/2016 03:00
  • Last Updated - 27/01/2016 14:43
MS Exchange Configuration Change (Resolved)
  • Priority - High
  • Affecting System - MS Hosted Exchange
  • Hi All,

    We are introducing a new aspect of our secure email flow this evening between 1900 // 2000. This may cause disruption to email flow in this period of time; however - everything will be queued and delivered once email flow is restored. We are not expecting this to last longer than 10mins.

    Kind Regards
    MTH Team

    ==
    Update: This is now complete
    ==

  • Date - 13/01/2016 19:00 - 13/01/2016 20:00
  • Last Updated - 21/01/2016 21:53
Connectivity issues affecting VoIP gateway (Resolved)
  • Priority - High
  • Affecting System - VoIP
  • We have been made aware of a problem affecting trunks on one of our gateways. This will be causing some trunks to deregister and loss of inbound and outbound service. Our systems engineers are currently looking into this issue as a priority and are working to resolve it as soon as possible.

    If you have any questions please do not hesitate to contact us via the support ticket portal.



    **UPDATE 16:51hrs**

    MTH engineers are still investigating connectivity issues affecting the affected gateway. This appears to be a widespread connectivity issue affecting multiple data centres across the UK. Regular updates will be posted here. Any customers with open tickets will be updated on resolution, however are advised to keep check on this status page.

    MTH apologise for the impact to our customers' service today.



    **UPDATE 18:10hrs**

    Engineers have confirmed that they have found a problematic route to one of our data centres. They are planning to implement a change of routing at 19:00hrs in an attempt to resolve ongoing connectivity issues.

    Another update will be sent after this routing change is complete. Engineers are continuing to monitor.



    **UPDATE 20:54hrs**

    Engineers have successfully re-routed traffic away from problematic routes. All alarms have now cleared, and call traffic is returning to normal. Engineers are continuing to monitor, but a recurrence is not expected.

    Once again, we apologise for the inconvenience caused today, and appreciate your patience.

    If you have any further questions, or if you see a recurrence of any issues, please do not hesitate to contact us.


    **UPDATE 22:50hrs**

    Further to this evening's earlier update of issue resolution, we have confirmd that the root cause of this issue was due to a UK wide BT Openreach issue - this also largely affected a number of mobile network providers as well as mobile broadband providers.
    A BBC article on EE and O2's issues can be seen at: http://www.bbc.co.uk/news/business-35286448

  • Date - 11/01/2016 15:56 - 11/01/2016 20:54
  • Last Updated - 11/01/2016 23:02
Exchange Server Introduction (Resolved)
  • Priority - Medium
  • Affecting System - MS Hosted Exchange
  • Hi All,

    We are introducing a new Exchange Server into the Cluster this evening (29/12/15).
    This will not cause any network or server issues, however - the mailbox's are replicating over to the Exchange Server, and this may have an effect on the speed of the system.

    Expected to be complete by: 29/12/15 @ 23:59:00

    Kind Regards
    MTH Team

    ==
    Update 18:00:00
    This has now started.
    ==

    ==
    Update 23:45:00
    This is now complete
    ==

  • Date - 29/12/2015 18:00 - 08/01/2016 23:33
  • Last Updated - 08/01/2016 23:33
Emergency Network Maintainance (Resolved)
  • Priority - Critical
  • Affecting System - MTH Core
  • Hi All,

    We will need to perform some emergency network maintainance to a set of our redundant routers in order to configure a redundany Layer 2 link to a new Data Center.
    This has the pottential to cause a 5s outage, however - could cause no impact at all.

    Target Date: Sunday 3rd January 2016 @ 02:00(am).

    Kind Regards
    MTH Team

  • Date - 03/01/2016 02:00 - 08/01/2016 23:32
  • Last Updated - 29/12/2015 18:44
Symantec.Cloud core issue (Resolved)
  • Priority - Medium
  • Affecting System - MS Hosted Exchange
  • Afternoon,

    Unfortunately we've been informed by Symantec (our spam and anti-virus filter providers for our exchange platform) that they're experiencing slow email throughput on their system. The cause is unknown at current and is only thought to be affecting a small percentage of domains routing through them. Emails are still flowing fine for most clients, however there is unfortunately a delay for a select few.

    We will update once we have some more information; unfortunately this issue is out of our hands.

    Kind Regards,

    MTH Team

    **UPDATE** 14:00
    We've had confirmation that Symantec have identified the issue.

    **UPDATE** 14:34
    Symantec are implimenting a temporary work around to improve mail flow.

    **UPDATE** 15:45
    Temporary work around is in place; some clients may still be experiencing issues, however this will improve over the next few hours.

    **UPDATE** 19:04
    Symantec have reported to us that the issue was due to a core switch stack failure in their core London datacentre causing slowness of outbound routing; a full RFI (Reason For Issue) will be issued to us by tomorrow morning and this will be updated on this network issue page.

    **UPDATE** 20:54
    We haven't had any further news back from Symantec yet, however testing carried out by MTH has shown large improvements with minimal if any delay. If you are still experiencing delays, please do let us know by raising a support ticket. We're not stating that this issue is resolved, however initial testing is showing promising results.

    **UPDATE** 00:25
    We have just finished in discussions with Symantec Engineers and the service is now fully restored. MTH testing shows this to be the case.
    Anyone with any further issues should contact support@monkeytreehosting.com

  • Date - 15/12/2015 13:00 - 16/12/2015 00:25
  • Last Updated - 16/12/2015 00:53
MTH UK Shared Server Issues (Resolved)
  • Priority - Critical
  • Affecting Server - UK Shared (N)
  • Hi All,

    We are seeing high-load and network issues on the MTH UK Shared cPanel server (uk.mthservers.com).
    We will update you as soon as we have a further update.

    Kind Regards
    MTH Team

    ==
    Update: 11:30
    This seems to be the result of several wordpress hacks; please take this time as a reminder to secure your wordpress installations.
    We have an ETA of 22:00 for resolution.
    ==

    ==
    Update: 16:24
    We are still working on restoring a few accounts and bringing the server back to a viable state. Please bear with us.
    ==

    ==
    Update 17:39
    We are starting to see the load on this server reduce and sites are beginning to load further.
    ==

    ==
    Update 23:00
    This issue is all now resolved, we are seeing sites return to normal speed.
    Clients experiencing any further issues should email "support@monkeytreehosting.com" in order to open a support ticket.
    ==

  • Date - 23/11/2015 06:00 - 24/11/2015 18:39
  • Last Updated - 24/11/2015 18:39
MS Hosted Exchange Issue (Resolved)
  • Priority - Critical
  • Affecting System - MS Hosted Exchange
  • Hi All,

    We are seeing large Exchange Delays with MS Hosted Exchange.
    We are investigating this at the moment, will update as we progress further.

    Kind Regards
    MTH Team

    ==
    Update: 10:34
    This is identified to be Disk I/O on HV3. We are restarting EX13-2 at the moment to kick exchange back to HV1
    ==

    ==
    Update: 11:04
    This is now resolved
    ==

  • Date - 20/11/2015 10:00 - 20/11/2015 11:51
  • Last Updated - 20/11/2015 11:50
MS Hosted Exchange - Emergency DR Testing (Resolved)
  • Priority - Critical
  • Affecting System - MS Hosted Exchange
  • Hi All,

    In light of recent issues that have occured with the MS Hosted Exchange Platform, we are bring forward some emergency disaster recovery testing in order to isolate and confirm no further problems exist.
    There is a posiblity for the following services to go offline during testing:

    - Outlook Web Access
    - Outlook Desktop
    - Mobile connectivity
    - Inbound / Outbound Mail

    All inbound mail will be buffered by out external delivery platform; and thus on the platform issues being resolved - any inbound email will be delivered.

    The plan will be as follows;

    1/ Suspend uk-mth-ex13-2 so that it appears it’s just fallen offline
    Ø  Test OWA for mailbox access
    Ø  Test Outlook for mailbox access
    Ø  Test Internal messaging
     
    Assuming all passes, bring back up uk-mth-ex13-2 and allow to re-sync (15 mins), before moving onto point 2; else resolve issues and re-test before moving forward.
     
    2/ Suspend uk-mth-ex13 so that it appears it’s just fallen offline
    Ø  Test OWA for mailbox access
    Ø  Test Outlook for mailbox access
    Ø  Test Internal messaging

    Should you have any issues please open a support ticket and raise one of the team,

    Kind Regards
    MTH Team

    ==
    Update: 20:49: This is now starting..
    ==

    ==
    Update: 21:09: UK-MTH-EX13 has been shutdown; there was a brief 2min of inactivity where the system was slow; but access was retained. Copy is just being re-sync'd across the two servers.
    ==

    ==
    Update: 21:24: UK-MTH-EX13-2 has been shutdown; there was a brief 2min of inactivity where the system was slow; but access was retained. Copy is just being re-sync'd across the two servers, further to this the system is back online completely now. DR Testing is complete and has been deemed as a success. A full report will be generated and reased before close of business on 16/11/2015.
    ==

  • Date - 13/11/2015 20:00 - 13/11/2015 23:59
  • Last Updated - 13/11/2015 21:30
MS Hosted Exchange - Outbound Route (Resolved)
  • Priority - High
  • Affecting System - MS Hosted Exchange
  • Hi all,

    Due to recent issues with outbound mail flow we have bought forward our upgrade to our outbound mail flow systems.
    This upgrade will allow us greater control and flexibility on the management and tracking of outbound mails on the MS Hosted Exchange infrastructure.

    This upgrade means that we will be moving over to pass mail through a new outbound cluster; that is security scanned by Symantec. Due to the change in outbound clusters; we will require a change in SPF records.
    During the period mentioned below; MTH will go through all clients with MS Hosted Exchange and change SPF records, however - any clients hosting external DNS will need to make this change themselves. MTH will endevour to find out these domains and contact you before Friday.

    The new SPF record is: v=spf1 +a +mx +a:spf.messagelabs.com ~all

    Please open a support ticket should you need any assistance with this,

    Kind Regards
    Steve

    ==
    Update: 21:02: New outbound route is now in place.
    SPF records are being updated by MTH; if you control DNS yourself please update your DNS to as above.
    ==

  • Date - 13/11/2015 21:00 - 13/11/2015 23:59
  • Last Updated - 13/11/2015 20:49
MS Hosted Exchange Route (Resolved)
  • Priority - Low
  • Affecting System - MS Hosted Exchange
  • Hi All,

    We are making an energency upgrade to our inbound spam filering platform with regards to recent issues that have occured.

    Part of this upgrade will involve the change of MX records used for the MS Hosted Exchange platform, as we route mail through our new Anti-Spam cluster.

    New MX records are:

    cluster8.eu.messagelabs.com - Priority: 10
    cluster8a.eu.messagelabs.com - Priority: 20

    MTH will be performing this update manually on all clients with DNS held locally at MTH; further to this - we will be contacting anyone with DNS held externally in order to make this change.
    This update will be performed on 12/11/2015 @ 19:00 and is expected to take 30mins to process all clients.

    We will be running both Anti-Spam systems in parralel for the following 2 weeks in order to allow everyone time to transisiton over, and to prevent the loss of any mail during change over.

    Please raise a support ticket if you have any queries,

    Kind Regards
    Steve

    ==
    Update: 12/11/2015 19:53
    This is now complete. All clients on MS Hosted Exchange have been migrated over to our new inbound cluster. Should we have any issues with your individual accounts we will be contacting you tomorrow AM.
    ==

  • Date - 12/11/2015 19:00 - 12/11/2015 19:30
  • Last Updated - 13/11/2015 00:38
Exchange Configuration Changes (Resolved)
  • Priority - Critical
  • Affecting System - MTH Hosted Exchange
  • All,

    Tonight we will be performing emergency maintainance on our Exchange Cluster.

    The below is the work that will be carried out:

    Separation of the MAPI Replication Network to an additional adapter
    Reschedule of the Shadow Reply
    Reschedule of the Exchange Index Route

    Kind Regards
    MTH Team

    ==
    Update; 11/11/2015 @ 01:35

    This work is all now compelete. No downtime was occured.
    ==

  • Date - 10/11/2015 23:59 - 13/11/2015 00:37
  • Last Updated - 13/11/2015 00:37
MS Hosted Exchange Outage (Resolved)
  • Priority - Critical
  • Affecting System - MS Hosted Exchange
  • All,

    During the period above - an outage was experienced on the MS Hosted Exchange Platform.
    A further RFO (Reason for Issue) will be issued once investigation work is complete.

    Kind Regards
    MTH Team

  • Date - 10/11/2015 13:05 - 10/11/2015 14:00
  • Last Updated - 10/11/2015 23:16
MTH Desktop - Improvements (Resolved)
  • Priority - High
  • Affecting System - MTH Desktop Infrastructure
  • All,

    On 16/10/2015 there will be a short outage of the internet from within the MTH Desktop System.
    Access to the service will remain available, but between the hours of 2100 - 2130; there will be no internet access to external applications.

    These improvements are in order to route the internet through a Threat Management Gateway; which will allow finer details and control of internet access through organisations.

    Kind Regards
    MTH Team

  • Date - 16/10/2015 21:00 - 10/11/2015 23:08
  • Last Updated - 16/10/2015 17:23
Issue on Hypervisor #3 (Resolved)
  • Priority - Critical
  • Affecting System - HV-WM-03
  • All,

    We have identified a stability issue with one of our hypervisors. Currently all nodes are stable and connection can be established, however - we have scheduled emergency maintainance for this evening in which will see the effected hypervisors restart with patch updates.

    Kind Regards
    MTH Team

    ==
    Update:
    This is now complete and the stability issues are resolved.
    ==

  • Date - 07/10/2015 12:22 - 16/10/2015 11:23
  • Last Updated - 16/10/2015 11:22
Hosted Desktop & Exchange Maintenance (Resolved)
  • Priority - Medium
  • Affecting System - Hosted Desktop & Exchange
  • Hi All,

    Following the recent issues we've had with the Hosted Desktop and Exchange platforms we've taken decision to implement some improvments to the system; this is also going to include 2 additional measures of failover to ensure future reliability is retained.

    The 1st maintenance period will be proceeding Saturday 29th August, 22:00hrs till Sunday 01:00hrs - through this, we expect to see around 20-30mins of intermittent downtime; due to our email buffering system, no incoming emails will be affected or bounced.

    The 2nd maintenance period will be proceeding Sunday 30th August 21:00hrs till Monday 03:00hrs - through this, we don't expect to see any downtime, however maintenance will be going on through this timeframe.

    Updates will be posted below following each maintenance completion period.

    Regards,

    MTH Team

    **Update 21:59hrs 29/08/15**
    Work has started.

    **Update 23:35hrs 29/08/15**
    Work has now all completed successfully with no further interruption to service.

    **Update 23:50 30/08.15**
    Work has been suspended for this evening. This is due to a routing issue identified by our pre-work tests. We have re-scheduled this work for next weekend.

    The 1st maintenance period will be proceeding Saturday 5th September, 22:00hrs till Sunday 01:00hrs - through this, we expect to see around 20-30mins of intermittent downtime; due to our email buffering system, no incoming emails will be affected or bounced.

    The 2nd maintenance period will be proceeding Sunday 6th September, 21:00hrs till Monday 03:00hrs - through this, we don't expect to see any downtime, however maintenance will be going on through this timeframe.

    **Update**
    The final phase of this maintenance upgrade work will be carried out on Tuesday 8th from 22:00hrs - Wednesday 9th 01:00hrs - during this time, we expect to see around 10-15mins downtime to the hosted desktop platform.

    **Update**
    This work is confirmed to be proceeding this evening - if anyone has any last minute questions, comments or requests, please do let us know via the support ticket system.

    **Update**
    This work has been returned to the testing phase due to unforseen circumstances. Issues that did not show up during phase 1 testing have become apparent and we are just returning to testing to check these senarios for any pottential complications.

  • Date - 08/09/2015 22:00 - 10/11/2015 23:08
  • Last Updated - 05/10/2015 11:15
West Midlands Datacentre Core upgrade (Resolved)
  • Priority - High
  • Affecting System - Core Networking Infrastructure
  • Dear Customers,

    For some time we have been experiencing a large number of network issues and outages at our West Midlands site, this issue has been with the redundant stack of gateway switches that form part of our core networking infrastructure at this site.


    We have been working tirelessly with Juniper Technical support to try and resolve the issues that we have been experiencing however we have reached the conclusion that the platform that we use at this site is no longer fit for purpose. 

    As a result within the last few weeks we have placed an order for a number of Juniper MX104 routers that will take over the main routing of our network at our West Midlands site. Once implemented all existing customers at the West Midlands site will be terminated on these MPLS enabled devices. We have been utilising the MX series routers at our core Docklands sites for a number of years where they have performed faultlessly and the MX104 is the latest generation in this range of devices. 

    We are going to be conducting a number of tests on these devices before migrating customers to the platform however given the recent issues with services at this site we are keen to bring about this migration swiftly. We anticipate a thirty second outage whilst migrating your services to this platform and this will be carried out between the hours of 02:00 and 05:00 GMT, an update will be posted below in due course with more information on when this upgrade will take place.

    If you have any questions with regards to the above, please raise a support ticket and we will happily speak with you.

    Kind Regards,

    MTH Team

  • Date - 21/08/2015 14:04
  • Last Updated - 05/10/2015 11:13
Network Maintenance (Resolved)
  • Priority - High
  • Affecting Other - MTH Core Infrastructure
  • We have had notification that during the following dates one of our fibre providers providing the fibre links between our West Midlands datacentre and London PoPs will be performing work on their network. This will result in one leg of our ring being unavailable for this time. 

    Due to the diverse nature of our network this should not be service impacting for any of our customers but you should be aware that our ring will be operating without it’s second redundant leg during this period and therefore during this time the network to our West Midlands datacentre will be deemed as “at risk”.

    Start Time 12/09/2013 – 11:00 GMT
    End Time 12/09/2013 – 13:00 GMT

    Start Time 20/09/2013 – 01:30 GMT
    End Time 20/09/2013 – 02:00 GMT

  • Date - 12/09/2015 11:00 - 12/09/2015 13:00
  • Last Updated - 05/10/2015 11:13
Hypervisor Updates (Resolved)
  • Priority - High
  • Affecting System - MTH HV1, 2 & 3
  • Hi,

    We have scheduled in an update patch for 3 of our Hypervisors on Friday 2nd October between the hours of 23:00 and 03:00 (Saturday 3rd October).

    During this period, each Hypervisor will be performing updates and patches, and then will restart 2 times.
    Each Hypervisor will be restarted in turn, so only one system will ever be offline at any time.

    Please let us know if yo have any queries,

    Kind Regards
    MTH Team.

    **Update**
    03/10/2015 - 1:05
    All updates and restarts have been completed and this Network Notification is now closed and all services are returning to normal.

    02/10/2015 - 23:06
    Work as described above is now starting - please call the office if you have any issues.

  • Date - 02/10/2015 23:00 - 03/10/2015 03:00
  • Last Updated - 05/10/2015 11:13
New / One off Direct Debit not completing (Resolved)
  • Priority - Medium
  • Affecting System - Direct Debit System
  • Afternoon,

    We've identified an issue with the GoCardless module we use for DirectDebit processing - this only affects those setting up new payments and trying to make one-off or upgrade payments.

    This issue is being looked into and a fix will be implimented soon, however payments affected will have to use another payment method such as Card or BACS.

    Apologies for this inconvenience; we will update this notification once we have any further information.

    Kind Regards,

    MTH Team

    ==
    Update: 24/08/2015 13:00

    This issue is now resolved, and all payments are sucesfully processing.
    ==

  • Date - 23/08/2015 13:33 - 24/08/2015 13:00
  • Last Updated - 27/08/2015 17:10
Virtualisation Issue (Resolved)
  • Priority - Critical
  • Affecting Other - HV 1
  • Morning All,

    Unfortunately, we've just experienced a major issue with one of our hypervisors which has taken some Virtual Machines offline at current.

    We apologies for this and we're working on this as we speak to get things back up and running.

    Regards,

    MTH Support Team

    **UPDATE 13:43hrs 19/08/15**

    This isssue is showing to be resolved now and was not related to the Hypervisor, however two specific Virtual Machines. This did take offline our Hosted Desktop service and Exchange, however service has been restored and all sessions have been retained.

    There will be some remedial works on the Hosted Desktop platform this evening out of hours, however everything is looking stable at current.

    This report will be updated once a full system check is complete this evening.

    **UPDATE 15:18hrs 19/08/15**

    The Hosted Desktop platform is still experiencing issues with user access. Mac users will be able to connect, however Windows users are unable to bypass a certificate warning. This is being worked on as we speak and will update the ticket when resolved.

    **UPDATE 15:14hrs 19/08/15**

    Hosted Desktop access is restored for Mac users, however Windows users are still experiencing certificate issues.

    **UPDATE 15:54hrs 19/08/15**

    The Hosted Desktop platform is back online and full service is restored. If anyone is experiencing any further connection issues, please report them ASAP to our support team and we will investigate, however our tests are showing full issue resolution; we can only apologise for the length of disruption to your business critical IT systems today.

  • Date - 19/08/2015 11:34 - 19/08/2015 15:54
  • Last Updated - 19/08/2015 19:47
UK Shared Crash (Resolved)
  • Priority - Critical
  • Affecting Server - UK Shared (N)
  • Hi,

    Unfortunately we've had the UK cPanel Shared server crash on us this afternoon.

    We're currently working to bring it back online - please standby for further updates.

    Our apologies for this,

    Regards,

    MTH Team

    **UPDATE**
    As of 15:00hrs this issue was resolved. A full RFO will be updated on this shortly.

  • Date - 05/08/2015 13:50
  • Last Updated - 07/08/2015 13:10
UK Shared Incoming Email issue (Resolved)
  • Priority - Critical
  • Affecting Server - UK Shared (N)
  • Hi,

    We have identified an incoming email issue with the UK Shared node with the hostname uk.mthservers.com
    This is currently being investigated and we will update this notice when we have a timeframe.

    Please open a suppport ticket if you have any queries,

    Kind Regards
    MTH Team

    ===
    **UPDATE** 14/7/15, 12:22pm

    We believe to have resolved this issue, however are running some tests to confirm resolution.
    ===
    **UPDATE** 14/7/15, 14:00hrs
    Issue confirmed resolved.

  • Date - 14/07/2015 12:00 - 14/07/2015 14:00
  • Last Updated - 21/07/2015 14:41
UK Shared hostname DNS (Resolved)
  • Priority - Medium
  • Affecting Server - UK Shared (N)
  • Hi all,

    Unfortunatley, we've encountered a DNS issue with the hostname uk.mthservers.com; this is currently resolving incorrectly.

    This shouldn't have a massive impact, however those of you who are accessing your reseller accounts, or using that URL for webmail/cpanel access will encounter blank screens.

    A temporary measure is to use your own domain in place ie domain.com/webmail or domain.com/cpanel and this will get yourself into where you require.

    As soon as we have an update, we will post this below.

    Regards,

    MTH Team

    **UPDATE**
    This issue has now been resolved and all should be back to normal. If you see any further issues, please open a support ticket.

  • Date - 21/07/2015 08:43 - 21/07/2015 12:40
  • Last Updated - 21/07/2015 14:40
UK Shared Server (Resolved)
  • Priority - Critical
  • Affecting Server - UK Shared (N)
  • Hi,

    We have identified a MySQL issue with the UK Shared node with the hostname uk.mthservers.com
    This is currently being investigated and we have a timeframe of roughly 10mins for resolution.

    Please open a suppport ticket if you have any queries,

    Kind Regards
    MTH Team

    ===

    **UPDATE** 14/7/15, 09:41hrs
    Our apologies, the resolution here is taking a little longer than expected. We're working as hard as we can to get all back up and running; thank you for your patience.

    ===

    ===

    **UPDATE** 14/7/15, 10:05hrs
    This is all resolved and all sites are back up. This was a major MySQL crash that we recovered, caused by a massive number of external connections. In order to combat this, we have limited external connections to known IP address' only.

    ===

  • Date - 14/07/2015 09:04 - 14/07/2015 10:05
  • Last Updated - 14/07/2015 10:22
MTH Core Network Issue (Resolved)
  • Priority - Critical
  • Affecting Other - MTH Core Network
  • We have a reported DC Network Outtage caused by an upstream router issue. We are investigating this issue at the moment and updates will follow shortly.

    MTH Team

    ==
    12:00 Update - Stability Increase
    We've seen increased stability for the last 10 mins; however - we are still waiting on final response from the DC on root cause and for them to give the all clear.
    ==
    12:15 Update - DC Cause
    We have identified this issue to seem to be a wide spread issue affecting much of the UK's connectivity at the London Internet Exchange (LINX).
    We have disabled our peering at LINX for now and all services should now be back online.
    ==
    12:34 Update - Complete 
    We have seen around 1 hour of uptime and stability now, and are confident that this issue has been resolves by the DC.
    ==

  • Date - 27/05/2015 11:05 - 27/05/2015 12:36
  • Last Updated - 27/05/2015 14:28
Hosted Desktop Maintenance Notification (Resolved)
  • Priority - Medium
  • Affecting Other - Hosted Desktop / Cloud IT
  • A small maintenance windows is required to impliment some additional performance and security measures into the file system of the Hosted Desktop Infrastructure.

    Normally this wouldn't incur any downtime, however we will need to reboot a cluster of servers for this enhancement to be picked up.

    Apologies for the short notification window, however if anyone has any issues with the scheduled time of these works, please do let us know ASAP by raising a ticket from your client area.

    Kind Regards,

    MTH Team

  • Date - 12/05/2015 00:00 - 12/05/2015 00:30
  • Last Updated - 26/05/2015 06:57
Planned Maintenance (Resolved)
  • Priority - Medium
  • Affecting System - Network
  • Dear Customer,

    Notification Type: Planned Maintenance - Coventry

    When:
    Maintenance Window Start Time: 09/04/2015 – 03:01 GMT +1
    Maintenance Window End Time: 09/04/2015 – 04:01 GMT +1

    Service Impacting?
    Due to the redundant nature of our network this maintenance should not be service affecting however due to the work involved the network is deemed at risk during this period.

    Detail of work being carried out:
    During this period we are going to be carrying out work on our switching platform at our Coventry site which involves breaking the virtual chassis stack of switches currently providing services to customers at this site and inserting a number of new switches into the stack.

    Service(s) Affected:
    All dedicated server, co-location and IP transit services at our Coventry site.

    Updates throughout this maintenance window will be posted on our status page.

    Please feel free to get in contact with our support team with any questions you may have.

  • Date - 09/04/2015 03:01 - 09/04/2015 04:01
  • Last Updated - 11/05/2015 11:50
Hosted Desktop Maintenance Notification (Resolved)
  • Priority - Medium
  • Affecting System - Hosted Desktop / Cloud IT
  • Hi All,

    We're performing some routine maintenance and upgrades to the Hosted Desktop platform Friday 27th evening.

    The works require us to unfortunately make the service unavailable for up to an hour (9pm to 10pm), however following this works there will also be noticeable performance improvements to the system.

    If anyone has any issues with this timing, please do let us know via a support ticket ASAP so we can accommodate.

    Kind Regards,

    MTH Team

    UPDATE 27/3/15, 19:14hrs
    Just a brief update to confirm this work is proceeding as planned. To confirm, the hosted desktop service will be unavailable this evening from 9pm for up to 1-hour max.

    UPDATE 27/3/15, 22:11hrs
    Completion update - all has been completed, tested and signed off as complete. Any issues, please open a support ticket.

  • Date - 27/03/2015 21:00 - 27/03/2015 22:00
  • Last Updated - 27/03/2015 22:14
Network Issues: Coventry (Resolved)
  • Priority - Critical
  • Affecting System - DC Core Network
  • All,

    We are experiencing network issues at our Coventry DC and the DC Technicials are rebooting a core piece of network equipment, we will provide updates soon.

    MTH Team

    ==
    14:02 UPDATE:

    This issue is now resolved. A full RFO will be available within the next 24 hours by request.
    ==

  • Date - 03/02/2015 13:23 - 03/02/2015 14:07
  • Last Updated - 03/02/2015 14:07
Backup Licensing issue (Resolved)
  • Priority - Medium
  • Affecting Other - Backup System
  • Hi All,

    For those of you who have upgraded to our R1Soft backup service, the R1Soft licensing server had some maintenance undertaken on it lastnight - this kicked out our license key and we've had to revalidate this today.

    Unfortunately this means backups from midnight until midday today have failed due to no valid licence key, however this is all back online and valid now - we've processed manual backup's to replace those which should have occurred lastnight.

    This issue is deemed to be fully resolved, but if anyone has any questions, comments or issues remaining, please do get in touch with out Support Team.

    Kind Regards,

    MTH Team

  • Date - 30/12/2014 00:02 - 30/12/2014 13:18
  • Last Updated - 30/12/2014 14:16
Internal Network Maintenance & Upgrade (Resolved)
  • Priority - Medium
  • Affecting System - All Coventry Services
  • Hi guys,

    We've got some internal network upgrades to impliment which will add a higher data throughput allowance to our network as well as increasing redundancy by adding an additional stacked switch configuration.

    Unfortunately, we do expect intermittent network downtime, however this will be kept to a minimum; expected is 5mins max network downtime.

    As shown in the Network Details, we've planned this work to be undertaken between midnight and 5am on Friday 12th December as to not interupt anyone in too much of a critical time. If anyone has any issues with this planned time, please do raise these with us and we'll ensure this is re-arranged around your requirements; whilest this cannot be guaranteed, we will do our best to do so. The sooner a request is received the more likely we'll be able to re-schedule.

    Kind Regards,

    MTH Team

    UPDATE: 02:38am, 12/12/14
    Switch and network maintenance has started. We don't expect any major outages, however there could be a few 10-15min blips in network connectivity.

    UPDATE: 05:00am, 12/12/14
    All switch and network maintenance has been completed - network connectivity was limited to less than 2min total.

    UPDATE: 03:18pm, 27/12/14
    Following the maintenance no issues or problems have been reported - network maintenance closed off as completed successfully.

  • Date - 12/12/2014 00:00 - 12/12/2014 05:00
  • Last Updated - 27/12/2014 15:20
MTH Mail: Upgrade & Maintenance (Resolved)
  • Priority - Low
  • Affecting System - MTH Mail System - mthmail.com
  • Hi All,

    We've taken our mthmail.com app offline to do some maintenance on it's server and also perform a few program updates.

    This should be back online Tuesday night for you, however email is still accessible via https://ukshared.monkeytreehosting.com/webmail

    Kind Regards,

    MTH Team

    ==
    Update: 03/12/14 - 00:10
    MTH Mail System is now back online. We've added features such as Dropbox Connections, and a place to upload files you send to people.
    For more information, please open a support ticket and one of our technical team will be more than happy to publish the changelog.
    ==

  • Date - 01/12/2014 08:45 - 02/12/2014 23:59
  • Last Updated - 03/12/2014 00:27
VPS Node 1 Hardware Issue (Resolved)
  • Priority - High
  • Affecting Other - VPS Node 1
  • Hi All,

    Unfortunately we've been notified by our monitoring systems of a hardware issue with VPS node 1. The implications aren't critical, however we'd prefer that client's servers on this node were not sitting "at risk", so have made the decision to migrate them across to another stable node this evening.

    The implications of this will be a minor unavailability of access as they will need to be shutdown, migrated, and re-started; depending on the size of HDD you have on your VPS, this could be between 30mins - 60mins.

    We apologise for the inconvenience, however we feel this is in the best interest of your continued service. Those of you that we've been able to contact to discuss this, we've initiated the migration already - those who we've been unable to contact will be done later on this evening.

    If anyone has any questions, concerns or general comments, please do open a support ticket and we'll respond to you shortly.

    Kind Regards,

    MTH Team

    ========

    19:11hrs UPDATE:
    5 of 13 VPSes have completed migration

    00:10hrs UPDATE:
    100% migration complete - all VPSes have been tested and showing live. If anyone has any issues, please raise a support ticket.

  • Date - 30/11/2014 17:00 - 01/12/2014 00:10
  • Last Updated - 01/12/2014 00:26
Google Service Disruption (Resolved)
  • Priority - Medium
  • Affecting Other - Google Apps for Work
  • Evening All,

    We've been made aware that Google are reporting some service disruption currently across a number of their services.

    Of course we'll keep this page updated with anything we have, but the best port of call is to keep up with the Google Status page:

    http://www.google.co.uk/appsstatus#hl=en&v=status

    Regards,

    MTH Team

    ========
    UPDATE
    Apologies for the delay in updating this network issue; Google have confirmed to have resolved this issue last week and this is also reflected on their status page linked above.

  • Date - 22/10/2014 07:01 - 27/10/2014 00:00
  • Last Updated - 06/11/2014 22:33
Emergency Switch Reboot (Resolved)
  • Priority - Critical
  • Affecting System - Core Network Stack
  • Notification Type: Un-Planned Maintenance

    Maintenance Window Start Time: 21/10/2014 14:10:00 GMT +1
    Maintenance Window End Time: 15/10/2014 14:15:00 GMT +1
    Service: Shared, VPS, Dedicated Servers, Co-Location, Rackspace, IP Transit

    Location: Coventry

    During the period above we had to place an emergency reboot on the Core Network Stack in our Coventry DC due to an update package as a result of attacks on the Network Infrastructure and advise taken from CISCO Engineers. This has patched all issues relating to the network, and we are now in the process of implimenting a dual-stack network Infrastructure, which will allow us to perform such actions to the network core without interuption to services.

    We apologise for the minor outage, although this was limited to 5 minutes network downtime.

    We have checked all services, and they are up and running - however; should you have any problems - please contact us by opening a support ticket.

    Kind Regards
    MTH Team

  • Date - 21/10/2014 14:10 - 23/10/2014 16:13
  • Last Updated - 21/10/2014 15:02
Planned Maintenance - Coventry - 15/10/2014 (Resolved)
  • Priority - High
  • Affecting System - Core Network Stack
  • Notification Type: Planned Maintenance

    Maintenance Window Start Time: 15/10/2014 03:00 GMT +1
    Maintenance Window End Time: 15/10/2014 07:00 GMT +1
    Service: Shared, VPS, Dedicated Servers, Co-Location, Rackspace, IP Transit

    Location: Coventry

    During this maintenance window we are going to be upgrading the JUNOS software on the redundant network stack of devices that form part of customers gateways at our Coventry site. For shared, VPS, dedicated server and co-location customers this should not be service affecting but due to the nature of the works all clients at this site should consider the network at risk during this period. Customers with rackspace/transit services at this site connected with a single uplink may notice downtime of up to ten minutes whilst the particular switch they are connected to reboots once upgraded.


    Full service updates will be posted at https://secure.monkeytreehosting.com/serverstatus.php throughout this maintenance window. This post will also be updated below as always, with time and date stamps too.

    If you have any questions regarding this maintenance please do not hesitate to submit to submit a support ticket via your client portal.

    Regards
    MTH Team

    UPDATE 15/10/14 04:20 GMT +1
    We are currently seeing lost connectivity to a number of racks in relation to the planned maintenance this morning. We hope to have service fully restored shortly.

    UPDATE 15/10/14 05:55 GMT +1
    Maintenance is now complete at our Coventry site. Full service has been restored.

    UPDATE 15/10/14 08:31 GMT +1
    All maintenance has been complete and all clients should be back online. Apologies to those affected by the network downtime. As always, if anyone has any issues, please raise a support ticket and we'll gladly assist you to resolve these.

    Please note, all servers remained online and live, only network connectivity was lost and subsequently restored.

    UPDARE 15/10/14 10:45 GMT + 1
    We have experienced a temporary network outage as a result of a fault of an upstream providor. This outage was experienced between 09:42 and 09:59. We are now seeing 100% connectivity through the network and all servers are online.
    If you have any questions regarding this maintenance please do not hesitate to submit to submit a support ticket via your client portal.


  • Date - 15/10/2014 03:00 - 15/10/2014 10:00
  • Last Updated - 15/10/2014 10:52
Planned Maintenance (Resolved)
  • Priority - Low
  • Affecting System - Planned Maintenance
  • We have had notification that during the following dates one of our fibre providers providing the fibre links between our Coventry datacentre and London PoPs will be performing work on their network. This will result in one leg of our ring being unavailable for this time. 

    Due to the diverse nature of our network this should not be service impacting for any of our customers but you should be aware that our ring will be operating without it’s second redundant leg during this period and therefore during this time the network to our Coventry datacentre will be deemed as “at risk”.

    Start Time 12/09/2013 – 11:00 GMT
    End Time 12/09/2013 – 13:00 GMT

    Start Time 20/09/2013 – 01:30 GMT
    End Time 20/09/2013 – 02:00 GMT

  • Date - 12/09/2014 11:00 - 20/09/2014 02:00
  • Last Updated - 08/10/2014 18:20
UK Node 2 Maintenance (Resolved)
  • Priority - Medium
  • Affecting System - UK VPS Node 2
  • Hey guys,

    We've got some BIOS and Firmware updates to do to the hardware node your Virtual Servers are situated on. We've already complete this work on UK Node 1 without any issues.

    This work requires 2x reboots of the server which we'll plan at 01:10hrs and 01:20hrs accordingly.

    Through the process we'll be updating this ticket, however don't expect more than 2x 10mins downtime. If anyone has any concerns, questions, or comments, please let us know by raising a support ticket.

    Regards,

    MTH Team

    13/09/14 UPDATE:
    Apologies, we've delayed this until Friday 19th, 1am.

    19/09/14 UPDATE:
    The BIOS update is currently running. First reboot expected around 12:30am

    19/09/14 UPDATE:
    BIOS update is complete and successful @ 00:36am
    Reboot complete and everything has been checked to be back online @ 00:45am

    19/09/14 UPDATE:
    RAID Card update complete and successful @ 01:30am
    Server is rebooted and back online.

    As always, if anyone has any issues, please raise a support ticket and we'll look into it immediately for you.

    MTH Team

  • Date - 19/09/2014 01:00 - 19/09/2014 01:30
  • Last Updated - 19/09/2014 01:32
UK VPS Node 1 - VPS Migration (Resolved)
  • Priority - Medium
  • Affecting Other - UK VPS Node 1
  • Following the issues we've had with UK VPS Node 1 recently, we've decided to give it a full re-build to be sure we can completely eliminate any risk factors associated with it.

    To enable us to do this, those clients who's VPSes are on this Node will be contacted shortly to arrange their migration across to one of our other VPS Nodes.

    If anyone has any questions with regards to this, or is experiencing any issues, please raise a support ticket.

    Kind Regards,

    MTH Team

    UPDATE 30/7/14 07:39hrs
    This is confirmed as still planned for this evening. As mentioned, we're expecting around 1-2hours of downtime from midnight till 2am. This ticket will be updated once complete.

  • Date - 26/07/2014 12:00 - 01/08/2014 00:00
  • Last Updated - 19/08/2014 10:22
UK Node 1 - SQL Issue (Resolved)
  • Priority - Critical
  • We currently have an issue with UK Node 1 and the SQL service. We are looking into it at the moment and will keep this System updated.

    MTH Server Team

    UPDATE 27/7/14 11:00hrs
    We've isolated this issue and believe to have resolved this. If anyone experiences any further issues, please raise a support ticket.

    Regards,

    MTH Server Team

  • Date - 25/07/2014 21:03 - 27/07/2014 11:00
  • Last Updated - 27/07/2014 11:20
UK VPS Node 1 - Downtime (Resolved)
  • Priority - Critical
  • Affecting Other - UK Node 1
  • Hi guys,

    We just experience about 1.5hrs downtime on our UK Node 1 server - this will have affected any VPS clients on this server as well as all shared hosting clients and email hosting clients.

    The reason for this was initially a drive failure in the machine which was in RAID-10 so we replaced it and it was re-building; mid-rebuild a 2nd drive failed which almost corrupted our array.

    We managed to quickly resolve this very rare situation and luckily retain everyone's data through a lot of hard work from the team.

    If anyone's VPS or website isn't loading, please raise a support ticket and we'll look into it for you.

    Cheers,

    MTH Team

  • Date - 25/07/2014 19:45 - 25/07/2014 19:45
  • Last Updated - 25/07/2014 19:52
UK VPS Node 1 - HDD Failure (Resolved)
  • Priority - High
  • Affecting Other - UK Node 1
  • Hi all,

    We've been alerted by our monitoring systems that we've had a Hard Drive failure in the RAID10 array of UK VPS Node 1.

    New hardware is on it's way to the DC and should be installed shortly.

    This server is currently running on "at risk" status until the new HDD's have been installed, however all your data and running should be unaffected.

    As always, if anyone has any issues they notice, please raise a support ticket and we'll respond for you shortly.

    Kind Regards,

    MTH Team

    UPDATE 24/7/14 16:03hrs
    The new HDD has been put in and is currently rebuilding. VPSes on this node may notice a slow HDD I/O speed while this is occuring, however will resume to normal once complete.
    This is currently at 6% complete.

    UPDATE 24/7/14 16:03hrs
    New HDD is still rebuilding nicely and is now at 11%, however a 2nd HDD has failed during this re-build, so we'll have to await the first HDD to finalise re-build and then replace the 2nd failed HDD and await this to rebuild. This is a very rare occurance, however one of the main reasons appart from performance that we use RAID10.
    We'll keep you updated.

    UPDATE 24/7/14 19:46hrs
    The new HDD is still re-building: 46% complete now. Assumining this continues at a linear rate, estimated completion is around midnight.

    UPDATE 24/7/14 21:27hrs
    The new HDD is still re-building: 65% complete now.

    UPDATE 25/7/14 00:04hrs
    The new HDD is still re-building: 86% complete now.

    UPDATE 25/7/14 03:00hrs
    The new HDD has completed re-build.
    The 2nd HDD failure is being replaced shortly and re-build will commence immediately upon installation; re-build is expected to take around 9-10hours.

    UPDATE 25/7/14 08:38hrs
    The 2nd HDD replacement has been done and rebuild is undergoing: 8% complete now.

    UPDATE 25/7/14 11:17hrs
    The 2nd HDD is still re-building: 49% complete now.

    UPDATE 25/7/14 12:41hrs
    The 2nd HDD is still re-building: 65% complete now.

    UPDATE 25/7/14 15:04hrs
    Rebuild complete.

  • Date - 23/07/2014 12:22 - 25/07/2014 15:04
  • Last Updated - 25/07/2014 19:50
Emergency Core Routing Platform Reboot (Resolved)
  • Priority - Critical
  • Affecting System - Network - Core Routers
  • Hi,

    We recently experienced a short amount of downtime; around 15mins. This should now be back online, the outage was caused by an emergency reboot of our core routing platform at our Coventry site as recommended by JTAC engineers due to an error we were seeing on these devices. If your service is not back online then please let us know.

    As always, any further issues, please raise a support ticket and we'll get this looked into ASAP for you.

    Kind Regards,

    MTH Support Team

  • Date - 17/07/2014 15:07 - 17/07/2014 15:22
  • Last Updated - 17/07/2014 13:32
Emergency Works - RAID Card battery failure (Resolved)
  • Priority - Medium
  • Affecting System - UK Node 1
  • Hi all,

    Our system has notified us of a RAID card battery failure within UK Node 1; this isn't affecting anything directly operational, but is purely RAID write protection upon unplanned shutdown of the Node.

    There's a new battery on order, however until this is delivered and installed, we're deeming this Node to be "At Risk" - however small the risk, we always take hardware issues seriously.

    Those affected are anyone hosted on UK Node 1:

    • UK Shared Hosting
    • UK Email Hosting
    • Some VPS Clients - to check, please log into your VPS control panel and it will show you which Node your VPS is on.
    As always, if you have any concerns, please raise a support ticket and we'll happily answer your questions.

    This ticket will be updated below with all progress.

    18 June '14 - UPDATE
    Apologies for the delay in updating this; we've finally got the new battery delivered and this should be installed shortly.
    Provisionally, 5am on Friday 20th June, however this is yet to be confirmed.

    19th June '14 - UPDATE
    This is a message to confirm that we're replacing the RAID card battery at 05:00hrs on Friday 20th June. (Tomorrow)
    Expected downtime should only be around 10-15mins, if that. A message will be added below when this work starts and is completed.

    20th June '14 - UPDATE
    The RAID card battery was replaced this morning, and the VPS node is back up and running 100% now.

    Issue resolved.

  • Date - 09/06/2014 22:18 - 20/06/2014 06:00
  • Last Updated - 25/06/2014 12:47
MTH: Planned Maintenance Notification (Resolved)
  • Priority - Medium
  • Affecting System - UK Node 1
  • Services affected:

    • MTH Website
    • MTH Mail (Email Hosting)
    • MTH Client Area
    • Web Hosting
    • VPS - if on "Coventry Node 1"
    Please note, your VPS will not be affected if you're on any other Coventry Node or any of our London Nodes.
    Please note, no Dedicated Server clients will be affected either.


    This is a planned maintenance notification for those clients who's service is hosted on our Coventry Node 1. We're currently in the process of upgrading our network infrastructure and this server is in the schedule for the beginning of May. We will be fully replacing the RAM in the server, doubling the resources available in the process and adding a new extensive monitoring card to the Server Node for increased monitoring systems.

    As our own website and Client Area are also hosted on this Node, unfortunately the Network Status page and Support system will be down for the upgrade process - All support requests for clients on our other nodes should be sent to info@monkeytreehosting.com

    While we've set aside 1-hour for this maintenance, we're positive it'll actually span only 30mins or less; the hour has been accommodated for if anything else happens during the upgrade.

    If you've got any concerns, worries, questions or comments with regards to this maintenance period, please do raise a support ticket or give us a call and we'll be more than happy to discuss this with you.

    Kind Regards,

    MTH Team

  • Date - 01/05/2014 02:00 - 01/05/2014 03:00
  • Last Updated - 01/05/2014 02:45
MTH: Planned Maintenance Notification (Resolved)
  • Priority - High
  • Affecting Server - UK Shared (old)
  • Hi All,

    This is a small maintenance notification that will only affect our cPanel hosting clients - we need to increase the virtual and swap resources on the server you're on so your accounts aren't strained to work with what they've got. Seeing as it's a VPS system, we've allocated the new resource settings already and this will be a 4min job.

    Unfortunately, to do this required a reboot of the shared server, however due to the equipment and setup we use, this will only be a 3-4min job - I've only put 10mins on the schedule to allow for any over-run, however we don't plan on needing this at all.

    99% of you won't even notice the reboot due to the speed of it, plus we'll be running it at 3am on Sunday 6th.

    As always, if there's any issues, questions, comments or queries, please raise a support ticket and we're more than happy to discuss this with you.

    Kind Regards,

    MTH Team

    UPDATE:
    07/04/14 08:00am
    Our apologies, this work was delayed. This will now be happening at midnight tonight.

    UPDATE:
    08/04/14 01:00am
    Following a longer than planned reboot, all services are back up and running.

  • Date - 06/04/2014 03:00 - 08/04/2014 01:00
  • Last Updated - 08/04/2014 01:21
Shared Hosting IP Blocked on some black-lists (Resolved)
  • Priority - Low
  • Affecting Server - UK Shared (old)
  • Hi All,

    This is a quick message to inform you that we've discovered that our shared IP has been added to some blacklists due to spam emails that some clients have been sending. We're currently in the process of narrowing down the account(s) which are doing this and then this will enable us to delist from the blacklists.

    UPDATE:
    27/03/14 22:30
    We've isolated the account that was doing this, suspended it, and are in the process of delisting from a number of blacklists.

    UPDATE:
    27/03/14 23:00
    We've been able to remove ourselves from 4 of 7 blacklists instantly.

    UPDATE:
    27/03/14 23:05
    Just received notification that 2 further blacklists have accepted our delist requests; this means we're awaiting for 1 more to give the all clear.


    UPDATE:
    28/03/14 10:00
    The final blacklist is unable to be instantly delisted from and unfortunately takes 7-days of "no reported spam" before it automatically delists our IP. The information about this final blacklist is shown below:

    IP Address: Listed
    Reason for listing: abusive activity (spam) seen within the last 7-days
    Number count: 1-hit (means 1 email)
    Last Impact: 27.03.2014 22:23hrs
    Expire time: 04.04.2014 01:00hrs (Thursday evening)

    UPDATE:
    04/04/14 09:00
    Removed from UCEPROTECTL1 4/4/2014 8:54:19 AM

    Issue resolved.

  • Date - 27/03/2014 11:30 - 04/04/2014 09:00
  • Last Updated - 05/04/2014 10:36
VPS Control Server (Resolved)
  • Priority - High
  • Affecting Server - VPS Control Panel
  • Morning guys,

    This morning we were informed that our VPS Control server had a HDD failure overnight. Unfortunatly this server needs to be shutdown for the new HDD to be installed and system restored to working condition.

    Due to this, the VPS control panel is currently in maintenance mode and unaccessible - in addition, your VPS metric stats in the MTH Client Area will be incorrect.

    Please note: Your VPS operation is not affected by this and should be working fine.

    The maintenance is undergoing this morning and is expected to be back up and running by early afternoon.

    As usual, if anyone has any queries, please do raise a Technical Support ticket from your client area and we'll address them swiftly.


    Regards,

    MTH Team

    140303 17:00 UPDATE: The server hard-drive has been replaced this afternoon and the system is currently in the restoration phase.

    140303 22:00 UPDATE: The server is up and running and VPS control panel is back online.

    140305 06:00 UPDATE: The VPS backup images have been updated and the backup schedule has been restored. Everything should be back up and running as normal. If anyone has any concerns or notices any problems, please raise a support ticket and we'll get it looked into.

  • Date - 03/03/2014 06:00 - 05/03/2014 03:00
  • Last Updated - 08/03/2014 15:01
PSU Failure (Resolved)
  • Priority - Medium
  • Affecting Server - UK Shared (old)
  • Afternoon,

    We've just been informed by our Datacentre staff that we've had a PSU failure on our cPanel Shared Hosting Server. This means the server is running "at risk" as we always run servers with Dual-PSU incase of hardware failure like this happens.

    A new unit has been ordered and will be installed in due course.

    If you have any concerns, please don't hestiate to raise a support ticket and we'd be more than happy to discuss.

    Regards,

    MTH Team


    140210 15:44 UPDATE: Our suppliers have shipped this out and the new PSU should be installed tomorrow morning.

    140213 12:00 UPDATE: New PSU unit installed - server is now back up to redundancy level.

  • Date - 10/02/2014 13:30 - 13/02/2014 12:00
  • Last Updated - 13/02/2014 15:29
Office Phones down (Resolved)
  • Priority - Low
  • Affecting Other - Office Phones
  • Hi guys,

    Our office phones have gone offline at current - we've got an engineer working on it and should be back up and running this evening.

    If anyone needs to get in touch with us, we're monitoring the support ticket system as always.

    Regards,

    MTH Team

    14/01/2014 - 20:00

    Update, these are now working now and we're fully contactable should you need us.

    Regards
    MTH Team

  • Date - 14/01/2014 14:00 - 16/01/2014 19:47
  • Last Updated - 16/01/2014 19:47
Very brief Network issue (Resolved)
  • Priority - Medium
  • Affecting System - UKNODE1
  • Hi guys,

    We've just been informed of a very brief network issue at our Midland Datacentre - this should now be resolved and we are looking into the cause. We will update this ticket with the cause information as soon as we receive our DC tech report.

    If anyone has been affected by this, please let us know if we can help.

    Kind Regards,

    MTH Team

  • Date - 02/01/2014 10:44 - 02/01/2014 10:47
  • Last Updated - 02/01/2014 11:15
Email issue (Resolved)
  • Priority - Low
  • Affecting Server - UK Shared (old)
  • Hi all,

    Since moving our server to the UK, for some reason Microsoft have picked up our new IP and blocked it as it has no previous history. Due to this, any email sent through our server gets junked in hotmail, outlook or office365 at current.

    We're currently working on resolving this issue, and as soon as Microsoft answer the phone, we'll have an update posted here.

    Regards,

    MTH Tech Team

    **UPDATE** 20:03hrs
    We have got through to Microsoft and submitted all the information they've requested. We're now awaiting on their response to whitelist us to get back to normal. Luckily, I believe it's all USA support, so they'll get the issue in their afternoon due to the time difference.

    **UPDATE** 00:40hrs Tuesday 17th Dec '13
    Just a quick update to this issue - We've been on the phone intermitently with Microsoft this evening for the last 7 hours. After many escalations, I believe we're getting somewhere.

    **UPDATE** 13:42 Tuesday 17th Dec '13
    Hi guys, so we've manually gone through all the accounts on the affected server IP and enabled DKIM and SPF Records - This activates email authentication and starts giving our IP a reputation. Following this, recipients should receive ALL emails, however since Microsoft are still filtering emails, they may still go to Junk folders until your domain develops it's reputation. To do this, recipients need to "Mark as safe sender" or select you email as "Not Junk"

    We are monitoring the IP reputation and it's been unblocked by Microsoft, however their automated filter is still junking it. - As soon as we see this improve, we'll inform you further. The more emails sent and marked as safe, the quicker this process will repair itself.

    Apologies to those affected by this, unfortunately it's out of our control as we've done as much as possible.

    **UPDATE** 22:15hrs Tuesday 17th Dec '13

    We've found a "work-around" for this issue at current until we find how Microsoft want to validate our IP.
     
    Use your domain email, ie "name@domain-name.com" from within a free Gmail account, the email goes through no problem straight to the inbox; I assume this would be the case for a Yahoo, Hotmail or Outlook.com account too as using any of their systems would bypass our server IP. I know this isn't an ideal solution or way to work in your environment, however it maintains your professional email image (using your domain) to the outside world (ie, clients)

    **UPDATE** 04:29hrs Wednesday 18th Dec '13
    So, finally Microsoft have reviewed our IP and removed it from their "blocklist 1". They've said to wait 1-2hours for this to take effect so we're going to run a few checks and tests around 6-7am and update the ticket following this!

    **UPDATE** 06:37hrs Wednesday 18th Dec '13
    I can confirm that blocklist 1 removal was successful and all emails successfully route into Microsofts network.

  • Date - 16/12/2013 18:00 - 18/12/2013 06:30
  • Last Updated - 18/12/2013 06:39
Server Migration (Resolved)
  • Priority - Medium
  • Affecting Server - UK Shared (old)
  • As part of our ongoing development, we're migrating all our shared hosting packages to our newly installed UK Server - In terms of uptime, unfortunately this means a small amount of downtime following the duplication and DNS changes.

    The the planned maintenance is to go ahead tonight (Saturday 14th December), and should take just over 60mins; data migration will start at 22:30hrs (GMT) and everything should be running from the new UK server by midnight.

    If you wish to work on your site, or make any alterations, please do so before 22:30hrs tonight - We will be taking a full backup prior to migration, but we expect everything to go smoothly.

    If anyone has any issues, questions or worries, please let us know and we'll be more than happy to assist with any personal backup's or answer any questions.

    Kind Regards,

    MTH Team

    -- UPDATE-- 23:28hrs
    Ok, so a successful migration has been done and our nameserver DNS changes have been made. We're now just waiting for the DNS to update itself for your sites to be fully live again. Unfortunately this isn't a job in our hands, but we estimate it to be no longer than 30mins as planned in the scheduled post this morning.

    --UPDATE-- 00:30hrs
    So, following the migration, the DNS has completed now too. All hosting is up and running on the new UK based server in a much faster environment than previously. We apologise for the downtime this evening, but hope you enjoy the increase in speed and response from the new server and location!

    If anyone has any problems following the migration, please raise a Technical Support ticket and we'll get onto it ASAP for you.

  • Date - 14/12/2013 22:30 - 15/12/2013 00:00
  • Last Updated - 15/12/2013 02:17