dogfrog4/doc some sites not responding (Resolved)

Affecting Server - doc_ourserver_US | Priority - High

Sites located on the second IP range on dogfrog4 are not responding due to a routing issue with the range.

We've temporarily changed IPs on the few sites on the range and are working to get service restored.

Only a handful of sites have been affected so far.

Will update with news when available, expect resolution within the hour ...

Date - 13/10/2016 08:20

Last Updated - 16/11/2016 14:24

Major rolling network and AWS outages (Resolved)

Affecting Other - AWS and internet | Priority - Critical

We're seeing major rolling outages across multiple internet based services.

At the time of writing, part of our datacentre in Sydney is affected.

Part of AWS in Sydney is also affected - some servers fine, others down.

We expect this to recover itself in the next hour -- no clear information is available yet.

UPDATE 4:51pm: Some more info and status available from here http://aussieoutages.com/status/aws-amazon-web-services�- still being worked on, affecting multiple providers including eWay etc

UPDATE 5:51pm: Most sites are back, though one ecommerce server and hooch are awaiting help from AWS support to get them rebooted (already urgently requested and should happen in due course)

It appears the outage was related to flooding in Sydney which caused power drops and internet backbone problems. �As part of this, AWS had a power failure in one datacentre.
News coverage:�http://www.theage.com.au/technology/technology-news/web-chaos-mostly-over-after-amazon-web-services-hit-by-power-outage-during-sydney-storm-20160605-gpc707.html

Date - 05/06/2016 15:26 - 05/06/2016 17:30

Last Updated - 16/09/2016 08:08

doc/dogfrog4 server down (Resolved)

Affecting Server - doc_ourserver_US | Priority - Critical

The doc/dogfrog4 server crashed overnight.  We're in the final stages of recovery and expect to have the server rebooted within the hour.

We've been unable to investigate the cause of the crash, so there is some concern that we may face some ongoing issues and we'll update here as soon as we have news.

Apologies for the downtime.

UPDATE: the server is now online and running and all seems well.  We're running some checks on it but so far haven't found any long term problems.

The outage appears to be related to a network interface going offline.  We're looking further into this.

Date - 16/09/2016 02:04 - 16/09/2016 07:42

Last Updated - 16/09/2016 08:05

dogfrog4 emergency maintenance Sat 23.4 2pm (Resolved)

Affecting Server - doc_ourserver_US | Priority - High

The datacentre housing dogfrog4/doc is doing some emergency router maintenance from 2pm to 2:30pm today.

There's a chance there will be up to 20 minutes downtime during this window; however, it's quite likely the maintenance won't be noticeable to users.  We're just letting you know so you don't worry!

Cheers,

WD3 team

Date - 23/04/2016 14:00 - 23/04/2016 14:30

Last Updated - 29/04/2016 13:53

k9 server non-responsiveness (Resolved)

Affecting Server - k9_ourserver_US | Priority - Critical

Our k9 server is not responding and technicians are investigating.

As soon as we know more we'll update here.

UPDATE: the server has recovered and we're watching it closely for a while.

Date - 29/04/2016 13:37 - 29/04/2016 13:48

Last Updated - 29/04/2016 13:52

Telstra: Connectivity to servers this morning (Resolved)

Affecting Other - Australia-wide internet | Priority - Critical

Around 8:40 this morning (Friday 18.3.2016)  internet connectivity to some of our servers from some ISPs was down.

All of our servers were up and running and available from many ISP (internet) connections but some were failing, notably Telstra.

Update 11:44am:

Information we've received from AWS indicates that the problem was with Telstra connectivity.  This will have impacted people on some other ISPs if their ISP used Telstra's backbone.

For more information, here are some articles about the Telstra connectivity issue:

http://www.itnews.com.au/news/telstra-customers-cut-off-from-aws-sydney-data-centre-417111
http://mashable.com/2016/03/17/telstra-outage-affects-aws/#TXxJQ_oHM5qi

Date - 18/03/2016 08:41 - 18/03/2016 09:39

Last Updated - 04/04/2016 14:48

Rover webserver down - 10 minutes (Resolved)

Affecting Server - lassie_ourserver_AU | Priority - Critical

Our AU-based Rover server (dogfrog5.net) was down for 10 minutes due to what we believe is a memory issue.

System Administrators are looking into the issue and will update this notice with more information as they investigate.

Date - 15/03/2016 16:06 - 15/03/2016 16:16

Last Updated - 15/03/2016 16:20

Datacenter Network Issue - AU Rover & AU (Resolved)

Affecting Server - lassie_ourserver_AU | Priority - Critical

Update 19/02/2016 - 1:45pm: The issue is now resolved; the network issue was caused by a large-scale DDOS attack on their network, but should be fine from now on.

Our AU-based Rover and Laika servers (dogfrog5.net & dogfrog6.net) are down due to a network issue with the datacentre.

They are looking into the issue and expect it to be resolved within 15 minutes.

Updates to follow.

Date - 19/02/2016 13:30

Last Updated - 19/02/2016 13:47

doc/dogfrog4 server move and upgrade (Resolved)

Affecting Server - doc_ourserver_US | Priority - High

Planned start of move:  10pm Friday 30th
Status: updated 6:45am: Move aborted due to unexpected technical issues

We have aborted the move due to a few issues that affected some sites.

Your site will continue to run unaffected on the old server for another week or so.

---

We are planning a move to a newer server (new hardware, new software) in a virtualized environment starting at 10pm on Friday 30th.

The move will start around 10pm on the night of Friday 30th October 2015 and will continue through the small hours.  Each site will be down only while it is moved (5-60 minutes, less for small sites and more for large sites).

All sites should be available by 8am on Saturday 31st.

More details are available in the notification email you should have received, and in the detailed FAQ at this link:
https://www.wd3.com.au/faq-server-move/

Date - 30/10/2015 22:00 - 31/10/2015 08:00

Last Updated - 12/02/2016 09:49

dogfrog4/doc slow response (Resolved)

Affecting Server - doc_ourserver_US | Priority - Critical

Update: The issue has been identified as intermittent slowness to the datacentre due to what appears to be a minor DDOS attack.  We're awaiting an update with further details.


Doc has been exhibiting some slow response this morning and we're currently investigating the cause.

Date - 16/01/2016 11:22 - 16/01/2016 13:00

Last Updated - 12/02/2016 09:49

Below is a real-time overview of our servers where you can check if there's any known issues.

View RSS Feed