We invite you to subscribe to our new status page at status.widen.com and receive email notification immediately if sites are experiencing issues.
June 21, 2018
9:02 a.m. CDT: The file uploading issue affecting certain customers has been resolved.
8:38 a.m. CDT: A fix has been implemented. Failed uploads are being reprocessed and we are monitoring the issue.
7:43 a.m. CDT: We are investigating an issue with files failing to upload for certain customers.
June 7, 2018
3:57 p.m. CDT: All sites experiencing issues are now available and working as expected.
3:43 p.m. CDT: Sites experiencing issues are beginning to come back online.
3:02 p.m. CDT: We are experiencing an issue with the availability of some sites and are actively investigating.
June 6, 2018
10:17 p.m. CDT: All sites experiencing issues are now available and working as expected.
9:48 p.m. CDT: Sites experiencing issues are beginning to come back online.
9:00 p.m. CDT: We have escalated the issue impacting availability of some sites to Amazon Web Services (AWS) and are working with AWS to resolve.
8:27 p.m. CDT: We have identified the issue impacting availability of some sites and are actively working toward a resolution.
8:02 p.m. CDT: We are experiencing an issue with the availability of some sites and are actively investigating.
May 31, 2018
6:14 p.m. CDT: All impacted sites are now working as expected. We will continue to monitor this issue.
4:30 p.m. CDT: We are seeing issues with multiple sites due to an issue with with Amazon Web Services (AWS) instances. We are working with AWS to correct the problem.
3:30 p.m. CDT: Some customers are still experiencing delayed uploading. We are currently investigating.
2:28 p.m. CDT: The index issue has been isolated and resolved for most customers. Some may still experience delayed uploading, however.
1:29 p.m. CDT: A secondary issue has been identified that impacts our indexing queue, and we are working on resolution of the issue.
12:15 p.m. CDT: The processing issue has been resolved. There are still delays with assets appearing after upload. We are continuing to investigate.
11:48 a.m. CDT: An issue with our processors has been identified, and we are working on resolution of the issue.
10:55 a.m. CDT: Customer sites may be experiencing issues with longer than normal upload times. We're currently investigating the issue.
May 17, 2018
7:00 a.m. CDT: All analytics indexes were verified for January 2017 for Insights.
3:50 a.m. CDT: All site search indexes were verified.
1:55 a.m. CDT: Full search reindex was completed across 100% of sites.
May 16, 2018
11:05 p.m. CDT: Full search reindexed completed for 75% of sites. Estimated time of completion of full reindex across all sites is 8 a.m. CDT. Insights data is still being restored.
10:00 p.m. CDT: Full search reindexed completed for 50% of sites. Estimated time of completion of full reindex across all sites is 8 a.m. CDT. Insights data is still being restored.
7:00 p.m. CDT: The restore of search indexes is completed. Search is functioning but may still have data from earlier today. A full search reindex of all sites is being completed across all customers. Estimated time of completion of full reindex across all sites is 8 a.m. CDT. Insights data is still being restored.
4:30 p.m. CDT: The restore of search indexes continues. Our estimated time of completion is now 8 p.m. CDT.
4:05 p.m. CDT: IP addresses were reset and URLs were corrected.
3:30 p.m. CDT: We are addressing stale IP addresses that are causing the incorrect redirect URL to sites.
3:15 p.m. CDT: Search is being restored, with an estimated time of completion across all customers of 5 p.m. CDT. During this time, searching may be intermittent and will contain search data from prior to 2:30 p.m. CDT on May 16.
3:00 p.m. CDT: Search is back online and indexes are being restored. We started restoring search indexes at 2:26 p.m. Insights is still unavailable.
2:45 p.m. CDT: Portals and Workflow are now available. Insights is still unavailable.
2:30 p.m. CDT: The issue has been identified and sites are beginning to become available. Insights, Portals, and Workflow are currently unavailable.
2:25 p.m. CDT: All sites are currently unavailable. Our Development team is investigating the issue, and we'll update this article when it is resolved.
May 15, 2018
8:53 a.m. CDT: The issue was identified as part of a new service that impacted return calls to our API and Portals. This service was removed and the issue has been resolved.
8:45 a.m. CDT: Customer sites may be experiencing issues that cause limited functionality with searching, the public API, and Portals. We're currently investigating the issue.
May 9-10, 2018
8 a.m. CDT (May 10) - The new service architecture was modified and released to prevent sending an excessive load to our search infrastructure. The issue has been resolved.
8 p.m. CDT (May 9) - 8 a.m. CDT (May 10): Customer sites that experienced issues with searching have search indexes regenerated.
7 p.m. CDT (May 9): When the 13.3 code upgrade began, we noticed intermittent search issues across several customer sites due to the search service. This is undergoing investigation.
May 9, 2018
11:55 a.m. CDT: We uncovered a recently deployed service that was adding excessive load to our search infrastructure. This service has been disabled while we investigate further.
11:45 a.m. CDT: We are currently investigating issues with searching and accessing categories.
March 26, 2018
5:30 p.m. CDT: The backlog of processing is complete. Normal processing times are expected.
4:30 p.m. CDT: Delays continue with processing as the backlog of requests needs to catch up.
12:30 p.m. CDT: The issue with Amazon Web Services in the US-EAST-1 region is resolved. Delays continue with processing as the backlog of requests needs to catch up.
10:02 a.m. CDT: Your Widen Collective site may be experiencing processing delays as a result of an issue with Amazon Web Services in the US-EAST-1 region.
January 2, 2018, and January 3, 2018
5:45 p.m. (Jan. 2) - 3 p.m. (Jan. 3) CST: Your Widen Collective site may have been unavailable for 15-50 minutes around this time as a result of an issue with Amazon Web Services.
We have taken steps internally to change our operating procedures to greatly mitigate the chance of these events occurring again in the future. We're also working with AWS to ensure that these types of events are communicated to us well in advance so we can inform you of any outage to your services.
Read more about the impact of this downtime and vulnerability remediation here.