May 17, 2018
7:00 a.m. CDT: All analytics indexes were verified for January 2017 for Insights.
3:50 a.m. CDT: All site search indexes were verified.
1:55 a.m. CDT: Full search reindex was completed across 100% of sites.
May 16, 2018
11:05 p.m. CDT: Full search reindexed completed for 75% of sites. Estimated time of completion of full reindex across all sites is 8 a.m. CDT. Insights data is still being restored.
10:00 p.m. CDT: Full search reindexed completed for 50% of sites. Estimated time of completion of full reindex across all sites is 8 a.m. CDT. Insights data is still being restored.
7:00 p.m. CDT: The restore of search indexes is completed. Search is functioning but may still have data from earlier today. A full search reindex of all sites is being completed across all customers. Estimated time of completion of full reindex across all sites is 8 a.m. CDT. Insights data is still being restored.
4:30 p.m. CDT: The restore of search indexes continues. Our estimated time of completion is now 8 p.m. CDT.
4:05 p.m. CDT: IP addresses were reset and URLs were corrected.
3:30 p.m. CDT: We are addressing stale IP addresses that are causing the incorrect redirect URL to sites.
3:15 p.m. CDT: Search is being restored, with an estimated time of completion across all customers of 5 p.m. CDT. During this time, searching may be intermittent and will contain search data from prior to 2:30 p.m. CDT on May 16.
3:00 p.m. CDT: Search is back online and indexes are being restored. We started restoring search indexes at 2:26 p.m. Insights is still unavailable.
2:45 p.m. CDT: Portals and Workflow are now available. Insights is still unavailable.
2:30 p.m. CDT: The issue has been identified and sites are beginning to become available. Insights, Portals, and Workflow are currently unavailable.
2:25 p.m. CDT: All sites are currently unavailable. Our Development team is investigating the issue, and we'll update this article when it is resolved.
May 15, 2018
8:53 a.m. CDT: The issue was identified as part of a new service that impacted return calls to our API and Portals. This service was removed and the issue has been resolved.
8:45 a.m. CDT: Customer sites may be experiencing issues that cause limited functionality with searching, the public API, and Portals. We're currently investigating the issue.
May 9-10, 2018
8 a.m. CDT (May 10) - The new service architecture was modified and released to prevent sending an excessive load to our search infrastructure. The issue has been resolved.
8 p.m. CDT (May 9) - 8 a.m. CDT (May 10): Customer sites that experienced issues with searching have search indexes regenerated.
7 p.m. CDT (May 9): When the 13.3 code upgrade began, we noticed intermittent search issues across several customer sites due to the search service. This is undergoing investigation.
May 9, 2018
11:55 a.m. CDT: We uncovered a recently deployed service that was adding excessive load to our search infrastructure. This service has been disabled while we investigate further.
11:45 a.m. CDT: We are currently investigating issues with searching and accessing categories.
March 26, 2018
5:30 p.m. CDT: The backlog of processing is complete. Normal processing times are expected.
4:30 p.m. CDT: Delays continue with processing as the backlog of requests needs to catch up.
12:30 p.m. CDT: The issue with Amazon Web Services in the US-EAST-1 region is resolved. Delays continue with processing as the backlog of requests needs to catch up.
10:02 a.m. CDT: Your Widen Collective site may be experiencing processing delays as a result of an issue with Amazon Web Services in the US-EAST-1 region.
January 2, 2018, and January 3, 2018
5:45 p.m. (Jan. 2) - 3 p.m. (Jan. 3) CST: Your Widen Collective site may have been unavailable for 15-50 minutes around this time as a result of an issue with Amazon Web Services.
We have taken steps internally to change our operating procedures to greatly mitigate the chance of these events occurring again in the future. We're also working with AWS to ensure that these types of events are communicated to us well in advance so we can inform you of any outage to your services.
Read more about the impact of this downtime and vulnerability remediation here.
November 29, 2017
2:40 p.m. CST: The issue has been resolved by the third party. Previews now display for share links and embed codes. We will be moving off of this third-party provider in mid-January 2018 and will no longer have this dependency at that time.
6:00 a.m. CST: There are some issues with previews displaying for share links and embed codes for PDF and document files, resulting from an outage at a third-party provider used for conversions and previews of these file types. Our development team is investigating the issue.