Dear Buzz community,
I am often reminded about how many of you are educational heroes. You have to confront so many challenges and you do so with courage, strength, and resilience. These challenges come in all shapes and sizes, and from every direction. Thank you for your determination!
We built Buzz to help alleviate your problems and challenges; to be your ally in supporting and strengthening students. We want Buzz to be a reliable companion in all of your learning and teaching experiences. Unfortunately, this week’s intermittent performance issues have not lived up to our (nor your) standard.
Starting on Sunday, January 28, we discovered that several resources were not loading correctly causing either pages to not display completely or preventing some users from accessing Buzz. While this issue has not been consistent, it has impacted users throughout the week, specifically on Sunday (very brief), Monday, Tuesday, and again on Thursday (very brief).
Since the issues first became apparent, we began collaborating with our great partners at Amazon Web Services (AWS), our cloud hosting provider, to explore why we may be experiencing these problems with their storage service (Amazon S3). We have been experimenting and exploring different solutions. While we are learning more with each experimentation, we still have not successfully resolved the root cause of the performance issues.
The gist: While we have learned how to mitigate and reduce the risk of the performance issues (and how to recover from it), we still have not completely resolved it. We are continuing to work closely with AWS to fully resolve these issues.
I apologize for the difficulties this has and could still cause you. I’ll continue to update this post with additional information as we continue to work out the remaining issues. If you have any questions or comments, add them below and I will answer everything I can.
Hope you have a great weekend,
Brad
VP of Product & Partnerships
Comments (1)
Happy Monday everyone! I wanted to provide an update regarding the issues we were experiencing last week.
While over the last week we have done several things to continue to optimize the system, we had not been able to address some of the critical causes of the problems. Then, this morning, one of our system architects woke up in the middle of the night with a theory of what was causing (or at least contributing) the problems. He got up, developed some tests to validate his theory, and successfully was able to reproduce the problem and a solution.
Just a couple of hours ago, after some additional testing, we updated Buzz with the latest changes to correct the problem we were able to reproduce. We are confident that with these latest changes that we have been able to fix some, if not all, of the problems that users were experiencing over the last week.
Through our findings we were able to identify some additional enhancements that we can make which will roll out over the next few weeks. Until then, we will continue to closely monitor system performance to confirm that last week’s problems do not surface again and, in the case they do, remedy them as quickly as possible.
Really appreciate your patience and trust with us! Let me know if you have any questions that I can help address.
Thanks,
Brad