skip to main content

TAMU Webmaster's Blog

Information and insight from the A&M Webmasters

Outage Notification and Review

March 28th, 2008 by Erick Beck

The gremlins scored a victory today. At about 2:00 this afternoon the Google Search Appliance experienced a significant problem which led to a service outage of almost an hour. Any query submission returned a 500 Error. As we were trying to find the cause of the problem seemed to correct itself and the appliance began serving out searches as normal.

We did learn a few lessons from the event. First, as great as Google is, it is not infallible. This rather disturbing news should force to to step back a moment and reevaluate the nature of the search service on campus. Hopefully this was a one-time incident and we can continue forward under our current plan, but we woud be remiss if we didn’t start to look at reliability issues and the need to bring on a fail-over machine if these sorts of things become more common.

The other important object lesson is on being prepared. Whenever we built the new WWW website we created it using modular lements that could be inserted and removed at a moment’s notice. One of these was the search form. When it became obvious that we weren’t going to resolve the outage quickly, we were able to quickly swap the form calling the GSA with one calling the public Google Custom Search. While this solution did help us for the main university site, we need to look at extending the principle so that we can just as quickly swap the form for other sites on campus as well.

While we will always give you plenty of advance notice if we have to bring down the Google Appliance under normal circumstances, it would be wise for you to be prepared to update your own sites in case we do experience further outages. The public search may not have the powerful customizations that our Google Search Appliance offers, but it is a good, free option to consider if the gremlins strike again.

Friday, March 28th, 2008 Search
Share this article

No comments yet.

Leave a comment