Field Journal from Tom Comeau - 3/13/96

OVERLOADING THE DATABASE SERVER

:-}     <--- (That's an "emoticon", which is computer nerd speak for an
        emotion icon.  You'll also hear them called "smileys". This
        particular emoticon means something like "pained grin".
        The conventional smiley, :-) or ;-) just means "smile".
        Since people can't see what your face looks like when you
        email them, you have to give them hints, or they might
        think you were mad at them when you're just trying to say
        something funny.

I'm a little peeved right now, because my parallelism has come back to haunt me.

Remember from my first journal, when I wrote about how the software changes I was making would allow DADS to work on more than one thing at a time? We used to have just five programs, and each did one step of processing HST observations in sequence. Our new version has seven programs, and lets us do an arbitrary number of jobs in parallel. I can now run eight or ten copies of each of those programs, and work on eight or ten HST pictures at a time!

I've completed all the changes, and I've been testing the complete new system. One thing each program does is open a "connection" to a database server -- another computer that does nothing but handle our database. The database describes all the HST data we have on Optical Disks in the Archive. Unfortunately, having all those extra copies of the programs means I've now overloaded the server, and I have to get the people who maintain that system to change their setup.

Imagine that you have one person who is real good at looking things up in the phone book. The phone book is a database (a paper database, but a good database nonetheless). The person who looks things up is the database "server". There is one "in box" for requests to look things up, and several "out boxes" for completed requests.

The first thing you do is "Open a connection" by marking one of the "out boxes" as yours, by putting your name on it. Only one "client" can use an "out box" at a time.

You get the "server" person to look numbers up by putting a 3x5 card with a request on the card in a little "in box" in front of the "server."

The "server" takes cards out of the "in box", finds the phone number, writes it on the card, and puts it in an "out box", where you pick up your card. There is a separate "out box" for everybody who uses the server.

You take the card out of your "out box", and it has the phone number you want on it. Then you take your name off the "out box," so somebody else can use it.

If there are only a couple of people putting cards in the "in box", the "server" can keep up. If there are 30 people putting cards in the "in box", the server gets behind and the "in box" fills up.

Also, if there are only five "out boxes", and a sixth person wants to use the "server", the "server" doesn't know what to do with the request once he's finished, because there are no "out boxes" left!

That's what's happened to me: My database server ran out of connections, and got flooded by requests.

So I'm getting the database people to "build a bigger 'in box'", to make sure that there is enough space to handle incoming requests, and make more "out boxes", so that I can have more connections to the server.

Which also reminds me how many people are working on developing, maintaining, and operating the Archive. Meanwhile, it's a nice day. It's in the mid-60s, which is great even if you don't consider that we had two inches of snow last Friday. Another member of the Archive Team has brought a baseball and a couple of gloves, so I think I'll go throw with him for a while. ;-)