May 29, 2009

Update

There's been a lot of mail suggesting that Tina and I could have done things better on Tuesday. We're looking into it. Our IT company has launched an investigation, we've got people sorting it... we'll see. We're more than open to constructive criticism, and it's all getting cleaned up. More details below.

But first.. I want to show you something. Look at this. This is the sign-up sheet for STITCHES, right after the sock summit. Now we're bigger - class wise, than Stitches. Scroll through that for a moment. Notice anything? We did. Something we've been noticing for a while. They aren't sold out. Neither is TNNA. The classes at Rhinebeck and Maryland don't always sell out either, and all of these are really big, really well established organizations (or corporations) that have done this a lot, and have nice big reputations. This was our first time out, and with the economy the way it is, we were worried about making it as big as we did. I mean, if those guys aren't sold out, and we make ours bigger... It was a risk - and a pretty arrogant one. (We relied on the power of the sock.)

There were 12 000 people on the "interested" list for Sock Summit. We pointed that out to anyone who would listen, and they all told us that you can expect about 10% of your mailing list to respond. "Ha" we thought. They don't know knitters, and so instead of expecting that 10%, which would have been about 1200, we expected four times that. About 5000.

For months now, Tina and I have been being treated like we're incredibly arrogant and awful human beings for preparing for that. Our friends, families, colleagues and associates have been acting like we smoked some pretty huge SS09 dope. When you rent a conference centre, you have to put down a down payment. It costs a lot of money. We did it anyway. We watched the news about the crashing economy and people losing their jobs and nobody spending any money and we were really, really scared about losing everything on this gamble, but we did it anyway. We believed. We're running an event with more spaces for students than anyone else.

Registration time loomed, and we got a server that people thought was a stupid waste of money. We got it anyway. We primed it to get way, way, way more hits than anyone thought was reasonable. "Thousands of people are going to try" we told them, and we didn't let it go. We made them believe - or at least do it anyway. There's a rumour out there that our IT company let us down... and I don't think so. We built a huge server and a pretty good system and we tested the snot out of it. All things considered, they handled the crash really well, because what happened to our server was so insane that when we saw the raw data we couldn't handle ourselves.

On Tuesday the 26th, at 10amPDT our server got upwards of THIRTY THOUSAND SIMULTANEOUS CONNECTIONS. That is, just to be clear... 30 000.

I am sorry for what happened next, which is that the server slowed to a crawl, then seized helplessly, then had to be rebooted several times (which was hard to do while it was still receiving that load) and some registrations were arsed as a result. When it was finally rebooted and working well (the techs had to break about 19 rules to get it "wide open") a lot of classes reappeared and we had a round two that was a lot better. Mostly, the problems we have now are due to people reacting to the server behaving badly. There are people that the server just didn't let complete the classes, but the people caught in a registration at the time (which was everyone- thousands and thousands) sort of freaked out. (I would have. We're not blaming anyone. When that stuff starts happening you start problem solving.) They used the back button (which was a no-no and gave some people extra classes) they registered twice because they weren't sure if the first one went through, and if we told you the number of mistakes (from rushing and frustration) in peoples registrations, you wouldn't believe it. Names, addresses, emails... lots of typos, and these create extra problems all their own.

------------------------

This means we aren't going to be putting up the page where you can search for your registration, because it isn't a "for sure" way of telling what's going on. When we were testing it we had a lot of trouble, and realized that if we put it up, what's going to happen is that people are going to search, not find themselves, figure they aren't there and freak out. Meanwhile, they are there, they just have a typo or glitch. We've decided to make sure that everything is accurate by doing it all MANUALLY, ourselves. That means that if you have trouble, you should write to us on the Contact us page with as many details as you can, and we'll sort it. If you already wrote us, we're on it. The first refunds went out today, and we have all our staff working only on this, and we hired more staff, and the IT company is lending us staff. It should go fast, but please be patient. We'll sort everybody out as best we can, as quickly as we can. The good news about duplicates is that there may be some open slots coming up on the website. Haunt the place, you might get lucky yet.
--------------------

Now. A final word. We are very sorry if you are disappointed that the server crash made things more complicated for you. We feel your pain, we assure you. It was our server, we accept responsibility for its behaviour and at our own expense, we will fix all of it's mistakes. (Great. Just what we needed. Another teenager.) I assure you however, that even with the server working perfectly (and we're hearing from IT types that they aren't sure if you can have any server perform well under that strain) 30 000 connections competing for (about) 4000 spots within 5 minutes meant that no matter what, there were going to be a lot of disappointed people. Servers that work perfectly don't make more spots appear.

In exchange for doing our level best, putting down the cash to start this up and sweating blood to make it happen, including getting a bigger, better tested server than anyone said we needed.... Tina and I have had some really, really terrible mail. I'm not going to go into details, I'm just going to let you know that we've both cried more in the last few days that we ever have. We agree childbirth is better, and we're both no-drug/homebirthers. This email isn't just disappointed. It's hateful, some of it is threatening, and we are asking nicely for it to stop. We want to hear from you. We will solve all of the problems to the best of our ability, but from now on, if we get an email that is overtly threatening, cruel, deliberately mean or wholly abusive, it will get moved to the bottom of the pile. This is not as punishment, it is because our staff (who are our friends) can no longer bear to read it, and we don't want to make them. Tina and I will do all of those ourselves.

We'd like you to remember, before you email... that your problem is that you didn't get into a knitting conference. It is actually not like Tina and I napalmed a village of orphan babies and then ate their puppies and it is up to you to exact revenge.

Some quick answers.

-No. We can't make it bigger. It's the biggest ever, it's really big, and there isn't more room at the Conference Centre, and we actually don't have a responsibility to make sure everyone can fit. A knitting conference for tens of thousands of knitters isn't a reasonable thing to ask of us. I couldn't get tickets for the last Prince concert even though it was really important to me and I was really, really looking forward to it. I have worked through my pain, and I did it without emailing the man and saying nasty things or implying that he should be better to me.

-No. We can't get the bigger ballroom at the Art Museum. It's is booked, and has been since we tried to book it before.

- No. We can't put more students in all the classes. Class size is dictated by the teachers. That's industry standard, and if the classes were any bigger you would just be sad that the classes were so big you couldn't learn in them.

- No. We can't get Barbara Walker to do some extra lectures. She's almost 80, and a retired and extraordinarily well respected matriarch of our community. We won't be exhausting her.

- No. we don't agree that we are horrible people because you didn't get what you wanted.
We are very, very sorry you are disappointed. We are even sorrier about the server crash, because it made what we now understand was inevitable - a lot of disappointed knitters, a lot of disappointed knitters who think that if the server hadn't crashed it would have worked out for them. The server slowed down for all of you. It crashed for all of you. Nobody got an advantage and we're heartbroken that you're sad. Write to us. We're helping everyone as best as we can. Really. While we don't think we're horrible, we know that this feels horrible, and we want to make as many people happy as we can.

It is a very good thing my friends, that we love knitters the way we do, because a few thousand bad skeins are sorely testing our patience. For every one of you who wrote to us with sympathy and intelligence (and there are way more of you than the other).. Thank you. We appreciate your patience, we are very sorry, we're working really fast, and we're getting it fixed - but demand outstripped supply way more than we could have realistically predicted. It's like the tickle-me-Elmo of knitting conventions, and you couldn't have surprised us more.
Thank you for the compliment. We'll earn it.

Now, if you don't mind, I'm putting this topic down long enough to go do a book event in Plano, Texas (I had Austin last night, what a gas) and tomorrow I'll tell you all about it. Austin kept up it's reputation for weird.

Posted by Stephanie at May 29, 2009 6:18 PM