Floods and Racks and My Back Oh My!11 Aug 2015, by Brad Collins.
I doubt many people noticed, but we've been down for a week. It all started around June 29 or 30 when the rains came, and they came and they came…. The building next door has been under massive renovation for the last month and a half. And since their building is four floors tall, and ours only two, the workers have made it their mission in life to drop as many bricks, scraps of wood and chunks of concrete onto our roof as possible. The fact that we have a corrugated steel roof amplifies the sound of impacts by orders of magnitude. So from 7am till 6pm six days a week it has felt like we've been under a constant aerial barrage.
When the rains hit and hit hard, the debris and damage to the roof quickly became evident. Rain in this part of the world during the rainy season tend to come late in the afternoon and end around sunset. Thursday was no different, except it started hard and kept getting harder for the next couple of hours. At about 7pm Ruben starts shouting for help from the front of the building. I rush out and find water literally pouring through several places in the front wall. After pulling desks and computers out of harms way, we tried to catch what we could in buckets and then continually mop the rest. Nearly two hours later it was over. For that night at least.
The next day we managed to climb onto the roof (no small feat) to find that all the debris had completely blocked up the gutters and the water damned into a pool that flooded the office.
That afternoon, the rains started again, but this time they lasted longer. My office in the back room began to have problems. This is mostly because the construction site had managed rip out all of their gutters and channel all of the water from their roof through a pipe in the wall that emptied onto a spot on our roof almost directly above my desk in my office. This meant that our roof had to not only handle heavy rains that the gutters could barely handle normally, but also handle all of the water from the roof of a very large building as well. The water overwhelmed the gutters and poured into our back patio and the debris clogged the drains forming a small swimming pool directly above my bedroom. So in the middle of the rain one of us had to stand on the patio with a stick and keep the drain clear so it wouldn't pool. Even then it wasn't enough, and water was cascading down the concrete steps like a waterfall.
Even when the rain stopped, the water from next door kept pour out for hours like it was draining a swimming pool. We later found out that in addition to the rainwater from the roof, a water pipe had burst and we had about 30 hours of water from the broken pipe that in addition to the rain.
Water had been seeping down the walls and through small holes from the damage to the roof to the point where my office began to flood as well. Between mopping the floor, keeping the drain clear on the patio we had to move most of the furniture out of my office and open up holes in the ceiling so it could pour down into large plastic trash bins.
We really thought we'd seen the worst of it and as long as one of us was here on flood watch we could handle it.
I moonlight as the night manager (or token white guy, take your pick) for Garage Bar which is down the street from our office near the riverside. Part of my duties (if you can call them that) is to go on a sunset cruise which the bar does every Saturday. Ruben was on flood watch, and the cruise was only two hours. What could go wrong?
Within five minutes of the boat pulling out onto the river, things looked amiss. There was an ominous black cloud down stream behind Nagaworld. The winds looked like it would stay on land and miss the river but it quickly became clear it was headed straight for us. Other boats on the river were tearing past us heading to dock before the storm hit, honking horns at us, waving their arms and pointing at the monster squall looming in front of us. We kept going. The wind wasn't bad, so the water remained relatively calm, but the rain came down harder and heavier than the last three nights combined.
There was no danger, we had a very strong boat and one of the best skippers on the river. Hell, his wife and five year old son was on the boat, so he wasn't going to take any chances. So just as the rains hit he nosed the boat into the embankment which was covered in two and a half meter tall elephant grass, and the customers and I went below deck to ride out the squall. The roof leaked in a dozen places, and water was leaking through the louvered windows, but the seven of us, four tourists and two expats from the bar, sat and drank beer and told stories and generally had a good time. It felt like we were in a colorized version of the African Queen. The tourists took it in stride and I am sure that they will remember the cruise much more than if there had been a proper sunset. It didn't let up for another hour.
I got through on the phone to Ruben on flood watch at the office and I could barely hear him. The patio was a swimming pool, water was flooding down the back stairs and didn't have anywhere to go but into our ground floor. Water was actually squirting out of the ground in our server room and my office upstairs was flooding all at the same time. He shouted through the noise, "I think the building is telling us to get the fuck out!"
Back on the boat, when it finally cleared and we got back under way, you could see the stairs on the embankment of the river looked like whitewater rapids. For that to have happened, the streets would have had to be at least a foot or more underwater to rise above the curb and then flood over the promenade to the stairs and down to the river.
Life had become a waking nightmare. We were exhausted and stressed to the breaking point over and beyond the regular stratospheric stress of a tech startup.
The servers didn't go down through all of this, which in itself was a bit of a miracle. At present, since we have just launched the cloud service, we only have a handful of customers and they all are using our service as a backup in case their main servers go down.
We met with the landlord the next day, who owns the building next door as well, but he refused to take any responsibility for what had happened. We couldn't see any way to prevent this from happening again in the future. So after talking to our customers we decided to make the hard choice of moving the server room, and to give up the building entirely. It as the only way that we could ensure that the cloud service would stay up and running, through hell and high water.
Two days later we took down the network, and moved it to its new home located on high ground in a beautiful swank building on the street behind the Royal Palace. We had the leased line moved and working the next day. That night we had our customers back up online. There has been so much work to do, installing electrical systems and coordinating the move that we didn't get our web sites back up until today.
We moved up our plans to move all of our R&D work to our new office at Prek Leap National College. We are about half moved in and will be up and running by the end of the week.
I managed to blow out my back yesterday (servers are frigging heavy) and was stuck in bed most of yesterday with what looks like a pinched nerve. I can move around today but I won't be doing any heavy lifting any time soon.
I was living under what I was calling our new patio paddling pool. My room didn't flood but got so damp that we had to dry half of our clothes outside in the sun the next day. So have had to find a new place to live as well. Our commercial kitchen downstairs is now largely in storage in my new flat, and we will decide later if we will set it up somewhere else or sell the equipment.
We should be back to business as normal by Monday. Ruben and I will be splitting our time between the office at the college and the Kinto offices. All in all it could have been a lot worse and I think that going forward we will actually be in better shape than we were before.