Chapter 102: The Toggle
[Noe Valley apartment Tuesday, January 20, 2016, 11:02 PM]
The phone buzzed on the counter next to a Blue Bottle cup he'd reheated twice. Diana's name on the screen, then a wall of text. He read it standing.
Ray Okonjo's evaluation committee, the one that had been "exploring options" for six weeks, wanted a live document generation demo at their scale by Thursday morning. Their scale was four thousand concurrent users across three document classes. Hooli was presenting to the same committee in the same window, which was why the RFP had moved up twenty two days.
He forwarded the text to Marcus. Then to Oyelaran. Then he sat on the one unpacked box he used as a chair, because he needed a number before he could decide anything.
Four thousand simultaneous users. Three document classes. At their throughput, that was roughly a hundred inference instances sustained for the demo window. A hundred A100 class GPUs. In January 2016, those did not exist to be rented, since the biggest public cloud topped out at maybe sixteen, if you booked two weeks ahead and knew someone.
He had a hundred. He had never used them.
The burst allotment, ten instances becoming a hundred for twenty four hours at triple cost, had sat in his account since the beginning. A safety net he'd never needed. The cost was roughly three hundred and ten thousand dollars for one day, billed against the monthly allotment. One shot per month. Use it or hold it.
He held the phone. The committee met in thirty six hours, so the choice couldn't wait.
The freight elevator was grinding at 11:35 when he walked in. Marcus was at the rack, cocooned in the surplus green sleeping bag with a laptop balanced on his knees, the fabric pulled to his chin. The monitor wall threw blue light across his face.
"Concurrent load test?" Marcus said, without looking up.
"Four thousand users. Three doc classes. Live demo, Thursday morning."
Marcus's fingers stopped. "That's a hundred instances."
"Yes."
"We have twelve."
"I know."
He turned in the bag. The zipper caught on the laptop edge. He freed it, looked at the rack, looked at Ethan, and did what he always did: the literal inventory of what was physically present versus what was being claimed.
"You're going to need ninety more by tomorrow night."
"By 11:40. Tonight."
The clock on the wall said 11:38. Marcus watched him, flat and patient, waiting for the part that made sense.
"Deployment constraints," Ethan said. "The provisioning window opens for about twenty minutes. After that it closes."
"What provisioning window."
"The kind I can't explain."
Marcus stared at him for four seconds. Then he pulled the laptop back onto his knees and opened the monitoring dashboard. "Wake me if the loss drops below ten or if anything catches fire. Literal or metaphorical."
"That's the line."
"It's always the line."
At 11:40 Ethan opened the browser on his phone. There was no URL bar, no page, just a menu he'd felt a hundred times and never opened. The instance count read 10. He pressed the allotment field, entered 100, and confirmed.
Nothing happened on his phone. Everything happened on the wall.
The monitor lit up in a cascade. Ten instances. Twenty. Forty. Marcus's laptop beeped as the dashboard auto scaled, lines climbing in real time. Sixty. Eighty. The counter settled at one hundred and the rack fan, the one that had been a gentle hum since installation, kicked to a pitch Marcus had never heard.
Forty minutes. One hundred instances. The loss curve was already initializing.
Marcus was standing. He hadn't stood up like that before, not once. The laptop was on the floor. He was looking at the rack the way a man looks at a wall that just produced a door.
"Where did those come from."
"Deployment constraints."
"You're going to say that a lot, aren't you."
"Probably."
He picked the laptop back up, sat down, and pulled the zipper up. "I'm going to pretend this is normal. For the next thirty six hours I'm going to pretend this is normal."
[Wednesday, January 21, 2016, 9:14 AM]
Oyelaran arrived with a legal pad, four teams' names written in columns, and the demo path broken into nine steps. He'd been there since 7 AM.
"Two features have to die," he said, tapping the legal pad. "The multilingual batch processor and the citation cross reference engine. They're demo irrelevant and they're the two things that break under load."
"Kill them."
"Already killed. Kai's team is on the front end scaffold. James has the inference pipeline. The API team has the document routing. I've got the contract layer." He flipped the page. "The problem is the human review layer. Full review at four thousand concurrent adds about nine seconds per document. The committee will see latency and read it as capacity failure."
"Can we pipeline it?"
"We can. But not in thirty six hours. What we can do is make the review layer a configurable toggle, on by default, but the customer can set it to batch mode or full bypass for their own workflows. Standard SaaS pattern. It ships in the demo as on, and it reads as flexibility, not omission."
Ethan looked at the legal pad. The toggle was one line in a contract. That layer was what he'd built the entire product philosophy around: the human in the loop, the safeguard. It was the whole reason the company had always called itself a documentation engine, not a misinformation engine.
The clock said thirty one hours.
"Make it a toggle," he said.
Oyelaran wrote it down without pausing. He was already on the next line, the API timeout config, the load balancer fallback, the failover path if instance forty seven dropped.
He was excellent at this. The passive scan confirmed it every time Oyelaran walked into a room: 9.2, steady, the second highest rating in the building. The man coordinated four teams off a legal pad, killed two features in ninety seconds, and negotiated the toggle into a clause without breaking stride. He did the job well. He did the job exactly as a 9.2 would.
Ethan let it go. The rating said the man was right, and the clock agreed with the rating. The toggle was still there, on by default, configurable by the customer. Standard.
Sarah arrived at noon with the pre flight checklist.
"Threat surface?"
"Demo path only. Two features cut."
"Legal?"
"Toggle clause, standard SaaS."
"Comms?"
"Diana has the prospect on a thirty six hour update cadence."
"Budget authorization?"
Ethan handed her the line item. She looked at the line: $310,000, one day, no vendor name.
"Deployment constraints," she read.
"That's what it says."
She held the page for a moment. The red marker was in her hand, uncapped, the tip hovering over the line. She didn't write anything. She set it down.
"Small victories," she said, and went to read the checklist out loud over the rack fan.
The demo shipped at 3:47 PM Wednesday, thirty four hours from Diana's call. One hundred instances held four thousand concurrent users across three document classes, and the latency profile made the committee's technical lead lean forward in his chair. Oyelaran ran the ship review in the second floor common area and the room applauded. Kai's team had the front end stable. James's pipeline held. The API team's routing didn't drop a single request.
Oyelaran took the applause with a nod and a half smile and went back to the legal pad to confirm the contract language. The toggle clause was in Section 4.2, Subsection C: "Human Review Layer configurable by customer administrator. Default: enabled. Options: enabled, batch mode, disabled."
Standard.
[Thursday, January 22, 2016, 4:20 PM]
The second floor was mid celebration. Someone had produced a prosecco that cost twelve dollars. Sarah's marker, he suspected. Marcus was back in the bag, the rack fan returned to its gentle hum, the hundred instances gone as quietly as they'd arrived.
Ethan stood near the window reading the countersigned contract on his phone. The room behind him was cheering a man he'd hired on a rating, and the cheering was earned. Oyelaran had done the hardest coordination of the arc in thirty four hours, and the room knew it.
The contract had one redline.
Section 4.2, Subsection C. The customer's counsel had moved the review layer toggle from a configuration setting, Gardner administered, default on, to a customer controlled setting. The customer's administrator could now change the review mode without Gardner's involvement or approval. The customer could turn it off.
Oyelaran had accepted it on the call. "It's standard," he'd said. And it was: customer administered settings were the industry default. Refusing would have killed the deal, because the committee's window was closing in minutes.
Ethan read the redline twice. The room cheered. The prosecco popped. The contract was signed, the account was won, the margin discipline that Hooli had tried to break was survivable. He had a real win on the board before the decision meeting.
He pocketed the phone without finishing the page. The toggle was on by default. The customer could turn it off. Standard.
Diana crossed the room with two glasses. She handed him one. "Ray Okonjo just sent the purchase order. Eight hundred K annual, three year initial term."
The figure was good. It covered the two accounts Hooli had taken and then some, so it was proof that holding price had been survivable.
He drank the prosecco. It tasted like twelve dollars. Behind him, Oyelaran was telling the legal pad story to Alex and Kai, and the story was funny. The man was warm and fast and excellent, and the passive scan said 9.2, and the scan was right.
He had not read to the end of the contract. The end of the contract was a toggle that a customer could switch off, in a regulated industry, at scale. Because Gardner had shipped in thirty four hours, the safeguard he'd built the product around was now one setting in someone else's admin panel.
The room cheered. He was tired and elated and holding a page he had not read to the end.
His phone buzzed. A text from Marcus, sent from the bag.
Two cloud vendors called me today asking who our provider is. I told them deployment constraints. They didn't think it was funny either.
✦ ━━━━━━━━━━━━━━━━━━━━━━━━ ✦
Read more chapters for free at unwrittenrealm.com
✦ ━━━━━━━━━━━━━━━━━━━━━━━━ ✦
TL;DR — Patreon has the chapters ahead.
Silver / 10 ahead
Gold / 15 ahead
Platinum / 20 ahead, no weekly wait
That's it. patreon.com/fanficwriter1
If reviews aren't your thing, no pressure — they do help though.