My Grand-Ulcer Named Ralph

[This account is not the original author of this piece. I had just been looking for a decent way to read a long post like this. The original post is a throwaway comment on the Hacker News thread “Who wants to be fired?” and can be found here: https://news.ycombinator.com/item?id=21141785. The author refers to themselves as “Throwaway” in the story. I have cleaned up some typos and typography.]

I didn’t know this format was a thing and am so very excited to discover it. I hope you folks enjoy reading horror stories.

I got a job as a Software Engineer in my current company four and a half years ago; friend-of-a-friend sort of thing. The company had an apparently disastrous piece of software that was their main LOB. They had gone through pretty much every local consulting agency at least once – on a few occasions they had gone back to one they had already used. It was about ten years old and consisted of a mix of VB6 (!), VB.NET, C#, F# and somehow now Node. At the time tackling a disaster like that sounded fun and I was miserable at a consulting gig. It was a 20k bump but no benefits (health or retirement), but as a single guy six months away from paying off his college debt I wasn‘t worried. I figured I’d dump a few years in then move on.

Three months in, I’m absolutely baffled at what the company does. I was told they handle insurance claims, basically acting as a TPA. (Important detail: I had no idea what a TPA was at the time. It's gonna matter later.) The software does handle claims, but they also have ten other projects that cover a bunch of random business use cases. Apparently the CEO is a self-described “idea man” and would task the previous developer to ‘prototype’ his ideas from time to time. The problem was his idea of a prototype was a fully-functional application that he could sell to investors and clients – until he got bored with it and shelved it. This ended up with the company having around a half-dozen actively used products in a half-dozen markets. In addition to the TPA side of the company that was about 50% of revenue, the other half was split over 1) check cashing software, 2) HR/onboarding software, 3) some sort of discount medical visit scam, 4) some sort of MLM scam that the CEO's brother-in-law co-opted him into, 5) a random cannabis and self-help website run by some yoga guru type dude the CEO knew and finally 6) a piece of software that helped churches organize events and donations that took about 50% of any transaction that was run through it as “fees” for our company. Now I could talk about any of those monstrosities at length, but this is already shaping up to be a wall so I’ll skip that.

One and a half years later. I’ve wrangled the mix of VB6, VB.NET, C#, F#, PHP4, PHP5, PERL, ASP.NET WebForms and MVC, SQL Server, Postgres, MySQL still using MyISAM, god knows what other horrors I've forgotten. All of this without version control – just folders copy-pasted over and over on a 10 year old server in the closet that has no redundancy, two failing disks and one PSU out of order. The last guy had started some positive changes: moving everything over to Azure, porting everything related to the claims business into a more modern MVC app. I finished his work. I squashed about a dozen Wordpress instances into a single, multi-tenant host. Squashed out all the other languages and databases into just C#, ASP.NET, SQL Server. Ended up reducing the Azure spend by about $2000 a month. Felt good! CEO loved me. COO (my direct manager) loved me. CFO was pleased. All throughout this, I had convinced the COO to cut out all the shady, near-illegal, morally bankrupt garbage we did. No more check cashing (awful, awful industry), no more MLM of any sort, no more stealing money from churches (we kept that going, just changed our fees to a nominal amount). All the work I had done lead to a decrease in onboarding time from two or three days to ten minutes and the TPA side of things was now about 85% of our revenue. Happy ending, right? Just you wait…

Somehow, I had not encountered a single brilliant “CEO Idea” for one and a half years. He decided to fix that on one delightful summer day in the mid-west by announcing that we would be acquiring a healthcare startup that a buddy of his ran. Now this pissed most of the folks at the company off and is probably a good point to talk a little about the structure of said company. As mentioned, we had a CEO, COO, CFO, and “Chief of Sales” (never heard of a COS myself, but who knows). We didn't call ourselves a startup and had none of that Bay-style of startupness; we were just a small business with some investors. After the C’s we had myself as the lone engineer, two sales guys, three admin-types and six or so customer service folks. None of which had healthcare or retirement benefits, mind you. So there was a bit of rancor when Mr. CEO started talking about dropping $5 mil to acquire this fancy new healthcare company. Somehow me, Mr. Software Engineer, ended up being the guy that needed to take this head-on (well, to be fair, the COO and I had great relationship). That's a tale in and of itself, but at the end of the day we ended up getting a 6% matching 401k and $500/$1000 single/family monthly reimbursement for health insurance, stopped three or four people from quitting, got me a whole lot of respect in the office and a fancy new title of “Chief Technical Officer” (not related to the benefits; CEO was just happy at how efficient I'd made everything) and 20k base salary increase. CTO at a company with one engineer. Neat. Happy ending, right? Just you wait…

We also got a brand new healthcare startup for about $2.5 milion in cash, $2.5 milion in stock. We got sheisted and it was our fault. While I'm no MBA, I know what due diligence is, and I intended to do it from the technical angle while our CFO handled it from the financial. Before we bought the company I made every effort to actually review what their software looked like, but was single-handled blocked by my own CEO. “We’re never going to do that, Throwaway,” he would say, “other CEO is my friend! I’ve known him for twenty years and if he says his software is solid, it is! Just trust me.” Diligence took about three months and despite dozens of arguments, I was denied any access to anything technical. All I ever got was: “Our software is in Node using MongoDB and is hosted in the cloud.” Great. I was never even allowed to meet or speak to their development team (apparently 5 engineers, all of which were phenomenal). The only human being I ever spoke to at this company was the CEO. So I tried other angles, the big one being: what the hell does your software actually do? Their big claim to fame was “modernizing concierge medicine using AI”. If you're like me and have no idea what concierge medicine is, it basically means your doctor comes to you because you're a rich yuppie and can't be bothered to leave your beach house to visit him. How do you enhance that using AI? I had no idea. Still don't. And so we bought the company with zero diligence done, though the CFO did say their books looked good, whatever that means. So the nightmare begins…

2 years in. We start onboarding people, I start onboarding the project itself. I am finally given direct developer contacts, which are a bunch of emails that don’t end in the same domain as the company we just bought? Pardon? They‘re all @BobsRandomConsultingCompany. I reach out, explaining who I am, that we just acquired Project X and I need access to the code, environment, engineers – the whole nine. I get a very lovely, professional response from a Project Manager over at Bob's who lets me know that they will be sending over a contract so we can get started right away, along with their rate sheet! I'm baffled! I thought Project X had 5 internal engineers, Mr. Other CEO?! At this point I promptly aged six months in six minutes and I felt the first twinge of an ulcer growing.

Contract arrives, I sit down with COO and CFO and explain that we have been duped. COO is angry, CFO is not concerned until I show him the contract that Bob's sent over. The contract ye olde healthcare startup signed apparently agrees to pay for five fixed resources (at $200/hr!) for 40 hours of work each, per week, for a period of a year. Now I'm not unfamiliar with being outsourced as a resource from a consulting company for a fixed amount per week – but never have I seen a contract that binds you for a year, much less for five resources, with not one deliverable mentioned anywhere. Maybe my five years of consulting weren’t enough, but that blew my mind. Additionally, they sent us the server bills (AWS) and informed us we paid directly for utilization in addition to a “HIPAA Monitoring and Compliance Fee” of $3000 per month. As I had not a year ago lowered our own cloud costs to about $800 a month, this number struck me as staggering. $3000/mo base plus around $2000 for the servers currently running. Also, “what the fuck is HIPA,” I said aloud, the only answer being the two confused shaking heads of my COO and CFO. Uh-oh…

Segue. The actual Project Manager of the acquired company (not the one from Bob’s Hair Care IT Consulting Nail and Tire Salon) has moved in and I’ve finally got a victim to victimize with my many, many questions. She already looks harrowed before I begin my interrogation. Are people actually using this? How much do we make per visit? Visits per month? I forget the answers to these, but the end takeaway was: we bring in about $10k per month net right now. I'm no accountant, but I’m fairly confident you can’t pay the expenses of a company plus a half dozen employees on $10k a month. PM agrees: they’ve burnt through about $7 million of investor cash over their 6 years of existence – no path to profitability is in sight.

Around the same time I’ve got the Project X repository (whew, at least they used source control) moved over into my world and have started reviewing the actual source. I’m no Node wizard, but I’m immediately confused as I see both Express and Hapi (two server frameworks, generally considered competition to one another) used in the same project. That’s… odd. Investigation intensifies: it’s a simple CRUD project that takes a form submission from a registered user, saves it in Mongo and slaps it into a queue for delivery to the given doctors email. That’s really it. There’s some back-end admin that allows the doctor to write some notes about their visit. Like a little baby EMR (though I had no idea what an EMR was at that time). Amusingly, it’s got an Angular front-end (1.x, because why not spread salt on my wounds) that hits an Express endpoint that then proxies the call to a Hapi endpoint. For no reason. I can’t find a single comment or piece of documentation explaining why. The icing on the cake? There is in fact authentication used from Angular to Express. The Hapi endpoints, however, are wide open – but surely not from the ELB, right? Certainly it’s just an idiotic architectural decision that isn’t actually exposed to the public? Nope. There’s a rule in the ELB. Sweet Baby Ray’s, someone help me, there is a publicly accessible, completely open API that anyone could discover that gives away patient and doctor information. Huh, I wonder if the US has any sort of regulation on that kind of stuff? I should really take some time to investigate that HIPAA thing I found earlier, maybe that’s got something to do with it…

Employment duration: unknown. My ulcer has had a baby. I think I may have had a psychotic break. I googled HIPAA. I simultaneously shat and pissed myself, which I didn’t think was possible during a panic attack, but the human body is an amazing thing. I took thursday and monday off from work to read through a PDF I found on this most enlightening “HIPAA” legislation. It says “SAMPLE” or “UNOFFICIAL” or some such on it, so I’m not sure how accurate it is, but whatever – I need to educate myself somehow. I spent a thrilling four days reading, re-reading, and summarizing what I understood of the several hundred page document – printed in three-column layout because why not make it more abysmal. It doesn't seem completely dire; it looks like there is some stuff we need to do if we are storing this mythical PHI, but it isn’t terribly complex (at least technically!). I had already been planning on encrypting everything we own, and all of our sites are already behind SSL, so this should be cake. Phew! Calm down, baby-ulcer, don’t think about grand-kids quite yet. Also I found a few great summaries of the Act which I could share with my COO – but really, we need to sit down with Legal and have them explain why this was never brought up. And let’s be honest, I’m not a lawyer – the professionals can handle this!

Legal has never heard of HIPAA. That’s not good. I convince COO to ask Legal to reach out to a different Legal who specializes in healthcare. We sit down with them a few days later and our new Legal turns white after I lay out everything we do, our concerns, and the simple question: “Do we need to do any of this stuff I read about?“ Turns out, having your CTO read a complex, many-hundred-pages legal document is not the best way to get accurate legal advice. We’re fucked. We’re a TPA filing insurance claims – we absolutely 100% must comply with this Act. Oh, and guess what? The Act has a delightful addition called an Omnibus, passed back in ’13, that makes any possible defense we might have had to not comply… completely null and void. We’re in what is called “Breach”! We have fucked up. Royally and legally. The icing? We’re all personally liable, at least to the letter of the law. But don’t worry – we didn’t know we fucked up, so the fees are an order of magnitude less. They’ll only bankrupt the company five times over, instead of ten! Hurray!

I decided to name my grand-ulcer Ralph.

Three years in. Somehow we aren’t in jail and have not been fined out of oblivion. Apparently, as long as no one reports you for being in Breach, nobody knows about it. Our new Legal has informed us the big companies breach all the time and they have entire teams of legal experts that know they’re violating the law and don't care. We shouldn't feel so bad. I feel bad. I'm not sleeping so much, what with the 14 hour days and the impending terror of a bankruptcy-inducing fine that could be dropped at any moment looming over me. Once we got CEO on-board with how utterly fucked we were, we put Project X on hold and went about unclustering ourselves. Everything is encrypted; everything is documented. We've got breach policies and firebins and security cameras and access policies and logs whose logs have logs. If someone sneezes anywhere near PHI, tiny assassin robots get deployed from the ceiling vents to incinerate the documents and murder the perpetrator. I own everyone's phones and laptops; I can remotely detonate them using satellites in outer space at any moment, or at least that's what it feels like. If the CEO even thinks about sharing a document outside the organization 15 different alarms go off. We’re locked down. We’re secure.

I get another raise. I'm now 29, pulling in about 165k. I now own 2% of the company in stock, though I have no idea what that really means. We got our 6% match bumped to 10%. Healthcare is now covered 100%. I’ve got a platinum plan. It still sucks though, because United States. I live in an area where I’m paying $600 a month in rent for a nice place. I hired two developers to unfuck Project X and moved it to Azure and cut our costs to about $300 a month. We didn’t get audited. Nobody reported a Breach. Life is good.

Our CEO decides to use one of our reporting tools to export all of our user data from our TPA platform (about 70k customers) and send it over to his buddy for a mailing campaign. His buddy runs an MLM. A pyramid scheme. He doesn't understand the reporting tool, so he accidentally also sends along any claims that were filed for said customers. It ends up being a many hundred thousand-line CSV file. How did he send it along, you ask? Don't you have all that fancy monitoring and prevention technology, Mr. CTO? Well, even if he had sent it via Gmail, nope I can't really stop it (unless Gmail has some fancy features I haven't found yet). But he didn't, since he knows I review all outbound emails on the regular. He slapped it on a thumb drive, drove over and physically handed it over.

And bless his heart, his MLM CEO buddy is on the ball; they had emails sent out to our 70k customers in less than 24 hours. Before I even knew what had happened. I cornered the CEO with the COO as backup and informed him what he had just done. I had sense enough to print off the fine table from the HHS before hand so he could really understand exactly how that breach would feel, financially, if it came back to kick us in the nuts. Second time in my life I got to see someone’s face lose color! After he understood the gravity of his fuck-up, he asked his MLM friend to destroy the drive. Said he did, but who knows.

Present day. There have been a few more fuck-ups, but I've mostly got us under lock and key. Nothing came of the Breach, except LOTS of angry customers asking why the fuck they got emailed by Shady MLM Company on behalf of Our Actual Company. Thanks for adding that in the campaign, Shady CEO. Nice of you.

If it wasn’t clear, we broke the law a lot. Unintentionally, but there were many instances where we should have said we had a Breach but didn’t. That was all on the advice of New Legal. I’m no lawyer, but I’m pretty sure he's fucking wrong. What do I know, though? I’m just an engineer with a fancy title.

I really hope some sort of divine event unfolds and we have to shut down, because as awful as this company is, it pays well and I just can't see myself abandoning that. With stingy living I'm set to retire at 40 and that blows my mind, though I have yet to factor in the sheer amount of dollars I'm going to lose to the impending mental health counseling I'll need.

I hope someone enjoys this. Took about two hours and three whiskeys to write. I’m sure I cocked up the timeline and details a bit as the ~5 years I’ve been here feel like 50, and also, whiskey. Feel free to hit me with questions, I’ll check back at some point.

[Again the link to the original post: https://news.ycombinator.com/item?id=21141785]