AI and Experience

Artificial Intelligence, via Wikimedia Commons

“People grow through experiences, if they meet life honestly and courageously. That is how character is built and young people recognize this ability to grow in those with whom they come in contact.”

Eleanor Roosevelt, My Day, 2-Aug-1941

Nowadays, most development projects follow an Agile methodology – or at least claim to, but that’s perhaps a topic for another time.

In the past, most developers were provided with specifications for which they needed to provide estimates. I once had a colleague who provided extraordinarily accurate estimates very quickly, and I assumed he was very good at breaking down a development project into components and then estimating the time required.

In fact, he followed an extremely simple heuristic. If the spec was only a few pages, he would mercilessly poke at the holes in it, then wait for the embarrassed analyst to re-write the now-shredded document.

In other cases, he simply counted the pages of the spec. One page = one day.

This is crazy.

But it worked astonishingly well, because a well-written spec would include an overview of the system, use-cases, and expected results. The more complicated the project, the more use-cases and details provided, so more pages and more work.

This technique was based on experience, of the type that is unappreciated by people until they have been “burned” a few times. The sort of experience that teaches you to recognize risky projects, or politically-charged situations, and also helps you understand that there is almost always an explanation for why a system was built (or evolved) in a particular way.

Which brings me to the US Social Security Administration (SSA). Formed in 1935, the SSA is an independent agency of the US federal government that has processed payments measuring in the tens of trillions of dollars (USD) over many decades. It’s also, as would obviously be expected, closely monitored by Congress, the Social Security Advisory Board, and the Office of the Inspector General (OIG).

So, even if Donald Trump and Elon Musk were generally honest, we should not simply accept outlandish claims without compelling evidence. When Trump claims that “we have millions and millions of people over 100 years old”, we should wonder. And when Musk posts things like

“Maybe Twilight is real and there are a lot of vampires collecting Social Security”,

or,

“Having tens of millions of people marked in Social Security as “ALIVE” when they are definitely dead is a HUGE problem. Obviously. Some of these people would have been alive before America existed as a country. Think about that for a second …”,

...we might wonder why something like that would not have been previously identified, if true.

And, even if you accept the (unfounded) assertion that trillions of dollars are lost to “waste” and “fraud”, and accept Musk’s promise that DOGE would identify $2 trillion (USD) in “savings” (which he later changed to $1 trillion (USD), then later to $160 billion (USD), and we are now finding that DOGE may have actually cost more than the claimed savings...), the idea that an issue of this type and magnitude was not already known is (at best) extremely implausible.

It’s the sort of thing one would expect to see in a document like the 2024 OIG Report on “Preventing, Detecting, and Recovering Improper Payments”, which describes the magnitude of improper payments – mostly erroneous overpayments to living people. It’s also something that the SSA and the US Treasury have been actively addressing for years.

Weird!

You might ask why Trump and Musk are saying things like this, particularly since Musk seems to think he is an expert on all things technology. In my opinion, anyone who truly understands technology, or data, or large organizations, or pretty much anything, would be asking the experts and trying to understand what is going on. This is probably difficult, though, when many have been fired, and others resigned en masse. And I can only assume that reading the many historical reports on these topics is unreasonable, for some reason...

Based on my own experience, my initial thoughts were to wonder how COBOL managed dates, how SSA managed dates, how SSA determined whether or not a person was alive, and what processes were in place that might also address these cases (if they existed). Apparently, Musk did not ask any of those questions, or asked the wrong people, and somehow concluded that the answer was fraud and abuse on a vast scale.

Shockingly (well, not really), there was a 2015 OIC Report on “Numberholders Age 112 or Older Who Did Not Have a Death Entry on the Numident”, which indicated that there were about 6.5 million records with no date of death recorded. However, about 6.4 million of these were issued to process benefit claims prior to March 1972. Others had dates of death listed elsewhere in SSA but not in the “master” file, or were affected by other data issues. There were a handful of records which were suspect and might indicate fraud, which were forwarded for investigation. In addition, a change was implemented in 2015 which automatically stops payments to people older than 115 years.

So, a few raindrops, rather than the vast storm claimed by Musk. A bit embarrassing, not to have checked beforehand, huh?

With this in mind, DOGE is now planning to “rebuild” the SSA system within a “tight timeframe of a few months”.

What could possibly go wrong? (I’ll focus on the technical side, and set aside the whole vast question of whether any of this is legal in the first place...)

In 2017, SSA published a document describing a strategy to migrate away from their legacy systems to a modern platform. This plan describes an investment of $677 million (USD) over a five-year period – which seems aggressive, but not unattainable. With an estimated 60 million lines of COBOL code, and millions more lines of Assembler, along with other legacy languages, this is not a small project.

Now, if DOGE were a large team, with extensive experience in migrating large systems, or managing large-scale projects, or addressing the legal and political issues around such migrations, or even extensive experience with SSA itself...

The timeframe would still be ridiculous. The work to create the strategy document noted above probably took “a few months”...

But what if we use AI?

Yeah. Sure. What kind of AI?

Which particular AI engine? How would it be trained? How would it be tested? What makes you think it would do what is needed?

Thinking you could migrate a system like SSA to a “modern” language in “a few months”, by using some undefined “AI” is so far beyond unreasonable that I find it hard to imagine an explanation other than pure lying. This isn’t inexperience – in my opinion, this is either deliberate falsehood, or overconfidence of such epic proportions that it’s impossible (for me, at least) to imagine.

But wait. Could AI help?

Yes. Absolutely.

It should be possible to create an AI system (or most likely multiple systems) to do a great deal of the work required for a migration. Interestingly, SSA is sufficiently large that it might actually be possible to use their datasets to effectively train these models.

One system could be trained on the existing SSA data structures, and be used to design the new data model, help rationalize/scrub the data, define data migration strategies, and possibly even write draft migration scripts/procedures. An enormous amount of human effort would would still be required, but AI could potentially speed things quite significantly.

Another system could be trained on the existing SSA COBOL code-base, and be used to generate specifications and templates for migration. When I looked into COBOL previously, IBM had a product which could potentially be used as a starting-point here.

And then there’s testing. By training an AI on the huge volume of past transactions and outcomes, it might be possible to create a system which could test the new system at such high volume that it could dramatically reduce the time required for human testing, while increasing the overall reliability of the testing.

That’s just three. Did I mention that – so far as I know - none of these currently exist, and each one could easily require multi-year projects to build?

Let us, for the sake of argument, assume that these systems exist and are ready to use right now. Could the DOGE team (consisting primarily of a half-dozen people in their early 20’s), complete the migration of a system like SSA in a few months?

No. Not a chance. Just the rollout would take months, even if everything else were done and perfect.

So, what’s the point?

That’s the interesting question. Unless the goal is to impair the operation of the SSA for some reason, the only option that seems plausible to me is that they are simply trying to gather as much data as they can for Musk and/or Trump to use as they see fit. I find it hard to imagine a scenario under which that is not a very bad thing...

The important thing, I think, is to recognize that the issue here is not with SSA, or COBOL, or the mythical wide-spread “fraud” and “waste”. The issue is that Musk and Trump are trying to grab data, bypass the checks and balances that keep things working effectively, and break the systems that support the operations of the US government.

We need to call out these illegal acts, and hold people accountable for them. We need to educate ourselves on what is happening, and we need to step up and say that this is NOT ok.

Cheers!