Feel the Burn!

Bernie Sanders doll, via Wikimedia Commons

People often call Bernie Sanders a (gasp!) socialist!

But then, many people on the “right” call anyone who disagrees with them a socialist, communist, anti-American, traitorous, or a variety of other terms which they (apparently) don’t understand and (often) can’t even coherently define. My suggestion would be to ask anyone who uses a term like “socialist” as a knee-jerk response to define it – if they can, respond appropriately. If they can’t, the best move is probably to end the interaction, as you’re most likely to receive near-automated responses that range from incoherent to incompatible with reality.

In Sanders’ case, however, they’re actually not too far off – at least regarding the name, though it doesn’t mean what they usually think. He is usually described (by himself and others) as a democratic socialist. Theory aside, in practice, this usually means that he works toward such “controversial” things as civil rights, criminal justice reform, and universal healthcare, but more frequently he is just trying to ensure that existing laws and policies are implemented fairly and equitably.

What a... scoundrel?

This, of course, is the point of agencies such as the Securities and Exchange Commission, the Food and Drug Administration, and others. While Sanders would probably love to overhaul the legislation and increase enforcement by these organizations, most of the time he’s just pushing people to let them regulate, when they are often trying to neutralize them instead.

So... he wants things to be better for people and wants things to work they way they were intended?

What a... revolutionary?

But he has an agenda!!

... to make things better for people and make it harder for the very rich to get away with whatever they want, simply because they’re rich?

Well, yes, but...

... and he wants people to be able to work and receive fair compensation for it? And affordable housing? And decent healthcare? The horror!!

An interesting quote:

“We had to struggle with the old enemies of peace—business and financial monopoly, speculation, reckless banking, class antagonism, sectionalism, war profiteering. They had begun to consider the Government of the United States as a mere appendage to their own affairs. We know now that Government by organized money is just as dangerous as Government by organized mob.”

Was that the “radical”, Sanders?

No. Actually, that was FDR, in 1936, in an address he gave in New York City. Interestingly, we are in a time when many of the same things are happening, and some are trying to dismantle many of his most impressive accomplishments, like Social Security. But that’s another story.

At any rate, this brings us to Burn.

Great! Let’s listen to Def Leppard’s 1983 album Pyromania!

Uh, no.

The song by Alan Parsons Project?

Sorry, no. That’s “Pyramania”, and actually refers to pyramids.

The song by Cascada?

No. Um, the AI framework for Rust.

I selected Burn mostly because it is Rust-based, in contrast to most other frameworks, which seem to be Python-focused. Since I’ve been trying to learn more about Rust, this seemed reasonable. While it doesn’t seem to appear on the general “top-ten” lists, Burn is definitely popular among the Rust-based frameworks.

Burn – well, AI frameworks generally – are libraries of tools for building and training AI models for a variety of purposes. The goal is usually to provide high performance, stability, and flexibility, so a project can be built in whatever way is needed.

For anyone interested in learning a bit about how these models actually “learn”, I highly recommend the series on Neural Networks, by 3Blue1Brown. I found it fascinating, and think it’s very good at providing a conceptual understanding, without overwhelming the audience with the math.

In order to build an AI model, the developer has several decisions to make initially, but it’s hard to even understand what they are without diving in. Ooh! POC time!

Fortunately, I’m not building anything for large-scale use or for posterity, so my initial decisions were relatively straightforward. CPU-only, as I don’t have GPUs available, and all I’m trying to do is build a trivial LLM.

Even so, there was a lot of iteration, and a lot of back-and-forth with the AI tools I was using to help. Now, as I have previously noted, I think that using AI to “vibe code” a system is not going to lead you anywhere good, but using it as a teaching tool and helper to learn can be very useful.

So, will AI make a good system? Probably not. But, in this situation, can it help me learn more about Rust and Burn? Mission accomplished!

Well, sorta.

I started out with a model at the “character” level, which essentially tries to predict the next letter and usually ended up with strings of identical characters, as the model “broke down”. Later, I shifted to a “word” level, which is somewhat more likely to generate somewhat-intelligible text.

One of the key dependencies is on the training data set. I started with “Tiny Shakespeare”, which is about 1MB, but then decided that the King James Bible, at 4MB might work better. Then I decided I wanted something a bit more current-sounding, and thought about a number of different data sets, until I realized that H. P. Lovecraft might generate some fun text!

I learned quickly that many AI models “learn” by taking random samples, rather than reading data sequentially. This is to avoid having the model “learn” patterns which are inherent in the way the data is presented, leading to bias in the data due to the order, rather than the inherent nature of the data.

But how big should the samples be? How much training do you need? How much memory is required to train a model?

That last one is easy: “A lot”.

I quickly realized that I need to build in features to monitor memory. Initially, this was to try and figure out why my computer kept crashing due to memory exhaustion, then later to figure out how to write my trainer so it could be restarted, or run more efficiently, or define options which were realistic for my situation.

Bottom line, I’ve learned a lot about Rust, a lot about Burn, and a lot about AI modelling/training.

Does that make me an expert in these things?

<REDACTED>, no!

But it has given me more perspective on both training and using AI models, and it has reinforced my opinions regarding the best use of AI in coding. Excellent for code-reviews and troubleshooting! Very good for accelerating the speed of learning, if used correctly! Very good for things which can be described in clear and unambiguous terms. Very good for finding patterns in data.

Designing innovative solutions for novel problems? Not so much.

Maybe someday, but maybe not – or at least not soon.

AI has already had a profound impact on our society, and that impact will certainly grow, so it’s very important to develop some understanding about what it can and cannot do.

But learn quickly, as the situation will be different tomorrow – That’s how fast it’s changing!

Fun, eh?

Cheers!