The Best C-Food!

Antarctic Krill, via Wikimedia Commons

The concept of biomass is both interesting and useful when thinking about life on Earth. In ecological terms, it is simply the total mass of the living biological organisms in a given area at a given time. That is useful for a number of reasons, but where it gets interesting is in comparing the proportions of different types of biomass, which puts things into perspective in a way that we don’t generally think about.

I found a fascinating graphical breakdown of the total biomass of Earth. It probably won’t surprise most people that over 82% of the Earth’s biomass consists of plants, but it might surprise some that animals represent only a tiny fraction of the remainder. Bacteria is the next largest group, representing about 12.8% of the total, followed by fungi, archaea, and protists, until you reach animals at a tiny 0.47%. Only viruses represent a smaller proportion of Earth’s biomass.

Zooming into the animals, you might think that domesticated animals would be pretty high on the list, but not so. As it happens, marine arthropods (eg, crabs, shrimp) represent 38.6% of the total animal biomass, followed by fish at 27%. Livestock represents a mere 3.86%, followed by humans at 2.3%. Interestingly, wild mammals and birds represent tiny percentages (0.27% and 0.08% respectively).

Looking at the human consumption of seafood (by export value), shrimp and prawns are more popular (16%) than any single fish category, except salmons, trouts, smelts (18%), even though we consume almost three times as much fish as crustaceans (67% vs 23%).

Part of the explanation for this is krill. These tiny crustaceans are usually about 1-2 cm long, though some grow as large as 6-15 cm. They are both food, and what food eats. These fascinating creatures are near the bottom of the food chain, and (depending on the species) feed on phytoplankton, algae, and sometimes zooplankton and fish larvae. They live in almost inconceivably vast numbers, and are a major food source for creatures ranging from fish and penguins, up to seals and baleen whales. To put their quantity into perspective, a single species (antarctic krill) represents more biomass than almost any other animal species.

Though not a significant part of the human diet, these crustaceans are probably the most popular seafood, and play a critical role in the lower part of the food chain – aside from such heroes such as Will the Krill, who strive to climb their way up the food chain...

Which brings me to Rust.

Huh? How the <REDACTED> did you get there?!

Well, Rust was designed as a general-purpose programming language, emphasizing performance, multi-tasking, and memory-safety.

Ok...

...and some Rust users refer to themselves as “Rustaceans”.

<GROAN>

...which made me think of seafood...

Oh, no...

...and, since Rust took a lot of concepts from the C programming language...

<EYEROLL>

At any rate, the first programming language I used in the “real world” was C (technically Oracle’s Pro*C). I have always loved the flexibility of the language, and the level of control which it allows. The language is beautiful and elegant, but its greatest strength is also its greatest weakness.

Simply put, the C compiler assumes that you – the developer - know exactly what you want to do, and exactly how to do it. This gives it enormous flexibility and power, but also means that it is notoriously unforgiving, and is not “memory safe”.

As an over-simplified illustration (pedants can direct their criticism to /dev/null), say you create two strings: A (6 characters) = ‘HELLO_’ and B (5 characters) = ‘WORLD’. The C program will store these as A = {‘H’, ’E’, ‘L’, ‘L’, ‘O’, ‘_’, ‘\0’} and B = {‘W’, ‘O’, ‘R’, ‘L’, ‘D’, ‘\0’}, probably one after the other, consisting of the characters and the ‘\0’ representing the end-of-string.

If you print A and B, you will get “HELLO_WORLD”.

Great!

Then, say you change A to be “HI_”. You will probably end up with A = {‘H’, ‘I’, ‘\0’, ‘L’, ‘O’, ‘_’, ‘\0’ }, as the C program will update the new characters and add the end-of-string, while ignoring the rest of the original string.

If you print A and B, you will now get “HI_WORLD”.

No problem.

Then, say you change A to “HI_TO_ALL_THE_”. You’ll probably end up with A = {‘H’, ’I’, ‘_’, ‘T’, ‘O’, ‘_’, ‘A’, ‘L’, ‘L’, ‘_’, ‘T’, ‘H’, ‘E’, ‘_’, ‘\0’}.

Then you try to print A and B, and get “HI_TO_ALL_THE_LL_THE_”, and then the program crashes.

Wait! No! You should get “HI_TO_ALL_THE_WORLD” – A and B, right?

Nope. When we updated A, we ended up overwriting B, because we “overshot” the space we expected for A.

But, there should have been an error when we tried to write the longer string, if it overwrote something else...

Nope.

But, if there was no error, the program should just have done what we wanted, right?

Nope.

The C compiler assumes you know exactly what you are doing, so it will happily write past the end of string A, right through string B, and into other memory which may or may not be used by something else. The string A might be fine, but that depends on whether the memory following A and B was being used. And, since B was overwritten, it will now reflect the last fragment of what was stored in A. The real problem, though, is that whatever was stored after the end of B has also been overwritten. If this memory is never used, the program could work “correctly”, but there’s basically a time-bomb in your code which might or might not cause some catastrophic issue.

Um... shouldn’t the compiler stop us from doing that? Or the programming environment? Or something?

There are tools that can catch some things like this, and ways that C (or it’s descendants) can be made “memory-safer” (or at least less memory-unsafe), but that would require a lot of changes to compilers and tools, and would not address issues in existing code – and there is a LOT of existing code.

Also, while there are projects out there to add memory-safety to C or C++, many C programmers don’t like things that interfere with their ability to do whatever they want, in any way that they want. (Coders can be a curmudgeonly bunch... ☺)

If only there was another way!

Rust is another way. It was originally created in 2006 by Graydon Hoare at Mozilla Research, and was intended as a general-purpose language. I’m starting to think of it as “C: The Next Generation”, as it deliberately prioritized good ideas from old languages, rather than “reinventing the wheel”.

While enormously powerful, Rust has a steep learning-curve and requires thought and discipline to use effectively. As with C, the greatest strength is also, arguably, the greatest “weakness”.

For example, Rust is strongly-typed, which means that it is necessary to be clear and explicit about variables and error handling. The example above would not “leak”. Depending on the details of the logic, it would fail to compile, “panic” and fail at runtime, or generate an error which can be handled. What it will NOT do, is give unpredictable results due to memory being overwritten.

Many languages “guess” at variable types, and automatically convert to the “appropriate” variable type, where “needed”, but this can lead to weird situations. As an example, in JavaScript, you can encounter things like ‘2’ + ‘2’ – ‘2’ = 20. JavaScript uses the “+” operator to concatenate ‘2’ + ‘2’ to ‘22’. But then, the “-” is not defined for strings, so JavaScript converts both “22” and “2” to integers and subtracts, resulting in 20.

Rust simply doesn’t permit this type of ambiguity. If you define a variable as a string, it’s a string, and will always be treated as such, unless you explicitly define otherwise. Trying to subtract from a string will generate a compile error, as will adding an integer to a string.

While this makes Rust more difficult for a beginner and creates a steeper learning-curve, it also eliminates whole classes of issues which have an enormous impact on both performance and security. Some estimates suggest that 70% of all security vulnerabilities at Microsoft and Google are caused by the type of memory issue Rust eliminates or dramatically reduces.

This is not to suggest that Rust is a silver bullet which will resolve all issues, but it does provide a tool which can improve the situation significantly, while being flexible and powerful enough for almost any application.

And, while I really love C, it may be time to let it ride into the sunset and let Rust be one of the major languages of the 21^st century.

Cheers!