This actually is niche optimization. The outer enum is using the niches available in the tag of the inner enum for its own discriminant.
The author seems to have a limited understanding of niche optimization.
I don't think it's actually "flattening" the enums discriminant spaces (though maybe it is). My guess is this is just 32-bit alignment requirement (ie. bytes 1:4 are padding) + sub-byte discriminant sizes. The surprising thing here is that the ordering of Outer's variants doesn't match the ordering as defined, instead having variant D's discriminant be 0 (0 << size_of::<Inner's discriminant>). size_of::<Inner> is actually 33 bits and size_of::<Outer> is 34 bits and then you just apply alignment requirements to each field. Actual size_of calls will account for alignment and padding, but the compiler knows the truth.
What's cool about this in general is nested match statements can be flattened into a jumplist (idk if rustc does this, but it's possible in theory).
Yeah, there are all sorts of nifty little tricks about compressing the enums, Appel dedicates a huge chunk of section "4.1. Data representation" on it in his "Compiling with Continuations" book about the internals of the SML/NJ compiler for the Standard ML, and ultimately concludes that the returns are quickly getting really diminishing with those, and since datatypes can actually be abstract in ML (in pretty much the same way they can be in Rust), the applicability of those tricks is generally restricted to unexported, intramodular datatypes.
This seems like it could be summarized as tag merging. When the member of an enum is another enum, then the two can be effectively merged, rather than tested, so that there is one discriminant tag. The tag values have to be displaced/renumbered so that their ranges do not overlap.
When was this implemented? I remember when people used to have to manually flatten nested enums in order to save space
I have actually requested an even more intelligent/aggressive size optimization for nested enums, where values are automatically reassigned to not coincide/conflict to enable tag-less differentiation. Consensus was that it’s a big ask though doable, but maybe possible to implement in a more restrictive version.
https://rust-lang.zulipchat.com/#narrow/channel/131828-t-com...
That's a great article. Niche optimization might seem minor, but it actually results in substantial memory usage savings in real-world Rust applications, because idiomatic Rust uses enums everywhere.
Does the niche optimization require the compiler to know things about the type? So that only certain specializations will work?
Also, how do I get some code to do the memory layout vizualizer, perhaps one that is a bit more advanced and knows what pointers are?
This is a great way to see why invalid UTF-8 strings and unicode chars cause undefined behaviour in Rust. `char` is a special integer type, known to have a valid range which is a sub-range of its storage type. Outside of dataless enums, this is the only datatype with this behaviour (EDIT: I neglected NonZero<...>/NonZeroXXX and some other zero-niche types).
If you manage to construct an invalid char from an invalid string or any other way, you can defeat the niche optimization code and accidentally create yourself an unsound transmute, which is game over for soundness.