Validity of wide pointer metadata

[RalfJung]

The discussions at #76 and #77 are about the validity invariant that all references have to maintain in general; this issue here is specifically about the validity invariant of the metadata component of any fat pointer -- which do not have to be references! Rc<[u8]> is also a fat pointer. The expected metadata depends on the type of the "unsized tail".

For slices, it seems clear that the invariant will be the same as that for usize. But for dyn Trait pointers, what should we require about the vtable? My guess would have been at least as much as an &[usize; 3] (a vtable has at least 3 slots: size, alignment and drop function, not necessarily in that order). But in this forum thread, the valid question came up why Weak::new does not work for dyn Trait, and indeed it could if we could "fake" the fat pointer component (though doing that generically seems hard).

And then there is the question how this interacts with eventual custom/user-defined DSTs.

[gnzlbg]

But for dyn Trait pointers, what should we require about the vtable?

Is there consensus about the validity invariant of the vtable pointer ? E.g. if the vtable pointer can be null, then there doesn't need to be a vtable at all. If the vtable is a reference, then whether there needs to be a valid vtable at all would depend on whether validity of references is transitive. Or do we want to be more strict about vtables ?

[hanna-kruppe]

Nit: The issue title talks about references, but the text (and Weak::new discussion) about pointers more generally, which is it?

[petertodd]

Food for thought example: creating a Weak<[T]>.

As long as the nightly-only Weak::as_raw(), Weak::into_raw() etc. exist we have to actually initialize the pointer metadata to something; we can't just "initialize" it with the equivalent of MaybeUninit. So we need some kind of Weak::new_unsized(metadata: <T as Pointee>::Metadata). This would also allow Weak::new() to to work with unsized types if the pointer metadata implements Default:

impl<T: ?Sized> Weak<T> {
    pub fn new() -> Weak<T>
        where <T as Pointee>::Metadata: Default
    {
        unimplemented!()
    }
}

(obvious disadvantage: unnecessary complexity confusing beginners in 99% of use-cases)

Equally, it might be reasonable to eventually (with arbitrary self types) change the slice API so that len() takes a pointer:

impl<T> [T] {
    pub const len(self: *const Self) -> usize {
        unimplemented!()
    }
}

...which in this case could actually be accessed via Weak::as_raw:

let w: Weak<[u8]> = Weak::new_unsized(42);
assert_eq!(w.as_raw().len(), 42);

Heck, you could even imagine a DerefPtr type trait. But I suspect the use-cases of it would be so narrow as to be unnecessary complexity.

As for how this would interact with trait objects, the compiler can probably implement Default for the vtable type automatically. And again, it'd be plausible to allow the calling of trait methods that take pointers.

[RalfJung]

Nit: The issue title talks about references, but the text (and Weak::new discussion) about pointers more generally, which is it?

Good point! That actually explains part of my confusion. I updated text and title.

[RalfJung]

I just realized that with the new title, this discussion would now also apply to fat raw pointers. But IIRC we agreed that fat raw pointers do not have to have valid metadata.

So actually maybe this thread should be about references? Weak internally uses NonNull which is a raw pointer, and IIRC there was agreement that raw pointers do not need to have valid metadata.

[RalfJung]

Similar to #72 (comment), I wonder if it would be worth to write up a (small) RFC that says that fat raw pointers (including NonNull) do not assume validity of their metadata, i.e., the validity invariant of their metadata is the same as that of usize? (It is already the case that a thin raw pointer itself has the same validity invariant as usize, as the two can be cast to each other in safe code.)

[petertodd]

@RalfJung So like usize they're assumed to not be undef bits, but beyond that there are no further assumptions?

Do we assume the metadata for all wide pointers is always safely castable to usize? Eg could there be a future wide pointer type where the metadata has a subset of valid bit representations, like, say, bool?

[gnzlbg]

Eg could there be a future wide pointer type where the metadata has a subset of valid bit representations, like, say, bool?

Do you have a concrete application in mind?

[RalfJung]

So like usize they're assumed to not be undef bits, but beyond that there are no further assumptions?

Exactly. And if we decide we are fine with uninitialized integers, then so would be wide raw pointer metadata.
This also matches *const T/*mut T, which have the exact same validity invariant as usize -- after all, they can be safely converted back and forth.

Do we assume the metadata for all wide pointers is always safely castable to usize? Eg could there be a future wide pointer type where the metadata has a subset of valid bit representations, like, say, bool?

I guess we should clarify that everything we are saying right now is only for wide pointer types that already exist (slices and trait objects). I do not want to unnecessarily constrain custom DST designs.

[petertodd]

Eg could there be a future wide pointer type where the metadata has a subset of valid bit representations, like, say, bool?

Do you have a concrete application in mind?

Having a hard time coming up with one, other than trait objects where it's plausible you'd want to call a method on the trait for a potentially invalid pointer. Basically, that'd allow a call like this to work: fn foo(self: *const dyn Foo)