Safe `if rank==0` execution mode

[manopapad]

In control-replicated execution mode, there are legitimate cases where we want to perform a computation on only one copy of the top-level program, e.g. writing out to a log file, or making a database operation.

A check like if get_rank() == 0: is unfortunately unsafe, because it leaves the door open to doing some operation that ends up launching Legion work (e.g. printing out a cuPyNumeric ndarray triggers an inline mapping), which is a violation of the control replication model.

We could perhaps provide this through something like an execution scope, that raises an exception if an unsafe operation is launched.

[lightsighter]

You can at least detect this with -lg:safe_ctrlrepl today so you won't hang.

[danielfrg]

This is great for the cases you mentioned, i have been bitten by those.

For the interactive apps/apis i am not sure if this is enough because in rank==0 we will also need to also trigger compute in the other nodes right?

Lets say in FastAPI:

import cupynumeric as cpn
from fastapi import FastAPI

[...]

def compute():
    cpn.array()
    [...]

@app.get("/compute")
def api_compute():
    compute()
    return "OK"

if rank == 0:
   start_server()

When a query comes to /compute it will just run in rank==0

[lightsighter]

I feel like if you want to do this, you should be writing Pygion code and not Legate code. Legate should be for scale-free computations. If you want to do non-scale-free things with a tasking system, then Pygion is a better level of abstraction on Legion than Legate is.

[manopapad]

I don't think we need to go to a different programming model for this. The actual computation is scale-free (what happens in compute()), we just need to have a single coordinator that spins up and waits on commands, then executes them across the cluster. I think this pattern just doesn't fit very well with control replicated execution, but it fits fine within the single-controller mode (you could do it under control replication by having the commands broadcasted to top-levels running on each process, but at that point you just have a single-controller mode with extra steps).

[magnatelee]

though Legate today is mostly about scale-free computations, there are mechanisms in Legate that help people do non-scale-free things (e.g., manual tasks, resource scoping). In my view, this use case is well within Legate's scope and we'd very much like to support this. Pygion is a non-starter for Legate and our users, as it's not controlled or maintained by NVIDIA and I don't think there's a feasible way to productize it.

[lightsighter]

I don't think we need to go to a different programming model for this. The actual computation is scale-free (what happens in compute()), we just need to have a single coordinator that spins up and waits on commands, then executes them across the cluster.

Why is it our responsibility to do this? The only place it should be our responsibility for doing this would be in a controlled environment like a Jupyter notebook where we can provide a custom backend that does it. I don't think we can safely handle all the ways that clients might want to create such patterns in normal python. We should provide a way for them to detect it when they mess it up and that's it. Legion's safe control replication check is already sufficient for that.

I think this pattern just doesn't fit very well with control replicated execution, but it fits fine within the single-controller mode (you could do it under control replication by having the commands broadcasted to top-levels running on each process, but at that point you just have a single-controller mode with extra steps).

Or you could just do like I said above and use Legion directly. It already supports this; always has and always will. I don't see why you'd need to do extra work to do something that's already been done.

though Legate today is mostly about scale-free computations, there are mechanisms in Legate that help people do non-scale-free things (e.g., manual tasks, resource scoping). In my view, this use case is well within Legate's scope and we'd very much like to support this.

You already know that I strongly disapprove of anything that's not scale-free in Legate. That is what Legate was born to be. Anything else is feature creep and risks the long term viability of the project. If people want lower-level control they can use Legion. Trying to handle all the different ways that clients might want to do all this rank == 0 stuff is a bottomless pit and will require diverting huge amounts of resources away from the other parts of Legate to make it work (especially when it breaks). The only place where I'm supportive of it is in controlled environments like Jupyter notebooks where the contract between the client and workers has already been established, codified, and is unlikely to change dramatically in the future.