Dealing with >4GB of data in SINGLE_FILE build

[413x1nkp]

I'd like to preface this by saying that i'm planning on sticking to wasm32 for the time being.

I have a build system targeting a single HTML file with all of the assets embedded in via --embed-file for maximum portability.

I'm kind of worried that with all the text, audio, and texture data, i will exceed the 4GB limit, so i wanted to look into how to store files without keeping them in memory at all times. OPFS sounds like a good solution, but i'm really not sure if i'd even be able to start up a routine to store embedded files in OPFS and then unlink the embedded files from memory if the total size of embedded files exceeds 4GB?

I know i could just ditch the SINGLE_FILE build and instead use an HTTP server for easier control of when data is loaded, but i'm not really sure if you can even start an HTTP server on android/iphone? I know firefox has some built-in thing into devtools on desktop, but even if it works on android - demanding all of that setup from the end user seems kind of unrealistic.

I'll greatly appreciate any help with this

[sbc100]

The --embed-file settings puts the data directly into the .wasm file, which in your case is then directly binary encoded into the html file.

The embedded files are already in the wasm memory on startup and they are directly accessed from there, so you should see almost zero overhead there. i.e. if you add 10Mb of file data your wasm memory usage should go up by exactly 10Mb.

Whether or not your browser engine can handle 4gb of binary encoded string data is another question.. that sounds like it might reasonable exceed some limit.

[413x1nkp]

I see! So, what are my options here, other than getting rid of -sSINGLE_FILE?

Is there some sort of way to use a custom routine for embed file, so that it loads the embedded files into OPFS on startup one by one?

[sbc100]

I don't know of anything like that in emscripten today no.

I imagine it would be fairly simple to write some code to preload all your files into OPFS before you start the module. Where would they be coming from though?

[413x1nkp]

I don't know of anything like that in emscripten today no.

I imagine it would be fairly simple to write some code to preload all your files into OPFS before you start the module. Where would they be coming from though?

I thought of just unlinking the embedded files once they're in OPFS, but that doesn't really solve the 4GB limit issue on its own.

I don't know, is fetching via URL by doing fetch("file://..."); possible, or is the module completely isolated from the real filesystem?