Show4DSTEM - Python widget for real-time 4D-STEM explorer and virtual detectors

[bobleesj]

What this PR does

A combination of react, anywidget, WebGPU to get real-time 4D-STEM explorer.

For developers:

Install Method 1. quantem.widget using conda

conda create -n qt-widget-env python=3.12 nodejs
conda activate qt-widget-env
pip install -e .
cd widget && npm install && npm run build
pip install -e .

Install Method 2. quantem.widget using uv with node already installed locally

uv sync --all-packages
cd widget && npm install
npm run build

API design:

from quantem.widget import Show4DSTEM
dataset = read_4dstem(
    '/home/bobleesj/datalamella_2_002_master.h5',
    file_type='arina'
)
widget = Show4DSTEM(dataset)
widget

Notebooks:

test_show4dstem_arina.ipynb

test_show4dstem_legacy_h5.ipynb

[bobleesj]

@dasol-yoon has agreed to test the API/features. Keeping it as a draft PR since folder clean-up is needed but it's a good starting point since the very basic features are done - virtual detectors, zoom, pan, color map, reset, zip, length, laster animation

[cophus]

Excellent work! I have several comments:
-I think the FFT panel it should be hidden by default (or a separate widget)
-rectangular scans don't work correctly
-we should add automatic center spot detection for the virtual detectors
-controls next to the appropriate panels, separate colorbars, separate reset buttons (and reset should just do the FOV / detectors, maybe not colormaps etc. too)
-need vmin and vmax color controls.
-in light mode, many of the cells "go dark" when activating the widget:

I'll make another comment with my suggested layout.

[cophus]

Suggested layout:

-Both sides have independent color controls, for cmap, vmin, vmax (and maybe linear/log/power scaling).
-Reset buttons reset view only.
-export button on both panels
-FFT panel either not shown by default or part of another widget
-Combine LAADF + HAADF into single ADF category (maybe a full ADF by default)
-auto-magic detector of BF center disk radius for virtual detectors

[bobleesj]

@cophus Thanks and superb feedback from the very first alpha tester!

Generous enough to even show mock designs as well.

Will do:

• FFT panel hidden by default (a new widget possible later)
• Fix rectangular scans
• Automatic center spot detection
• Separate reset buttons (and reset should just do the FOV / detectors, maybe not colormaps etc. too)
• vmin and vmax color controls.
• Check light/dark mode (configurfe)

Another layer of comments

• Both sides have independent color controls, for cmap, vmin, vmax (and maybe linear/log/power scaling).
• Combine LAADF + HAADF into single ADF category (maybe a full ADF by default)
• Suggested design layout

[bobleesj]

@cophus All done.

@dasol-yoon I will upload v2 tmr or the day after.

[bobleesj]

v2 updated

[gvarnavi]

Hey @bobleesj -- nicely done, very impressive!

Had a quick look on a large-ish Arina dataset (256,256,192,192) and here are my quick thoughts (video-real time):

compressed_demo.mov

• It takes a considerable amount of time to load.. I suspect this is a data transfer bottleneck? Are you using binary buffers and traitlets.Bytes?
• There should be an option for [hot-pixel filtering](def filter_hot_pixels() (very common for Arina datasets)
• I think the DP scale default should be log, or perhaps using quantile-clipping before the histogram otherwise the range is a hard to use
• Circular virtual imaging is also very slow..
• It's not obvious to me what the reset buttons ought to do, but I wasn't expecting it to atleast reset the position, possibly even the virtual detector type
• I think the FFT panel would be better added as a third column?
  • There also seems to be a bug with the boundaries? Perhaps apply a hanning window before the first?

[bobleesj]

@gvarnavi Thanks George for testing! 🙏

Ah, I should have checked with unbinned Arina data. Will fix:

george-v1 after ophus-v1

• Fix loading speed
• hot-pixel filtering
• Circular virtual imaging is slow
• FFT on third column (agree, since now the size layout is 1 to 1)
• FFT scale default log (I will see if I can do clipping thing)
• FFT check boundaries - hanning window

Regarding the reset button, for now, based on the following feedback (we can discuss of course)

Reset buttons reset view only. @cophus

[bobleesj]

@gvarnavi

I think it's working mostly with full ~9 Gb dataset. Will discuss more today's dev call

[gvarnavi]

@bobleesj this looks much better indeed! Will try and check on the gold dataset before the dev call

[bobleesj]

@gvarnavi indeed, much better with feedback. -

Testing on my macbook now - it can't handle 9 Gb... but for smaller datasets (48x48x192x192(, VI is real-time

[gvarnavi]

Hmm.. hot_pixel_filter seems to be wrong on the dataset I'm testing -- I get an empty dataset on load

I tried switching it off as follows but that didn't work either

import traitlets

class MyWidget(widget.Show4DSTEM):
    # Overrides the default value to False while keeping metadata
    hot_pixel_filter = traitlets.Bool(False).tag(sync=True)

What's the best way to override these defaults? Perhaps add them as kwargs to the Show4DSTEM class?

[bobleesj]

I will report back - loading full arina dataset to macbook/laptop

for overriding defaults - I will make it easy to toggle on and off sth like this:

# # Manual calibration (overrides dataset values)
# widget_config = Show4DSTEM(
#     dataset.array,
#     pixel_size=2.39,      # Å (real-space)
#     k_pixel_size=0.46,    # mrad (k-space)
#     center=(96, 96),
#     bf_radius=24,
#     log_scale=True,
# )
# widget_config

I think it's eaiser than subclassing a widget since i haven't done subclassing yet

[bobleesj]

@gvarnavi works on your dataset. I will test a bit more.

Tested across ~4 data types. I will wait for @dasol-yoon's feedback on her dataset type.

[CSSFrancis]

@bobleesj I think that looks great! But I want to maybe poke at it a little bit to make sure this is the right approach

1. I think that it might be worthwhile to consider if we need to use typescript and compile to javascript. It's another layer/abstraction that makes another barrier for contributions. There is at least a discussion to be had about if we can do everything here with vanilla javascript. Typescript is great but it's probably overkill unless we want to create a full library.
2. I don't think that react is necessary. React is a nice way to manage different documents within the widget, but it's also something that we can do ourselves. Most of what is happening is rendering to the two canvases. Other things like the picker and the rois are just overlaid canvases so it's fairly easy to target exactly what needs to be updated.
3. I'm not sure I'm even really confident in this. But maybe we should consider a modular approach.... The way I see it we have a 1D and a 2D viewer (and a 3D viewer). Each can have static markers or dynamic widgets. Below is just a simple example where I have a seperate 1D and a 2D viewer linked via some widget. (I know, I know very hyperspy-esque) but you can add markers to the plot and label them as well. Ideally, we could also make them a single widget, or have a generic template for the 1/2D canvases to really standardize things.

Screen.Recording.2026-03-04.at.11.27.50.AM.mov

The main thing is that right now the barrier for making a new widget is probably too high and I don't think that it has to be. The _esm definition for anywidget is pretty easy and approachable (javascript on the other hand is a completely insane language). The good thing is that there are literally billions of examples of javascript out there so LLM's are fantastic at writing it, anything that takes out a compliation step should be considered.

Anyways... I'm really not sure. That being said it would be nice to have consistent widgets, plotting across different packages, orix is a big one that needs better visualization tools. Maybe abtem would benefit as well. Additionally it would be nice to improve Hyperspy's visualization as well.

[bobleesj]

@CSSFrancis As we spoke on the phone for a bit. We can revisit the points you've mentioned. Before I only had two constraints in terms of widget infrastructure: 1) make it universally fast and so it solves sicentific problems (denoise, drfit correciton, etc.) 2) do not affect quantem as carefully designed with @gvarnavi.

Now, you are addressing the third point of 3) How to make it scalable so other serious widget devs. I hope you make some powerful real-time widgets like Show4DSTEM where anywidget is truly needed and where human interactivity is highly appreciated (not just sliders, plotting, etc) Let's revisit this. I will keep this PR open and perhaps love to hear your thought in the following dev meeting.

[bobleesj]

(in other words, please feel free to try what works best for you in designing widgets that require lowest latency possible and human interactivity, as if we are using a smartphone)