[OpenVINOQuantizer] Tests for the OpenVINOQuantizer

[daniil-lyakhov]

Summary

Tests were borrowed and adopted from the directory https://github.com/pytorch/executorch/tree/main/backends/xnnpack/test/quantizer

TODO

• Fix [Torch FX] Map Namespace Names to Metatype openvinotoolkit/nncf#3237
• Unskip the skipped dynamic shape / embedding models tests and update the version of the NNCF in the requirements

CC: @cavusmustafa

[cavusmustafa]

Is this command below the right way to execute the tests?
python test_runner.py --build_folder ../../../cmake-openvino-out/ --test_type quantizer
If this is the right way to execute, we will need to add quantizer option into --test_type argument in test_runner.py file.

But tests were failing with the error below on my side if I execute this way.

ERROR: test_static_linear (test_representation.TestPT2ERepresentation.test_static_linear)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/intel/Workspace/env/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3120, in wrapper
    method(*args, **kwargs)
  File "/home/intel/Workspace/et_01/executorch/backends/openvino/tests/quantizer/test_representation.py", line 99, in test_static_linear
    self._test_representation(
  File "/home/intel/Workspace/et_01/executorch/backends/openvino/tests/quantizer/test_representation.py", line 46, in _test_representation
    model = prepare_pt2e(model, quantizer)  # pyre-ignore[6]
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/intel/Workspace/env/lib/python3.12/site-packages/torch/ao/quantization/quantize_pt2e.py", line 96, in prepare_pt2e
    model = quantizer.transform_for_annotation(model)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/intel/Workspace/env/lib/python3.12/site-packages/executorch/backends/openvino/quantizer/quantizer.py", line 346, in transform_for_annotation
    nncf_fx.transformations.fold_constant_except_qdq(model)
    ^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'nncf.experimental.torch.fx' has no attribute 'transformations'

To execute this test, run the following from the base repo dir:
    python test_representation.py TestPT2ERepresentation.test_static_linear

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

I saw this error on two different systems. Am I running tests correctly?

[daniil-lyakhov]

I tried to run test locally with updated nncf version, run without an error:

python test_runner.py --build_folder ../../../cmake-openvino-out/ --test_type quantizer
/home/dlyakhov/Projects/executorch_ov/executorch/openvino/executorch_install/python/openvino/runtime/__init__.py:10: DeprecationWarning: The `openvino.runtime` module is deprecated and will be removed in the 2026.0 release. Please replace `openvino.runtime` with `openvino`.
  warnings.warn(
WARNING:nncf:NNCF provides best results with torch==2.6.*, while current torch version is 2.7.0.dev20250131+cpu. If you encounter issues, consider switching to torch==2.6.*
./home/dlyakhov/Projects/executorch_ov/executorch/env/lib/python3.10/site-packages/torch/export/_unlift.py:81: UserWarning: Attempted to insert a get_attr Node with no underlying reference in the owning GraphModule! Call GraphModule.add_submodule to add the necessary submodule, GraphModule.add_parameter to add the necessary Parameter, or nn.Module.register_buffer to add the necessary buffer
  getattr_node = gm.graph.get_attr(lifted_node)
/home/dlyakhov/Projects/executorch_ov/executorch/env/lib/python3.10/site-packages/torch/fx/graph.py:1790: UserWarning: Node t target t t of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
  warnings.warn(
........sINFO:nncf:2 ignored nodes were found by types in the NNCFGraph
INFO:nncf:Not adding activation input quantizer for operation: 8 linear
INFO:nncf:Not adding activation input quantizer for operation: 9 linear_1
INFO:nncf:2 ignored nodes were found by names in the NNCFGraph
INFO:nncf:Not adding activation input quantizer for operation: 8 linear
INFO:nncf:Not adding activation input quantizer for operation: 9 linear_1
INFO:nncf:5 ignored nodes were found by patterns in the NNCFGraph
INFO:nncf:Not adding activation input quantizer for operation: 8 linear
INFO:nncf:Not adding activation input quantizer for operation: 9 linear_1
..s....../home/dlyakhov/Projects/executorch_ov/executorch/env/lib/python3.10/site-packages/torch/ao/quantization/observer.py:1333: UserWarning: must run observer before calling calculate_qparams.                                    Returning default scale and zero point 
  warnings.warn(
.../home/dlyakhov/Projects/executorch_ov/executorch/env/lib/python3.10/site-packages/torch/fx/graph.py:1188: UserWarning: erase_node(batch_norm) on an already erased node
  warnings.warn(f"erase_node({to_erase}) on an already erased node")
/home/dlyakhov/Projects/executorch_ov/executorch/env/lib/python3.10/site-packages/torch/ao/quantization/utils.py:400: UserWarning: must run observer before calling calculate_qparams. Returning default values.
  warnings.warn(
/home/dlyakhov/Projects/executorch_ov/executorch/env/lib/python3.10/site-packages/torch/ao/quantization/observer.py:1333: UserWarning: must run observer before calling calculate_qparams.                                    Returning default scale and zero point 
  warnings.warn(
.s/home/dlyakhov/Projects/executorch_ov/executorch/env/lib/python3.10/site-packages/torch/export/_unlift.py:81: UserWarning: Attempted to insert a get_attr Node with no underlying reference in the owning GraphModule! Call GraphModule.add_submodule to add the necessary submodule, GraphModule.add_parameter to add the necessary Parameter, or nn.Module.register_buffer to add the necessary buffer
  getattr_node = gm.graph.get_attr(lifted_node)
/home/dlyakhov/Projects/executorch_ov/executorch/env/lib/python3.10/site-packages/torch/fx/graph.py:1790: UserWarning: Node weight target weight weight of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
  warnings.warn(
..../home/dlyakhov/Projects/executorch_ov/executorch/env/lib/python3.10/site-packages/torch/fx/graph.py:1188: UserWarning: erase_node(batch_norm) on an already erased node
  warnings.warn(f"erase_node({to_erase}) on an already erased node")
./home/dlyakhov/Projects/executorch_ov/executorch/env/lib/python3.10/site-packages/torch/fx/graph.py:1188: UserWarning: erase_node(batch_norm) on an already erased node
  warnings.warn(f"erase_node({to_erase}) on an already erased node")
/home/dlyakhov/Projects/executorch_ov/executorch/env/lib/python3.10/site-packages/torch/ao/quantization/utils.py:400: UserWarning: must run observer before calling calculate_qparams. Returning default values.
  warnings.warn(
/home/dlyakhov/Projects/executorch_ov/executorch/env/lib/python3.10/site-packages/torch/ao/quantization/observer.py:1333: UserWarning: must run observer before calling calculate_qparams.                                    Returning default scale and zero point 
  warnings.warn(
......
----------------------------------------------------------------------
Ran 35 tests in 21.387s

OK (skipped=3)
OpenVINO backend tests completed successfully

requirements.txt are updated