Implement tf_idf_transform (TF-IDF transform)

[rasmushenningsson]

Implement tf_idf_transform (TF-IDF transform)

Summary

Wires the TF-IDF (term frequency-inverse document frequency) transform into the public API.
tf_idf_transform was declared public on this branch but had no binding, so the feature
was unreachable (its siblings logtransform/sctransform are implemented; tf_idf_transform
was left as an advertised-but-missing name).

SCP.tf_idf_transform([T=Float64,] counts; scale_factor=10_000, annotate=false, kwargs...) -> Job

Returns a DataMatrix Job with the transform log(1 + scale_factor * tf * idf), where the term
frequency is tf = counts ./ max.(1, sum(counts; dims=1)) and the inverse document frequency is
idf = nobs ./ max.(1, sum(counts; dims=2)). T controls the eltype of the transformed matrix
(Float32 to lower memory usage).

Design decisions

• Projection — full ID-remapping (like sctransform). The idf is estimated from the base
  data and frozen. Under projection, tf_idf_idf_pr intersects the base variable IDs with the
  projected ones and reindexes idf; a matching var_ind
  (create_find_matching_ind_job(:; project_ids=:intersect)) keeps the matrix rows aligned. So
  projecting onto a dataset whose variables are reordered or a subset works correctly, and the
  idf is reused (never recomputed on the projected data). This matches main's behavior.
• annotate exposed; var_filter omitted. annotate adds the idf as a var annotation,
  defaulting to false to match this branch's sctransform. var_filter/var_filter_cols are
  intentionally not supported, consistent with how ProjectionRefactor simplified logtransform
  (filter before transforming instead).

Implementation

Mirrors the sctransform structure, but much simpler (the frozen parameter is a single idf
vector rather than a params DataFrame, and there is no low-rank term):

• src/SCPCore/transform.jl — tf_idf_idf (the idf vector) and tf_idf_matrix (the
  element-wise kernel, natural log, ported from main's tf_idf_transform_impl).
• src/Impl/transform.jl — the frozen + ID-remapped idf Projectable, the projectable matrix
  transform (per horizontal block via hblock_map), and the DataMatrixFunction dispatcher
  (Mat/Var/Obs); the base nobs is fetched so it is not affected by projection.
• src/transform.jl — public tf_idf_transform + docstring.

tf_idf_transform was already in the public list, so no module change was needed.

Testing

tf-idf testsets added to run_transform_tests (test/transform.jl), validated against the
simple_idf/simple_tf_idf_transform ground truth already present in test/test_setup.jl:

• Eval matches exactly for Float64/Float32 and scale_factor 10 000 / 1 000.
• annotate=true adds a correct idf column; off by default.
• Projection onto the same variables reuses the frozen base idf (verified it differs from
  recomputing idf on the subset).
• Projection onto a scattered variable subset (name > "L", non-contiguous) — idf and matrix
  rows remap correctly by ID.

Full run_transform_tests: 448/448 on both the New Disk Cache and Reused Disk Cache
passes (no regressions to logtransform/sctransform; caching is correct). The package
precompiles cleanly and tf_idf_transform is exposed via public.

🤖 Generated with Claude Code