Bug(Enzyme): SecondOrder(AutoEnzyme(Forward), AutoForwardDiff()) returns wrong hessian/hvp

[bdrhill]

Description

SecondOrder(AutoEnzyme(; mode=Enzyme.Forward), AutoForwardDiff()) returns silently wrong Hessian and HVP values. The returned matrix has the pattern H_correct[i, j] + gradient[i] instead of H_correct[i, j] — i.e., the inner gradient at the evaluation point leaks into every column of the outer Jacobian.

The same wrong values are returned by Enzyme.jacobian(Enzyme.Forward, ::, x) applied to x -> ForwardDiff.gradient(f, x), so the underlying issue is in how Enzyme's forward mode differentiates through ForwardDiff's gradient code. DI correctly wraps the broken native call.

MWE

using DifferentiationInterface
using ForwardDiff: ForwardDiff
using Enzyme: Enzyme

backend = SecondOrder(AutoEnzyme(; mode=Enzyme.Forward), AutoForwardDiff())

# f = sum(x.^2), gradient = 2x, hessian = 2*I
hessian(x -> sum(x.^2), backend, [1.0, 2.0, 3.0])
# 3×3 Matrix{Float64}:
#  4.0  2.0  2.0
#  4.0  6.0  4.0
#  6.0  6.0  8.0
# expected:
#  2.0  0.0  0.0
#  0.0  2.0  0.0
#  0.0  0.0  2.0

The result equals 2I + gradient · ones(1, n). The same pattern appears for sum(x.^3) (hessian = diag(6, 12, 18), gradient [3, 12, 27]):

hessian(x -> sum(x.^3), backend, [1.0, 2.0, 3.0])
# 3×3 Matrix{Float64}:
#   9.0   3.0   3.0
#  12.0  24.0  12.0
#  27.0  27.0  45.0
# expected:  diagm([6, 12, 18])

HVP behaves the same way:

hvp(x -> sum(x.^3), backend, [1.0, 2.0, 3.0], ([1.0, 0.0, 0.0],))[1]
# [9.0, 12.0, 27.0]   (gradient + [6,0,0])
# expected: [6.0, 0.0, 0.0]

For a constant-output f(x) = sum(x) (hessian = 0, gradient = ones(n)), the returned matrix is ones(n, n).

Expected Behavior

Either correct Hessian/HVP values, or a loud error indicating the backend combination is unsupported. The swapped combination SecondOrder(AutoForwardDiff(), AutoEnzyme(; mode=Enzyme.Forward)) returns the correct Hessian.

Actual Behavior

Returns H_correct + gradient[i] * ones(1, n) silently — the inner-gradient value leaks into every column of the outer Jacobian.

Native Backend Comparison

The bug reproduces with raw Enzyme:

using Enzyme, ForwardDiff
h(x) = sum(x.^2)
g_via_fd(x) = ForwardDiff.gradient(h, x)
x = [1.0, 2.0, 3.0]

Enzyme.jacobian(Enzyme.Forward, g_via_fd, x)
# ([4.0 2.0 2.0; 4.0 6.0 4.0; 6.0 6.0 8.0],)
# expected: ([2.0 0.0 0.0; 0.0 2.0 0.0; 0.0 0.0 2.0],)

Enzyme.autodiff(Enzyme.Forward, g_via_fd, Enzyme.Duplicated(x, [1.0, 0.0, 0.0]))
# (([4.0, 4.0, 6.0],),)
# expected: (([2.0, 0.0, 0.0],),)

So the silent-wrong value comes from Enzyme's forward-mode differentiation of code that itself contains ForwardDiff Dual arithmetic, not from DI's SecondOrder plumbing.

Backend

• Backend: SecondOrder(AutoEnzyme(; mode=Enzyme.Forward), AutoForwardDiff())
• Works with other backends: SecondOrder(AutoForwardDiff(), AutoEnzyme(; mode=Enzyme.Forward)) and SecondOrder(AutoForwardDiff(), AutoForwardDiff()) return the correct Hessian.
• Native API gives same result: yes — wrong values come from Enzyme.jacobian(Enzyme.Forward, ::, x) applied to a function that internally uses ForwardDiff.gradient.

Environment

• Julia 1.12.5
• DifferentiationInterface v0.7.18
• ForwardDiff v1.3.3
• Enzyme v0.13.147

Full environment

julia> using Pkg; Pkg.status()
  [a0c0ee7d] DifferentiationInterface v0.7.18
  [7da242da] Enzyme v0.13.147
  [f6369f11] ForwardDiff v1.3.3

julia> using InteractiveUtils; versioninfo()
Julia Version 1.12.5
Platform Info:
  OS: Linux (x86_64-unknown-linux-gnu)
  CPU: 4 × Intel(R) Core(TM) i5-2520M CPU @ 2.50GHz
  WORD_SIZE: 64
  LLVM: libLLVM-18.1.7 (ORCJIT, sandybridge)

---

🤖 I am a robot. This is an experiment in agentic bug-catching under the supervision of @adrhill and @gdalle (#1008). Contents may be hallucinated.

[gdalle]

Manually reproduced this and filed an issue upstream: EnzymeAD/Enzyme.jl#3346