Description
As @stdlib continues to expand its machine learning and statistical primitives, the repository currently provides classical scalar activations like expit (Sigmoid), relu, and log1pexp (Softplus). However, it is missing the standard activation functions that power modern deep learning architectures.
To support developers building JS-based ML inference engines natively in @stdlib, I propose adding GELU and SiLU to the @stdlib/math/base/special namespace.
Motivation:
- GELU (Gaussian Error Linear Unit): The de facto standard activation function for modern Transformer architectures (GPT, BERT, Vision Transformers).
- SiLU (Sigmoid Linear Unit / Swish): Heavily utilized in modern vision models (YOLO, EfficientNet) and modern LLMs (LLaMA).
Proposed Solution:
I propose creating two new purely scalar mathematical packages:
1. @stdlib/math/base/special/gelu
-
Formula: $x \cdot \Phi(x) = \frac{x}{2} \left[ 1 + \text{erf}\left( x \cdot \sqrt{0.5} \right) \right]$
-
Implementation: Pure JavaScript scalar composition relying on the existing
@stdlib/math/base/special/erf and @stdlib/constants/float64/sqrt-half.
2. @stdlib/math/base/special/silu
-
Formula: $x \cdot \sigma(x) = \frac{x}{1 + e^{-x}}$
-
Implementation: Pure JavaScript scalar composition relying on the existing
@stdlib/math/base/special/expit.
Edge Case Handling (IEEE 754 Compliance):
Both implementations will strictly adhere to standard @stdlib floating-point handling:
-
NaN inputs will return NaN (validated via @stdlib/math/base/assert/is-nan).
-
$-\infty$ inputs will explicitly return
0.0 to avoid JavaScript's default behavior where $-\infty \times 0.0$ evaluates to NaN.
-
$+\infty$ inputs will return $+\infty$.
Related Issues
No.
Questions
No.
Other
Prior Art:
- PyTorch:
torch.nn.GELU and torch.nn.SiLU
- TensorFlow:
tf.keras.activations.gelu and tf.keras.activations.swish
Implementation Notes:
This RFC strictly covers the exact mathematical definitions for the primary namespace. Fast approximations (e.g., the standard tanh approximation for GELU) can be addressed in separate, future PRs targeting math/base/special/fast/gelu to keep this initial scope tight.
If this proposal is accepted, I am ready to handle the implementations, generate the test fixtures, and write the JSDoc documentation for both packages.
Checklist
Description
As
@stdlibcontinues to expand its machine learning and statistical primitives, the repository currently provides classical scalar activations likeexpit(Sigmoid),relu, andlog1pexp(Softplus). However, it is missing the standard activation functions that power modern deep learning architectures.To support developers building JS-based ML inference engines natively in
@stdlib, I propose adding GELU and SiLU to the@stdlib/math/base/specialnamespace.Motivation:
Proposed Solution:
I propose creating two new purely scalar mathematical packages:
1.
@stdlib/math/base/special/gelu@stdlib/math/base/special/erfand@stdlib/constants/float64/sqrt-half.2.
@stdlib/math/base/special/silu@stdlib/math/base/special/expit.Edge Case Handling (IEEE 754 Compliance):
Both implementations will strictly adhere to standard
@stdlibfloating-point handling:NaNinputs will returnNaN(validated via@stdlib/math/base/assert/is-nan).0.0to avoid JavaScript's default behavior whereNaN.Related Issues
No.
Questions
No.
Other
Prior Art:
torch.nn.GELUandtorch.nn.SiLUtf.keras.activations.geluandtf.keras.activations.swishImplementation Notes:
This RFC strictly covers the exact mathematical definitions for the primary namespace. Fast approximations (e.g., the standard
tanhapproximation for GELU) can be addressed in separate, future PRs targetingmath/base/special/fast/geluto keep this initial scope tight.If this proposal is accepted, I am ready to handle the implementations, generate the test fixtures, and write the JSDoc documentation for both packages.
Checklist
RFC:.