Sigmoid vs ReLU Activation Functions: The Inference Cost of Losing Geometric Context

Sigmoid compresses inputs into a narrow range between 0 and 1.

That compression pushes values away from decision boundaries, making them indistinguishable and causing loss of geometric context across layers.

ReLU preserves magnitude for positive inputs, so distance information can propagate through the network.

A two-moons experiment compares how Sigmoid and ReLU differ in forward-pass signal propagation and representation geometry.

The experiment uses a make_moons dataset that is noised and standardized with StandardScaler and considers inference efficiency and scalability.