muon-newton-shultz-research
muon-newton-shultz-research
Newton-Schulz Iteration Analysis
An experiment was conducted to visualize the effect of the newton_shultz function on a random 4x4 matrix. The function applies an iterative method to orthogonalize the matrix.
The results are captured in the following image:
Analysis
The visualization clearly demonstrates that the newton_shultz function progressively orthogonalizes the input matrix over 5 iterations. This is evident from two key indicators in the plot:
-
Convergence of M @ M.T to Identity: The middle row of heatmaps visualizes the product of the matrix and its transpose (
M @ M.T). The plot for Step 0 shows a scattered result, but by Step 5, it has converged to a distinct diagonal line, which is the visual representation of an identity matrix. This behavior directly satisfies the definition of an orthogonal matrix (Q @ Q.T = I). -
Convergence of Singular Values to 1: The bottom plot tracks the singular values of the matrix at each iteration. The values start scattered (Step 0) but rapidly converge towards 1.0 with each step, as shown by the lines clustering around the red dashed line. A matrix is orthogonal if and only if all its singular values are equal to 1.
What is a Singular Value? (An Intuitive Explanation)
Think of a singular value as a measure of how much a matrix stretches or squishes things.
Imagine you have a circle. If you apply a 2x2 matrix transformation to every point on that circle, it will become an ellipse. The two singular values are the lengths of the two main axes of this new ellipse. They tell you the "stretch factors" of the transformation.
This idea generalizes to higher dimensions. The number of singular values equals the number of rows or columns of the matrix (its rank).