The gradient-based optimization method for deep machine learning models suffers from gradient vanishing and exploding problems, particularly when the computational graph becomes deep. In this work, we propose the tangent-space gradient optimization (TSGO) for probabilistic models to keep the gradients from vanishing or exploding. The central idea is to guarantee the orthogonality between variational parameters and gradients. The optimization is then implemented by rotating the parameter vector towards the direction of gradient. We explain and test TSGO in tensor network (TN) machine learning, where TN describes the joint probability distribution as a normalized state $|\psi ⟩$ in Hilbert space. We show that the gradient can be restricted in tangent space of $⟨\psi |\psi ⟩=1$ hypersphere. Instead of additional adaptive methods to control the learning rate $\eta$ in deep learning, the learning rate of TSGO is naturally determined by rotation angle $\theta$ as $\eta =tan\theta$. Our numerical results reveal better convergence of TSGO in comparison to the off-the-shelf Adam.