Unitary equivariance is a natural symmetry that occurs in many contexts in physics and mathematics. Optimization problems with such symmetry can often be formulated as semidefinite programs for a d^(p+q)-dimensional matrix variable that commutes with U^(⊗p)⊗\bar{U}^(⊗q), for all U∈U(d). Solving such problems naively can be prohibitively expensive even if p+q is small but the local dimension d is large. We show that, under additional symmetry assumptions, this problem reduces to a linear program that can be solved in time that does not scale in d, and we provide a general framework to execute this reduction under different types of symmetries. The key ingredient of our method is a compact parametrization of the solution space by linear combinations of walled Brauer algebra diagrams. This parametrization requires the idempotents of a Gelfand--Tsetlin basis, which we obtain by adapting a general method arXiv:1606.08900 inspired by the Okounkov--Vershik approach. To illustrate potential applications, we use several toy examples from quantum information: deciding the principal eigenvalue of a quantum state, quantum majority vote, asymmetric cloning and transposition of a black-box unitary.
