Fritz edged out Alexander Bublik in three sets, while Paul made light work of Mikhail Kukushkin with a straight-sets victory. However, Bublik returned with partner Aleksandr Nedovyesov to down Jack Sock and Rajeev Ram in the doubles and reduce the overall score to 2-1.
"It was an electric atmosphere" - Jack Sock on playing against Andy Murray during the USA-Great Britain Davis Cup tie
History has proven Bublik to be a tricky opponent for Fritz, but it was an early break in the deciding set which swung momentum in Fritz's favor as he went on to claim victory in one hour and 36 minutes.
"Today I think a lot of it was just accepting in a third set that I was just going to have to battle and grind and play longer rallies," said the Indian Wells champion. "My game style is to be very aggressive and dictate. I just had to sit back and be a little more patient and I think that's what got it for me in the end."
Carlos Alcaraz gets beaten in first match as number one by Felix Auger-Aliassime in Davis Cup
Paul needed one hour and 25 minutes to get past Kukushkin in a match that was tighter than the score line suggests, making it consecutive wins for the American who saw off Great Britain's Dan Evans on Wednesday.
"I felt pretty comfortable, even though the games were close," Paul said. "The first set I felt like I was in full control. He wasn't putting too much pressure on me, so I felt like I could dictate everything in the first set. The second set got tricky."
Back-to-back victories for the USA puts them in a strong position to advance to the knockout phase ahead of their final match with the Netherlands on Saturday. Meanwhile, Kazakhstan know they have to beat Great Britain on Sunday to stand any chance of progressing.