World Cup runner-up Brozovic scored four minutes into injury time in Genoa after Radja Nainggolan and Kwadwo Asamoah both had goals ruled out for Inter, with Sampdoria's Gregoire Defrel also denied by VAR.
Inter Milan coach Luciano Spalletti was also banished from the touchline by officials for his wild celebration of the winning goal.
"I turned towards the camera to shout 'goal' and the fourth official told me I had done it with too much force," Spalletti told streaming service DAZN.
"It was too important a goal, like a liberation, and they interpreted it as something with too much emotional reaction," added Spalletti. "I didn't do anything wrong."
The win followed Inter's 2-1 Champions League comeback victory in the final minutes against Tottenham on Tuesday.
"The team had demonstrated their character in the previous game (Tottenham) and did it again tonight, always remaining in the game and fighting, even with some risk.
"But we are Inter and we have to play until the win."
The three points give Inter just their second win of the Serie A season and move Spalletti's side up to seventh -- equal on seven points with Sampdoria -- after five games.
Both sides had early chances with Inter goalkeeper Samir Handanovic saving Fabio Quagliarella's overhead kick, with Asamoah preventing top-scorer Defrel breaking through.
Matias Vecino nodded an Antonio Candreva cross just over the bar.
But there was controversy just before the half-time whistle when Belgian Nainggolan volleyed in from a free-kick.
His goal was disallowed after a long consultation with VAR because captain Mauro Icardi was offside in the build-up.
After the break, Asamoah was also denied because the ball had gone out of play before being pulled back by Danilo D'Ambrosio, while Defrel missed the chance to take sole possession of top spot in the Serie A scorers' charts when what would have been his fifth goal of the season was ruled out for offside.
Minutes later, Brozovic made no mistake when he collected a deep cross from fellow Croatian Ivan Perisic in the fourth added minute, beating Emil Audero in the Sampdoria goal with a low, deflected strike.
"I think this week will be the turning point in our season," said Brozovic.
Earlier Fiorentina moved second with a 3-0 win in Tuscany, spoiling SPAL's promising start to the campaign.
Ferrara outfit SPAL had been riding high in second, three points behind champions Juventus, before the tie in Florence with just one goal conceded in four games.
Italian international Federico Chiesa completed the scoring in the 56th minute as Fiorentina bounced back from last week's 1-0 defeat by Napoli to climb to two points behind Juventus who travel to promoted Frosinone on Sunday.
Former Arsenal winger Gervinho scored a stunning goal as Parma beat Cagliari 2-0 to secure back-to-back wins after they shocked Inter Milan at the San Siro last weekend.
"I don't know if it's my best goal, as I've scored many in my time," said Gervinho.
Gervinho was substituted five minutes from time to a standing ovation at the Stadio Tardini, as Parma moved up to seventh following their first Serie A win at home since 2015.