With just four minutes remaining, the African nation were cruising to what was a deserved place in the last 16 thanks to goals on Wednesday (Thursday AEST) from Sunderland's Habib Diarra and a brilliant effort from Crystal Palace forward Ismaila Sarr.
However, Belgium's all-time record scorer Romelu Lukaku's 92nd international goal gave Rudi Garcia's side hope when there had been none and Tielemans headed home Leandro Trossard's cross two minutes and 38 seconds later.
Twenty minutes earlier, the pair had been separated by teammate Nicolas Raskin as they exchanged heated words as frustrations boiled over at the hydration break.
With a shootout looming, Dodi Lukebakio clipped the crossbar only for VAR to spot a foul by Lamine Camara on Tielemans in the build-up and after a seven-minute delay a penalty was awarded.
In January, Senegal walked off in the Africa Cup of Nations final in protest at Morocco being awarded a late spot kick and although they returned and went on to win the game, they were subsequently stripped of the title.
There were no such antics this time with Tielemans holding his nerve to send Belgium through with the latest goal in World Cup history, with 124 minutes and 44 seconds on the clock.
It was harsh on Senegal, who took a first-half lead when Sadio Mane's inswinging cross was flicked on by Sarr's twisting header and Diarra followed in as the ball bounced back off the upright.
His goal was the eighth scored by a Sunderland player at this summer's tournament, with only Real Madrid and Paris Saint-Germain players having contributed more.
Senegal's second was both simple and brilliant: Sarr superbly controlled Moussa Niakhate's long ball on his chest, having exploited the space between centre-backs Arthur Theate and Brandon Mechele, before firing home.
