Well, this is only my GUESS, because I don't know your email server, your configuration, etc.
But let's discuss the most probable case:
1) Tor (the application) chooses random Tor Exit Node from public list, makes virtual circuit to it and creates localhost:9050 that acts like socks5 proxy. Pretty much the same as "SSH Tunneling" (plink.exe or 'ssh -D'), if you are familiar with this technique.
2) Tor Browser uses Socks5 proxy to access Internet (Firefox: Options -> Advanced -> Connection). Tor application receives that traffic, routes it to Tor Exit Node, and Tor Exit Node makes request to target website ("Email server").
3) Email server responds to Tor Exit Node with login page. You see that page in your browser.
4) You type in Login and Password, press Enter, Tor Exit Node submits your form to Email server.
5) Email server responds with HTML view of your mailbox.
As long as you are inside Tor Browser, all traffic continues to travel through Socks5 proxy on localhost -> Tor Exit Node -> Email server. Your real IP is hidden.
Now let's discuss your Email server. For example, imagine there are only 3 users on it: alice, lis, martin. Let's send them 3 different letters:
Hello, Alice, please click here: http://ourserver.com?user=1
Hello Lis please click here: http://ourserver.com?user=2
Hello Martin, http://ourserver.com?user=3
Now let's wait some time and then see ourserver's logs. If we found in logs somebody was visiting "ourserver.com?user=3", we would definitely know this is Martin. Because of user=3.
I don't know what server do you actually use. In Gmail link to download original attachment looks like this:
If you are Gmail admin, you can find the account & letter I took this link from. Just search values 'tx0derq50qxd' and '28f31aa19ba3f28e' . If I have downloaded this attachment before, you can also find my IP.
Do you understand how?
So now, what we have:
6) You used IDM to download some link. And that link was tied to your mailbox. (I am pretty sure your Email server doesn't let strangers to download random attaches, so every link contains Session ID or something similar). And I assume your IDM was not configured to use proxy, so it went directly from your IP.
So now what we have in Email server logs?
IP Address of Tor Exit Node, that logs in to mailbox.
Credentials (username, password).
Session ID that was assigned to Tor Exit Node after successful login.
And then some another IP Address that downloaded attach during the session with this Session ID.
Session ID was valid, session was active, just IP was different. And that was YOUR REAL IP.
PS: There are many chances they do not log every download, only sign ins. Or, they could log both, but in different places and without cross reference. Probably, session ID's are not logged, and all session data is discarded after session expires... Who knows? But in theory you gave them enough information to de-anon you.