So, Where is The Oldest Mail Link?
It's a bit sneaky what Google has done with this particular change, but they didn't take away the ability to get to your oldest mail. They are starting to realize that Gmail has way to many buttons, so for this option they went ahead and put it into hiding. What does that mean? Well, all you need to do is hover over the message number that you are on and you will see the outline of a box that will get you to where you need to go.
Here is what the message count looks like normally in the new interface update.