For future searches as well, I had an mp4 file that I downscaled with Handbrake using handbrake-gtk from apt-get, e.g. sudo apt-get install handbrake-gtk. In Ubuntu 14.04, the handbrake repository doesn't include support for MP4 out of the box. I left the default settings, stripped the audio track out, and it generates an *.M4V file. For those wondering, they are the same container but M4V is primarily used on iOS to open in iTunes.
This worked in all browsers except Safari:
I changed the mime-type between video/mp4 and video/m4v with no effect. I also tested adding the control attribute and again, no effect.
This worked in all browsers tested including Safari 7 on Mavericks and Safari 8 on Yosemite. I simply renamed the same m4v file (the actual file, not just the HTML) to mp4 and reuploaded to our CDN:
Safari I think is fully expecting an actually-named MP4. No other combinations of file and mime-type worked for me. I think the other browsers opt for the WEBM file first, especially Chrome, even though I'm pretty sure the source list should select the first source that's technically supported.
This has not, however, fixed the video issue in iOS devices (iPad 3 "the new iPad" and iPhone 6 tested).