Google recently changed how 4K works on YouTube's homepage and Safari doesn't support it, but you can still view 4K YouTube videos on a Mac!
Google has made some changes to the way 4K videos are coded on its home page. YouTube uses the VP9 video coding format, while Apple's Safari web browser supports H.264. It's a quirky little change, of which AppleInsider goes into more detail. The basic gist is, Safari doesn't support VP9 and Google will only stream H.264 4K videos on YouTube's homepage.
So, why doesn't Apple just add VP9 support in Safari? You could argue about it for days. The simple truth is, we don't know why Apple doesn't add VP9 support in Safari, just like we don't know why Google removed support for H.264. But, since VP9 is open source and royalty free, it is likely that Apple wants to make sure that Safari is outfitted with a licensed official video compression standard.
There is no real choice, right now, for viewing 4K videos on YouTube's home page while using Safari. Hopefully, either Apple or Google will adopt the other's system, or a third-party developer will add a Safari extension that will allow us to play 4K video from YouTube's home page similar to the way Player for YouTube allows you to view YouTube videos with Flash.
For now, the only way you can watch 4K videos directly on YouTube's home page is by using another web browser that supports VP9, which are as follows.
You could also dual-boot Windows 10 onto your Mac and run the Microsoft Edge browser.