When Safari is in receive only mode and has not started local media, then it refuses to render incoming remote audio/video due to overly restrictive permissions.
If you do not start local media, which is perfectly reasonable given you are not using it, then the user is not asked permission for the camera and mic, and the remote media does not render even though the connection is successfully established. If you start local media but do not use it, then the user is asked permission for the camera and mic as usual, and remote media renders as expected.
While starting local media is a work around for this issue, it results in an unexpected user experience where the mic and/or camera is active even though the connection is negotiated in SendOnly mode. That said, this is the only work around for this issue.
We have filed a bug with the Apple Developers Bug Reporter and will update this posting as we hear more from the Safari team.