In Drupal 8/9, when a file is attached to an entity, it receives some assumptions regarding its general access. That access generally follows the permission access rules of the entity that it’s attached to. While this has its benefits, and certainly made sense back in the days when files were directly attached to nodes, it can create a situation where a private file can be directly accessed because its parent entity type is publicly accessible.
In my case, a paragraph type that allowed for embedding different media entities was set to be viewable by all. Even though the parent node of the paragraph was set to a private access control, the extra embed levels of paragraphs and media items created an access override.