I agree. This would be a nice feat.
The developers have tried to add an “invisible” and “no access” permissions. Yet it seems to be difficult to implement and have thus abandoned the idea - for the time being as I understood.
If Seafile was otherwise flawless and feature-saturated, I would agree that this should be a priority. But - imho - I would invest my priorities elsewhere.