Trimming Viva Engage default “All Company” community membership

Hey

Question/problem statement: 

I’m looking in if there is a possibility of trimming the Viva Engage default “All Company” community membership list/count?

Background:

We have a lot of “external/contractors” that are fully O365 E3 licensed, and it also semes disable user if still with in the count (this is just a speculation), that we wish to remove from the membership list/count.

My thoughts:

Is the trimming the membership count just removing the Viva Engage Core (but all so removing the “external/contractors” access, which is fine)

Or is it at all possible to trim the member list?