Filter to support Fallback language

Hi I have the following filter where i have to match the language and category id to get the Articles.

So for example get me all Articles that have en-GB languageBranch and Category ID 20 Or en-GB languageBranch and Category ID 21 and so forth e.t.c.

How can i modify this filter to handle fallback language for example en-AU language branch or other fallback languages? If the Article that doesnt exist in en-GB with Catgegory ID 21 , than return the Article with en-AU and Catgeory ID 21. No matter what i do i get duplicate articles for Catgeory Id 21 (One in en-GB and one in en-AU)

public static ITypeSearch<T> MatchCategories<T>(this ITypeSearch<T> search, string languageBranch, List<int> categories) where T : PageData
{
var taxonomyBuilder = search.Client.BuildFilter<T>();

foreach (var category in categories)
{
taxonomyBuilder = taxonomyBuilder.Or(x => x.LanguageBranch.MatchCaseInsensitive(languageBranch) & x.Category.Match(category));
}

return search.orFilter(taxonomyBuilder);
}