The gender earnings gap can be attributed either to the different distribution of males and females across jobs or to within job biases in favour of men. The latter is frequently called the wage structure effect, and it may be interpreted as wage discrimination against women. In this paper we focus on this second source of the gap. In particular, we study the heterogeneity of the wage structure effect by looking for the main drivers of it. On Hungarian matched employer-employee data we identify those firm-worker profiles that exhibit extremely high gender wage differentials We apply the Causal Forest methodology, borrowed from the conditional average treatment effect (CATE) literature, which has been utilized in several observational studies, recently. Our findings show that those firms that pay relatively high wages tend to discriminate against women most strongly, and especially with respect to women who have spent a longer time in the same firm. But this tendency is moderated by regional effects; where demand side competition is strong the wage structure effect tends to be smaller. These findings are, by and large, in accordance with the view that relative bargaining power is relevant for wage-setting, or, alternatively, firms practice third degree wage discrimination.