The quality of fermented milk can be improved by the exopolysaccharides (EPSs) produced by probiotics. Bifidobacterium breve H4-2 and H9-3 with different EPS production were used as co-fermented probiotics to ferment goat milk in the present study. The physicochemical properties of fermented goat milk and the interaction between goat milk casein and CEPS (EPSs separated from starter culture and B. breve H4-2 fermented goat milk) were investigated. Results indicated that both B. breve H4-2 and H9-3 have similar gene clusters for EPS synthesis, but B. breve H4-2 exhibits a higher ability in EPS productions. Compared to the starter culture group, combining B. breve H4-2 or H9-3 with a starter culture significantly increased the levels of EPSs ((724.16 ± 1.54) and (607.15 ± 4.55) mg/L) and lactic acid ((14.62 ± 0.02) and (16.87 ± 0.01) mg/mL) at the late storage period. Co-fermentation also enhanced the water holding capacity, firmness, consistency, cohesion, and viscosity index, reduced dehydration in fermented goat milk. Moreover, the interaction study revealed that CEPS reduced the particle size and increased the ζ-potential absolute value of the CEPS and casein mixture system. Additionally, CEPS effectively mitigated casein aggregation, thereby enhancing the stability of fermented goat milk. These findings present a novel perspective for the development of a natural starter.