Williams, whose sister crashed out of the event earlier in the week lost the opening set in a lengthy tie-break, but hit back to claim the second with ease.
But Candian Bouchard held firm in the final set to claim the match 7-6(6) 2-6 6-4 and set up a quarter-final against either Jelena Jankovic or Ajla Tomljanovic.
Third seed Sara Errani came safely through her third round match, holding her nerve to beat China's Peng Shuai 7-6(6) 7-6(5).
