Listed below are file formats that are compatible with Apple TV. To be able to stream VLC videos from Mac to Apple TV (through Airplay) your video format needs to be one of the file formats that are supported by Apple TV.
Using Airplay to Stream MP3/MP4 videos from Mac to Apple TV.