I. Open VLC. On the menubar go to VLC > Preferences (⌘,) to open the preferences window. On the bottom left-hand corner, there are two radio buttons, one for Basic options and the second for All options. Select All and it will automatically open all preferences.
II. Click on the Video tab. If it does not expand automatically, click on the left-arrow button (►).
III. Under the 'Output modules' tab find and click on 'Scene filter'. 'Directory path prefix' will save the caps to whichever directory you specify. If you do not specify a directory, it will save to your Pictures folder. If there is a specific folder you want the caps to be saved to, it will look something like this:
/Users/Username/Desktop/FolderName. I usually set 'Recording ratio' between 20-40 depending on what I am capping. The lower the number the more screencaps.
IV. To enable scene filter select the Filters tab and in the 'Video filter module' textarea type in scene.
V. Now Save your preferences. Start playing your video and it should start to screencap automatically. If the changes do not take effect immediately, quit and reopen VLC.
VI. To turn off automatic screencapping, go back to Video > Filters and delete scene from the 'Video filter module' textarea and Save changes again.