Use gamma 0.7. Pixel textures require a higher bitrate than filtered ones but from the few pixels I could actually see in yours it seemed fine. I would highly recommend using H264 through x264 though, it is vastly superior to XviD.
https://www.quaddicted.com/forum/viewtopic.php?id=6 can give you some ideas but make sure to read through it all as it is more of a diary than a guide.
I simply use
ffmpeg -i "$1" -acodec libmp3lame -ab 192k -vcodec libx264 -preset slow "$1.mkv" for 1280x720 captures and the quality seems good.
Capturing in full quality in-game and encoding it later on is best.
Alternatively to trying to figure out how to capture with Fitz, maybe just wait until christmas for something special :)