Problem-1: Detecting identical thumbnails
Fortunately, we have a good solution--libpuzzle. It would be ideal to have a filter in ffmpeg for that.
Problem-2: Detecting black/dark frames
I'm still keenly looking for a solution tied up with ffmpeg. When Google turned no help, Heleena (my wife) solved it by calculating average luminance value of the frame and ignoring the frames based on certain preset threshold value--which worked better at that time.