How do you remove the knob? - I already removed the center cap and there is no retainer visible.
Just unscrew counterclockwise. Instead of steady twist, try a quick forcefull twist & it will pop loose. Also oil on a toothpick, let it seep down behind the knob & let it soak if still won't budge.
2) I am sectioning that piece of harness, hopefully I have that one my parts car, is there a better way to deal with this?
You could buy new wire, it's your choice. But the headlight switch connector is what you want from another harness, I'd splice one about 3 inches away from the existing connectors end & use heat shrink tubing to make it look nice after I soldered each wire. A tip here, don't cut all the wires at the exact same spot, spread the cuts apart about 1 or 2 inches so the spliced points of the wires with heatshrink will not bunch up as big in one spot, looks nicer.
3) why did this happen (root cause) - I had a similar issue with the tail light harness a month ago, which I repaired by sectioning with an donor piece. Is it possible that the two are related?
Guessing here, can't diagnose that, over the internet, but maybe corrosion/humidity in the wiring... if it's green/grey or discolored wires when you cut into fix it, keep cutting back till the copper is clean for a few inches. Previous owner, who knows ?
4) why did I not blow fuses, not enough current went through any one fuse, because not every 12 volt source in your car is fused, so shorts could occur without blowing a fuse, and it the short only pulls a few amps, it may not blow a fuse.
5) should I add a fuse to this circuit (ie headlamp)
Shouldn't be necessary, but inline fuseholders & fuses are inexpensive protection if your not sure what's going on there, just make sure you place it where it is easy at nite & find it, it's your headlight fuse.
6) I really don't want to embark on a total rewiring - what would be the second best option?
Fix the problem areas like you are doing. Clean common ground points like under the heater box etc... if you havent' looked at em in years, they may have green corroison etc... Stuff like that. & try to fix one area at a time, it will be easier to troubleshoot, as you may make an error etc....