So after taking the clutch cover off to adjust the clutch, I found a whole world of issues.
1. Shift shaft. (Why it wasn't "catching" when down shifting)
2. Kick start has a chip in the gear (still works fine)
3. Clutch basket, inner hub and pressure plate are worn out.
4. Fibers and plates are done for.
5. Nut holding the basket and inner hub in was almost impossible to take off because of previous attempts to take it off.
6. Quite a few shavings in the oil 😅
7. Almost forgot this one, shift shaft bolt snapped.
I'll try to upload pictures, seeing as it won't let me due to file size