Yep you can do that, just start carefully unbolting stuff until you can get access to what you need. I had to do that once to get one of mine off (the race failed on me when I was trying to remove it and spat balls through everything so needed to completely strip it to recover one offending and very shy and retiring part).
It's not a very complicated setup, I can't remember if I had to pull the swivel hub unit and CV or whatever to do that job as it was about 2005 and I did the swivel hub bearings and seals as a separate job so had it all apart anyway.
Consider doing those bearings and seal anyway, as you are 90% of the way there and it removes the possibility of those bits failing shortly afterwards. The swivel bearings come with shims that align the hub on the axle, I got lucky on one side and slapping everything together same as it came out worked but the other side needed to be reshimmed and the ride height reset (probably it had been dicked with in the past and not set up again right, which is why it fell apart on me).
If you wanted to be argumentative with the inner race and didn't want to strip it apart, you could protect the axle and run a short bead of weld around the race and expand it. It will just pull straight off like that. Having said that, if you don't have the gear to heat the inner race and remove it you probably don't want to start welding on the thing!
Bookmarks