The inner sleeve will move to the side, but if the bearings have been squeezed tight to it then you will need to force it aside. One thing you could do is drill a 1/4" hole in the side of your carrier and use a smaller punch in the hole to drive it to the side. When you're done, you can install a grease fitting in the hole. When you get the sleeve out, drill a 1/4 inch hole in it too so that when you grease it in the future, some grease can get to the axle and make it easy to get out.