In some new versions of Ubuntu, apt-get remove will not work. We have to use the underlying dpkg of apt-get:
sudo /etc/init.d/oracle-xe stop
sudo dpkg -P oracle-xe-universal
It should remove all config files and /usr/lib/oracle stuff (unless some files, dirs are held by some process).
To make sure clean uninstallation,
rm -rf /usr/local/oracle
Then reboot
sudo shutdown -r 0
Now it should be done! You can re-install Oracle xe again:
sudo dpkg -i oracle-xe*.deb
sudo /etc/init.d/oracle-xe configure
good luck,