You could do this by mounting the remote folder as a file-system using sshfs. To do this, first some pre-requisites:
sudo apt-get install sshfs #for Debian based OS, use yum or zypper depending on your OS
sudo adduser <username> fuse
Now, do the mounting process:
mkdir ~/remoteserv
sshfs -o idmap=user <username>@<ipaddress>:/remotepath ~/remoteserv
After this, just go into the mounted folder and use your own local customized vim.