Use rsync, and pass -u if you want to only update files that are newer in the original directory, or --ignore-existing to skip all files that already exist in the destination.
rsync -au /local/directory/ host:/remote/directory/
rsync -a --ignore-existing /local/directory/ host:/remote/directory/
(Note the / on the source side: without it rsync would create /remote/directory/directory.)