Windows is notorious for this issue. SQLite handles the problem by retrying the delete operation every 100 milliseconds up to a maximum number.
I believe if you are sure that you have no open handles, doing this in your implementation will save you some headaches when things like antivirus software open the file.
For reference, the comment from SQLite source:
/*
** Delete the named file.
**
** Note that windows does not allow a file to be deleted if some other
** process has it open. Sometimes a virus scanner or indexing program
** will open a journal file shortly after it is created in order to do
** whatever it does. While this other process is holding the
** file open, we will be unable to delete it. To work around this
** problem, we delay 100 milliseconds and try to delete again. Up
** to MX_DELETION_ATTEMPTs deletion attempts are run before giving
** up and returning an error.
*/