When to use assert() and when to use try catch?

Try... catch - for exceptional conditions, i.e. conditions which aren't caused by malformed code, but which may just alter the normal control flow by external unpredictable events.

Assertions for catching invalid code, i.e. checking if an invariant is held in the function, checking if an internal method is called with right arguments (for public API you might still want an exception for that), etc.

Those are my basic guidelines, but the conventions vary from situation to situation and from language to language.