You might want to review the linear algebra text book of freshman's course in most college.
First of all, let's make some vectors based on your given points:
b=B-A
c=C-A
d=D-A
, In other words, we simply let A be the original point. Vectors b, c, and d are connecting from A to B, to C, and to D, respectively. We are going to answer your questions based on these vectors.
To check if A, B, and C are not in a straight line, we simply need to check if vector b and c are independent. We use the rank of matrix with verctor b and c to determine:
r = rank([b c])
Here we assume b and c are column vectors. (If b and c are row vectors, r = rank([b;c]) in Matlab) Now, If r=2, vector b and c are independent, in other words, A, B and C are not in one straight line, they can determine a plane.
We assume b and c are independent now. We use the following st