There are two sides to my app, on one side I'm using C++ in order to read the frames from a camera using Pleora's EBUS SDK. When this stream is first received, before I convert the buffer to an image, I am able to read the stream 16 bits at a time in order to perform some calculations for each pixel, i.e. there exists a 16 bit chunk of data for each pixel.
Now the second half is a Django web app where I am also presented this video output, this time via an ffmpeg, nginx, hls stream. When the user clicks on the video I want to be able to take the current frame and the coordinates of their click and perform the same calculation as I do above in the C++ portion.
Right now I use an html5 canvas to capture the frame and I use canvas.toDataURL() in order to convert the frame into a base64 encoded image, I then pass the base64 image, the coordinates, and the dimensions of