The information for this is VERY very sparse, so here is a summary of what I have found.
Install libx264
Get the libx264 package from here: http://www.videolan.org/developers/x264.html
Extracts the bz2 file
tar -xjvf /path/to/x264-snapshot-20080805-2245.tar.bz2
And now the useual
cd /path/to/x264-dir
./configure --enable-shared
make
make install
ldconfig
Get a nice error message:
[root@dev01 x264-snapshot-20080805-2245]# ./configure
No suitable assembler found. Install 'yasm' to get MMX/SSE optimized code.
If you really want to compile without asm, configure with --disable-asm.
You do want MMX/SSE at somepoint for the time being I am disabling this though (lack of time to find a valid RHEL source). so I added the disable asm line.
Install ffmpeg
For this I use subversion.
cd /path/to/where/I/want/sources
svn checkout svn://svn.mplayerhq.hu/ffmpeg/trunk ffmpeg
cd /path/to/where/I/want/sources/ffmpeg
./configure --enable-libx264 --enable-gpl --enable-shared
make
make install
et voila ffmpeg is now installed with libx264 (h264).
error while loading shared libraries: libavdevice.so.52: cannot open shared object file: No such file or directory
To fix this:
vi /etc/ld.so.conf.d/custom-libs.conf
Add the line : /usr/local/lib
And run ldconfig.
UPDATE: I am writing a full set of notes blog entry for installing ffmpeg with codecs on RHEL4