Add a system call to the linux kernel in Ubuntu

1) Install the tools to build a kernel:

sudo apt-get install fakeroot build-essential crash kexec-tools makedumpfile kernel-wedge
sudo apt-get build-dep linux
sudo apt-get install git-core libncurses5 libncurses5-dev libelf-dev asciidoc binutils-dev
sudo apt-get build-dep --no-install-recommends linux-image-$(uname -r)
apt-get source linux-image-$(uname -r)

2) Get the kernel to edit:

sudo apt-get install linux-source
mkdir ~/src
cd ~/src
tar xjvf /usr/src/linux-source-<version-number-here>.tar.bz2
cd linux-source-<version-number-here>
_{*or go to kernel.org and download version you wish, uncompress it and store in /usr/src(see notes below)}

3) Edit the kernel:

These are files that will have to be edited or created for the system call to work.
a) change syscall table
folder: arch/x86/syscalls/

edit syscall_32.tbl for 32-bit processors:
at end of file add: 350 i386 myservice sys_myservice
(might not be 350 for your linux kernel)

*or edit syscall_64.tbl for 64-bit processors:
at end of non-specific system call numbers add: 313 64 myservice sys_myservice


b) add header file to include folder:
folder: include/linux/syscalls.h
at very end of file(after #endif) add: asmlinkage int sys_myservice


c) create the system call:
folder: /kernel
create the system call and save as myservice.c:

#include <linux/kernel.h>
#include <linux/init.h>
#include <linux/syscalls.h>
#include <linux/linkage.h>

asmlinkage int sys_myservice(void) {
	printk(KERN_EMERG "my service is running");
	return 0;
}

d) create the Kconfig file:
folder: /kernel
create the Kconfig and save as Kconfig.myservice:

config MYSERVICE
        bool "prints my service is running"
        default y
        help
                This will print my service is running			

e) edit the Makefile of Kernel directory:
folder: /kernel
edit Makefile: add myservice.o to obj-y list


f) edit Makefile of kernel
folder: linux-<version-number-here>
edit Makefile at .extraversion line: EXTRAVERSION = .syscall(or whatever you want)

4) Create the config file:

cp -vi /boot/config-`uname -r` .config(this should be different config name, so look up /boot/configsomething instead of uname -r)
make menuconfig

5) Build the kernel:(this will take a while so maybe make a cup of tea or coffee. also this is safer in fakeroot)

export CONCURRENCY_LEVEL=3 #this is for dual processors quadcore would be...
make-kpkg clean # only needed if you want to do a "clean" build
fakeroot make-kpkg --initrd --append-to-version=-some-string-here kernel-image kernel-headers

echo vesafb | sudo tee -a /etc/initramfs-tools/modules
echo fbcon | sudo tee -a /etc/initramfs-tools/modules

6) Install the kernel:

sudo dpkg -i linux-image-2.6.20-16-2be-k7_2.6.20-16_i386.deb(your version not 2.6.20...)
sudo dpkg -i linux-headers-2.6.20-16-2be-k7_2.6.20-16_i386.deb(same here)

sudo cp /usr/share/doc/kernel-package/examples/etc/kernel/postinst.d/initramfs /etc/kernel/postinst.d/initramfs
sudo mkdir -p /etc/kernel/postrm.d/
sudo cp /usr/share/doc/kernel-package/examples/etc/kernel/postrm.d/initramfs /etc/kernel/postrm.d/initramfs

7) Change grub to show your new kernel at boot:

cd /boot
sudo mkinitramfs -k -o initrd.img-2.6.32.15+drm33.5-mylucid 2.6.32.15+drm33.5-mylucid(your initrd.img-3.5.something....)
sudo update-grub2

8) Boot into new kernel:

a) restart computer and hold down shift to get to grub menu
b) select Advaced options for Ubuntu
c) select your version of the kernel

9) Create the file to test system call:

a) create the source file to save system call
create file named test_myservice.c:

#include <stdlib.h>

int myservice() {
        int ret;
        __asm__("movl $350,%eax");
        __asm__("int $0x80");
        __asm__("movl %eax, -4(%ebp)");
        return ret;
}

int main()  {
        int ret;
        printf("invoking system call myservice...\n");
        ret = myservice();
        if(ret<0) {
                printf("not working");
                exit(1);
        }
        printf("system call executed.\n");
        return 0;
}

b) compile the source and 
gcc -o test_myservice.c myservice

c) run the service
./myservice

10) Check the call was logged in the kernel error log:

folder: /var/log/
open: kern.log
search for: "my service"

Helpful links:

http://blog.techveda.org/index.php/adding-system-calls-linux-kernel-3-5-x/
https://help.ubuntu.com/community/Kernel/Compile
http://kernel.org/
http://kernelnewbies.org/KernelBuild
http://www.wiley.com/college/sc/silberschatz/projectch2.pdf

Notes:

*the method presented is a mix and match of the web pages listed above and a few others
you might need to mess with it to get it to work correctly for your system

*make sure computer is connected to internet

*to enter directory as root:
alt+F2
gksudo nautilus /directory
or
alt+F2 gksu nautilus

*to compile something with multicore processors and speed up build:
example for dual core:
export CONCURRENCY_LEVEL=3
*or -j 3
number should be 1 + number of cores your processor has

*to list kernel version:
uname -r

*edit source file:
install VIM