To change how the table headers appear you need to subclass NSTableHeaderCell, perform your own custom drawing in one of its drawing methods, then replace the header cells of each column with an instance of your subclass.
You may also find that you need to subclass NSTableHeaderView to draw the any space where no header cells are visible, and to replace the cornerView of
the table view.
This should get you started:
for (NSTableColumn *column in [tableView tableColumns])
initTextCell:[[column headerCell] stringValue]]
And here’s a starting point for a subclass of NSTableHeaderCell: