Sorting is generic and comparison operators are defined in catalog:
// gets pointer to function via pg_operator
SelectSortFunction(Oid sortOperator,
bool nulls_first,
Oid *sortFunction,
int *sortFlags);
// returns negative, zero, positive
ApplySortFunction(FmgrInfo *sortFunction,
int sortFlags,
Datum datum1, bool isNull1,
Datum datum2, bool isNull2);