-or vs -our
Those that do end in -our for us, like colour or harbour, always do, but we do have words like horror or governor that don't, so there's no one absolute rule, sadly.
-ise vs -ize
This one is even less concrete. I went an looked it up as I've never been sure of the precise rule here, and apparently whilst many Brits may tend to view -ize as Americanized and wrong, that's actually an incorrect assumption and either is in fact acceptable and it is a stylistic preference that varies between publishing houses (for example the Oxford University Press apparently favours -ize, and consequently so does the Oxford English Dictionary, whilst the Cambridge University Press favours -ise).