LMAX Group, a London-based institutional trading platform, is exploring strategic options including a potential sale or public listing that could value the company at up to $5 billion, according to people familiar with the matter.