Create a comprehensive and well-thought-out plan. Involve both the new system vendor and your old provider in the process. Map the data from your old system to the new one. Clean up your data before you convert. Stage and complete a mock conversion and a conversion readiness check.
The key success measure of a core system modernization is that everything works as expected. Customers don't choose to bank with you because of the core system you use. Minimize customer impact.
Make all the decisions before you begin. Manage risk and avoid last-minute heroics. Train and let employees practice on the new system before conversion.
Limit the scope and be clear on what is included and what is not. Consider the amount of customization to your existing core system and how much retrofitting will need to occur with the new system. Do not make product changes or introduce new products as part of the change. Narrow the scope as much as possible by resisting the temptation to simultaneously upgrade other systems. Identify areas of risk and include mitigants in the execution plan.
There is no perfect system, but there are some great ones available. You can find a close fit if you know your requirements well. And remember: You are selecting a new vendor as well as a new system.