Star Wars has already addressed how Palpatine survived his fall at the end of Return of the Jedi, but we still don’t know why Luke wasn’t able to feel his presence through all those years.
As the official novelization for The Rise of Skywalker revealed, Palpatine was working on a contingency plan in case the Empire fell. When Luke and Vader duelled each other on the second Death Star, the Sith Eternal was constructing a new body for Darth Sidious on Exegol. When the redeemed Anakin Skywalker threw his master down the reactor shaft, Palpatine used the remainder of his strength to transfer his subconscious to the new body. Alas, the phantom Emperor was too weak to do anything at the time, so he bided his time and slowly regained power and influence across the galaxy.
But how did he elude Luke for so many years? Well, in Chuck Wendig’s Aftermath trilogy, it’s revealed that Palpatine had established many secret bases across the galaxy called “Observatories,” and each of them served a specific purpose. Anakin once stumbled upon one of them in The Clone Wars and even Luke found one shortly after the Battle of Endor. The latter was a facility dedicated to the mass-production of Sentinel droids, programmed to convey the Emperor’s final wishes to his followers. While Luke disrupted one of these facilities, many more sent out these Sentinels and they began concealing Palpatine’s hidden network throughout the galaxy.
The Sentinels commanded the remnants of Imperial forces to commence Operation Cinder, a terrible scorched-earth scheme that had the Star Destroyers launch devastating and genocidal attacks on countless planets. Though according to Alexander Freed’s recent novel Shadow Fall, this operation served another purpose: to tactfully target worlds that housed Palpatine’s secret Observatories.
Of course, the plan wasn’t entirely successful. It all culminated in the Battle of Jakku, where the New Republic fought the Empire one last time. The site also held another one of Palpatine’s secret bases, one that he had used to chart the Unknown Regions. Luke managed to break into the lab, which is probably why he also started searching for the Sith Wayfinder to quench his suspicions.
As the last Star Wars movie explained, though, the young Skywalker couldn’t finish his mission. Sensing the danger that Luke represented, Palpatine kidnapped Lando’s daughter and even turned Ben Solo onto the dark side, making sure that the Last Jedi would never find out about his whereabouts on Exegol.