Israel froze the entry of Palestinians to the Al-Aqsa Mosque because the Noble Sanctuary officials refused entry as long as metal detectors and cameras were installed on doors.
The noon prayer was held outside Al-Aqsa.
Previously, Israel said it will gradually reopen the Jerusalem holy site on Sunday after taking the rare step of shutting it down following a deadly assault there that sparked concerns of a fresh round of violence.
For the first time in decades, Israel closed the site on Friday after an attack by three Arab citizens of Israel who opened fire from the sacred site with automatic weapons and killed two police officers before being shot dead inside the compound.
Benjamin Netanyahu said that following consultations with security officials the site would be reopened Sunday afternoon with increased security measures that included metal detectors at the entrance gates and additional security cameras to make sure that no weapons are smuggled inside again.
Israeli forces conducted a security sweep early Sunday and began installing the detectors. Given the large number of worshippers expected to enter the site, the new measures could slow movement and spark tensions.
The attack triggered a rare phone conversation between Netanyahu and Palestinian President Mahmoud Abbas, who condemned the attack and called for the site to be reopened.
Jordan, a custodian of the sacred compound, also called for its immediate reopening.
Netanyahu acted quickly to allay Muslim fears, saying that the status quo at the Muslim-administered site "will be preserved." But Gaza's Hamas rulers called the act a "religious war" and urged Palestinians to carry out more attacks.
Early Sunday, Israeli police said security forces shot dead a Palestinian assailant behind a pair of recent shooting attacks.
Spokesman Micky Rosenfeld said police tracked down the 34-year-old suspect in a joint operation with the military. The suspect opened fire with an automatic weapon, prompting the troops to return fire, killing him.