Maryland led by as many as 23 in the second half before Nebraska closed to 92-86 with 1:37 to play. Damonte Dodd and Layman combined to make three free throws down the stretch to help seal the win.
Layman went 6 for 9 from 3-point range. The previous season high for the senior forward was 19 points.
Next up for the third-seeded Terrapins is Michigan State on Saturday. The second-seeded Spartans advanced with an 81-54 victory over Ohio State.
Maryland (25-7) went 20 for 28 from the field on its way to a 54-37 halftime lead. It made nine 3s in 10 attempts before the break.
Nebraska (16-18), which defeated Rutgers and Wisconsin to reach the quarterfinals, fought back behind the hot shooting of Andrew White III, who made five 3-pointers and scored 25 points. Shavon Shields added 19 points.
The Terrapins placed five players in double figures on their way to the Big Ten Tournament scoring record. The previous high was 94 set by four teams.