In D&D: Dungeon Master's Guide, the Abyss is one of the Outer Planes. It "has a seemingly endless number of layers" and is associated with the chaotic evil alignment. However, that cosmology may not be canonical for Critical Role.
When Scanlan Shorthalt scried through his magical poo, his consciousness transcended the Material Plane, traveled across the Astral Sea, and arrived at his scrying destination on the Material Plane.
The Elemental Planes of Fire, Water, Air and Earth intersect with the Material Plane. The druidic Ashari guard the portals at those connecting points.
The Ethereal Plane overlaps the Material Plane, but they're out of phase with each other.
The Far Realm is another dimension that connects to the Material Plane.
Clarota told Vox Machina that, should they try to enter through the rift to the Far Realm that they found while in the Underdark, their minds would break.
The Shadowfell, like the Feywild, overlaps the Material Plane. The only known portals are the three magic siphons that were set up throughout Exandria, though it is unclear if these are still active[presumed].