Universe B does not (yet) have magic mirrors.

Universe A magic mirrors do not connect with a corresponding Universe B magic mirror. Sort of like In Mary Poppins, Bert's sidewalk chalk painting does not connect with a sidewalk chalk painting at the destination - Bert, Mary and the kids enter the chalk painting and emerge in the countryside of the alternate dimension. The cartoon residents of that dimension cannot come into our dimension the same way. Your mirrors connect similarly - some way such that people come thru and go back the way they came, but not vice versa. B people cannot use this route to leave B.

In Mary Poppins they were pulled back to their own world when the chalk painting was destroyed by rain. Maybe something has to happen to the mirror in A to pull the travelers back through - like the lights go out. It would be a good story way to ensure you were not trapped - your magic mirror is set in the garden, and you have until sundown. Unless there are artificial lights...

-

Later in your story the Universe B people, realizing that magic mirrors are possible, eventually make their own. They too are one way and they are more Universe B; different, possibly sinister.

Because everyone knows they would die

In Universe A these mirrors are well known and studied by some very smart people. Even though the average Joe cannot easily obtain a mirror, it is well known to the average UA resident that if you try to bring anything living from Universe B back into UA it will die. Everyone knows this in UA, it's a simple fact that everyone has heard. I mean have you ever met anyone on the street not from this universe?

If anyone from UB asks to come with the characters back to UB, the character from UA will have to tell them that unfortunately it just is isn't possible. As everyone in UA knows, a UB'er will not survive the trip into UA. You can even add horrible details of how painful they heard the death was.

But....

Through a series of events someone from UB either decides to risk it, accidentally falls into the mirror, or is faced with a dire situation anyways, so hey why not. Much to their surprise they open their eyes to UA, and conveniently for them they are still breathing.

Turns out those smart people had a reason to misinform people, and due to the rarity of the mirrors, and the high cost of proving them wrong those smart people were successful.

Disease.

Before travelling from Universe A to Universe B, you have to get a set of vaccines to prevent you from contracting common Universe B illnesses. If you do show signs of contagion, there are procedures in place for returning to the Universe A medical center and getting treatment in quarantine. All Universe A citizens are well informed that allowing a Universe B resident to travel into Universe A has the potential to trigger a plague that could kill off millions of people.

Later, you could have someone "find a cure" for the disease (or determine Universe A residents couldn't contract it in the first place), retracting the travel ban. Or you could determine that it's only actually contagious if you show certain signs within the last three weeks, or a Universe B resident stole a mirror and released the plague already, sending all Universe A citizens running to the doctor for a vaccine they wouldn't normally bother to get. At this point, why bother closing the barn door after the cows have gotten out?

The Evil One

Once, long ago, a great evil arose in Universe A. After a Great War, in which many heroic deeds were done and many horrible things happened, the mages of Universe A united in one last, desperate attempt to defeat The Evil One (whose name shall not be mentioned). They were unable to kill this being, but managed to imprison him in his home realm, Universe B. This prison is secured with seven great seals, kept hidden lest they be discovered and destroyed, freeing The Evil One to wreak havoc upon Universe A once more.

Now, millenia later, the Great War is but a distant memory, and few people actually believe it happened, or that there ever was an Evil One. Dramatic progress has been made into dimensional travel, but the strange fact that only one-way travel to Universe B is not entirely understood. Experts suspect that it is caused by a magical shield, which is anchored somehow by physical objects, but they are not entirely sure how or why it was created. To solve this dilemma and to allow free passage and trade with the disadvantaged people of Universe B, they find an unlikely hero and his ragtag band of misfits, and task them with a great quest: to find a way to unite the peoples of Universe A and Universe B in harmony and great justice.

Meanwhile, a Great Evil broods and watches, yearning to once more spread his malign influence over the delicious feast once denied him...

The level of ambient magic in Universe B could simply not be high enough for its natives to pass through a portal. Natives of Universe A are not creating a connection when traveling back to Universe A, but are just dropping their current connection.

Perhaps the existance of the portals is actually bleeding magic from Universe A into Universe B, and eventually, the ambient magic of Universe B will rise high enough to allow B-to-A travel.

One such possibility is that the laws of physics are different between the universes, and universe B biology is incompatible with universe A physics.

For example, suppose that Universe A does not have quantum tunnelling. Thus, the lifeforms there don't need it. If they travel to Universe B, they are largely unaffected. However, The lifeforms in Universe B depend on quantum tunneling. If they were to travel over, they would die, or face medical problems, as you decide.

This gets somewhat easier to justify if magic is involved. There could be incompatible magics, much like incompatible blood types.

The planar travel is possible only if some particular restraints are met.

Let's say that in the universe B there is a big abundance of metals in the environment, so that the majority of its inhabitants are slightly poisoned by heavy metals (think of mercury poisoning: it could also give an explanation of why the world B is a dystopic version of A).

Planar travel is possible only if the person has "low content" of metals, otherwise he/she would simply slam against the glass; people of A have no problem, but people of B would simply contain too much of forbidden substances to cross the mirror.
People of A, while in the universe B, would need to give big attention to what they eat, avoiding food that could contain metal, otherwise they couldn't come home. This would be a stimulus for the wizards of world A to improve the magic of the mirrors, reducing the limits that can prevent the travel, to the point that eventually even people of B will be able to cross the mirrors.

Mirrors are linked to their travelers

Any one can safely travel through a Mirror from World A (I'll refer to all worlds merely by letter from now on) to B, and thence, back through that mirror return to A. However, Travelling through a Mirror from B will send the traveler to C. Unfortunately C is desolate, inhospitable and unpopulated. Normally anyone who travels there perishes since there are no mirrors to return to B through. Since no one ever comes back from B everyone just assumes it's lethal to travel by mirror originating from B. Eventually someone does bring a mirror through from B to C after which it's discovered that mirror travel from C to A is possible, and travelers from B can return from C to B through the one mirror that was brought there.

They are missing a spacial dimension, such as in the tale of Flatland or Interstellar (2014 movie).

Members of Universe A can interact with Universe B by ignoring their extra spacial dimension(s), but members of B cannot gain the dimensions that would allow them to enter or perhaps even properly perceive A.

Explanation:

The "magical lock" is actually placed on mirrors in Universe B by a secret order of humans tasked with preventing the other "savage" non-human races from traveling over to Universe A and obtaining technology that would be able to obliterate humans.

Source of Magic:

The source of magic in each universe is the combined wisdom and magical ability of all magic-wielding inhabitants. Universe A has many more highly adept magic users, owing to its relatively stable and functional society.

The Lock:

The strength of the lock is directly proportional to the magic abilities of the lock creator(s) and maintainer(s). In the case of Universe B, the human lock builders were very powerful and the secret order has maintained knowledge of and skill with magics that have been otherwise lost to the rest of the inhabitants. There has to this point never been an individual or group of magic users with enough ability to override a lock (many have tried). Killing one or more of the secret order diminishes the strength of the locks until/unless the order is replenished.

Travel:

Travelers from A->B maintain a magical connection back to the point of origin that allows them to return from any mirror in Universe B. In all but the rarest of cases, they do need to keep a talisman/device on their person that taps into the source of Universe A magic. The talisman temporarily unlocks (but does not destroy) any Universe B lock. The talisman itself is magic- and gene-locked, which means it cannot open a mirror portal without a willing, living magic user (i.e., no chopping off a hand or holding a gun to the talisman-holder's head).

Resolutions:

• A human from the secret order goes haywire and (magically or otherwise) forces the other members to release one or more locks.


• A human from the secret order has a child with a non-human and over time, diplomacy prevails and the multiple species become allies. The locks are released.


• A human from the secret order falls in love with a non-human and gives them a way to "overthrow" the humans.


• An inhabitant of Universe A has a child with an inhabitant of Universe B. The child grows up with a connection to the source of magic in both universes. Whenever someone travels from A->B, the child is able to tap into the vast magic reserves of Universe A. Over time, they become the most powerful magic user in Universe B.


• An inhabitant of Universe A has a child with an inhabitant of Universe B. The mother or father or both are killed at some point, leaving behind a talisman. The child's magic "profile" and genes match those of its parent from Universe A, allowing it to use the talisman.

Anyway, variations on a theme. I hope something in here helps you.

Racism

Humans in universe B have all existing portals to universe A.

They refuse to travel to Universe A because they let all these subhuman species mingle together, also, they aren't terribly keen on letting their resident non-humans escape oppression via mass exodus through these mirrors, so the mirrors to Universe A are kept under close observation.

Likewise, Universe A isn't too keen on being a destination for refugees from this backwater hell hole of a universe - all the humans are bigoted jerks, and if they started taking in the non-humans it'd get to be a serious immigration problem. Similarly, if any of their own people went over, their either weirdo human supremacists, or they were insane enough to travel to a world where their own kind is totally oppressed, so in general Universe A isn't terribly keen on letting anyone from Universe B pop on over.