Ryfujk

I want to get an intuitive idea of the operation being "well-defined" for quotient groups. So, let's say I have a group $G$, with subgroup $H$, and let's say my set of left cosets is $G/K$. My lecture note says this: If $H$ is normal, then $G/H$ is a group under binary operation $aH circ bH = (ab)H$. So let's say I am asked whether $G/H$ forms a group where $H$ is not normal, and I have already determined that. My professor referred to the following which I am not sure if I totally understand:

Show that there are $a, b, a', b' in G$ such that $ aH = a'H$ and $bH = b'H$ but $aH circ bH = (ab)H neq a'H circ b'H = (a'b')H$, and you are done, and I did what she suggested, but I am not sure what is going on. So,

$1.$ What did I exactly show by proving what my professor suggested?

$2.$ In general, is it a strategy that every time you have to prove $G/H$ does not form a group for a non-normal $H$, you show that the operation is not well-defined?

Also, a quick google search also showed me that the theorem that talks about $G/K$ forming a group is an "if and only if" statement and not difficult to prove as well. Still, any help on my questions above would be great.

In general, mathematicians use the phrase "well defined" when a definition is written in a form that depends (or, rather, seems to depend) on some more or less arbitrary choice. If you make such a definition, you are obligated to show that another choice that satisfied appropriate conditions would lead to the same result.

In a group the product $abc$ is well defined to be $(ab)c$ because its value does not depend on your choice of where to put the parentheses: associativity guarantees $(ab)c = a(bc)$. This fact is so intuitively clear that it's often not made explicit in a beginning algebra course.

When considering quotient groups, you want to define the multiplication of two cosets by choosing an element from each, multiplying them together, and taking the coset of the product. This coset product will be well defined only when the coset of the product of the two group elements does not depend on which ones you happened to choose. The sum of any two odd numbers will be even, so the product of cosets
$(2mathbb{Z} + 1) circ (2mathbb{Z} + 1)$ is $2mathbb{Z}$.

There is an alternative definition. You can define the product of two cosets $A$ and $B$ as
$$
A circ B = { ab | a in A text{ and } b in B }.
$$
This definition does not make any arbitrary choices, but you don't know that the set so defined is really a coset until you prove it.

Perhaps a concrete example will make it clearer?

We need a non-abelian group. Let's take the simplest one there is, namely $G=S_3$.

We need a non-normal subgroup. Let's take $H={e,(12)}$.

We need two different cosets, and it won't work with H itself, so we have to take
$$ a=(23) qquad aH = {(23),(132)} = a'H qquad a'=(132) $$
$$ b=(13) qquad bH = {(13),(123)} = b'H qquad b'=(123) $$

Now, if we had a quotient group what should the product ${(23),(132)}circ{(13),(123)}$ be?

From one perspective we have
$$(23)Hcirc(13)H =^? (123)H$$
But we could also say
$$(123)Hcirc(132)H =^? eH = H$$

But the product of the coset ${(12),(132)}$ with the coset ${(23),(123)}$ cannot be allowed to depend on what we choose to call those cosets. And here we have two calculations that say they should be two different things! So we're in trouble.