In material medium, it's useful to think of it composed of molecules. For example, water molecules, or the molecules of the fibers of a rope. These molecules are attached to each other through intermolecular forces. These forces can be thought of as small springs or rubber strips that connect them.
When you move one of these molecules in a perpendicular direction, away from the rest of the molecules, they drag their neighbours, creating the wave front. These intermolecular forces eventually push the molecule that moved back to its original position, and past it, starting the oscillatory motion.
The neighbour molecules, however, have now moved, too, and they drag their own neighbours on the other side. This is how the wave front advances perpendicularly to the movement you first gave.
The electromagnetic waves is a completely different matter, and probably beyond what you need. I think it's enough to say that the magnetic field H is itself very strange (it doesn't have "charges", like the electric or gravitational fields), interacts in very strange ways with the electric field E. When we apply the energy conservation to these fields, eventually we can arrive at the following equation:
#vec S=vec E xx vec H#
Where #vec S#, called "Poynting vector" is the directional flux of energy, and it's perpendicular both to the electric and magnetic field.