If there's anything I can do well, it's make a wavy edge when I'm whipping with yarn. After doing several rugs, especially ones with a curved edge that needed a lot of steaming to get them to lay flat, I came up with a solution.
This may not be the 'Right' way to finish, but it works for me. It can be used on small mats or wall hangings where the piece will not get a lot of wear and tear. I use a more conventional method on rugs that may be used on the floor.
First I steam my finished piece to get it to lay flat. Then I carefully measure around the mat and mark it 1/2 inch from the hooked loops. Make marks every inch or inch and a half and then connect the dots to make a circle. I then cut on the line. I serge or zigzag with the machine to finish the edge. If you don't have a machine available, you could use glue on the edge. BUT...it would have to be a glue that stays flexible, because the edge is going to be turned up.
I usually use yarn to whip the edge, doubled in a large needle. Now for the solution part- I put a piece of cord in the edge under the whipping. This is cotton cord that I buy in the hardware store in the section where 'packing supplies' are sold.
I then lay it on the edge of the mat and roll the serged edge up over it. I then take my first stitch with my yarn, going down from the top, right against the edge of the hooked loops. Leave a 5 inch tail of cord.
Roll the serged edge up over the cord and the tails and take another stitch right on top of the first one.
Pull your stitch taut and continue on in this manner. After I get about 5 or 6 inches done, I get to the Not Laying Flat Anymore part! See how it's starting to get wavy!
When you get to the end of your yarn, flip your mat to the underside and take a long stitch back under the yarn you just whipped and cut it off.
Continue in this manner until about 2 1/2 inches of unwhipped edge remain. Pull the cord if necessary to make the edge lay perfectly flat. I'm sorry I forgot to take a picture of this part!!!
The cord I used has 3 or 4 strands twisted together. I separate one strand out on each end and tie these two strands together with a square knot. I do it this way to eliminate bulk where I tie the knot. I clip the tied ends off to about 1/4 inch and clip the remaining strands off so they lay flat but not overlapping inside the rolled edge. Continue whipping until the whole edge is covered. Then flip the mat and take a long stitch back through the already whipped yarn and cut off the ends.