From a basic standpoint, all of these protocols can be split up into three categories, with certain protocols living in each category.
Layer 1: These protocols use Cat5 cabling simply for the cabling. They do not work with standard switches, hubs, etc. as the protocol differs from traditional Ethernet data. The advantage to this is that they're usually very easy to configure and deploy. A few examples of the protocols that exist in this category are:
AES50 Behringer and Midas (notably the main users)
Digico also uses a proprietary format for the SD9, SD11, and S21 when used with a D-Rack
Layer 2: In my opinion, these protocols are the easiest to implement since they generally use inexpensive switches, hubs, and cabling. They sometimes even play nice on existing data networks with computer traffic. Configuring redundancy can be somewhat challenging, but for most purposes these work exceptionally well. Some examples of these protocols include:
CobraNet Many manufacturers have products with CobraNet, one of the oldest and most widely used.
Ethersound Much less prevalent than CobraNet; Yamaha used Ethersound before switching to Dante.
AVB (selected versions) A newer standard that will hopefully have better interoperability between audio and video equipment manufacturers in the future.
Layer 3: These protocols are typically found in the most high-performance, redundant, and low latency systems. Cabling and networking hardware is usually far more expensive as it must be able to support the higher bandwidth requirements of the robust feature sets. Consumer-grade networking hardware will not generally work. A few examples of these protocols include:
Dante Undoubtedly the most popular new format, most manufacturers feature a product that uses this natively, or converts/interfaces with it in some way, shape, or form.
AVB (selected versions) The same details mentioned above apply here, but there are versions of AVB intended for lower latency, higher sample rates, etc. that require Layer 3 infrastructure.
Q-Lan Found in many QSC installation products.
Which format is right for you? There is no one right answer, as it depends on the application and the direction of primary equipment you want to purchase. For instance, if you are considering a Midas M32, it's best to stick with AES50 since it is natively supported by the console. If you are planning on a Yamaha based system, Dante is the way to go.
The most important tip I can impart is to do your research and fully understand whichever protocol you decide to use for your implementation. Yamaha, for instance, provides an excellent and comprehensive "Dante Network Design Guide" that is available on their website. It offers advice on cables, switches, topology, and overall configuration. Similar guides for each of the protocols I have described can be found online with a simple search.