mathematics of speciation

In standard evolutionary theory, every species is trying to optimize its fitness function, within a certain environment. But in reality, most of the environment is determined biotically by other species. Let's assume that for simplicity, for now. The situation is each species is trying to optimize its multivariate fitness function, where it only has control over a few variables (geographic location, DNA, etc) and the other species control the rest of the variables (food availability, CO2 concentration, predation, etc). It seems to me that there is only one fitness function since when two species diverge they have the same fitness function and so there is no reason to speculate more than one function with different domains. But one can consider a function of only the variables under a particular species' control, call it the reduced fitness function. These functions can be different for each species, and they are time dependent. The definition of a species is a domain around a local maximum of the reduced fitness function, separated from other maxima by mountain passes. This avoids all the problems encountered when one defines species only by reproductive isolation.Speciation can occur in two different ways.

In the first, there is a temporary reduction in fitness as an emergent species goes downhill for a while, through a mountainpass to another peak.

In the second, two populations which have the same value of the fitness function but are separated a bit by one or a few of the control variables have their reduced fitness function change in different ways due to other species attempted optimization of their reduced fitness function. They both go uphill at all times, with no need to reduce their fitness, unlike in the first case.

It would be neat to come up with actual numerical examples of how this could happen. Perhaps there are topological constraints making certain things impossible. Perhaps it is necessary to introduce an abiotic element (meteors, solar flares, etc) in order for the second situation to be possible.