Pretty sketchy hypothesis, would need much fleshing out to be an actual argument. To exhaust soils you need to plant them intensively, and of course intensive planting IS agriculture. Soil exhaustion is a fairly common consequence of sedentary agriculture, which allows large enough harvests to increase human population and thus demand on limited acreage, that limit being how far its practical to walk to a field and work it and perhaps protect it from animals.

There are many examples of early agricultural societies either exhausting their soils or 'salting" them through irrigation to the point of societal collapse or migration.