HOMER calculates payback by comparing one system with another. In general, payback tells you how many years it takes to recover an investment. You invest a certain amount of money initially, then earn income from that investment. The payback is the number of years it takes for the cumulative income to equal the value of the initial investment. HOMER can also calculate other economic metrics such as Internal Rate of Return (IRR), present worth, and return on investment.
Sometimes, the "income" of a power system can be defined intrinsically, without a reference for comparison. If, for example, a grid-sales system does not need to serve any load, but simply sells power to the grid, you can define the income of the system as the money it makes by selling power to the grid minus the expense of operating the system. In that case, you can compare the initial cost of the system with its income to calculate the payback.
Distributed power systems are typically not as simple, and the income must be defined relative to some alternative. Say you are designing a system to provide electricity to an off-grid house. A pure diesel system might have low capital cost and high operating cost, whereas a PV-diesel-storage system might have high capital cost and low operating cost. Neither system produces any income. In both cases you spend money initially to build the system, then you keep spending money each year to operate the system. The concept of payback has meaning only if you compare one system to the other. You can define the payback of the additional investment required for the PV-diesel-storage system by comparing the difference in capital cost with the difference in operating cost. HOMER does exactly that under the Compare Economics tab.
To calculate the payback of one alternative, we must compare it to another alternative. Even to calculate the payback of the simple grid-sales system mentioned above involves an implicit comparison with the base case, which is to do nothing. The do-nothing alternative is easy to compare because all its costs are zero. However, if an electric load must be served, the do-nothing alternative is not acceptable.
A renewable power system designer often uses a non-renewable power system as the base case, but HOMER lets you choose any base case you want. You can compare a system with two wind turbines to a system that has one wind turbine to calculate the payback of the second wind turbine. Or you can compare a PV-diesel-storage system with a PV-diesel system to calculate the payback of the storage.