Heating oil, basically diesel fuel was cost effective and has a high BTU (heat) output per gallon used, the only down sides are that the furnace requires annual servicing, the heating oil can smell and the need for a fuel tank.
Above ground tanks were not that common and are easily removed if no longer required, indoor tanks were often placed in the basement before the framing was installed and underground (buried) tanks were generally installed close to the foundation.
Then there are the buried tanks, these are a huge problem. First of all, any fuel remaining in the tank (and the generally is) must be removed, failing tanks can often be filled with a mixture of fuel and water which increases the cost.
Prices range from a few hundred dollars to remove a empty above tank to thousands of dollars to have a buried tank removed.