We propose a new algorithm to estimate the phase of speech signal in the mixture of audio sources under the assumption that the magnitude spectrum of each source is given. The pre- vious method, multiple input spectrogram inversion algorithm (MISI), often performs poorly when the magnitude spectro- grams estimated are not accurate. This may be because it im- poses a strict constraint that the summation of source wave- forms should be exactly the same as the mixture waveform. Our proposing algorithm employs a new objective function in which this constraint is relaxed. In this objective function, the difference between the summation of source waveforms and the mixture waveform is the target to be minimized. The perfor- mance of our method, modified MISI is evaluated on two dif- ferent experimental settings. In both settings it improves the audio source separation performance compared to MISI.