I'd imagine you're not doing anything wrong at all, and what you're seeing is simply the nature of the beans given their origin and the way they were fermented. Most of Vietnamese farmers don't know how to ferment their beans - and while there are a number of concerted efforts to improve the quality, it's a slow process. Vietnamese beans can be very good - they'll be close to west african beans in character. It's impossible to create sourness during roasting/grinding - so it's unusual that you'd detect sour increasing over time.
Remember that fermentation is a very, very complex science - thousands of components are formed. Sourness is from the acids - and there are dozens of acids that are formed, and not all of them are volatile (ie not all of them disappear during roasting). if the beans don't smell / taste of vinegar, then my bet is that you've got beans that are low in volatile acidity, but high in organic acidity (and those don't evaporte no matter what you do) simply due to the way they were fermented. Addition of some baking soda during conching might help mitigate that.
The beautiful thing about chocolate is there's so many variables to play with. If you've still got some beans left, roast them at 20 degrees higher temp for 20% longer time and see what you get. One of the other challenges is that we may not be speaking the same sensory 'language' (ie you may say bitter but mean something completely different than what i mean). It's very, very hard to trouble shoot sensory over the internet w/o first having a shared and agreed upon lexicon of what the words actually mean.