Pītha Jātaka (No.337)
The Bodhisatta was once an ascetic in the Himavā. One day, having gone to Bārāṇasī for salt and vinegar, he entered the city for alms and went to the house of a merchant with a reputation for piety. However, the merchant was away at the court, and as the ascetic saw no one in the house, he turned and went away. On the way he was met by the merchant, who apologised for his absence and invited him to return to his house.
The story was told to a monk, who, on joining the Order, inquired who looked after the monks. On being told that Anāthapiṇḍika and Visākhā did so, he went to their houses very early the next day. It was so early that no one attended to him. When he returned later it was too late and the food had all gone. Thereupon he started abusing the two families. J.iii.118 ff.