Should One Honor a Verbally and Physically Abusive Parent?

Hello Honorable Rabbi, I am married +3, even before the marriage my father was not a simple man. He and my mother would beat me and my sister on a daily basis, abuse us, and prevent us from having friends etc. in childhood. Around 10 years ago he kicked me out of the house, and I had to live for a few days with someone who nowadays is my wife (however then we were not married). This caused us financial distress without any support from my father throughout the years. The weird thing is that despite the fact that he kicked me out of the house, my father comes every day to my house (a short distance from his), without warning (the frequency of visits increased to twice a day when my mother passed away seven years ago), he knocks on the door and I of course unfortunately, because of the mitzvah of honoring ones parents, open the door and let him into my home (my wife also forwent on her honor, however it is reaching a breaking point). From the moment my father enters the door begins a terrifying experience – he curses, speaks evil of us and others, insults my wife (a Jewish olah chadasha) calling her a gentile, refuses to go despite our requests and continues to fight with us while he is there. My eldest son unfortunately grew up in this atmosphere, learned harsh curse words from his grandfather, and all the time hears arguments (child on the autism spectrum). My eldest brother protects his home and distances his children from their grandfather (he lives in another city and there his wife from the first day ensured distance between him and the family, and like him my younger brother also cut ties with him). I unfortunately, because of the situation that we have reached, am still in the same city and neighborhood as my father, therefore he is always with us. My youngest daughter is now 3 and growing up with the atmosphere of curses and fights that my father brings up about people that aren’t related to us, and we ask him to stop but it does not help. In the last week I decided that I am not opening the door for him anymore, but I want to ask if I am not transgressing the mitzvah of honoring one’s parents (I’ll remind you that my father’s visit causes problems at home, in my relationship, and in my children’s education, it will already be difficult to fix their education with the words that they learned). The issue is that to get to him we need to go up to the third floor with no elevator, and therefore it hurts me that I am not opening the door for him – but once again, this is an unexpected visit, at random hours, with no consideration for us or our children at all. My father is not a nice person, there is almost no person in our city that hasn’t merited an argument with him, including close neighbors – he even brings up and curses the positive relationship that we have cultivated with the neighbors in our building. What should I do? Thanks in advance.