If you don’t want to do the settlement quests but are the type of person that finds unfinished quests sitting in their log, next time you turn in the quest, as he starts talking about the next quest, just turn and walk away from him. That’s it. He’ll shut up and you won’t get another quest.