He had heard of the Jedi, he just thought they were charlatans, e.g., “A lot of simple tricks and nonsense.” And there could be trillions of people, like Din and Carga, who know nothing about the Jedi or the Force. The stories just tend to focus on people who are better informed.