Elementary arithmetic is arithmetic without sets, or, more precisely, arithmetic with no notation for sets. We can refer to sets indirectly, by means of the expressions which could be used to define them, but we can’t name a set and we can’t quantify over sets. This prevents us expressing concepts like our minimum principle, that every non-empty set of natural numbers has a least member.
We now move on to set theory. Set theory, like first order logic, is generally used as a base for other, more interesting theories. Just as in first order logic we didn’t enquire too closely into the meanings of variables and predicates, in pure set theory we mostly avoid the question “sets of what?” Sets are sets of members. For now that’s all we need to know.
Set theory is weird. To be more precise, it’s weird in two ways. One is that various statements each of which individually seem to be intuitively obvious turn out to be logically inconsistent when combined. This means that any choice of axioms for set theory will necessarily have some unexpected consequences. The other way that it’s weird is that the particular set of axioms which the mathematical world has converged on has somewhat more unexpected consequences than strictly necessary.