A quick search tells me you need to open the directory’s properties dialog and click on the icon on the top left; if you want to set directory icons programmatically maybe you can do that, then try and see what Nemo does in order to change it?

Otherwise I’m afraid you’ll need to seek some documentation for Nemo or GNOME Files (since the former is a fork of the latter), or at worst, Nemo’s code itself.