A hunting permit shouldn't have anything to do with it. Public conservation land would be the same as any other public place.
Unless it's a doc rule specific to firing a rifle when not hunting, which it doesn't appear to be, then this will be the relevant law:
https://www.legislation.govt.nz/act/.../DLM72991.html
This touches on it. Carrying a firearm.
https://www.herengaanuku.govt.nz/kno...i-carry-a-gun/
Just don't endanger, annoy, or frighten any one.
Bookmarks