Use separate setting for incognito mode:
Enable referrers in incognito sessions

Select "Allow in incognito" to access incognito preferences