If you have a dataset which you have evaluated, you can still edit it and make it private again. We should make a decision on if this should be allowed or not (especially considering Gentlecat's evaluation stuff, which I think he said could have private datasets)