[Debate] Should pretests be strong?

Hello dear codeforces

Today, someone unburied a cadaver for me, that is, "Well, I think pretests should not be a guarrantee" and what not. And I think finally there should be a clear blog post where we should be to debate specifically which is "better*": weak pretests or strong pretests.

I await to see how each matter is argued, hopefully I and others could have something to learn from this,

Francisc

*: from a contest quality and overall quality perspective

← Rev. 4 →  

0

My opinion in this matter is that pretests should be strong (and as such I would really like to see a 'reasonable' argument for weak pretests).

My argument is the following: generally, in problemsetting, there is a saying "The tradeoff to allow those with slightly slow technically optimal solutions while also allowing those with brutally optimised technically unoptimal solutions". The general idea when this is employed is that those that have the right path to the problem should always be encouraged by getting as many points as possible (if one could also encourage further optimisation to that person to cutoff "technically unoptimal" solutions, i.e. by employing a subtask with n = 1e5 and another with n = 5e5 it would be great). I have seen this principle applied to the preparation of very many problems ranging from unimportant to highly important (i.e. TST) by plenty of highly experienced problem setters.

Now, how does this principle apply to the matter at hand? Well, why should anyone discourage further development on the contestant party by weak pretests? What happens when you add weak pretests (**intentionally) devolves into two cases:

First, the case was intended to break a certain foreseeable bug, thing which will inevitably lead to the discouragement od the contestant. Although one should always be aware of what mistakes they can do, it is not fair to imply that everyone should always loow out for bugs after they get Pretests Accepted. Because, then it only begins to be an unnecesarry gamble of time where you have to factor in what is the chance that you do have a bug vs the chance that you are wasting your time and should proceed (because checking later is never an option as codeforces takes the time penalty of the last submission). This leads to a very depressing moment to him who hasn't gambled their time to find the bug, and a frustrating one to him who gambles too much time to yield nothing, or him who gambles time and observes that the pretests did not cover the case which he found.

Second, the case is a very minimal thing that breaks something by pure chance, nothing purely intended in that scope. In this case, I find we can give the problemsetter the benefit of the doubt whether it is a small thing (like a random hash collision coincidence that could have never been thought of) of a big thing (*like not putting some god forsaken tests with maximal limit*). Of course, this is subkective, but the pure fact that it was not done with (maybe ill) intent could be absolved as a mistake and we should carry on.

Another reason why we should enforce good pretests is that if we dont land in the former case (where by some pure chance our code fails), we open the gate to hacking, which is one of the most evil things I ever find imaginable. The reason for this is that when hacked, the test is autocontained for the hacker and the hacked, giving only the hacked the opporunity to autoreflect and to submit again. This creates a significantly big disproportion in ulterior performance only based on factors that are not controllable by us, but by luck (to be assignes into what room). This is yet another gamble which has nothing to do with the essence and true meaning of this sport.

TL;DR, FSTs are the most depressing thing that can happen to anybody. If you fail pretests and ulteriourly not solve the problem, as opposed to FST it, at least you aren't lied to that you did a great job, only to be gut punched in the last moment. The most tragic case is when you realise your mistake only when you find the nature of your counter-example, and the correction was to the degree of the trivial. Getting WA puts anyone I've ever met into a more cautious and alert state, rather than (maybe fake) AC