Evaluating Program Coverage for Code-Model Training

In recent years, CodeLLMs have revolutionized the way developers interact with code. One notable application has been program translation, such as converting COBOL to Java or C to Rust. A critical challenge in this domain is ensuring that CodeLLMs are trained on programs that cover all syntactic features of the target language. This issue is especially pronounced for legacy languages like COBOL and ABAP, which are syntactically rich and have limited availability of open-source programs. In this paper, we present a tool for evaluating the syntactic coverage of COBOL programs. At the core of our approach is a representation called the Coverage Tree, which compactly and intuitively captures the syntactic constructs covered by a set of programs. Additionally, the tool can generate code statements to address uncovered syntactic gaps. Experimental results with COBOL benchmarks demonstrate the effectiveness of the tool.