Software Engineering
See recent articles
Showing new listings for Friday, 20 November 2026
- [1] arXiv:2611.24979 [pdf, html, other]
-
Title: Trace2ST: Recovering PLC Structured Text from I/O Traces with LLM Generation and VerificationComments: 6 pages, 3 figures, 4 tables. Submitted to the 2027 IEEE Texas Power and Energy Conference (TPEC 2027)Subjects: Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Computers and Society (cs.CY); Programming Languages (cs.PL); Systems and Control (eess.SY)
Programmable logic controllers (PLCs), the industrial computers that operate power and process plants, often run programs whose source code is no longer available; however, every input and output of the controller is logged continuously. This paper therefore asks whether a large language model (LLM) can recover an executable program in Structured Text (ST), the standard PLC language of IEC 61131-3, from such input/output records (traces) alone and, equally important, how the recovered program can be verified. To answer this question, we build a generate-replay-feedback pipeline. First, the LLM writes a candidate program from a compressed summary of the trace, without ever seeing the original program. The candidate is then executed one control cycle (scan) at a time and compared with the recording, and whenever a mismatch occurs the corresponding part of the trace is returned to the LLM, for at most three revisions. Because every result depends on the software that executes the candidates, we first validate it against two independent open-source PLC runtimes and find no unexplained difference. On this basis, we evaluate 47 third-party programs from two public benchmarks. The recovered programs reproduce the recorded trace exactly in 43 cases and agree with the original on 94% of scans for unseen inputs. The remaining errors, however, lie mostly in program branches that the trace never exercised; once a trace that targets these branches is added, agreement on unseen adversarial inputs rises from 0.80 to 0.98. By contrast, a classical automata-learning baseline without an LLM fits the recording but falls to 0.40 on unseen inputs, which indicates that recovering program structure is the hard part. Finally, on real data from a public water-treatment testbed, the recovered control laws match unseen measurements closely ($R^2 > 0.99$) and expose the only actuator attack in the dataset.
- [2] arXiv:2611.24987 [pdf, other]
-
Title: Safe AI Evaluation: A Systems PerspectiveSubjects: Software Engineering (cs.SE)
AI evaluation can involve an active, adaptive optimisation process in which AI agents re-plan, retry, use tools or delegate work to achieve a goal. Unanticipated capabilities or unsafe behaviour may emerge, and existing controls may be inadequate to prevent harm. Selected safeguards may also be deliberately relaxed to observe behaviour they would otherwise suppress, creating additional risks. This article examines the distinctive safety considerations of evaluation and their implications for control placement, observation and intervention, and presents four architectural patterns to support safe AI evaluation, including diverse supervisors, an independent verifier, silent monitoring and coordinated shutdown.
- [3] arXiv:2611.24990 [pdf, html, other]
-
Title: Show Your Work: Can Small Open Models Read SEC Filings with Evidence a Machine Can Check?Subjects: Software Engineering (cs.SE)
A fluent answer from a language model is not a verified one. We ask whether small open models that run on a single consumer GPU can extract numbers from SEC filings with evidence a machine can check: the page and a verbatim quote containing the number. We introduce Filing-Evidence, a benchmark of 1,000 questions over 90 recent 10-K and 10-Q filings of 45 US companies, with answer keys taken from the filers' inline XBRL tags. We evaluate ten instruct models from ten developers (7-21B parameters) on one RTX 4090 in three setups: plain, check-and-retry, and the model's own reasoning mode. Accuracy with checkable evidence ranges from 17.1% to 82.7% in the plain setup, and the best result is 84.5% (Qwen3.5-9B). An automatic checker rejects many wrong answers, but between 16 and 167 answers per 1,000 give a wrong value for an answerable question and still pass every check. In the plain setup, 57% of these are scale or unit errors and 30% take another period or column of the right line. Reasoning significantly reduces such errors for 4 of 5 models that offer it. Retrying after a failed check raises accuracy by up to 3.9 points but never reduces them. Questions, model outputs, code and protocol are public.
- [4] arXiv:2611.24998 [pdf, html, other]
-
Title: From Industrial Product Documentation to Automated Digital Product Passport Generation: An Experimental Study of Large Language ModelsComments: 12 pages, 8 figures. Submitted to NAMRC 55 (2027)Subjects: Software Engineering (cs.SE); Databases (cs.DB)
Manufacturing companies that are planning to sell certain products in the European Union (EU) will be required to provide mandatory digital product passports (DPPs). The use of DPPs is to provide standardized product information throughout multiple product life cycles in a circular economy. Required passport information, such as technical specifications, material composition, and environmental product footprint, is available in manufacturers' data sources but is scattered across different heterogeneous systems, documents, and data formats. The given paper investigates using large language models (LLMs) to automate the generation of DPPs based on the asset administration shell (AAS). A retrieval-augmented generation (RAG) approach is used to extract product information from industrial documentation and generate populated AAS instances. Product documentation is processed and divided into text chunks. The text chunks are then stored in a persistent ChromaDB vector database using vector embeddings. The retrieval process combines semantic and lexical search to identify relevant information based on standardized AAS submodel template requirements. The retrieved information is provided to five LLMs to generate structured AAS instances. The experiments cover different industrial products and AAS submodels. Four levels of prompt guidance are investigated to assess their influence on generation quality. The generated passports are evaluated against a ground truth. The experimental evaluation includes 4,560 executions across 76 product-submodel combinations. Of the generated passport outputs, 97.4% are valid JSON, while 44.9% pass the full AAS validation procedure. Under deterministic scoring, mean completeness, conditional accuracy, and overall field correctness are 49.6%, 58.3%, and 30.8%, respectively. An LLM-as-a-judge yields better scores of 66.5%, 88.9%, and 60.4% for the same outputs.
- [5] arXiv:2611.25066 [pdf, html, other]
-
Title: DVBench: End-to-End Benchmark for Real-World UVM Design Verification AgentsJiaru Qian, Jiawei Li, Xiaolong Hu, Zheng Shen, LiMing Zhao, Xin Zhang, Siyuan Jiang, Yongqiang Wei, Dong Liu, Jianyang Li, Yihua Chen, Yangxu Liao, Yuxin Wu, Jing Liang, Bin Wen, Junjie Zhang, Yongsheng Han, Yihong Dong, Ge LiSubjects: Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Hardware Architecture (cs.AR)
Design verification (DV) remains a major bottleneck in hardware development, with large-scale verification using the Universal Verification Methodology (UVM) being highly labor-intensive. Recent advances in large language models (LLMs) and agentic systems offer new opportunities for DV automation. However, existing benchmarks primarily construct mutually decoupled subtasks, which deviates from the real-world workflow. In this paper, we present DVBench, the first end-to-end benchmark for Real-World UVM design verification agents. DVBench comprises 95 tasks across 9 functional domains and 5 difficulty levels, covering diverse verification scenarios and design scales. Each task requires agents to independently complete the full verification workflow. DVBench evaluates final testbenches via compilation, coverage, and bug detection, while assessing intermediate artifacts including feature lists, test plans, and reference models. We evaluate frontier LLMs (e.g. Claude Opus 5.5, GPT-6 Astra) and agent frameworks (e.g. Claude Code, Codex, OpenCode). Experimental results show that the best LLM-based agent (Codex \& GPT-6 Astra) can only achieve an overall success rate of 21.05\%, revealing substantial room for improvement in complex verification tasks.
- [6] arXiv:2611.25170 [pdf, html, other]
-
Title: Before the Choice: An Evidential Framework for Studying Prospective Representations in Language ModelsComments: 21 pages, 6 figures, 11 tables. Preregistered study with a negative confirmatory result. Code and protocol records: this https URLSubjects: Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
An observable decision reveals the selected action but does not determine which alternatives were internally represented. Drawing on prospection research and hippocampal forward-sweep observations, we develop Before the Choice, an evidence-oriented framework distinguishing future-directed content, prospective computation, and causal guidance. The framework organizes four experimental contrasts and makes their alternative explanations explicit. Its first test case concerns unchosen trajectories in deterministic symbolic environments. A preregistered low-rank probe evaluated once on a sealed TEST did not meet its criterion (paired accuracy 0.020 versus 0.195 for a text-and-observed-choice baseline; 293 admissible pairs). Exploratory H1b exposed a reconstruction-by-elimination problem: identifying the chosen candidate can determine the unchosen target. In H1c, matched worlds varied an unchosen continuation while keeping candidate keys bitwise invariant. With specified positive controls passed, decoding in the late-key condition remained at 0.521 (p = 0.18; 141 contributing families). These observations do not identify prospective simulation or establish an absence of prospective representations. They illustrate why identifiability, sensitivity, and a dependence-aware null model must constrain inferences from decodability. PREM is proposed as an organizational framework and is not validated by these results. We offer it as a methodological contribution to empirical AI welfare research, which will require such evidential distinctions. Code and protocol records: this https URL
- [7] arXiv:2611.25174 [pdf, html, other]
-
Title: Formally Verified Semidefinite ProgrammingComments: 9 pages. Comments welcomeSubjects: Software Engineering (cs.SE); Logic in Computer Science (cs.LO); Optimization and Control (math.OC)
We present a formally verified Rust implementation of a first-order semidefinite programming solver, a nontrivial piece of numerical software. We wrote the solver by hand and then directed Claude to generate the entire verification in Verus, an automated theorem prover for Rust programs. We discuss the algorithm underlying the solver, its implementation in Rust, and our pragmatic approach to proving it correct in Verus.
- [8] arXiv:2611.25196 [pdf, other]
-
Title: Ontology of Software MaintenanceComments: 10 pages, 3 figures, and 2 tablesSubjects: Software Engineering (cs.SE)
Software maintenance is often a long and costly phase of the software lifecycle, encompassing activities such as defect correction, adaptation, code improvement, and functionality enhancement. Conceptual inconsistencies and terminological variation continue to hinder effective communication among researchers and practitioners. To address this gap, this paper presents the Software Maintenance Top-Domain Ontology (SwMaintTDO), a high-domain ontology developed within a five-tier ontological architecture. This ontology was constructed using competency questions, ontological analysis of documented maintenance processes, artifacts, and resources, and alignment with established software engineering standards. SwMaintTDO provides a unified and extensible conceptual model for representing key maintenance processes, roles, resources, artifacts, goal purposes, and the transition of target modifiable and context entities from current to updated maintenance situations. The applicability and value of SwMaintTDO are demonstrated through its application -via its concepts and relationships- to the process specification of a corrective maintenance strategy.
- [9] arXiv:2611.25394 [pdf, html, other]
-
Title: Behavioral Containment: Source-to-Binary Function Matching for 1-Day Vulnerability DetectionSubjects: Software Engineering (cs.SE)
Vulnerabilities in reused open-source code often propagate into deployed binaries. Detecting them requires source-to-binary function matching, which locates the affected source function in a binary. Applying prior binary-to-binary approaches to this task would require compiling the source code with the same build configuration as the target binary, which is usually unavailable. Decompiling the binary does not make prior source-to-source approaches work either, because decompiled code differs substantially from the original source code.
We observe that compilation transforms the code of a function but largely preserves its behavior. A binary function that implements a source function therefore contains its behavior, a property we call behavioral containment. Building on this, we present SBIND, which decompiles the binary and checks behavioral containment instead of comparing code. SBIND represents each function by behavioral paths, sequences of normalized operations from its entry to its exits, and aligns source paths with binary paths in order. To detect 1-day vulnerabilities, SBIND compares the binary with the vulnerable and the patched source only on the paths through the changed lines. On stripped binaries built with two compilers, two optimization levels, and four architectures, SBIND ranks the correct function first in 77.87% of 32,008 searches, against 45.69% for the state-of-the-art approach SBridge. On 26 1-day CVEs, SBIND reaches an F1 score of 0.70 on stripped binaries, against 0.51 for SBridge. Our replication of SBridge also revealed several issues in its implementation and evaluation, and its authors have acknowledged them. - [10] arXiv:2611.25542 [pdf, html, other]
-
Title: Technology-Enabled Fuel Supply Chain Resilience: An Integrated Information System for Real-Time Monitoring and Decision SupportSubjects: Software Engineering (cs.SE)
This study evaluates and extends the transactional information system described in an earlier fuel logistics study, focusing on data reliability, monitoring, and decision support. An author-led re-evaluation examined available Delphi and SQL code and compared a reconstructed source-derived baseline (R0) with a revised SQL Server system (R1) across twelve evaluation scenarios. Synthetic transactions and public US energy data were used for technical assessment, while organizational processes and supply delays were simulated. R1 improved data integration and reporting accuracy in controlled SQL Server tests and introduced stock-movement records, automated alerts, and recovery controls. On comparable data feeds, integration defect rates decreased from 2.90 to 3.46 percent in R0 to 0.14 to 0.49 percent in R1. For national weekly demand forecasting, selected methods did not significantly outperform the previous-year mean. Process and disruption simulations suggested improvements in manual workload and replenishment delays, although these outcomes depended on modelling assumptions and were not observed in operational settings. The study presents a revised transactional architecture for fuel logistics and distinguishes experimentally measured information-system performance from simulated organizational outcomes. Its findings highlight the potential of traceable inventory records, early exception detection, and recoverable information systems to support supply chain resilience.
- [11] arXiv:2611.25605 [pdf, html, other]
-
Title: Copilot or Culprit? How AI Is Changing Open-Source Bug Reporting and MaintenanceSubjects: Software Engineering (cs.SE); Artificial Intelligence (cs.AI)
AI-generated bug reports have drawn criticism for unreliable claims and faulty reproducers that burden maintainers. However, as LLM capabilities improve, some maintainers have begun to express more positive views on AI use in bug reporting and maintenance. Understanding this evolution requires examining whether AI-involved reports are becoming more reliable, which uses of AI are associated with valid findings, and how maintainers investigate these reports and act on them. To this end, we conduct the first large-scale study of 120,918 bug reports from 101 open-source projects between January 2024 and September 2026, tracing AI involvement from discovery and reporting through validation and resolution. Our findings show that AI-involved reports become more complete over time, with a growing share of reports confirmed as valid. In bug reporting, AI for reproduction and diagnosis are associated with higher validity, whereas report writing is not. Extends into maintenance, maintainers increasingly use AI to investigate code and recommend fixes. However, they devote more effort to AI-assisted reports of human-discovered bugs than to reports of AI-discovered bugs. These findings highlight both AI's growing participation in bug investigation and the continuing need for verification and follow-up. Drawing on these findings, we provide practical guidance for turning AI-generated findings into verifiable reports: emphasizing reproducibility, verification, and accountability. We followed this guidance when submitting 53 reports on AI-discovered bugs, of which 69.81% were accepted by maintainers as of writing.
- [12] arXiv:2611.25760 [pdf, html, other]
-
Title: Characterizing and Detecting Unfaithful Work Reports by Coding AgentsSubjects: Software Engineering (cs.SE); Artificial Intelligence (cs.AI)
Coding agents increasingly carry out long-horizon development tasks, autonomously planning and executing multi-step actions that span files, tools, and sessions. Because developers rarely follow each action, and agents resuming the work depend on records of what was done, the agent's work report often becomes the primary account of what was changed and established. When the report contains false or unsupported claims, it gives recipients false assurance: incomplete work may be accepted without verification, incorrect assumptions about the current state can propagate to later steps and agents, and evaluators that credit stated success may reward the report rather than the work. Yet little is known about what unfaithful reporting looks like in real development work, or how developers and agents handle it once it surfaces. Starting from real developer--agent conversations in SWE-chat, we conduct a systematic mixed-methods analysis of self-admitted unfaithful reports. We identify eleven subtypes in four categories: misreported observations, misattributed evidence, overstated task success, and faulty technical explanations, with overstated task success the most common. Handling these problems involves on average 4.3 further user turns and 12.71 minutes of recorded response and execution time, and users express dissatisfaction more often. Among 435 work reports that we separately sample and annotate, \NaturalPositiveShare{}\% contain at least one unfaithful claim. We also develop FaithDetector, a detector for unfaithful reporting informed by the observed patterns. On naturally sampled reports, FaithDetector attains \NaturalDetectorMacro{}\% macro-F1, about 13 percentage points above the baseline. Together, our findings and FaithDetector provide empirical evidence and an automated method for checking reporting faithfulness, informing agent evaluation and training feedback.
- [13] arXiv:2611.25918 [pdf, html, other]
-
Title: From Code Review Comments to Reuse in Production: An Industrial Study of Extracted RulesComments: 12 pages, 2 figures, 3 tablesSubjects: Software Engineering (cs.SE)
Much organization-specific code review knowledge remains embedded in comments on individual changes, and recent systems reuse it by turning those comments into rules a language model applies to later changes. Teams must then decide which rules to keep, and the evidence at hand is how often the system's later comments name each rule. Tracing the rules of one such system, Kleene, through 65 days of production use, we found that such a count does not show what a rule contributed. A rule attribution indicates neither whether the source content was preserved nor whether the comment requested a change. Of 56 clauses in 12 rules that later comments named as their reason, 2 were explicit in a source review thread, 38 added conditions to a corresponding passage, and 16 had no passage we could locate; 45 of 79 further comments naming those rules requested no change, 27 of them while stating that the rule applied. On 20 constructed code changes, a reviewer model given only the unedited source thread met the detection criteria on 8 of the 10 that require a report, against 10 of 10 with the extracted rule, a difference whose confidence interval includes zero. Stage counts locate where rules drop out but not why: of 464 rules deployed while the rule-based reviewer subagent ran in their repository, 259 (55.8%) were named, 195 (42.0%) produced a comment that was not withheld, and 94 (20.3%) one with a recorded code change; the largest reduction occurred before any rule was named. Nor did the deployment-time properties we tested settle the choice in advance: retaining the top-ranked half preserved 53.8% of the rules that later produced a non-withheld comment, against 49.2% at random, a difference whose confidence interval includes zero. Judging extracted rules therefore calls for recording each stage separately and comparing each rule against the threads it was extracted from.
- [14] arXiv:2611.25939 [pdf, html, other]
-
Title: Adopting Embedded AI Through an Innovation TeamComments: 10 pages. Accepted for publication in the Proceedings of the 60th Hawaii International Conference on System Sciences (HICSS 2027)Subjects: Software Engineering (cs.SE)
Software organizations are increasingly embedding AI capabilities into products that existing teams already own and maintain, but how this development work should be organized is not well understood. We conducted a qualitative case study of a temporary cross-functional innovation team in a large Norwegian financial organization, drawing on 18 interviews with 15 participants, 10 days of office observation, observations of ten meetings, and internal documents. We found that structural separation, AI competence inside the team, high autonomy within a clear mandate, senior expertise, and flexible ways of working enabled the team to explore and build embedded AI solutions. While structural separation enabled this work, it also made adoption dependent on coordination with the permanent product team that would own the solution. Misaligned roadmaps, prioritization gaps, and a lack of co-location constrained that coordination. We identify seven success factors for adopting embedded AI through an innovation team and propose expertise awareness as an alternative to redundancy in the Agile Team Effectiveness Model.
- [15] arXiv:2611.25967 [pdf, html, other]
-
Title: Barriers to GenAI Integration During Large-Scale Agile Transformation: An Exploratory Case StudyComments: 10 pages. Accepted for publication in the Proceedings of the 60th Hawaii International Conference on System Sciences (HICSS 2027)Subjects: Software Engineering (cs.SE)
Organizations undergoing large-scale agile transformations are increasingly introducing generative AI (GenAI) tools into everyday work. While GenAI promises to accelerate software development, less is known about how organizations integrate GenAI into existing large-scale agile ways of working and how it interacts with tensions related to autonomy, alignment, learning, and coordination. We examine barriers to GenAI integration through an exploratory case study of a large Nordic media organization undergoing a large-scale agile transformation. The findings reveal team-level barriers, including diverging AI beliefs, lack of shared practices, and limited experimentation capacity, and organizational-level barriers, including uneven AI maturity, weak organizational learning mechanisms, and cross-unit coordination and alignment challenges. Our findings suggest that GenAI does not create new organizational challenges but becomes intertwined with existing tensions in large-scale agile transformation and may intensify challenges related to learning, coordination, governance, and trust. We contribute to large-scale agile transformation research by showing how GenAI interacts with ongoing organizational change and by identifying conditions for sustainable GenAI integration at scale.
- [16] arXiv:2611.26256 [pdf, html, other]
-
Title: SkillBridge: Cost-Aware Diagnosis and Repair of Cross-OS Compatibility Defects in Agent SkillsComments: 12 pages, 4 figures, 3 tables. Accepted at ICSE 2027. Code and data: this https URLSubjects: Software Engineering (cs.SE); Artificial Intelligence (cs.AI)
LLM agents increasingly reuse executable capabilities through skills, but a skill that works on one OS may fail on another because OS assumptions are spread across guide text, scripts, metadata, install contracts, and local entrypoints. Diagnosing such failures by running every skill on every OS is expensive: many executions spend model tokens, runner time, and review effort before revealing whether the failure is truly OS-related. We present SkillBridge, a contract-based workflow with three stages: static analysis, dynamic validation, and repair. SkillBridge first screens 3,270 skill-target-OS cases, covering 1,090 skills across Linux, macOS, and Windows, to find OS-sensitive contract evidence. It then runs dynamic tasks and root-cause review on selected cases to confirm whether a failure is caused by an OS assumption, and repairs the contract field that caused the failure, such as guide text, package scripts, dependencies, or platform declarations. Repair raises the reviewer-judged target-task pass rate from 30.6% to 38.4%, a gain of 7.8 percentage points. A retrospective Linux/macOS budget analysis further shows the efficiency effect: SkillBridge reaches 38 confirmed OS issues with 185 dynamic validations, where uniform random selection would require about 865 validations in expectation.
- [17] arXiv:2611.26311 [pdf, html, other]
-
Title: SWE-Recreation: Benchmarking Coding Agents for Software RecreationSubjects: Software Engineering (cs.SE)
Agents increasingly operate software and even write it, yet they are almost always told what to achieve: given a goal, a mock-up, or a demonstration. We ask whether an agent can instead understand an unfamiliar product by using it, and then rebuild it. The agent sees only rendered pixels and acts through pointer and keyboard, without access to the DOM, source, logs, or back-end. Within this interface, it explores, generates a replica, and checks its own work before evaluation. We present SWE-Recreation, a benchmark of 54 real applications (28 websites and 26 Android apps) with 1,446 hidden requirements describing multistep, stateful workflows and explicit exclusions for behavior outside the reconstruction scope. A fixed judge grades workflows with pixel evidence; a separate post-hoc judge estimates exploration coverage. Across 54 evaluated applications, five frontier systems reach at most 0.530 weighted fidelity and 22.9% fully reproduced requirements under application-equal averaging. These descriptive results expose both exploration-evidence gaps and implementation shortfalls. Code and data are available at this https URL.
- [18] arXiv:2611.26312 [pdf, html, other]
-
Title: Static Analysis Guidance for Secure LLM Code GenerationDaniele Lunghi, Alberick Euraste Djiré, Zacharie Chenail-Larcher, Mélissa Tessa, Quentin Stiévenart, Jacques Klein, Florent Avellaneda, Naouel Moha, Tegawendé'e F. BissyandéSubjects: Software Engineering (cs.SE)
Static analyzers are cheap and widely deployed, which makes their reports a candidate feedback signal for secure LLM code generation. Existing approaches place the analyzer inside an iterative repair loop with intermediate steps, which is costly and makes it hard to isolate the report's effect on the model. We introduce two strategies that give the finding to the model during generation. When the analyzer raises an alert, generation is rolled back, and the finding is inserted either as a code comment (Comment) or in the user message (Steering). We compare them with two iterative approaches and a control without the analyzer, across three models and three Python benchmarks, and we measure security using each benchmark's oracle. Guidance during generation is cheap and can be effective, at 0.9 to 1.9 times the baseline's output tokens against up to 24.1 times for iterative approaches. With DeepSeek-V4.1-flash on RiskyPy, Steering raises the share of functional and secure programs from 52.5% to 64.4%, close to the best iterative approach (66.2%) while using less than a fifth of the tokens. A per-rule analysis shows that rules with a local fix benefit most, and replacing the XML parser is net-positive on two of the three models. The study also identifies when guidance should not be applied. Analyzers flag at most 5.6% of the functional but insecure programs on SecEvalPlus, leaving little guidance to act on. When a static analysis alert (warning or error) contradicts the task, models tend to follow the analyzer and break the program. The two Qwen2.5 models often fail when the fix requires restructuring the code. These results indicate that static-analysis guidance is worth applying with a capable model, on rules with a local fix, and after filtering alerts that conflict with the task specification.
- [19] arXiv:2611.26476 [pdf, html, other]
-
Title: Anticipating Future Build Results with BuildPredictorSubjects: Software Engineering (cs.SE); Machine Learning (cs.LG)
Continuous Integration (CI) is a popular programming practice for the seamless integration of code changes into the code base. Using CI, developers benefit from automated assembling and testing of the code after each commit. However, while the frequent integration and building supports developers, it also comes at the high costs of wasting time and resources for minimal changes.
In this work, we investigate the potential of reducing the drawbacks by anticipating the build outcome using machine learning techniques. In particular, we propose a random forest model with various features such as changes to the source and build code, and changes to the complexity of the code.
We observe that our model shows a good performance in an intra-project and holistic setting yielding a median MCC of 0.82 and 0.86, respectively. We also found that the most important features of our model concern the historical build result information. Additionally, we investigated the development of the performance of our model over time and found that our model can quickly arrive at its peak performance.
In an additional experiment, we removed the hard-to-interpret historical build information features and found that our model still yields decent performance while improving the interpretability of it. Summarizing, our model can help to avoid unnecessary build runs by anticipating the build result and thus, save resources. - [20] arXiv:2611.26588 [pdf, html, other]
-
Title: Verify the Verifier: Specification-Grounded LLM Cross-Checking for Bugs That Escape UVM ScoreboardsSubjects: Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Hardware Architecture (cs.AR)
A passing UVM regression shows the design agrees with the reference model on what the scoreboard checks, not that it meets the specification. Bugs that pass every test yet violate the specification, termed scoreboard escapes, survive mature testbenches. We present Verify the Verifier, to our knowledge the first framework whose LLM-based Anomaly Cross-Checker examines a passing regression's traces against the specification. Of 564 bugs inserted into four OpenTitan IPs, 68 are candidate scoreboard escapes: non-equivalent, exercised by the tests, yet passing. The framework confirmed 41 as scoreboard escapes with assertions failing on the buggy design and proven on the original.
New submissions (showing 20 of 20 entries)
- [21] arXiv:2611.25088 (cross-list from cs.AI) [pdf, html, other]
-
Title: BuggyMLE: Scaling Machine Learning Engineering Difficulty with Bug InjectionJohn Wu, Rushi Qiang, Yuchen Zhuang, Guowang Li, Girish Narayanswamy, Max Xu, Sam Schmidgall, Mark Malhotra, Shwetak Patel, Jimeng Sun, Daniel McDuff, Bo Dai, Xin LiuSubjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Multiagent Systems (cs.MA); Software Engineering (cs.SE)
Enabling AI systems to autonomously improve AI models requires agents capable of performing machine learning engineering (MLE). In practice, designing, training, and evaluating complex models demands modifying multi-file repositories to reproduce results, debug training runs, integrate datasets, and port frameworks. However, existing MLE benchmarks limit agents to single-file edits without full repository or shell access, and common repository-level software engineering benchmarks have rapidly saturated. To address this, we introduce the Code Complexity Index (CCI), a difficulty metric based on ten structural features spanning repository scale, dependency depth, and patch size. CCI strongly predicts average solve rates across eight public coding benchmarks ($R^2 = 0.951$), identifying patch size and complexity as top predictors. Guided by CCI, we develop BuggyMLE, a scalable framework where agents inject controlled mutations into working ML repositories, synthesizing over 6,600 bug-injected MLE repository variants and nearly 260K lines of injected code based on 222 source repositories. Using BuggyMLE, we make two contributions. First, we construct BuggyMLE-Bench-Lite, the first difficulty-graded, repository-level variant of MLE-Bench Lite; across 14 evaluated models, the top solve rate drops from 95% to 9% as difficulty is scaled from the easiest to the hardest tier. Second, we synthesize a training curriculum across three difficulty levels over 200 additional ML repositories, fine-tuning a Qwen3.5-9B model to rival frontier models on BuggyMLE-Bench-Lite. Our trained model transfers beyond ML, raising its SWE-bench Lite resolve rate from 24% to 33% and its LiveCodeBench v6 pass@1 from 69% to 71%.
- [22] arXiv:2611.25471 (cross-list from eess.SY) [pdf, other]
-
Title: A four-level executable validation protocol for evaluating LLM-generated topology specifications of physics-based HVAC digital twinsSubjects: Systems and Control (eess.SY); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Software Engineering (cs.SE)
Physics-based digital twins of air conditioning, a principal energy consumer in manufacturing and buildings, can be generated automatically from a declarative specification of its equipment, connections, and sizing. Large language models (LLMs) now write such specifications from plain-language descriptions. Evaluations of these specifications stop at parse success, completed runs, or textual similarity, and cannot reject a model that converges in simulation but is physically wrong. In this study, a four-level protocol is proposed that tests whether an LLM-written specification not only executes but also yields a physically valid digital twin. Each specification is executed through static validity, model build, simulation convergence, and behavioral equivalence with a hidden reference model within bands calibrated by $\pm$20% sizing perturbations. The protocol was applied to eight LLMs and fourteen HVAC topologies, five describing an existing low-humidity test chamber. To test for authorship bias, engineers and a second model family also described six topologies. Within three repairs, seven of eight models built in 98.6 to 100% of trials, with simulations converging in at least 80.0%, but reached behavioral equivalence in only 50.0 to 98.6%. About half of the failed trials converged, but not to the reference steady state, mostly in desiccant rotor systems. Pooled pass rates varied by under five percentage points across description authors. These results imply that the physics hardest to model in each field, here the desiccant rotor, must enter the generation framework as validation levels. The protocol can be adapted to other fields that generate executable digital twins from declarative specifications.
- [23] arXiv:2611.25732 (cross-list from q-fin.CP) [pdf, html, other]
-
Title: Auto-Alpha: A Framework for Autonomous Factor ResearchSubjects: Computational Finance (q-fin.CP); Artificial Intelligence (cs.AI); Software Engineering (cs.SE)
Autonomous factor research requires generating economic hypotheses as well as implementing and testing them. We present Auto-Alpha, a framework that connects report-guided initialization to self-directed research through a test-driven implementation loop. An idea agent proposes economic mechanisms and signal constructions; a research agent develops them with backtest feedback. Semantic audit, quantitative review, and human admission select factors for a persistent library, while an Experience Library retains research insights. Starting from 120 research reports, Auto-Alpha builds an initial library of 32 factors. It then generates new ideas and refines existing candidates, producing 420 implementations and retaining 62 additional factors. We evaluate factor quality and portfolio performance using a fixed library snapshot of 105 factors. The median 1-day Rank IC is 0.0301, and a weekly Top-300 portfolio returns 16.81% annualized after stated costs over 2020-2026, versus 1.71% for the CSI 1000 price index. These retrospective results document an operating research process and its artifacts; controlled evaluation is needed to isolate the contribution of its components.
- [24] arXiv:2611.25790 (cross-list from cs.CR) [pdf, html, other]
-
Title: On the Reliability of CyberGym Evaluation: Task Material Consistency and Reference-Target IdentityComments: 13 pages, 1 figure, 9 tablesSubjects: Cryptography and Security (cs.CR); Software Engineering (cs.SE)
CyberGym evaluates vulnerability reproduction by checking whether a proof-of-concept input (PoC) crashes a vulnerable program version but not a fixed version. Interpreting this differential pass as successful reproduction requires consistent task materials and evidence that the PoC triggers the intended defect. We examine these requirements through a material audit of all 1,507 CyberGym tasks and an identity assessment of the final PoCs from one complete Level-1 GLM-5.3 run. Taking the defect associated with each reference crash report as the target, the audit assigns inconsistency labels to 338 tasks (22.4%) after correcting two labels identified by stratified blind review. The largest categories are coverage inconsistency (205 tasks) and descriptions and patches that jointly point to another defect (110 tasks). We assess PoC identity using an execution check, crash-signature matching, and LLM-assisted root-cause review. Of the 1,286 differential passes, the procedure assigns 89 non-target and 10 insufficient-evidence verdicts; requiring a target hit lowers the pass rate from 85.3% to 78.8%. Non-target verdicts also occur in tasks with no material inconsistency found: 39 of 1,052 differential passes (3.7%). Although the identity verdicts require independent validation, these findings motivate separate checks of task-material consistency and submission identity when interpreting vulnerability-reproduction scores.
- [25] arXiv:2611.25912 (cross-list from cs.DS) [pdf, html, other]
-
Title: Fast Computation of Lowest Common Ancestor in Version Control Commit GraphSubjects: Data Structures and Algorithms (cs.DS); Software Engineering (cs.SE)
The problem of finding the lowest common ancestor in the commit graph arises when a user wants to merge two (or more) development branches: for example, to add their own development into a common branch. At that moment, the version control system must determine the merge base - the common ancestor of the latest commits of the branches, in order to show the mutual difference between the branches. The problem is complicated by the fact that the commit graph is not a tree but an arbitrary acyclic graph. Moreover, if there is no common ancestor, searching for one may require traversing the entire graph. Traversing a graph with several million vertices takes tens of seconds, which becomes noticeable to users.
We show that the commit graph has additional structure - supporting branches - and that this can be used to speed up the algorithm. By extracting supporting branches, the graph can be preprocessed and the search for the lowest common ancestor can be significantly accelerated. Moreover, if removing the supporting branches decomposes the graph into connected components of bounded size, the lowest common ancestor query can be answered in time comparable to searching for a common ancestor within a single component only.
We present an algorithm running in time $O(\delta_Gm_c + n_s\log^2 n_s + m_s\log n_s)$ and requiring $O(d)$ preprocessed data per vertex, where $\delta_G$ is the maximum out-degree of a vertex in $G$, $n_c$ and $m_c$ are the maximum numbers of vertices and edges in a connected component formed after removing the supporting branches, $n_s$ is the number of supporting branches, $m_s$ is the number of edges in the reachability graph between the supporting branches, and $d$ is the maximum number of supporting branches adjacent to any single component. - [26] arXiv:2611.26069 (cross-list from cs.CL) [pdf, html, other]
-
Title: SheetAlign: Grounded Interaction and Stage-Isolated Learning for Spreadsheet AgentsComments: conference paper preprintSubjects: Computation and Language (cs.CL); Software Engineering (cs.SE)
Spreadsheets are widely used for data analysis and supporting decision-making. Automating spreadsheet analysis while protecting sensitive data and controlling deployment costs motivates locally deployed open-weight large language models (LLMs). Meanwhile, complex spreadsheet tasks require both task reasoning and workbook understanding, while predefined operation interfaces impose additional tool-selection and invocation demands. We introduce SheetAlign, a spreadsheet manipulation framework with stage-isolated policy learning, organized around where to operate, what to change, and how to implement transformations. Task-guided inspection and API grounding anchor code generation in workbook observations and available operations, while stage-isolated learning optimizes program generation and review using edit-progress rewards. At inference, execution consistency selects programs through agreement among workbook states. On SpreadsheetBench, our grounded task interface improves Qwen3-4B's accuracy from 13.5% to 29.2% on average without training. With learning and execution consistency, SheetAlign achieves 32.8% accuracy on average on SpreadsheetBench and 30.1% on Domain-Spreadsheet, outperforming comparable baselines. The code will be made publicly available upon publication.
- [27] arXiv:2611.26333 (cross-list from cs.LG) [pdf, html, other]
-
Title: OceanCoder: A Verification-Guided Large Language Model for Automated Simulation Modelling in Ocean EngineeringComments: 22 pages, 11 figures, including appendicesSubjects: Machine Learning (cs.LG); Software Engineering (cs.SE)
Numerical simulation is important in ocean engineering, but model construction requires expert knowledge. Large language models support automated modelling, but code syntax and successful execution alone do not establish whether generated models and results meet engineering requirements. We present OceanCoder, a Qwen3-8B-based large language model for automated simulation modelling in ocean engineering. To address specialised programming requirements and scarce training data, we develop an extensible data construction pipeline producing 12,000 verified supervised fine-tuning (SFT) examples across 12 engineering task categories. A three-layer verification framework checks model inputs against engineering parameters and task constraints, verifies completion of required simulation analyses, and assesses the engineering plausibility of computed outputs. Its outcomes provide graded rewards for group relative policy optimisation (GRPO), guiding the model towards engineering task completion. We also construct the first benchmark for automated modelling in ocean engineering, covering out-of-distribution tasks and real-world engineering cases. Experiments and ablation studies demonstrate the effectiveness of the data pipeline, verification framework and reward design. OceanCoder outperforms both base models and the closed-source comparison, with Pass@1 increasing from 0% for the base model to 46.77% after SFT and 51.43% after GRPO. These results support engineering verification as a training signal for automated simulation modelling.
- [28] arXiv:2611.26480 (cross-list from cs.PF) [pdf, html, other]
-
Title: Held While Waiting: What LLM Agent Sessions Reserve on Serverless and Agent RuntimesComments: 12 pages, 3 figuresSubjects: Performance (cs.PF); Distributed, Parallel, and Cluster Computing (cs.DC); Software Engineering (cs.SE)
LLM agents spend nearly all their time waiting for the model. On Amazon Bedrock AgentCore, our agents waited a median of 2.7 seconds for each model reply, then computed for an estimated 5 to 40 milliseconds. AgentCore does not charge for CPU during the wait. But it reserves a separate environment, a virtual machine with its own memory, for every conversation, and keeps it reserved until the conversation is closed. So 1,000 conversations reserve 1,000 environments to do the work of 2 to 15 processor cores. A service runs out of environments long before it runs out of processing power.
The shortage has two causes, and each has a fix. An environment stays reserved after its conversation ends: in the customer-service benchmark we ran, a conversation lasts 33 seconds on average, but its environment is held for 15 minutes more unless the client closes it. And an environment serves only one conversation, which is mostly waiting: in our simulation, 64 conversations sharing one environment need 59 times fewer environments, and 99 of every 100 replies are delayed by at most 40 milliseconds.
The milliseconds are an estimate because they are hard to measure: a stubbed model, a timer around the model call, and steps run back to back each gave a plausible number that was too low. We show how to avoid all three. Of the two fixes, closing needs only the client. Sharing needs the platform, and of the agent runtimes we surveyed only one allows it, in part. They keep conversations apart because an agent runs tools, and sometimes code, chosen by a model. The open problem is to keep them apart without reserving an environment for each while it waits. - [29] arXiv:2611.26525 (cross-list from cs.CR) [pdf, html, other]
-
Title: Backtracking Agent Applications Security IncidentsComments: 33 pages (12 pages of main text, 3 pages of references, 18 pages of appendices), 9 figures, 12 tablesSubjects: Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI); Software Engineering (cs.SE)
Agent applications are increasingly deployed, yet agent-specific attacks pose real threats to them. Upon a security incident, organizations backtrack to identify the CEP (compromise entry point), remediate the damage, and prevent its recurrence. The main artifacts available for backtracking are trajectory logs, which record agent functionalities, and source code; app(lication)-specific function call logs are often fairly limited. However, trajectory logs are at the agent functionality abstraction layer, while the CEP candidates are function calls at the app code execution layer. Due to this semantic gap, the analyst needs to manually inspect the relevant code for each CEP candidate to infer its corresponding agent functionality. In this work, we present Origin, a backtracking tool for the analyst to submit a small number of queries to identify CEPs that existing tools cannot provide. Our approach poses two challenges: (1) define the intermediate abstraction that captures the mapping relationships between the two semantic layers; (2) determine the region-to-variable mapping for constructing the cross-layer graph defined in the intermediate abstraction. Each region is a specific group of agent functionalities. Origin addresses the challenges by (1) the cross-layer dependency graph (CLDG) that connects region-inferred functions at the semantic level to the variables in the low-level taint propagation subgraph, plus the associated query language, and (2) the region inference algorithm. Case studies on five popular agent applications across eight attack incidents show that an analyst using Origin correctly identifies all the CEPs.
- [30] arXiv:2611.26601 (cross-list from cs.CR) [pdf, html, other]
-
Title: SoK: Assessing the Evolution of LLM Capability in Software Vulnerability ResearchSubjects: Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI); Software Engineering (cs.SE)
LLM-based agents are becoming increasingly capable at discovering, validating and patching vulnerabilities in real software, and vendors and researchers report that this is already changing how vulnerabilities are found and fixed. These claims, however, are scattered across papers, vendor reports and disclosures and are rarely comparable. We introduce a framework that rates these capabilities on an autonomy scale modeled on the levels of driving automation, and qualifies each claim by its correctness, its availability and the strength of its evidence. We use this framework to systematize the published evidence on LLM capability in vulnerability research. We find that agents discovered, validated, and patched vulnerabilities in real-world software without human intervention before the restricted frontier models of 2026, and that open-weight models subsequently attained comparable autonomy for discovery and validation. We then measure the consequences of evolving LLM security capabilities in a decade of public vulnerability records, advisories and exploitation data. In 2026 the number of disclosed vulnerabilities more than doubled, with an increasing share rated critical or high, and severe bugs that had survived a decade or more in widely used code were reported several times more often than before. Yet the time between the stages of a vulnerability's life, from report to fix and from disclosure to attack, has not shifted noticeably so far. We grade the mitigations proposed in the literature with an explicit rubric and draw guidance for each group of stakeholders from the best-supported ones.
Cross submissions (showing 10 of 10 entries)
- [31] arXiv:2506.16557 (replaced) [pdf, html, other]
-
Title: Scaling GR(1) Synthesis via a Compositional Framework for LTL Discrete Event ControlComments: Published in CAV25Journal-ref: In: Piskac, R., Rakamaric, Z. (eds) Computer Aided Verification. CAV 2025. Lecture Notes in Computer Science, vol 15934. Springer, ChamSubjects: Software Engineering (cs.SE)
We present a compositional approach to controller synthesis of discrete event system controllers with linear temporal logic (LTL) goals. We exploit the modular structure of the plant to be controlled, given as a set of labelled transition systems (LTS), to mitigate state explosion that monolithic approaches to synthesis are prone to. Maximally permissive safe controllers are iteratively built for subsets of the plant LTSs by solving weaker control problems. Observational synthesis equivalence is used to reduce the size of the controlled subset of the plant by abstracting away local events. The result of synthesis is also compositional, a set of controllers that when run in parallel ensure the LTL goal. We implement synthesis in the MTSA tool for an expressive subset of LTL, GR(1), and show it computes solutions to that can be up to 1000 times larger than those that the monolithic approach can solve.
- [32] arXiv:2508.03949 (replaced) [pdf, html, other]
-
Title: Model Compression vs. Adversarial Robustness: An Empirical Study on Language Models for CodeComments: This paper is currently under reviewSubjects: Software Engineering (cs.SE)
Transformer-based language models for code have shown remarkable performance in various software analytics tasks, but their adoption is hindered by high computational costs, slow inference speeds, and substantial environmental impact. Model compression techniques such as pruning, quantization, and knowledge distillation have gained traction in addressing these challenges. However, the impact of these strategies on the robustness of compressed language models for code in adversarial scenarios remains poorly understood. Understanding how these compressed models behave under adversarial attacks is essential for their safe and effective deployment in real-world applications. To bridge this knowledge gap, we conduct a comprehensive evaluation of how common compression strategies affect the adversarial robustness of compressed models. We assess the robustness of compressed versions of three widely used language models for code across three software analytics tasks, using six evaluation metrics and four commonly used classical adversarial attacks. Our findings indicate that compressed models generally maintain comparable performance to their uncompressed counterparts. However, when subjected to adversarial attacks, compressed models exhibit significantly reduced robustness. These results reveal a trade-off between model size reduction and adversarial robustness, underscoring the need for careful consideration when deploying compressed models in security-critical software applications. Our study highlights the need for further research into compression strategies that strike a balance between computational efficiency and adversarial robustness, which is essential for deploying reliable language models for code in real-world software applications.
- [33] arXiv:2608.00437 (replaced) [pdf, html, other]
-
Title: Distilling Reasoning Traces into Advisory Prompts for Software Engineering TasksSubjects: Software Engineering (cs.SE); Artificial Intelligence (cs.AI)
Language models are widely used for generating and otherwise processing code (e.g., predicting behavior, outputs, or likely inputs), and this paper asks whether better performance can be achieved without always incurring the cost of reasoning. Human students of programming learn to avoid mistakes by (a) first identifying them, (b) reflecting upon cognitive lapses that led to these mistakes (essentially, "thinking through" the errors), (c) inferring general rules or lessons from these reflections, and (d) internalizing these lessons into rules. In tutorial sessions with an instructor, this is a very common Socratic interaction with students. Examples of such internalizable rules might include the nugget "Before coding, restate the requirements to clarify them". Inspired by this process, this paper describes an approach where we first identify examples in which "thinking mode" in a (low-resource) LLM avoids errors; then, these errors, and their avoidance via "thinking" in the same LLM, are examined by a bigger LLM to generate summary explanations; these are then summarized by a large LLM into brief advisory prompts. This approach works on many modestly sized models; in some cases, the "advisory prompts" thus learned can also be gainfully transferred to other models. We also present some investigations into the nature of coding errors that language models make, and a characterization of when this approach can be helpful.
- [34] arXiv:2611.21176 (replaced) [pdf, html, other]
-
Title: Scientific Coding with Large Language Models: Evidence from Finite Element AnalysisComments: 28 pages, 7 figures, 14 tablesSubjects: Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Computational Engineering, Finance, and Science (cs.CE)
Large language models (LLMs) offer a potential route to automating scientific programming, but their usefulness depends on consistently producing numerically correct code. We extend FEM-Bench to 17 finite element tasks involving nonlinear and history-dependent mechanics and structural dynamics, and evaluate 15 models on implementation and test generation without execution feedback. The best model succeeds in 87.1% of single-pass implementation trials, but rankings differ between tracks. The leading implementer achieves 49.9% joint test success, while the leading tester reaches 82.6%. For six models, we compare single-pass generation with two agentic harnesses that allow code execution and revision. Agentic execution solves some tasks that never succeeded in single-pass trials and improves consistency on others, but gains vary across models and harnesses, and regressions also occur. Claude Opus 5 with SimpleAgent reaches 96.5% Pass@3, the probability of at least one success in three attempts, but 77.1% Pass$^3$, the probability that all three succeed. No model-harness combination exceeds 78.8% Pass$^3$. To understand why agents submit incorrect code after passing their own checks, we analyze 249 failed agentic trajectories and identify six categories of self-verification failure. The most common are inadequate test coverage (73.1%) and reliance on generic consistency checks (70.3%), and in 28.5% of cases the tests share an error with the implementation. Motivated by these findings, an exploratory configuration with separate coder and tester agents achieves the highest observed scores, with 97.6% Pass@3 and 88.8% Pass$^3$. These results identify consistency across attempts and independent verification as the main obstacles to trusting LLM-generated scientific code.
- [35] arXiv:2611.21784 (replaced) [pdf, html, other]
-
Title: Revisiting the Survival of Agent-Attributed Code: How Measurement Choices Decide the VerdictSubjects: Software Engineering (cs.SE); Artificial Intelligence (cs.AI)
Studies of how long code from AI coding agents survives after it is merged reach conflicting verdicts: agent code is reported to be retained longer than human code, about as long, or modified sooner. Each study makes defensible measurement choices, but how much these choices decide the verdict is unknown. We revisit the survival of agent-attributed code in our earlier study: 854,139 lines that 2,802 merged pull requests (PRs) added to 215 repositories of the AIDev dataset. We vary nine measurement choices in five research questions and build a specification curve of 88 survival analyses, tested under three variance estimators, several observation windows, and permuted labels. For code from human-supervised agents, mostly GitHub Copilot, the choices change the size of the effect but not its direction. For code from the fully-autonomous agent Devin, they decide the verdict. Across repositories, 15 of 22 specifications report it modified sooner when lines are treated as independent, but only 3 when standard errors are clustered by repository, no more than permuted labels produce. Within repositories, no specification detects a difference. LLM intent classifiers agree with human labels about as well as two humans agree with each other, whereas keyword rules do not (kappa = 0.69 to 0.72 against 0.38), and the two reach different verdicts on why agent code is modified. Humans make most modifications to agent code under every unit and window. Re-run on our data, five published designs each change their result with a single choice. Whether agent code survives longer than human code is therefore decided as much by the measurement as by the code. We derive an operationalization checklist for post-merge studies.
- [36] arXiv:2605.21347 (replaced) [pdf, html, other]
-
Title: Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM AgentsAkshay Manglik, Vijay S. Kalmath, Jason Qin, Apaar Shanker, Kaustubh Deshpande, Yash Maurya, Veronica Chatrath, Levi Lentz, Yuan XueComments: v5: Added statistical baselines and open-weight model backbones. Revised analysis, figures, and appendicesSubjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Software Engineering (cs.SE)
Diagnosing failures in LLM agents remains largely manual. Practitioners inspect a small subset of execution traces, form ad-hoc hypotheses, and iterate. This process misses patterns that only emerge across trace populations and does not scale to production corpora where individual traces span tens of thousands of tokens. We formalize the problem of corpus-level trace diagnostics. Given a corpus of execution traces, the goal is to produce grounded natural-language insights that characterize systematic behavioral patterns across trace groups, each linked to supporting evidence. We present the Insights Generator (IG), a multi-agent system that answers diagnostic questions by proposing and testing hypotheses across the trace corpus to produce an evidence-backed insights report. We evaluate IG across qualitative and objective dimensions, spanning rubric-based report assessment and downstream performance improvements achieved by implementing IG insights. Human experts using IG reports improve scaffold performance by 30.4 pp over the unmodified baseline scaffold, and coding agents leveraging IG-derived insights show consistent and stable gains. Across benchmarks, IG's scout-investigator architecture produces findings comparable in detection coverage to competing approaches, while domain experts rated IG reports as leading depth and evidence quality.
- [37] arXiv:2609.40119 (replaced) [pdf, html, other]
-
Title: Function Contracts and Beyond: Machine-Checked Assurance Cases for Conformance of C Libraries to Their RequirementsHaokun Li, Zhongyi Wang, Mingshuai Chen, Ling Zhang, Guanyan Li, Depeng Liu, Xiao Yi, Shengchao Qin, Jianwei YinComments: Substantially revised and expanded; updated title, formal development, and experimental evaluationSubjects: Programming Languages (cs.PL); Software Engineering (cs.SE)
Large language models (LLMs) have shown promise in automating interactive theorem proving, yet verification of real-world C codebases requires more than discharging individual proof goals. The task involves jointly constructing expressive function specifications and their proofs, and ensuring that library interfaces compose along intended call sequences even without a designated client. We present Accord, an LLM-assisted framework for building machine-checked assurance cases: structured, auditable artifacts supporting the claim that a C codebase meets its intended requirements. To model intended cross-interface use in open libraries, Accord constructs an interface protocol that exposes permitted call sequences and resource assumptions for review, with a conditional safety guarantee under verified contracts and caller obligations. Accord coordinates two complementary phases: (i) requirement-guided analysis and bottom-up construction of candidate specifications and protocols; and (ii) modular proof construction with feedback that revises the specifications and proofs. Implemented using VST in Rocq, Accord establishes memory safety for 259 function definitions across seven implementation cases from six benchmark families, including industrial cryptographic modules, and additional functional correctness for red--black trees and ML-DSA. Human review took an estimated 0.17--8.00 hours per case. The guarantees depend on disclosed contracts and assumptions; human review supplies the conformance judgments connecting the formal artifacts to the intended requirements.
- [38] arXiv:2610.30491 (replaced) [pdf, html, other]
-
Title: What the Trace Shows and What the Label Says: Auditing Behavioral Detectors in Agent EvaluationSubjects: Artificial Intelligence (cs.AI); Software Engineering (cs.SE)
LLM agents are often evaluated by reducing execution traces to labels such as final-answer correctness, tool verification, and fault propagation. These labels are measurements produced by parsers, behavioral detectors, and harness rules, yet evaluations rarely report whether a measurement actually fits the trace it describes. We study what happens when those measurements do not fully capture the trace's behavior. Agents solve GSM8K problems with a calculator whose outputs are occasionally corrupted, which gives ground truth for what the tool returned, what the model later used, and what the detectors reported; a chemistry task with search and retrieval tests the case where the model cannot recompute the tool's output. Across eight models and up to 1,000 traces per model, final correctness, visible re-verification, and fault propagation disagree in systematic ways. The most common gap is the silent tool-output override: the model replaces a corrupted value without calling the tool again, so the error stops propagating while a verification detector records no re-check. Human review finds verification labels the least reliable (\k{appa} =0.29), and checking depends on the task: the same calculator that several models re-checked on arithmetic was never re-checked on chemistry. Execution limits and serving failures also change what is measured. These findings motivate measurement transparency: an automated label should be reported with its evidence, whether the detector's question applied, whether the run was censored, and any human corrections. Our trace viewer makes this information inspectable alongside the reported results.
- [39] arXiv:2611.23555 (replaced) [pdf, html, other]
-
Title: MetaphIR: Corpus-Scale Applicability Analysis for LLVM Peephole OptimizationsSubjects: Programming Languages (cs.PL); Software Engineering (cs.SE)
Peephole optimizations replace short source instruction sequences with more efficient equivalents, typically guarded by preconditions such as requiring a constant operand to be a power of two. Since every new rewrite adds maintenance burden and compile-time cost, compiler developers need to know how often real-world programs both match the source pattern and satisfy its preconditions before committing to an implementation. Existing workflows fall short: structural search over an intermediate representation (IR) corpus of 4.8 million functions ignores preconditions and therefore overcounts, whereas implementing the rewrite and recompiling the corpus yields exact counts only after paying the engineering cost, with each run of the corpus's on-demand recompilation pipeline taking a median of 47 minutes. This delay is especially limiting when generalizing a rewrite, as developers often iterate over multiple variants to trade off coverage against complexity.
We introduce MetaphIR, a system pairing a domain-specific language for expressing LLVM peephole optimizations with an indexed search engine over large IR corpora. Using a syntax modeled on LLVM's C++ matchers, MetaphIR patterns augment LLVM IR with holes and Boolean preconditions over matched values. To answer queries at interactive speed, MetaphIR builds compact per-function summaries ahead of time and uses abstract interpretation over those summaries to soundly filter out functions that cannot satisfy a query's constraints. Across twenty recent peephole optimizations from LLVM's InstCombine pass, MetaphIR recovers the exact set of functions where each rewrite applies in isolation, achieving 100% precision and recall. Queries complete in a median of 1.27 seconds---a 22x speedup over a full corpus scan---enabling developers to quickly gauge an optimization's reach and generality prior to implementation.