Top suggestions for Benchmarking Code GenerationExplore more searches like Benchmarking Code GenerationPeople interested in Benchmarking Code Generation also searched for |
- 850×1100
Explore more searches like
BenchmarkingCode Generation
+remains+an+open+question.+We+find+that+existing+benchmarks+are+poorly+aligned+with+real-world+code+repositories+and+are+insufficient+to+evaluate+the+coding+abilities+of+LLMs.%250A++To+address+the+knowledge+gap%252C+we+propose+a+new+benchmark+named+DevEval%252C+which+has+three+advances.+(1)+DevEval+aligns+with+real-world+repositories+in+multiple+dimensions%252C+e.g.%252C+code+distributions+and+dependency+distributions.+(2)+DevEval+is+annotated+by+13+developers+and+contains+comprehensive+annotations+(e.g.%252C+requirements%252C+original+repositories%252C+reference+code%252C+and+reference+dependencies).+(3)+DevEval+comprises+1%252C874+testing+samples+from+117+repositories%252C+covering+10+popular+domains+(e.g.%252C+Internet%252C+Database).+Based+on+DevEval%252C+we+propose+repository-level+code+generation+and+evaluate+8+popular+LLMs+on+DevEval+(e.g.%252C+gpt-4%252C+gpt-3.5%252C+StarCoder+2%252C+DeepSeek+Coder%252C+CodeLLaMa).+Our+experiments+reveal+these+LLMs%27+coding+abilities+in+real-world+code+repositories.+For+example%252C+in+our+experiments%252C+the+highest+Pass%25401+of+gpt-4-turbo+is+only+53.04%2525.+We+also+analyze+LLMs%27+failed+cases+and+summarize+their+shortcomings.+We+hope+DevEval+can+facilitate+the+development+of+LLMs+in+real+code+repositories.+DevEval%252C+prompts%252C+and+LLMs%27+predictions+have+been+released.&cdnurl=https%3a%2f%2fth.bing.com%2fth%2fid%2fR.d2e84af4ed6f2ffae56f2aa166c653c3%3frik%3dUPFssikub4liKA%26pid%3dImgRaw%26r%3d0&exph=630&expw=1200&q=Benchmarking+Code+Generation&FORM=IRPRST&ck=CF97195EB1842DA25100C4DE62EB6BAC&selectedIndex=30&itb=0)
Some results have been hidden because they may be inaccessible to you.Show inaccessible results
Feedback
