Harper 疑难句子语料库用 “Difficult sentences” 压测分词标注器与 Linter 的边界能力【免费下载链接】harperOffline, privacy-first grammar checker. Fast, open-source, Rust-powered项目地址: https://gitcode.com/GitHub_Trending/har/harper本文以 Harper 仓库中的 harper-core/tests/text/Difficult sentences.md 语料文件为主体完整梳理这份“疑难句子”测试语料的组织结构与选句策略并结合 harper-core/tests/pos_tags.rs、harper-core/tests/linters.rs 与 harper-core/tests/snapshot.rs 说明它如何驱动快照测试读者读完能掌握 Harper 的词性标注测试体系、标签记号规范以及如何验证标注器行为回归。一、语料定位一份专门为难分词器的句子集Difficult sentences.md是 Harper 核心测试套件harper-core/tests/text/目录中的一份文本语料其文件头对自己的定位写得很明确A collection of difficult sentences to test Harpers ability to correctly tag unusual/uncommon but correct sentences. 一组疑难句子用于测试 Harper 正确标注“非常见但正确”句子的能力。文件同时给出了两条重要边界说明部分词语目前可能标注不正确原文Note that some word may not be tagged correctly right now.——即这份语料既是回归守卫也是已知缺陷的“留档现场”例句大部分取自 Wiktionary 词条采用 CC BY-SA 4.0 许可。选择对象是英语中最高频的功能词a、at、by、for、from、in、of、on、to、with十个字母/介词小节。这些词恰恰是标注器最难处理的对象——它们兼具介词、副词、名词、动词、连词等多种身份且大量存在于习语、古语、体育/法律/航海等专门语境中。语料按“字母小节 词性子类”组织例如At下分Preposition / Noun / Verb三节In下分Preposition / Verb / Adverb / Noun / Adjective / Unit六节On下分Adjective / Adverb / Preposition / Verb四节To下分Particle / Preposition / Adverb三节。各小节选句策略与代表例句以下按原文件骨架逐节展开保留其全部小节划分与关键例句完整语料见 源文件。A不定冠词的非常规用法With one attack, he was torn a pieces.古语用法a相当于inI brush my teeth twice a day.基线对照句AtPreposition一节覆盖了从空间到抽象语义的全谱系地点Caesar was at Rome; a climate treaty was signed at Kyoto in 1997.I was at Jims house at the corner of Fourth Street and Vine.at the bottom of the page; sitting at the table; at church; at sea目标/时间Target at five miles. Prepare torpedoes!Look out! UFO at two oclock!动作指向Dont pick at your food!My cat keeps scratching at the furniture.I was working at the problem all day.情感对象He shouted at her.We laughed at the joke.She was mad at their comments.习语men at work; children at playThe two countries are at war.She is at sixes and sevens with him.At my request, they agreed to move us to another hotel.He jumped at the sudden noise.She pointed at the curious animal.NounThe at sign. 符号Verb(In online chats:) Dont me! Dont at me!聊天场景把 当动词ByPreposition是全文最大的介词群之一覆盖邻近The mailbox is by the bus stop.、经过He ran straight by me.、期限Be back by ten oclock!Well find someone by the end of March.、被动施事The matter was decided by the chairman.The boat was swamped by the water.、方式I avoided the guards by moving only when they werent looking.We went by bus.I discovered it by chance.The electricity was cut off, so we had to read by candlelight.、出处There are many well-known plays by William Shakespeare.By Pythagoras theorem, we can calculate the length of the hypotenuse.、因果I was aghast by what I saw.The invention was born of necessity.类、程度/增量Our stock is up by ten percent.We won by six goals to three.His date of birth was wrong by ten years.、逐次We went through the book page by page.We crawled forward by inches.His health was deteriorating by the day.、计量sold by the yard; cheaper if bought by the grossThe pickers are paid by the bushel.、乘号与尺寸It is easy to invert a 2-by-2 matrix.The room was about 4 foot by 6 foot.The bricks used to build the wall measured 10 by 20 by 30 cm.、血统Shes a lovely little filly, by Big Lad, out of Damsel in Distress.以及两条专门标注的方言/宗教用法Are you eating by Rabbi Fischer? (at the house of)、By Chabad, its different. (with, among)。AdverbI watched the parade as it passed by.There was a shepherd close by.Ill stop by on my way home from work.The women spent much time after harvest putting jams by for winter and spring.存储义Adjectivea by path; a by room偏僻的by catch; a by issue附带的ForConjunctionI had to stay with my wicked stepmother, for I had nowhere else to go.“因为”古语用法Preposition覆盖约五十条语义簇包括方向The astronauts headed for the moon.Run for the hills!、受益I have something for you.Were having a birthday party for Janet.、用途This is a new bell for my bicycle.This medicine is for your cough.、时长Ive lived here for three years.They fought for days over a silly pencil.The store is closed for the day.、距离I can see for miles.、代替/代表I will stand in for him.I speak for the Prime Minister.Make way for the president!、不定式引导It is unreasonable for our boss to withhold our wages.、追求Hes going for his doctorate.Im saving up for a car.、比较/评价Fair for its day.Shes spry for an old lady.For all his expensive education, he didnt seem very bright.、列举For one thing, I cant even swim. / For another, we dont have any equipment.、命名He is named for his grandfather.以及体育记分In term of base hits, Jones was three for four on the dayAt close of play, England were 305 for 3.板球“305 分 3 出局”。From19 句全部保留Paul is from New Zealand.I got a letter from my brother.You cant get all your news from the Internet.He had books piled from floor to ceiling.He departed yesterday from Chicago.This figure has been changed from a one to a seven.Face away from the wall!The working day runs from 9 am to 5 pm.Tickets are available from 17th July.Rate your pain from 1 to 10.Start counting from 1.You can study anything from math to literature.Its hard to tell from here.Try to see it from his point of view.The bomb went off just 100 yards from where they were standing.From the top of the lighthouse you can just see the mainland.Ive been doing this from pickney.方言词Your opinions differ from mine.He knows right from wrong.InPreposition约 38 句覆盖容纳Who lives in a pineapple under the sea?The dog is in the kennel.、组织归属Shes in an orchestra.、时间My birthday is in the first week of December.Easter falls in the fourth lunar month.Will you be able to finish this in a week?、变化方向Less water gets in your boots this way.、状态John is in a coma.My fruit trees are in bud.The company is in profit.、对象What grade did he get in English?、媒介Please pay me in cash — preferably in tens and twenties.Beethovens Symphony No. 5 in C minor is among his most popular.His speech was in French, but was simultaneously translated into eight languages.When you write in cursive, its illegible.以及分词短语引导In replacing the faucet washers, he felt he was making his contribution to the environment.In trying to make amends, she actually made matters worse.。Verb古语He that ears my land spares my team and gives me leave to in the crop.“收获”义AdverbSuddenly a strange man walked in.Would you like that to take away or eat in?They flew in from London last night.For six hours the tide flows in, then for another six hours it flows out.The show still didnt become interesting 20 minutes in.NounHis parents got him an in with the company.内情、门路AdjectiveIs Mr. Smith in?Skirts are in this year.the in train (incoming train)You cant get round the headland when the tides in.He is very in with the Joneses.Im three drinks in right now.I was 500 dollars in when the stock crashed.以及法律古语in by descent; in by purchase; in of the seisin of her husband。UnitThe glass is 8 inches./The glass is 8in.单位缩写Of22 句全部保留Take the chicken out of the freezer.He hasnt been well of late.Finally she was relieved of the burden of caring for her sick husband.He seemed devoid of human feelings.The word is believed to be of Japanese origin.Jesus of NazarethThe invention was born of necessity.It is said that she died of a broken heart.What a lot of nonsense!Ill have a dozen of those apples, please.Welcome to the historic town of Harwich.Im not driving this wreck of a car.Im always thinking of you.He told us the story of his journey to India.This behaviour is typical of teenagers.He is a friend of mine.We want a larger slice of the cake.The owner of the nightclub was arrested.My companion seemed affable and easy of manner.Its not that big of a deal.Ive not taken her out of a goodly long while.After a delay of three hours, the plane finally took off.OnAdjective16 句All the lights are on, so they must be home.We had to ration our food because there was a war on.the shows still onYoure on next!Are we still on for tonight?Its on now.责任落到头上Game on!Climbing up that steep ridge isnt on.不可行The captain moved two fielders to the on side.板球Ponsonby-Smythe hit a thumping on drive.板球术语He always has to be on, its so exhausting.Adverbturn the television onThe lid wasnt screwed on properly.Put on your hat and gloves.The policeman moved the tramp on.Drive on past the railway station.From now on things are going to be different.and so on.He rambled on and on.Ten years on, nothing had changed in the village.Preposition约 58 句覆盖接触面A vase of flowers stood on the table.The painting hangs on the wall.、方位on the left, on the right, on the side, on the bottom.、交通工具on a bus, on a train, on a plane, on a ferry, on a yacht.、日期Born on the 4th of July.On Sunday Im busy. Ill see you on Monday.、媒体I saw it on television.My favorite shows are on BBC America.、对象/影响They planned an attack on London.Her words made a lasting impression on my mind.What will be the effect on morale?、依赖I depended on them for assistance.Youve been on these antidepressants far too long.、工作语境Hes on his lunch break.Im on nights all this week.Im on question four.、惯用语The drinks are on me tonight, boys.The meal is on the house.A curse on him!Please dont tell on her and get her in trouble.The Tories are on twenty-five percent in this constituency.Smith scored again on twelve minutes, doubling Mudchester Rovers lead.等。VerbCan you on the light? (switch on)动词化ToParticleI want to leave.He asked me what to do.I have places to go and people to see.To err is human.Who am I to criticise? Ive done worse things myself.Precisely to get away from you was why I did what I did.I need some more books to read and friends to go partying with.If he hasnt read it yet, he ought to.省略 to 的不定式I went to the shops to buy some bread.PrepositionShe looked to the heavens.We are walking to the shop.The water came right to the top of this wall.I gave the book to him.His face was beaten to a pulp.I sang my baby to sleep.Whisk the mixture to a smooth consistency.It was to a large extent true.We manufacture these parts to a very high tolerance.There are 100 pence to the pound.Three to the power of two is nine.数学幂Whats the time? – Its quarter to four in the afternoon (or 3:45 pm).时间“差一刻”AdverbPlease push the door to. (close)WithPreposition约 30 句覆盖对手He picked a fight with the class bully.、伴随He went with his friends.She owns a motorcycle with a sidecar.、状态Jim was listening to Bach with his eyes closed.、独立主格结构The match result was 10-5, with John scoring three goals.With a heavy sigh, she looked around the empty room.With their reputation on the line, they decided to fire their PR team.、问题There are a number of problems with your plan.What on Earth is wrong with my keyboard?、工具I water my plants with this watering can. This is the watering can I water my plants with.cut with a knife、疾病/情绪The sailors were infected with malaria.overcome with happinessgreen with envy; flushed with success.、雇佣She was with Acme for twenty years before retiring last fall.、省略With what/whose money? I have nothing left to buy groceries (with).等。AdverbDo you want to come with?二、快照测试机制语料如何进入 CI 门禁语料文件本身不是测试真正让它产生约束力的是 harper-core/tests 下的两套快照测试。二者共用 harper-core/tests/snapshot.rs 中的通用脚手架snapshot_all_text_files扫描tests/text/目录下所有.txt/.md文件即Difficult sentences.md等语料按文件名后缀尝试方言覆盖——try_get_dialect_override会把Spell.US.md这类文件名的US解析为Dialect因此Difficult sentences.md无覆盖后缀走默认路径用 rayon 并行对每个文件生成快照与tagged/、linters/子目录中已有快照逐字节比对任何不一致都会写回快照并使测试失败Snapshot mismatches!/No snapshot!从而在 CI 中拦截标注器或 Linter 的行为回归。文件头注释也写明了使用方式向tests/text添加新文档后直接运行测试快照会自动生成更新既有快照同样只需重跑测试。2.1 词性标注快照pos_tags.rsharper-core/tests/pos_tags.rs 的test_pos_tagger将每份语料经FstDictionary::curated()词典与Document::new_markdown_default解析后输出到tests/text/tagged/子目录对应文件为tagged/Difficult sentences.md。快照格式是每行原文配一行对齐的标签 I told her how I had stopped off in Chicago for a day on my way East . # ISg V I/J/D NSg/C ISg V V/J NSg/V/J/P NPrSg/V/J/P NPr C/P D/P NPrSg J/P D J NPrSg/J .标签体系在源码 format_word_tag 中定义注释自称“受 Penn Treebank 词性标签集启发”其记号规则完整继承自源文件文档注释记号含义记号含义N名词V动词L系动词、X助动词、B原形、P过去式/分词、Pr/g进行体、Pt过去式、Pp过去分词、3第三人称单数N后缀Pl/Sg复数/单数J形容词C比较级、S最高级NPr专有名词R副词N上标ᴹ/质量名词 / 质量可数C连词I代词Pl/Sg/$复数/单数/所有格D限定词dem指示、q数量K含撇号但词典无标注缩合词P介词W?词典查无此词#/#d/#r数字 / 年代 / 罗马数字B脏话swear.标点名词短语成员np_memberAmBrCaAu/NoAmComm方言标注及其分组/多标签表示不确定如N/V/J兼名词、动词、形容词HeadingStart、Url等其余 TokenKind 直接打印变体名关键设计是不确定度支持词典中的DictWordMetadata允许一个词同时具备多种词性快照中用/并列输出这正是语料选择“一词多义”句子的意义所在。2.2 Linter 快照linters.rsharper-core/tests/linters.rs 的test_most_lints对同一批语料跑LintGroup::new_curated若文件名未指定方言则调用Dialect::try_guess_from_document从文档内容猜想方言猜不出则回落到Dialect::American。每条 lint 输出类型、优先级、行号定位^~~下划线与修改建议写入tests/text/linters/*.snap.yml。三、解读Difficult sentences的实际标注快照打开 tagged/Difficult sentences.md可以看到语料暴露出的真实标注行为以下均引自该快照标题词也被标注# Difficult sentences行输出为Difficult → VB/J、sentences → NPl/V3——形容词/分词与名词/动词三重身份并存。高频功能词的复合标签With one attack, he was torn a pieces.中标签为With → P、one → NSg/I/J、attack → NSg/VB/J、he → NPr/ISg、was → VLPt、torn → VB/J、pieces → NPl/V3。几乎每个实词都带有“名词/动词”复合标签印证了词库层面的多义存储。已知缺陷留档介词in在大量句子Who lives in a pineapple under the sea?等中标为NPr/J/R/P——专有名词成分被错误地保留在候选集中at普遍标为NSg/P介词候选正确但多挂了名词。这正是原文件头部“some word may not be tagged correctly right now”所指的内容。未知词与特殊 token方言词pickneyIve been doing this from pickney.标签为裸?URLhttps://en.wiktionary.org/整体被切为Urltoken不参与拼写检查oclock被标为R。数字与缩写10-5中的数字标#17th July标为# NPr。再看 linters/Difficult sentences.snap.yml同一份语料同时充当 Linter 的回归样本快照中可验证的典型条目方言敏感拼写Spelling, 63 优先级由于语料未带方言后缀、实际以美式方言运行armour → armor、travelling → traveling、neighbours → neighbors、favour → favor、behaviour → behavior、programme → programmed/programmes、centimetres → centimeters、criticise → criticize均被报出专有名词缺口地名Harwich被建议改为Norwich/Warwick人名Ponsonby-Smythe被逐段报拼写板球队名Mudchester Rovers被建议Manchester——语料中的虚构/生僻专名直观展示了专有名词词典的覆盖边界语法与风格Agreement/Style/Grammar, 31 优先级a thumping on drive触发“thumpingis a mass noun”Its not that big of a deal建议删ofyour kind of body size建议改为youre格式Formatting, 63 优先级10-5被建议改为 en dash–大写Capitalization, 31/127 优先级多个小写开头的片段句men at work、cut with a knife等被要求首字母大写标题则被要求转为 Title Case127 优先级更高冗余Redundancy, 31 优先级All of the responsibility建议简化为All the。这些快照说明语料的双重价值对标注器它是“一词多义 罕见用法”的压力集对 Linter 群它是“正确英文不应被过度报修”的误报探针——凡是快照中出现的报错要么已被接受为既有行为要么就是待修复缺陷的精确现场。四、如何运行与扩展这套测试在仓库根目录Harper workspace下执行# 运行词性标注快照测试会重写 tests/text/tagged/ 下的快照文件 cargo test -p harper-core pos_tags # 运行 Linter 快照测试会重写 tests/text/linters/ 下的 .snap.yml cargo test -p harper-core linters两点适用前提需要注意快照测试是写文件型测试运行后会更新工作区内的快照文件因此应在允许修改检出副本的环境下运行比对失败时请自行审查 diff 后决定是否提交本文仅为说明不修改当前仓库方言行为取决于文件名后缀如Spell.US.mdDifficult sentences.md无后缀Linter 测试走try_guess_from_document猜测或美式默认这解释了为何快照中集中出现“英式拼写被报为拼写错误”的条目。扩展方式是按 pos_tags.rs 与 linters.rs 头部注释所述进行的把新语料放入harper-core/tests/text/跑一次对应测试即可自动生成快照此后 CI 便会对该语料的标注与 lint 结果持续回归。若要加入新的疑难介词/功能词小节建议沿用本文件的“字母小节 词性子类”组织方式并优先收录词典多标签冲突、古语/方言用法与专有名词缺口三类句子——这正是Difficult sentences.md覆盖的三大难点。五、小结Difficult sentences.md表面上只是 469 行例句实际上是 Harper 质量工程的关键部件它以十个高频功能词的多义用法为骨架同时驱动词性标注快照tagged/与 Linter 快照linters/两套 CI 门禁标签记号规范、方言覆盖机制、快照失败即回归的实现都沉淀在 harper-core/tests/snapshot.rs 及配套测试代码中。对想研究离线语法检查器如何评测“长尾正确用法”的读者这份语料及其快照是仓库内最直接可验证的样本。【免费下载链接】harperOffline, privacy-first grammar checker. Fast, open-source, Rust-powered项目地址: https://gitcode.com/GitHub_Trending/har/harper创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考