HarmonyText: Harmonious Scene Text Editing via Reverse-to-Refined Pair Synthesis and Region-Decoupled Frequency Learning | 华南理工大学;Kolors Team, KlingAI Research | arXiv:2611.17733
Figure 2. Overview of the proposed HarmonyText framework. (Left) RePair constructs high-quality paired data from unpaired real scene text images. A base editing model first produces flawed edits with color shifts and incoherence. The editing direction is then reversed, treating flawed outputs as corrupted inputs and original real images as clean supervision targets, training the model to reconstruct artifact-free scene text. The resulting pairs are further curated via text verification, region matching, and quality filtering to form STE25K. (Right) Region-Decoupled Optimization (RDO) assigns distinct objectives to edited and non-edited regions: non-edited regions are anchored to the source image for background preservation, while edited regions are supervised against the pseudo-target for accurate text generation. Cascaded High-Frequency Texture Guidance (CHFTG) further applies region-decoupled wavelet constraints across two frequency levels, preserving fine-grained textures and enforcing crisp text strokes. Overview of HarmonyText. The left side shows the RePair data synthesis pipeline from real scene-text images through flawed edits, reverse self-refinement, forward generation, verification, region matching, and filtering. The right side shows Region-Decoupling Optimization and Cascaded High-Frequency Texture Guidance for edited and preserved regions.
Figure 3. Qualitative comparisons of scene text editing between our HarmonyText and other SOTA methods. Arrows indicate regions where the edited result deviates from the original appearance in color, texture, or coherence. Our HarmonyText achieves the best visual harmony, with superior texture preservation, color consistency, and text-background coherence. Best viewed with zoom-in. Qualitative comparison grid for several Chinese and English scene-text editing prompts. Each row shows the source image and outputs from multiple baselines and HarmonyText; arrows mark color, texture, or coherence artifacts. HarmonyText more closely preserves the source appearance while producing the requested text.
Fig. 2. (Left) MusicFlux-Concat: The target latent x_t and source audio latent s are concatenated along the sequence dimension (2L frames), while the binary mask m is concatenated along the channel dimension. (Right) MusicFlux-CAR: Condition-Aware Reparameterization uses lightweight auxiliary networks to shift the noise start-point z_0 and target endpoint z_1. Diffusion operates directly in the shifted space over L frames, reducing complexity.
Figure 5: ReDiverse overview. The student loses the teacher’s variation across noises at the collapsed block ℓ \ell , while the later blocks can still render it; ReDiverse restores it there, by training (a) or at inference (b). Below each diagram, the activations (blue: teacher, red: student, green: ReDiverse) and samples, before and after. (a) ReDiverse-FT regresses the student’s activation onto the teacher’s for the same noise, updating only LoRA adapters. (b) ReDiverse-TF finds the teacher’s directions of variation by PCA and moves the student along them.
Figure 8: Qualitative results on Wan 2.1. Rows are seeds, each clip shown by its first and last frame, same noise for all three columns. Best viewed as videos available in supplementary material.
本文贡献:Just Video Transformer(JVT)用大 patch 的 DiT 直接预测干净像素视频,不依赖预训练 VAE、辅助细节头或级联。研究比较 patch embedding、register token 与自适应层归一化;在相机条件任务中联合输出 RGB 与视差,视差可按相机参数反投影成点云。
Figure 2: JVT architecture and register token design . The network predicts a clean video x ^ 0 \hat_{0} from a noisy input x t x_{t} . Register tokens are prepended to visual tokens after the first L 1 L_{1} blocks, and then discarded after the remaining L 2 L_{2} blocks, where L = L 1 + L 2 L=L_{1}+L_{2} .
Figure 5: Joint RGB-disparity generation and point-cloud reconstruction on RealEstate10K . We show generated RGB frames alongside aligned disparity maps. Colored point clouds are obtained by unprojecting depth recovered from disparity using the camera parameters, without an additional reconstruction model at inference.
Figure 3: Token-wise selection with tile-wise execution. (a) An SVD of the second-moment square-root product gives matched projections P q , P k P_{q},P_{k} from 128 to 32 dimensions, in which queries and keys are clustered per token. Each query keeps the shortest prefix of whole key clusters that meets its token budget (colored); a query cluster executes the union of its members’ selections (outlined), and each remaining cluster contributes one centroid term per query (grey, ⋄ \diamond ). (b) The original Q Q multiplies the union keys, packed into tiles across cluster boundaries; Q P q QP_{q} multiplies the centroids of the unselected clusters, and an LSE merge yields one softmax per query. Clusters, selections and plans are refreshed every R = 4 R=4 calls; reuse calls use the current Q Q , K K , V V . No N × N N\times N tensor is materialized.
Figure 1: Higher dense-reference fidelity at comparable generation speed. (a) H3 I2V sweep on a common VBench-I2V subset with 95% bootstrap PSNR intervals; green labels are nominal budgets. Near 1.49 × 1.49\times on this subset, Token2Tile improves PSNR by 4.9–5.6 dB. (b) Crops of lettering from the dense video and three sparse methods at 1.47 1.47 – 1.50 × 1.50\times (whole-video PSNR, request time). (c) Dense-left/Token2Tile-right splits of four T2V examples at 1.5%.
Figure 3 : Overview of structural self-teaching. Phrase-conditioned support masks { A p } \{A_{p}\} extracted from the frozen understanding path provide two training-time signals for the LoRA-tuned generation path: phrase-local contrastive alignment between reference and generated phrase representations, and a structural scaffold that modulates an intermediate spatial feature F t ℓ F_{t}^{\ell} . The scaffold is applied with sample-wise cosine-decayed dropout and removed at inference.
Figure 1 : Unified multimodal models can localize phrases they fail to generate. Given an image-text pair that satisfies a compositional prompt, the understanding path produces phrase-wise attention maps that localize the requested concepts. However, generation from the same prompt can omit objects or entangle attributes, as shown by the base UMM. Structural self-teaching transfers these internal supports during training, improving phrase coverage and compositional fidelity.
本文贡献:RSE-TTS 的 Reconstructive Speaker Encoder 与生成端共享 Global Transformer;八个学习查询把参考文字和音频压成连续向量,再结合说话人嵌入条件生成。Local 模块逐帧生成离散 codec,避免为参考编码另训一套同规模骨干。
Figure 1: RSE-TTS uses the same Global transformer to encode reference text and speech and to generate a new utterance. Eight learned queries produce \mathbf{Z} , which is combined with a projected speaker embedding to form ¯ \bar{\mathbf{Z}} . During generation, Global consumes frame embeddings E ( i ) E(\mathbf{c}_{i}) , while Local produces discrete codec frames i \mathbf{c}_{i} . Repeated Local and decoder blocks illustrate successive generation steps, not separate parameters.
Figure 3: Model size versus intelligibility and speaker similarity. WER and SIM are macro-averaged with equal weight across Seed-TTS, CV3-eval, and MiniMax. Parameter counts include the generator and active waveform decoder. Lower WER and higher SIM are better; the horizontal axis uses a logarithmic scale. “Ref.” denotes reference conditioning and “cont.” denotes acoustic continuation.
本文贡献:CrossPairs 将现有视觉推理数据转成“同图、不同提示”的偏好对,和常规“同提示、不同图”的人类偏好一起训练奖励模型,减少审美差异对构图判断的干扰。奖励分数再用于 best-of-n 候选筛选或 DiffusionNFT 强化学习后训练;best-of-n 是生成 n 张后挑分数最高的一张。
Figure 1 : (Left) Existing reward models exhibit poor visual understanding. VLM backbones possess strong visual reasoning, but this is eroded during preference fine-tuning, even with robust fine-tuning techniques. (Center) We introduce CrossPairs – preference pairs sharing a common image but differing in prompt – to isolate text-image alignment signal by eliminating aesthetic confounds. Mixed with standard pairs during training, our method achieves a Pareto improvement. (Right) The resulting reward model guides generations with superior compositional accuracy.
Figure 6 : Generated samples after post-training with different reward models. The first three columns are prompts from Pick-a-Pic; the last three columns are prompts from GenEval.
Figure 2: Benchmark construction and evaluation. The upper panel shows skill-guided prompt construction, followed by quality control. The lower panel illustrates the evaluation protocol to produce an overall score, per-skill scores, and reasons.
Figure 4: Results of human pairwise evaluation of generated scoring rationales. C o ttage is compared against Qwen2.5-Omni-7B, the SFT-only variant, and the proprietary model Gemini-3.1 Pro.
评论 (0)