Question:
Which of the following statements about the datasets used in the MSIT framework are correct?
A. The AGTD employs in-context learning with a small set of seed attributes to generate new attributes from both images and texts.
B. The CTTD introduces contrastive samples with balanced positive and negative attributes to prevent overfitting during self-correction.
C. AGTD is constructed by merging modality-specific attributes generated separately from images and texts to reduce semantic gaps in multimodal fusion.
D. CTTD incorporates a structured 5-step reasoning process to validate attribute relevance through image-based verification and text-based common-sense analysis.

Answer: AC

3.1
Attribute Generation Tuning Data
Raw Dataset.
We expand two raw unimodal
datasets, WOAM (Xu et al., 2023) and OAMine
(Zhang et al., 2022), to multimodal datasets.
The two datasets encompass several prevalent e-
commerce product categories such as Tea, Vita-
min, Sofa, and Phone Case. We collect the product
images from Amazon.com corresponding to the
respective products.
Seed Dataset. For each type, the seed set includes
several applicable attribute types. We manually
construct and annotate the seed dataset to ensure
consistency with product characteristics. This al-
lows us to refine unclear or coarse-grained attribute
types into newly defined fine-grained types.
In-Context Learning Generation. We employ
In-Context Learning (Zhang et al., 2023a) to con-
struct Attribute Generation Tuning Data (AGTD).
Specifically, a small set of seed attributes and corre-
sponding values is selected to guide the generation
of new attributes and values by GPT-4 (OpenAI,
2023). The prompts for attribute generation are
listed in Appendix A.1. In the initial experiments,
we observe that GPT-4 would mostly generate the
attributes from text information if the images and
texts are input simultaneously. To mitigate the bias
of modality attention, images and text descriptions
are input separately to generate potential attributes.
The generated attributes are manually reviewed to
filter out incorrect attributes, ensuring high-quality
data for further processing. Finally, we employ
GPT-4 to merge filtered attributes of images and
descriptions as shown in Appendix A.2.
3.2
Chain-of-Thought Tuning Data
To address the limitation of lacking an explicit rea-
soning process, we construct Chain-of-Thought
(CoT) Tuning Data to be jointly trained with AGTD.
Recent works (Zhang et al., 2023c) find that vanilla
form of CoT which directly lets LLM to indiscrimi-
nately output the reasoning process would decrease
the performance. The phenomenon mostly results
from the generation of hallucinated rationales. To
alleviate the problem, we divide the reasoning pro-
cess into 5 steps and specify the output targets of
each step. The prompts for generating CoT data
are stated in Appendix A.3.
Product Type Range Narrowing:
Firstly,
MLLM should judge the type of current product
given the corresponding image and text to narrow-
ing down the range of product attributes. This step
provides context for subsequent reasoning.
Reasoning with Internal Common-sense
Knowledge:
The second step utilizes internal
common sense alongside preliminary screening to
determine whether a to-be-judged attribute applies
to a product type. For instance, when evaluating if
a specialized term like ’screen_size’ is appropriate
for vitamins, consider the common sense context of
the product. Moreover, if the attribute’s relevance
is unclear, it should initially be considered unsuit-
able. If the meaning is clear, then common sense
should guide the preliminary judgment of whether
the attribute generally fits the product type.
Image-Based Attribute Validation: This step
assesses whether the attribute originates from the
image. If the attribute is deemed valid after initial
filtering, the model infers the attribute value from
the image to confirm its presence.
Text-Based Attribute Verification: This step
evaluates whether the attribute is derived from the
text. If the model preliminarily determines that the
product may have this attribute, then it will infer
whether the attribute can be judged from the text.
Final Evaluation and Decision-Making: The
final step summarizes the reasoning from the pre-
vious steps and decides whether the attribute is
derived from the given data, concluding with a yes
or no answer.
Contrastive Chain-of-Thought Tuning Data. As
the above steps employ the manually reviewed at-
tributes to construct the CoT tuning data, the final
decision for each attribute is ‘yes’. To prevent over-
fitting during tuning, we introduce Contrastive CoT
Tuning Data. Attributes from different product
types are selected as negative samples. These sam-
ples undergo a rigorous manual selection process
to ensure reliability and effectiveness in training.
In addition, we ensure that the number of positive
and negative samples is balanced.

Open-World Attribute Mining for E-Commerce Products with Multimodal
Self-Correction Instruction Tuning
Anonymous ACL submission
Abstract
001
In e-commerce, effective product Attribute
002
Mining (AM) is essential for enhancing product
003
features and aiding consumer decisions. How-
004
ever, current AM methods often focus on ex-
005
tracting attributes from unimodal text, under-
006
utilizing multimodal data. In this paper, we
007
propose a novel framework called Multimodal
008
Self-Correction Instruction Tuning (MSIT) to
009
mine new potential attributes from images and
010
texts with Multimodal Large Language Mod-
011
els (MLLMs). The tuning process involves
012
two datasets: Attribute Generation Tuning Data
013
(AGTD) and Chain-of-Thought Tuning Data
014
(CTTD). AGTD is constructed utilizing in-
015
context learning with a small set of seed at-
016
tributes, aiding the MLLMs in accurately ex-
017
tracting attribute-value pairs from multimodal
018
information. To introduce explicit reasoning
019
and improve the extraction accuracy, we con-
020
struct CTTD, which incorporates a structured
021
5-step reasoning process for self-correction. Fi-
022
nally, we employ a 3-stage inference process to
023
filter out redundant attributes and sequentially
024
validate each generated attribute. Comprehen-
025
sive experimental results on two datasets show
026
that MSIT outperforms state-of-the-art meth-
027
ods. We will release our code and data in the
028
near future.
029
1
Introduction
030
In the realm of e-commerce, product attributes
031
enrich product selling points, helping consumers
032
make informed decisions (Xu et al., 2019; Yan
033
et al., 2021; Shinzato et al., 2023).
However,
034
with the constant emergence of new products, e-
035
commerce often struggles with incomplete attribute
036
data. To this end, Open-World Product Attribute
037
Mining (AM) (Zhang et al., 2022; Xu et al., 2023)
038
technology addresses this need by extracting new
039
potential attributes from product profiles. Although
040
numerous works have demonstrated outstanding
041
performance in AM, they still face the following
042
limitations:
043
“ Organic
Matcha Green
Tea;100%
Pure Culinary
Grade ...”
MLLM
Finetune
{ "Attributes":
       {"Tea_Type": "Green",
        "Product_Shape": "Powder",
        "Weight": "1lb"
        ... }
}
Batch Attribute Generation
Filtering Repeated Attributes
Sequential Attribute Inference With COT
(c)  Our Work
Attribute
Generation
Tuning Data
CoT
Tuning Data
(b) Absence of explicit reasoning
(a) Underutilizating multimodal information
(a) Underutilizating multimodal information
Invalid Attribute
Invalid Attribute
Valid Attribute from Image
Valid Attribute from Image
Valid Attribute from Text
Valid Attribute from Text
"Organic Matcha Green
Tea; Japanese Everyday
Drink | 100% Pure
Culinary Grade | Sugar
Free, Eco-friendly Bag..."
{ "Attributes":
       { "Tea_Type":
"Green",
        "Sugar":
"Sugar-free"
          ... }  }
PLM
PLM
"Organic Matcha Green
Tea; Japanese Everyday
Drink | 100% Pure
Culinary Grade | Sugar
Free, Eco-friendly Bag..."
{ "Attributes":
       { "Tea_Type":
"Green",
        "Sugar":
"Sugar-free"
          ... }  }
PLM
{ "Attributes":
       { "Harvest":
 "Premium First Harvest"
         "Marketing Claims":
 "Calm and clarity"
        ... }
}
MLLM
“ Organic
Matcha Green
Tea;100% Pure
Culinary
Grade ...”
{ "Attributes":
       { "Harvest":
 "Premium First Harvest"
         "Marketing Claims":
 "Calm and clarity"
        ... }
}
MLLM
“ Organic
Matcha Green
Tea;100% Pure
Culinary
Grade ...”
Figure 1: Comparison of current methods and our work.
(a) Current methods rely on textual data, missing out
attributes present in images. (b) Existing approaches
lack explicit reasoning, leading to extracting invalid at-
tributes. (c) Our work leverages multimodal data and a
chain-of-thought process for accurate attribute extrac-
tion.
1) Underutilization of multimodal informa-
044
tion. Recent AE methods (Zhang et al., 2022; Xu
045
et al., 2023) rely solely on textual data, extracting
046
potential attributes from given descriptions or titles
047
in the product profiles. However, product images
048
also offer valuable attributes that can enhance the
049
shopping experience for consumers. As illustrated
050
in Figure 1(a), current models often overlook key
051
visual information in product images. In Figure
052
1(c), attributes like ‘Product_shape’ and ‘Weight’
053
are visible on the packaging, but are not extracted
054
by models that only use textual data. By integrating
055
both textual and visual data, a more comprehensive
056
set of attributes can be extracted.
057
2) Absence of explicit reasoning. Earlier works
058
(Ghani et al., 2006; Zheng et al., 2018a; Mehta
059
et al., 2021; Fu et al., 2022) treat AM as a clas-
060
sification task, leveraging pre-trained models to
061
implicitly derive classification results. More recent
062
researchers (Zou et al., 2024; Shinzato et al., 2023;
063
Zhang et al., 2023b; Li et al., 2023b; Khandelwal
064
et al., 2023) utilize generative language models,
065
which typically generate results directly without
066
an explicit reasoning process. As shown in Figure
067
1
1(b), models without explicit reasoning capabilities
068
may extract attributes like ‘Marketing Claims’ with-
069
out justifying their relevance or correctness. The
070
absence of an explicit reasoning process means
071
these models cannot effectively validate and refine
072
their outputs based on context and common sense,
073
leading to suboptimal results.
074
To address these limitations, we propose a novel
075
framework called Multimodal Self-Correction In-
076
struction Tuning (MSIT) for the task of Open-
077
World E-commerce Product Attribute Mining. Our
078
approach leverages generative Multimodal Large
079
Language Models (MLLMs) to mine new potential
080
attributes from both images and texts. The tuning
081
process involves two datasets: Attribute Genera-
082
tion Tuning Data (AGTD) and Chain-of-Thought
083
Tuning Data (CTTD). AGTD is constructed uti-
084
lizing in-context learning with a small set of seed
085
attributes. AGTD aids MLLMs in accurately ex-
086
tracting attribute-value pairs from multimodal in-
087
formation. To address the limitation of lacking
088
explicit reasoning, we construct CTTD to guide the
089
MLLMs in self-correction. CTTD is created by
090
leveraging the attributes generated in AGTD and
091
incorporating a structured 5-step reasoning process.
092
In the inference phase, our approach employs a
093
3-stage process to extract attributes accurately.
094
The primary contributions of our work are as
095
follows:
096
• To the best of our knowledge, we are the pi-
097
oneers in exploring extracting potential at-
098
tributes with MLLMs, extending Attribute
099
Mining to multimodal settings.
100
• We propose a comprehensive framework that
101
can discover attributes from both textual or
102
visual information, followed by a 5-step chain-
103
of-thought reasoning process to self-correct
104
the generated attributes.
105
• We expand two unimodal datasets to multi-
106
modal datasets. The experimental results on
107
two datasets demonstrate the superiority of
108
our method compared to existing methods.
109
2
Related Work
110
Multi-modal Large Language Models. Multi-
111
modal Large Language Models (MLLMs) extend
112
Large Language Models (LLMs) by integrating
113
non-textual modalities for various tasks. BLIP-
114
2 (Li et al., 2023a) achieves state-of-the-art per-
115
formance in vision-language tasks by leveraging
116
frozen pre-trained image encoders, language mod-
117
els, and lightweight query transformers. Instruct-
118
BLIP (Dai et al., 2023) improves upon BLIP-2 by
119
performing vision-language instruction tuning, out-
120
performing the Flamingo model (Alayrac et al.,
121
2022) in zero-shot tasks. LLAVA (Liu et al., 2023)
122
uses GPT-4 (OpenAI, 2023) to generate multi-
123
modal instruction-following data and trains large-
124
scale models for general visual and language un-
125
derstanding.
126
Attribute Mining. Research in product attribute
127
mining, particularly in e-commerce, has gained
128
significant attention (Shinzato et al., 2022; Kara-
129
manolakis et al., 2020; Zheng et al., 2018b). Open-
130
tag (Zheng et al., 2018a) uses neural networks and
131
active learning to identify missing attributes, but
132
does not expand attribute frameworks. LATEX-
133
Numeric (Mehta et al., 2021) extracts numerical at-
134
tributes via distant supervision and multitask learn-
135
ing, eliminating manual labeling. CMA-CLIP (Fu
136
et al., 2022) models attribute completion as a clas-
137
sification task but assume a closed-world scenario.
138
Other approaches (Roy et al., 2021; Xu et al., 2023)
139
treat attribute completion as a generative modeling
140
problem, using large language models to generate
141
attribute values, but are limited to specific prod-
142
uct categories and do not consider personalized
143
attribute generation.
144
3
Proposed Method
145
In Multimodal Open-World Attribute Mining task,
146
the i-th product Pi is composed of a text (title and
147
bullet point) Ti and an image Ii. The text con-
148
sists of wi tokens Ti =

si
1, si
2, . . . , si
wi

. Our
149
goal is to extract a set of relevant and applicable
150
attributes Ai = {a1, a2, ..., ak} from the given in-
151
puts, where k is the number of mined attributes.
152
These attributes can range from basic features such
153
as color and size to more complex attributes like
154
material and style. Each attribute corresponds to
155
a specific value, denoted as Vi = {v1, v2, ..., vk}.
156
The Open-World setting requires all the attributes
157
need to be extracted, not limited to a pre-defined
158
schema. Our method leverages a generative Multi-
159
modal Large language model to output a sequence,
160
where the mined attributes and corresponding val-
161
ues are presented in a unified format, such as
162
{a1 : v1, a2 : v2, ..., ak : vk}. The overall architec-
163
ture of the model is shown in Figure 2. Our method
164
utilizes a structured process for constructing high-
165
quality tuning data and employs a 3-stage inference
166
2
Merge
Instruction-Tuning Dataset
Instruction-Tuning Dataset
Image
Image-Based Reasoning Process
The following reasoning to determine if
this product has the attribute of "...."
From the image, I can infer that...
Attribute From Image
Image-Based CoT Tuning Dataset
Image-Based CoT Tuning Dataset
Text
Text-Based Reasoning Process
The following reasoning to determine if
this product has the attribute of "...."
From the Text, I can infer that...
Attribute From Text
Thai Tea Blend No.
1 Thai Tea Original
(50 Bags)
Attribute Generation Tuning Data Construction
Attribute From Image
Attribute From Text
"Brand": "Micro
Ingredient"
...
"Tea Type": "Green"
...
Attribute From Image
Attribute From Text
"Brand": "Micro
Ingredient"
...
"Tea Type": "Green"
...
Thai Tea Blend No. 1 Thai
Tea Original (50 Bags) Made
in Thailand Tastes...
Image
Text
Positive Examples:
Attributes randomly
chosen from Tea
Negative Examples:
Attributes randomly
chosen from other product
Thai Tea Blend No. 1
Thai Tea Original
(50 Bags) Made in
Thailand.
Thai Tea Blend No. 1
Thai Tea Original
(50 Bags) Made in
Thailand.
"Tea Type": "Green"
"Shape": "Powder"
…
"Brand": "Thai"
…
“ Organic Matcha Green Tea; Premium
First Harvest Japanese Everyday Drink |
100% Pure Culinary Grade | Sugar Free,
Eco-friendly Recyclable Bag...”
For the attribute of [brand], here are  the
inference results:
…
Yes[Marking this attribute is valid]
“ Organic Matcha Green Tea; Premium
First Harvest Japanese Everyday Drink |
100% Pure Culinary Grade | Sugar Free,
Eco-friendly Recyclable Bag...”
For the attribute of [brand], here are  the
inference results:
…
Yes[Marking this attribute is valid]
Given this product, please generate its attribute-
value pairs
Product Attribute Inference
Chain of Thought Tuning Data Constraction
Incontext
Learning
Generation
"Tea Type": "Green"
...
"Brand": "Micro
Ingredient"
...
Attribute From Image
Attribute From Text
"Tea Type": "Green"
...
"Brand": "Micro
Ingredient"
...
Attribute From Image
Attribute From Text
"Tea Type": "Green"
...
"Brand": "Micro
Ingredient"
...
Attribute From Image
Attribute From Text
Modality Separated Label
LLaVa
v1.5
LLaVa
v1.5
LLaVa
v1.5
MLLM
Filtering
Repeated
Attributes
Sequential
Attribute
Inference
with 5-step
CoT
Attribute
Batch
Generation
Attribute
Batch
Generation
...
{ "Attributes":
     {
    "brand": "Micro Ingredients",
    "Tea Type": "Green",
    "Product Shape": "Powder"}
}
{ "Attributes":
     {
    "Tea Type": "Green",
    "Shape": "Powder"}}
{ "Attributes":
     { "compatible device":
"iphone13",
    "Material": "plastic"}}
{ "Attributes":
      {
        "brand": "Micro Ingredients",
        "Tea_Type": "Green",
        "Product_Shape": "Powder",
        "Weight": "1lb",
        "Type": "Green",
        "Product_Brand": "Micro Ingredients"}
}
{ "Attributes":
      {
        "brand": "Micro Ingredients",
        "Tea_Type": "Green",
        "Product_Shape": "Powder",
        "Weight": "1lb",
        "Type": "Green",
        "Product_Brand": "Micro Ingredients"}
}
{ "Attributes":
      {
        "brand": "Micro Ingredients",
        "Tea_Type": "Green",
        "Product_Shape": "Powder",
        "Weight": "1lb",
        "Type": "Green",
        "Product_Brand": "Micro Ingredients"}
}
 "brand": "Micro Ingredients"
Valid?
 "brand": "Micro Ingredients"
Valid?
 "brand": "Micro Ingredients"
Yes!
 "brand": "Micro Ingredients"
Yes!
Product Image
Product Text
Seed Dataset
GPT4
GPT4
Extract the attribute values from the
Image/Text of the product
respectively.
Here are some examples from other
product:...
Seed Dataset
GPT4
Extract the attribute values from the
Image/Text of the product
respectively.
Here are some examples from other
product:...
Please determine if this product possesses
this attribute through the text or
image. Let's think in the following 5
steps, and provide your reasoning process.
Here are the attributes from other product
Text-Based CoT Tuning Dataset
Text-Based CoT Tuning Dataset
Step 2 Reasoning with Internal Common-sense Knowledge: “use common sense
judge whether the attribute matches type...”
Step 1 Product Type Narrowing: “determine the type of the product...”
Step 3 Image-Based Attribute Validation:”please use the picture to confirm
that the product indeed has this attribute”
Step 4 Text-Based Attribute Verification: “... If you inferred an attribute
value from the text, give the exact attribute value.”
Step 5 Final Evaluation and Decision-Making: “please combine the reasoning and
draw a Conclusion...”
Step 2 Reasoning with Internal Common-sense Knowledge: “use common sense
judge whether the attribute matches type...”
Step 1 Product Type Narrowing: “determine the type of the product...”
Step 3 Image-Based Attribute Validation:”please use the picture to confirm
that the product indeed has this attribute”
Step 4 Text-Based Attribute Verification: “... If you inferred an attribute
value from the text, give the exact attribute value.”
Step 5 Final Evaluation and Decision-Making: “please combine the reasoning and
draw a Conclusion...”
Figure 2: The overall framework of MSIT. MSIT is divided into three main components: Attribute Generation
Tuning Data (AGTD) Construction, Chain-of-Thought Tuning Data (CTTD) Construction, and Product Attribute
Inference. In the first component, we construct AGTD by separately extracting attributes from images and text
using in-context learning and merging them. The second component involves creating CTTD to guide the model in
a structured 5-step reasoning process. Finally, the Product Attribute Inference component utilizes a 3-stage process
to generate, filter, and validate attributes from multimodal data.
procedure to accurately extract attributes from both
167
images and texts. The data construction phase in-
168
volves generating attributes using in-context learn-
169
ing, followed by creating Chain-of-Thought (CoT)
170
tuning data with a 5-step reasoning process. Dur-
171
ing inference, the 3 stages generate attributes in
172
batches, filter out redundant attributes, and sequen-
173
tially validate each attribute.
174
3.1
Attribute Generation Tuning Data
175
Raw Dataset.
We expand two raw unimodal
176
datasets, WOAM (Xu et al., 2023) and OAMine
177
(Zhang et al., 2022), to multimodal datasets.
178
The two datasets encompass several prevalent e-
179
commerce product categories such as Tea, Vita-
180
min, Sofa, and Phone Case. We collect the product
181
images from Amazon.com corresponding to the
182
respective products.
183
Seed Dataset. For each type, the seed set includes
184
several applicable attribute types. We manually
185
construct and annotate the seed dataset to ensure
186
consistency with product characteristics. This al-
187
lows us to refine unclear or coarse-grained attribute
188
types into newly defined fine-grained types.
189
In-Context Learning Generation. We employ
190
In-Context Learning (Zhang et al., 2023a) to con-
191
struct Attribute Generation Tuning Data (AGTD).
192
Specifically, a small set of seed attributes and corre-
193
sponding values is selected to guide the generation
194
of new attributes and values by GPT-4 (OpenAI,
195
2023). The prompts for attribute generation are
196
listed in Appendix A.1. In the initial experiments,
197
we observe that GPT-4 would mostly generate the
198
attributes from text information if the images and
199
texts are input simultaneously. To mitigate the bias
200
of modality attention, images and text descriptions
201
are input separately to generate potential attributes.
202
The generated attributes are manually reviewed to
203
filter out incorrect attributes, ensuring high-quality
204
data for further processing. Finally, we employ
205
GPT-4 to merge filtered attributes of images and
206
descriptions as shown in Appendix A.2.
207
3.2
Chain-of-Thought Tuning Data
208
To address the limitation of lacking an explicit rea-
209
soning process, we construct Chain-of-Thought
210
(CoT) Tuning Data to be jointly trained with AGTD.
211
Recent works (Zhang et al., 2023c) find that vanilla
212
form of CoT which directly lets LLM to indiscrimi-
213
nately output the reasoning process would decrease
214
the performance. The phenomenon mostly results
215
from the generation of hallucinated rationales. To
216
alleviate the problem, we divide the reasoning pro-
217
cess into 5 steps and specify the output targets of
218
each step. The prompts for generating CoT data
219
are stated in Appendix A.3.
220
3
Product Type Range Narrowing:
Firstly,
221
MLLM should judge the type of current product
222
given the corresponding image and text to narrow-
223
ing down the range of product attributes. This step
224
provides context for subsequent reasoning.
225
Reasoning with Internal Common-sense
226
Knowledge:
The second step utilizes internal
227
common sense alongside preliminary screening to
228
determine whether a to-be-judged attribute applies
229
to a product type. For instance, when evaluating if
230
a specialized term like ’screen_size’ is appropriate
231
for vitamins, consider the common sense context of
232
the product. Moreover, if the attribute’s relevance
233
is unclear, it should initially be considered unsuit-
234
able. If the meaning is clear, then common sense
235
should guide the preliminary judgment of whether
236
the attribute generally fits the product type.
237
Image-Based Attribute Validation: This step
238
assesses whether the attribute originates from the
239
image. If the attribute is deemed valid after initial
240
filtering, the model infers the attribute value from
241
the image to confirm its presence.
242
Text-Based Attribute Verification: This step
243
evaluates whether the attribute is derived from the
244
text. If the model preliminarily determines that the
245
product may have this attribute, then it will infer
246
whether the attribute can be judged from the text.
247
Final Evaluation and Decision-Making: The
248
final step summarizes the reasoning from the pre-
249
vious steps and decides whether the attribute is
250
derived from the given data, concluding with a yes
251
or no answer.
252
Contrastive Chain-of-Thought Tuning Data. As
253
the above steps employ the manually reviewed at-
254
tributes to construct the CoT tuning data, the final
255
decision for each attribute is ‘yes’. To prevent over-
256
fitting during tuning, we introduce Contrastive CoT
257
Tuning Data. Attributes from different product
258
types are selected as negative samples. These sam-
259
ples undergo a rigorous manual selection process
260
to ensure reliability and effectiveness in training.
261
In addition, we ensure that the number of positive
262
and negative samples is balanced.
263
3.3
Model Training
264
We fine-tune three strong and widely used MLLMs
265
LLAVA-7B (Liu et al., 2023), Qwen-VL (Bai et al.,
266
2023) and InternLM (Dong et al., 2024) with the
267
Attribute Generation Tuning Data and Chain-of-
268
Thought Tuning Data. We adopt the Low-Rank
269
Adaptation (LoRA) fine-tuning method. The core
270
idea of this method is to freeze the language model
271
and tune only the rank-decomposition module of
272
the Transformer layer.
273
Formally, given the parts of instruction tuning
274
data D, our training objective is to obtain a fine-
275
tuned model Mˆθ:
276
arg min
ˆθ
E(Ii,Ti)∈D
h sj
X
s=1
log PMˆθ
si
w | Ii, si
1, . . . , si
w−1

#
,
(1)
277
where Ti = si
1, si
2, . . . , si
w and ˆθ denotes the
278
parameters of LoRA in MLLM.
279
3.4
3-Stage Attribute Inference
280
Batch Attribute Generation. In this stage, the
281
fine-tuned MLLM generates attributes for a given
282
sample. The model leverages its understanding of
283
both images and text to produce a batch of relevant
284
attributes. We extract a set of formalized attributes
285
and values from the output texts.
286
Filtering Repeated Attributes. To reduce compu-
287
tational costs, we filter out repeated attributes of
288
the first stage. For instance, attributes like ‘type’
289
and ‘product type’ are identified as duplicates. A
290
rule-based system is employed to eliminate these
291
redundancies, streamlining the attribute list.
292
Sequential Attribute Inference with 5-step CoT.
293
The final stage sequentially inputs each filtered
294
attribute into the MLLM for inference using the 5-
295
step CoT process. Whether an attribute is reserved
296
is determined by the yes or no output in the last
297
step of CoT.
298
We summarize the prompts for each inference
299
stage in Appendix B.
300
4
Experiments
301
4.1
Experiment Setup
302
Datasets. We evaluate our approach on two AM
303
datasets: the WOAM dataset (Xu et al., 2023) and
304
OAMine dataset (Zhang et al., 2022). The WOAM
305
dataset covers four product categories: Tea, Vita-
306
min, Sofa, and Phone Case, with over 9,000 En-
307
glish descriptions per category. The seed attribute
308
set contains 16.5 attribute types and 22 values per
309
type on average. The OAMine dataset includes 100
310
product types, with 1,943 manually annotated test
311
products across 10 types, averaging 11.5 attributes
312
per type and 48.1 unique values per attribute. Both
313
datasets are expanded to multimodal settings with
314
images collected from Amazon.com. We use 1,000
315
AGTD and 300 CTTD samples for training and
316
1,000 samples for testing.
317
4
Model
WOAM
OAMine
Similar Match
Exact Match
Similar Match
Exact Match
Precision
Recall
Precision
Recall
Precision
Recall
Precision
Recall
VisualGLM-6B
40.32±1.24
5.25±0.11
17.80±0.65
2.83±0.34
44.12±0.28 18.98±0.27 17.46±0.89
9.38±0.00
Instructblip-vicuna-7B
39.79±0.58
6.62±0.00
13.87±3.18
2.75±0.23
40.16±0.23 19.78±0.30 19.78±0.30
5.76±0.05
Qwen-VL-chat-7B
36.86±0.29 35.38±1.82 10.29±0.19 17.61±1.42 43.06±0.11 58.09±0.08 18.63±0.08 36.15±0.36
DeepSeek-VL-7B
42.93±0.25 36.51±0.45 12.15±1.33 19.95±1.49 53.00±0.02 56.05±0.47 25.03±0.47 36.10±0.56
InternLM-XComposer2-7B 52.19±0.28 31.99±0.69 23.16±0.82 19.14±0.80 69.31±0.00 45.12±0.37 36.23±0.37 27.18±0.27
LLAVA-7B
40.49±1.50 56.21±1.60 10.61±1.31 26.22±4.46 47.07±0.08 54.99±0.08 16.71±0.02 26.82±0.03
LLAVA-13B
41.22±3.29 64.62±1.59 10.52±1.35 37.72±4.91 51.87±0.21 58.59±3.27 21.46±0.99 37.75±1.19
GPT-4
52.03±1.33 65.35±4.23 15.51±2.01 41.60±5.14 64.92±0.14 55.75±2.36 29.25±2.36 33.72±0.03
OA-Mine
42.27±1.29 53.62±1.59 17.52±1.35 26.72±1.91 53.85±0.15 47.71±0.01 18.15±0.01 35.69±0.67
Amacer
51.27±0.82 58.30±0.17 21.91±0.53 29.23±0.90 58.41±0.26 51.65±0.12 22.98±1.04 38.84±0.05
MSIT(QWen)
59.95±0.40 41.60±0.57 20.00±0.11 23.75±0.34 65.01±0.19 52.44±0.13 24.61±0.01 38.13±0.08
MSIT (InternLM)
63.40±0.34 46.80±0.34 25.00±0.26 25.53±0.23 70.79±0.28 46.99±0.13 37.52±0.09 27.37±0.05
MSIT (LLAVA-7B)
66.90±0.53 66.99±2.13 35.34±2.71 52.50±3.93 74.50±0.17 63.06±0.45 54.33±0.45 51.54±0.05
Table 1: WOAM and OAMine performance of visual language models. Results are reported in precision and recall.
Evaluation Metrics. We evaluate performance
318
using two metrics: 1) Precision, the ratio of true
319
positives to total positive predictions, and 2) Re-
320
call, the proportion of true positives among all ac-
321
tual positives. We report Exact Match and Similar
322
Match as in previous work (Roy et al., 2021; Zheng
323
et al., 2018a). Exact Match requires strict consis-
324
tency with the gold standard, while Similar Match
325
allows for synonyms, treating attribute predictions
326
as correct if they match any synonym of labels.
327
Implementation Details. We leverage Pytorch and
328
one Tesla A100 GPU to implement our framework
329
and conduct experiments. The optimizer is Adam
330
and the learning rate is set to 3e-4. LoRA is em-
331
ployed to fine-tune the three MLLMs as stated in
332
Section 3.3. We train our models for 10 epochs.
333
During inference, we employ top_p sampling as
334
our type of decoding. The temperature and top_p
335
are set to 0.2 and 0.7 respectively. We report the
336
means and standard deviations of 5 independent
337
trials. For each trial, we utilize a random seed to
338
ensure fairness.
339
Baselines. We compare our model with several
340
strong MLLMs: VisualGLM-6B (Ding et al.,
341
2021), a multimodal conversational model support-
342
ing images and both Chinese and English; Instruct-
343
BLIP (Dai et al., 2023), which uses the BLIP-2
344
architecture for visual instruction tuning; Qwen-
345
VL-chat-7B (Bai et al., 2023), a model for image-
346
related reasoning in text-oriented tasks; DeepSeek-
347
VL (Lu et al., 2024), an open-source model for real-
348
world vision-language understanding; InternLM-
349
200
400
600
800
(a) Data Size of AGTD
20
30
40
50
60
70
Percentage (%)
100
200
300
(b) Data Size of CTTD
20
30
40
50
60
70
Percentage (%)
Similar Match Precision
Similar Match Recall
Exact Match Precision
Exact Match Recall
Figure 3: Performance comparison with different data
size of AGTD and CTTD on WOAM dataset.
XComposer2 (Dong et al., 2024), which special-
350
izes in text-image comprehension; LLAVA (Liu
351
et al., 2023), another open-source MLLM of two
352
scopes; and GPT-4 (OpenAI, 2023), a multimodal
353
model by OpenAI. We also compare with exist-
354
ing AM baselines: OA-Mine (Zhang et al., 2022),
355
which uses meta-classifiers and clustering for at-
356
tribute values, and Amacer (Xu et al., 2023), which
357
improves generalization with self-supervised regu-
358
larization.
359
4.2
Main Results
360
Table 1 presents the results of our experiments
361
on the WOAM and OAMine datasets. We fine-
362
tuned several multi-modal language models, includ-
363
ing LLAVA-7B, Qwen, and InternLM, and com-
364
pared MSIT with these baseline models. It can be
365
observed that MSIT significantly outperforms all
366
baseline methods in both Similar Match and Exact
367
Match metrics across both datasets. On the WOAM
368
dataset, MSIT achieves the highest Similar Match
369
precision of 66.90%, compared to the next best
370
5
GPT-4-image
GPT-4-bullet
GPT-4-title
MSIT-image
MSIT-bullet
MSIT-title
0
20
40
60
80
Percentage (%)
Similar Match Precision
Similar Match Recall
Exact Match Precision
Exact Match Recall
Figure 4: Performance comparison with GPT-4 over
different input.
model, GPT-4, which achieves 52.03%. MSIT also
371
excels in recall, with a score of 66.99%, far sur-
372
passing the second-best score of 65.35% by GPT-4.
373
In terms of Exact Match, MSIT shows a substan-
374
tial improvement with a precision of 35.34% and a
375
recall of 52.50%, compared to GPT-4’s precision
376
of 15.51% and recall of 41.60%.
377
On the OAMine dataset, MSIT maintains its su-
378
perior performance with a Similar Match precision
379
of 74.50% and recall of 63.06%. The second-best
380
model, InternLM-XComposer2-7B, achieves a pre-
381
cision of 69.31% and recall of 45.12%. For Exact
382
Match, MSIT achieves a precision of 54.33% and a
383
recall of 51.54%, significantly higher than Amacer,
384
which records a precision of 22.98% and recall of
385
38.84%.
386
It could be found that MSIT improves the per-
387
formance of LLAVA-7B by a large margin from
388
10.61% to 35.34% on the precision of Exact Match.
389
This result demonstrates that the multimodal in-
390
struction tuning significantly elicits the attribute
391
mining ability of MLLMs. OA-Mine and Amacer,
392
which focus on self-supervised learning and meta-
393
classifier techniques, show competitive perfor-
394
mance but are still outperformed by MSIT, par-
395
ticularly in the Exact Match metrics.
396
4.3
Ablation Study
397
Ablation Study Results. Table 2 shows the results
398
of our ablation study on the WOAM and OA-Mine
399
datasets, evaluating the impact of different MSIT
400
components: Attribute Generation Tuning Data
401
(AGTD), Chain-of-Thought Tuning Data (CTTD),
402
and the 3-stage inference process.
403
Training Components. Using AGTD alone leads
404
to moderate improvements in Similar Match and
405
similar_pre
exact_pre
similar_rec
exact_rec
(a) Sofa
0.2
0.4
0.6
InternLM-XComposer2-7B
Amacer
OA-Mine
GPT4
MSIT
similar_pre
exact_pre
similar_rec
exact_rec
(b) Phone
0.2
0.4
0.6
similar_pre
exact_pre
similar_rec
exact_rec
(c) Tea
0.2
0.4
0.6
similar_pre
exact_pre
similar_rec
exact_rec
(d) Vitamin
0.2
0.4
0.6
Figure 5: Domain Adaptation performance of different
methods with cross-validation on WOAM dataset.
Exact Match metrics. For instance, on the WOAM
406
dataset, it achieves a Similar Match precision of
407
62.15% and recall of 62.52%. CTTD also improves
408
performance, but less significantly (e.g., Similar
409
Match precision of 50.15% and recall of 45.88%
410
on WOAM). When combined, AGTD and CTTD
411
produce the best results, with Similar Match pre-
412
cision reaching 63.89% and recall at 67.42% on
413
WOAM.
414
Inference Components. Stage 1 alone provides
415
a baseline improvement, but its effectiveness in-
416
creases when combined with AGTD and CTTD
417
(e.g., 63.89% Similar Match precision on WOAM).
418
Introducing Stage 2 further enhances performance,
419
reducing computational costs while improving pre-
420
cision. For example, combining Stage 2 with Stage
421
1, AGTD, and CTTD results in 63.72% Similar
422
Match precision and 67.91% recall. MSIT with-
423
out Stage 2 shows slightly better recall but signif-
424
icantly lower precision. The addition of Stage 3,
425
incorporating 5-step chain-of-thought reasoning,
426
boosts both precision and recall. The full combina-
427
tion achieves the highest performance, with Similar
428
Match precision of 66.90% and recall of 66.99%.
429
4.4
Discussion on Output Format of AGTD
430
We also conduct experiments on the impact of out-
431
put format of AGTD as shown in Table 3. The re-
432
sults indicate that including both attributes and their
433
corresponding values in the output format of AGTD
434
significantly enhances model performance. Specif-
435
ically, when the format is changed to outputting
436
attributes only, the Exact Match Precision and Re-
437
call would decrease 28.68% and 38.37% respec-
438
tively. This demonstrates that providing attribute-
439
6
Components
WOAM
OA-Mine
Training
Inference
Similar Match
Exact Match
Similar Match
Exact Match
AGTD CTTD S1 S2 S3 Precision
Recall
Precision
Recall
Precision
Recall
Precision
Recall
40.49±1.50 56.21±1.60 10.61±1.31 26.22±4.46 47.07±0.08 54.99±0.08 16.71±0.02 26.82±0.03
✔
✔
✔
✔63.72±0.83 67.91±1.79 29.47±2.14 53.69±4.27 71.48±0.13 63.82±0.00 46.93±0.14 52.82±0.00
✔
✔✔
62.15±1.46 62.52±3.19 28.27±3.86 46.85±5.48 72.07±0.15 61.80±0.15 51.35±0.01 49.58±0.00
✔
✔
✔
60.37±1.93 64.46±2.89 23.30±2.30 49.03±6.45 68.14±0.10 62.79±0.00 41.56±0.06 51.32±0.00
✔
✔✔✔50.15±2.23 45.88±0.50 22.01±3.30 29.97±4.58 47.98±0.38 36.48±0.02 21.81±0.07 19.52±0.05
✔
✔
✔✔
63.89±1.50 67.42±2.06 32.61±1.06 52.86±4.06 70.79±0.17 63.18±0.27 49.87±0.00 51.81±0.02
✔
✔
✔✔✔66.90±0.53 66.99±2.13 35.34±2.71 52.50±3.93 74.50±0.17 63.06±0.45 54.33±0.45 51.54±0.00
Table 2: Ablation Study on different components of MSIT on WOAM and OAMine datasets.
Format
Similar Match
Exact Match
Precision
Recall
Precision
Recall
Attri-only
58.19±4.23
50.81±1.26
28.68±0.45
38.37±0.34
Attri-Value
66.90±0.53
66.99±2.13
35.34±2.71
52.50±3.93
Table 3: Performance comparison on output format in
AGTD.
value pairs rather than attributes alone improves
440
the model’s ability to extract product attributes.
441
4.5
Analysis of Training Data Size
442
In this section, we analyze the impact of different
443
training data sizes on the performance of our MSIT.
444
Figure 3 (a) illustrates the performance metrics on
445
the WOAM dataset with varying sizes of AGTD.
446
As the data size increases from 200 to 900, Similar
447
Match precision improves from 54.76% to 56.70%.
448
This result indicates that increasing AGTD size
449
generally enhances the model’s ability to identify
450
similar attributes. Both precision and recall met-
451
rics in Exact Match show a consistent improvement
452
with increasing data size. Precision increases from
453
26.13% to 30.85%, while recall rises from 40.31%
454
to 43.46%. The narrow error margins suggest ro-
455
bustness in these trends. Figure 3 (b) presents the
456
performance metrics on the WOAM dataset with
457
different sizes of CTTD. For Similar Match, pre-
458
cision rises from 63.36% to 66.90%, and recall
459
increases from 53.88% to 66.99% as CTTD size
460
grows. These trends suggest that CTTD size has a
461
substantial impact on improving the model’s abil-
462
ity to accurately identify similar attributes. Exact
463
Match precision improves slightly from 35.21% to
464
35.34%, and recall rises from 42.89% to 52.50%.
465
The improvement indicates that explicit reasoning
466
through CoT data significantly aids in identifying
467
exact attribute matches.
468
93.3%
6.7%
1.4%
98.6%
Positive Judgments
Negative Judgments
Misjudged Negatives
Self-corrected Negatives
Figure 6: Visualization of Self-Correction after 3-stage
inference.
4.6
Analysis of Attribute Mining from
469
Different Input
470
The bar chart in Figure 4 illustrates the perfor-
471
mance of MSIT compared to GPT-4 across dif-
472
ferent input types: images, bullet points, and titles.
473
As illustrated in Figure 4, our MSIT method gen-
474
erally exhibits superior performance across most
475
metrics. For instance, MSIT achieves a notable
476
Similar Match Precision of 69.21% for images,
477
compared to GPT-4’s 63.81%, and an Exact Match
478
Precision of 40.96% versus GPT-4’s 23.81%. Fur-
479
thermore, MSIT outperforms GPT-4 significantly
480
in bullet points recall, achieving a Similar Match
481
Recall of 58.37% compared to GPT-4’s 47.19%.
482
Additionally, from the recall distribution it could
483
be observed that images consistently yield a high
484
number of extracted attributes. This observation un-
485
derscores the validity of our multimodal approach,
486
affirming that leveraging multiple modalities can
487
uncover an extensive range of product attributes.
488
4.7
Analysis of Domain Adaptation
489
Our domain adaptation experiment is designed to
490
evaluate the generalizability of our model in real-
491
world scenarios, where it may encounter previ-
492
ously unseen product categories. We conduct cross-
493
7
(Product Type Judgement) The product is
identified as \"Honey Sticks, Mango\" .
(Judge With External Knowledge): The attribute
'Packaging Type' refers to the way the product is
packaged or presented.
(Judge form Image)However, the image clearly
shows the product packaged in a pouch.
(Judge from Text)The text provided does not
explicitly mention the packaging type.
Conclusion: Yes
Product: "English Tea Store Honey Sticks,
Mango, 1.3 Pound"
Product Text: Add a tropical treat to your
tea, Wonderful snacks for the kids, Honey
sticks are 6 inches long, Gluten Free
(Sequential Attribute Inference)
"Packaging Type"
Conclusion: Yes
(Sequential Attribute Inference)
"Return Policy"
(Product Type Judgement) The product is
identified as \"Honey Sticks, Mango\" .
(Judge With External Knowledge): The attribute
'return_policy' refers to the conditions under which
the product can be returned and refunded.
(Judge form Image)From the image, there is no
explicit indication of a return policy.
(Judge from Text)The text provided mentions a \
"90 DAY MONEY BACK GUARANTEE\" and \
"BUY WITH CONFIDENCE.\" This indicates that
the product has a return policy.
(Product Type Judgement) The product is
identified as \"Honey Sticks, Mango\" .
(Judge With External Knowledge): The attribute
'return_policy' refers to the conditions under which
the product can be returned and refunded.
(Judge form Image)From the image, there is no
explicit indication of a return policy.
(Judge from Text)The text provided mentions a \
"90 DAY MONEY BACK GUARANTEE\" and \
"BUY WITH CONFIDENCE.\" This indicates that
the product has a return policy.
Product: "SckoonCup Beginner Choice -
Made in The USA"
Product Text: "...90 DAY MONEY BACK
GUARANTEE: Only Sckoon can
guarantee the quality. Return for a refund in
90 days., BUY WITH CONFIDENCE:
from Sckoon Organics an innovative..."
(b)
(a)
Figure 7: Case study on Attribute Mining with MSIT.
validation on the WOAM dataset across four cat-
494
egories: Sofa, Phone, Tea, and Vitamin. For each
495
experiment, we train the model using data from
496
three categories and evaluated its performance on
497
the fourth category. The results, depicted in Fig-
498
ure 5, reveal that MSIT consistently outperforms
499
other models—InternLM, Amacer, OA-Mine, and
500
GPT-4—across all categories. MSIT achieves the
501
highest Similar Match Precision and Exact Match
502
Precision in every category, with notable perfor-
503
mances in the Sofa (69.21% and 40.96%, respec-
504
tively) and Tea (69.21% and 40.96%, respectively)
505
categories. The results confirm the robustness of
506
MSIT in attribute extraction.
507
4.8
Effectiveness of Self-Correction
508
The results shown in Figure 6 demonstrate the ef-
509
fectiveness of MSIT in the final inference stage by
510
evaluating the distribution of attribute judgments.
511
Out of a total of 1028 attributes, a significant pro-
512
portion (93.3%) was judged as positive, with a
513
small percentage (6.7%) labeled as negative judg-
514
ments. Within these negative judgments, 98.6%
515
were accurately self-corrected, while only 1.4%
516
were misjudged. This indicates that our method
517
effectively self-corrects nearly all incorrect nega-
518
tive judgments, thus enhancing overall accuracy.
519
Although there is a slight reduction in recall, MSIT
520
achieves the minimal trade-off in recall for substan-
521
tial precision gains.
522
Batch Generation
Ours
Product Text:
"Chiquita Banana
Bread Mix 13.7 oz (1
Box makes 12
Delicious Slices) by
Chiquita"
Product Text:
"Chiquita Banana
Bread Mix 13.7 oz (1
Box makes 12
Delicious Slices) by
Chiquita"
Sequential
Attribute Inference
Sequential
Attribute Inference
{ "Attributes":
      {
  "brand": "Chiquita",
  "product_name": "Banana Bread
Mix",
  "type": "Banana Bread Mix",
  "net_weight": "13.7 oz (380g)",
  "quantity": "13.7 oz ",
  "ingredients": ["Bananas",
"Flour", ..., "Salt"],
  "flavor": "Hearty"
}
Repeated Attribute
Filter
Repeated Attribute
Filter
{ "servings": "Unknown"
    "special_features":
"Unknown"
    "mfg_date": "Unknown",
    "best_by_date":
"Unknown"...}
Attribute Generated
by Qwen
{ "servings": "Unknown"
    "special_features":
"Unknown"
    "mfg_date": "Unknown",
    "best_by_date":
"Unknown"...}
Attribute Generated
by Qwen
{  "Package Type": "Stand-up
pouch with a resealable zipper"
    "Colors": "Predominantly
yellow"
    "Brand History": Chiquita is
a well-known for bananas.}
}
Attribute Generated by
GPT4
{  "Package Type": "Stand-up
pouch with a resealable zipper"
    "Colors": "Predominantly
yellow"
    "Brand History": Chiquita is
a well-known for bananas.}
}
Attribute Generated by
GPT4
Figure 8: Case study on comparison with other methods.
4.9
Case Study
523
Figure 7 presents two examples of multimodal
524
attribute mining with MSIT. In the ‘Packaging
525
Type’ scenario, the system correctly identifies the
526
attribute using visual cues, as the text lacks ex-
527
plicit information. Conversely, in the ‘Return Pol-
528
icy’ example, textual evidence, such as ‘90 DAY
529
MONEY BACK GUARANTEE’, is used to con-
530
firm the attribute when the image is insufficient.
531
These examples highlight MSIT’s capability to han-
532
dle real-world e-commerce scenarios where prod-
533
uct information may be incomplete or ambiguous.
534
As shown in Figure 8, we compare MSIT to other
535
models, such as Qwen and GPT-4. For the prod-
536
uct ‘Chiquita Banana Bread Mix’, Qwen struggles
537
to extract relevant attributes, often returning ‘Un-
538
known’ for key information such as ‘servings’ and
539
‘mfg_date.’ GPT-4 performs better by identifying
540
attributes like ‘Package Type’ and ‘Colors’, but it
541
misses important details about the product’s con-
542
tents and characteristics. In contrast, MSIT accu-
543
rately extracts a comprehensive set of attributes,
544
including the brand, product name, etc., addressing
545
the limitations observed in Qwen and GPT-4.
546
4.10
Conclusion
547
This paper presents Multimodal Self-Correct In-
548
struction Tuning (MSIT), a novel framework for
549
Open-World E-commerce Product Attribute Min-
550
ing. By utilizing both textual and visual data, MSIT
551
addresses the limitations of current methods, par-
552
ticularly the lack of multimodal information and
553
explicit reasoning. MSIT self-corrects the false pos-
554
itive attributes through a 5-step chain-of-thought
555
reasoning. Extensive experiments on the WOAM
556
and OAMine datasets demonstrate that MSIT sig-
557
nificantly enhances precision and recall compared
558
to state-of-the-art methods. Our framework also
559
shows robustness in domain adaptation scenarios,
560
highlighting its potential for real-world applica-
561
tions. In future work, we aim to explore the inte-
562
gration of additional modalities of MSIT to other
563
domains that require detailed attribute analysis.
564
8
Limitations
565
The main limitations of our work are related to
566
the scope of MLLMs fine-tuning.
Due to re-
567
source constraints, we conducted fine-tuning on
568
three 7B-parameter MLLMs (LLAVA, Qwen and
569
InternLM), without extending our efforts to larger-
570
scale MLLMs such as those with 13B parameters
571
or beyond.
572
References
573
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc,
574
Antoine Miech, Iain Barr, Yana Hasson, Karel
575
Lenc, Arthur Mensch, Katherine Millican, Malcolm
576
Reynolds, Roman Ring, Eliza Rutherford, Serkan
577
Cabi, Tengda Han, Zhitao Gong, Sina Samangooei,
578
Marianne Monteiro, Jacob L. Menick, Sebastian
579
Borgeaud, Andy Brock, Aida Nematzadeh, Sahand
580
Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira,
581
Oriol Vinyals, Andrew Zisserman, and Karén Si-
582
monyan. 2022. Flamingo: a visual language model
583
for few-shot learning. In NeurIPS.
584
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang,
585
Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou,
586
and Jingren Zhou. 2023. Qwen-vl: A frontier large
587
vision-language model with versatile abilities. CoRR,
588
abs/2308.12966.
589
Wenliang Dai, Junnan Li, Dongxu Li, Anthony
590
Meng Huat Tiong, Junqi Zhao, Weisheng Wang,
591
Boyang Li, Pascale Fung, and Steven C. H. Hoi.
592
2023. Instructblip: Towards general-purpose vision-
593
language models with instruction tuning. In NeurIPS.
594
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng,
595
Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou
596
Shao, Hongxia Yang, and Jie Tang. 2021. Cogview:
597
Mastering text-to-image generation via transformers.
598
CoRR, abs/2105.13290.
599
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao,
600
Bin Wang, Linke Ouyang, Xilin Wei, Songyang
601
Zhang, Haodong Duan, Maosong Cao, Wenwei
602
Zhang, Yining Li, Hang Yan, Yang Gao, Xinyue
603
Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui
604
He, Xingcheng Zhang, Yu Qiao, Dahua Lin, and
605
Jiaqi Wang. 2024. Internlm-xcomposer2: Master-
606
ing free-form text-image composition and compre-
607
hension in vision-language large model.
CoRR,
608
abs/2401.16420.
609
Jinmiao Fu, Shaoyuan Xu, Huidong Liu, Yang Liu, Ning
610
Xie, Chien-Chih Wang, Jia Liu, Yi Sun, and Bryan
611
Wang. 2022. CMA-CLIP: cross-modality attention
612
clip for text-image classification. In ICIP, pages
613
2846–2850. IEEE.
614
Rayid Ghani, Katharina Probst, Yan Liu, Marko Krema,
615
and Andrew E. Fano. 2006. Text mining for product
616
attribute extraction. SIGKDD Explor., 8(1):41–48.
617
Giannis Karamanolakis, Jun Ma, and Xin Luna Dong.
618
2020. Txtract: Taxonomy-aware knowledge extrac-
619
tion for thousands of product categories. In ACL,
620
pages 8489–8502. Association for Computational
621
Linguistics.
622
Anant Khandelwal, Happy Mittal, Shreyas Sunil Kulka-
623
rni, and Deepak Gupta. 2023. Large scale generative
624
multimodal attribute extraction for e-commerce at-
625
tributes. In ACL (industry), pages 305–312. Associa-
626
tion for Computational Linguistics.
627
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H.
628
Hoi. 2023a. BLIP-2: bootstrapping language-image
629
pre-training with frozen image encoders and large lan-
630
guage models. In ICML, volume 202 of Proceedings
631
of Machine Learning Research, pages 19730–19742.
632
PMLR.
633
Yanzeng Li, Bingcong Xue, Ruoyu Zhang, and Lei Zou.
634
2023b. Attgen: Attribute tree generation for real-
635
world attribute joint extraction. In ACL (1), pages
636
2139–2152. Association for Computational Linguis-
637
tics.
638
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae
639
Lee. 2023. Visual instruction tuning. In NeurIPS.
640
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai
641
Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhu-
642
oshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng,
643
Hanwei Xu, Zhenda Xie, and Chong Ruan. 2024.
644
Deepseek-vl: Towards real-world vision-language
645
understanding. CoRR, abs/2403.05525.
646
Kartik Mehta, Ioana Oprea, and Nikhil Rasiwasia. 2021.
647
Latex-numeric: Language-agnostic text attribute ex-
648
traction for e-commerce numeric attributes. CoRR,
649
abs/2104.09576.
650
OpenAI. 2023.
GPT-4 technical report.
CoRR,
651
abs/2303.08774.
652
Kalyani Roy, Pawan Goyal, and Manish Pandey. 2021.
653
Attribute value generation from product title using
654
language models. In Proceedings of The 4th Work-
655
shop on e-Commerce and NLP, pages 13–17.
656
Keiji Shinzato, Naoki Yoshinaga, Yandi Xia, and Wei-
657
Te Chen. 2022. Simple and effective knowledge-
658
driven query expansion for qa-based product attribute
659
extraction. In ACL (2), pages 227–234. Association
660
for Computational Linguistics.
661
Keiji Shinzato, Naoki Yoshinaga, Yandi Xia, and Wei-
662
Te Chen. 2023. A unified generative approach to
663
product attribute-value identification. In Findings of
664
the Association for Computational Linguistics: ACL
665
2023, Toronto, Canada, July 9-14, 2023, pages 6599–
666
6612. Association for Computational Linguistics.
667
Huimin Xu, Wenting Wang, Xin Mao, Xinyu Jiang, and
668
Man Lan. 2019. Scaling up open tagging from tens
669
to thousands: Comprehension empowered attribute
670
value extraction from product title. In Proceedings
671
9
of the 57th Conference of the Association for Compu-
672
tational Linguistics, ACL 2019, Florence, Italy, July
673
28- August 2, 2019, Volume 1: Long Papers, pages
674
5214–5223. Association for Computational Linguis-
675
tics.
676
Liyan Xu, Chenwei Zhang, Xian Li, Jingbo Shang, and
677
Jinho D. Choi. 2023. Towards open-world product
678
attribute mining: A lightly-supervised approach. In
679
ACL (1), pages 12223–12239. Association for Com-
680
putational Linguistics.
681
Jun Yan, Nasser Zalmout, Yan Liang, Christan Grant,
682
Xiang Ren, and Xin Luna Dong. 2021.
Adatag:
683
Multi-attribute value extraction from product profiles
684
with adaptive decoding. In Proceedings of the 59th
685
Annual Meeting of the Association for Computational
686
Linguistics and the 11th International Joint Confer-
687
ence on Natural Language Processing, ACL/IJCNLP
688
2021, (Volume 1: Long Papers), Virtual Event, Au-
689
gust 1-6, 2021, pages 4694–4705. Association for
690
Computational Linguistics.
691
Xinyang Zhang, Chenwei Zhang, Xian Li, Xin Luna
692
Dong, Jingbo Shang, Christos Faloutsos, and Jiawei
693
Han. 2022. Oa-mine: Open-world attribute mining
694
for e-commerce products with weak supervision. In
695
WWW, pages 3153–3161. ACM.
696
Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. 2023a.
697
What makes good examples for visual in-context
698
learning? In NeurIPS.
699
Yupeng Zhang, Shensi Wang, Peiguang Li, Guanting
700
Dong, Sirui Wang, Yunsen Xian, Zhoujun Li, and
701
Hongzhi Zhang. 2023b. Pay attention to implicit
702
attribute values: A multi-modal generative frame-
703
work for AVE task. In ACL (Findings), pages 13139–
704
13151. Association for Computational Linguistics.
705
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao,
706
George Karypis, and Alex Smola. 2023c.
Multi-
707
modal chain-of-thought reasoning in language mod-
708
els. CoRR, abs/2302.00923.
709
Guineng Zheng, Subhabrata Mukherjee, Xin Luna
710
Dong, and Feifei Li. 2018a.
Opentag: Open at-
711
tribute value extraction from product profiles. CoRR,
712
abs/1806.01264.
713
Guineng Zheng, Subhabrata Mukherjee, Xin Luna
714
Dong, and Feifei Li. 2018b.
Opentag: Open at-
715
tribute value extraction from product profiles. In
716
KDD, pages 1049–1058. ACM.
717
Henry Peng Zou, Gavin Heqing Yu, Ziwei Fan, Dan
718
Bu, Han Liu, Peng Dai, Dongmei Jia, and Cor-
719
nelia Caragea. 2024. Eiven: Efficient implicit at-
720
tribute value extraction using multimodal llm. arXiv
721
preprint arXiv:2404.08886.
722
10
A
Prompts for data generation
723
A.1
Prompt for Generation of text attributes
724
and image attributes
725
The prompt generated by the data in this stage is
726
mainly divided into three parts. The first part is the
727
Task Description, the second part is In-context
728
Learning, and the third part is to provide product
729
text or image information. Since we generate text
730
and image attributes separately, the information in
731
the third part is provided separately. The follow-
732
ing shows the prompt that enables GPT4 to extract
733
text attributes.To generate image attributes, we only
734
need to extract image attributes in the task descrip-
735
tion and change the third part of the information to
736
image information.
737
738
Task Description:
739
You are a world-class algorithm for extracting
740
information in structured formats.
741
There are some product descriptions, and your
742
task is to extract the attribute values from the text
743
information of the product in a JSON format.
744
Please provide me with the corresponding at-
745
tribute value of the attribute. If there is no corre-
746
sponding attribute value in the information I pro-
747
vide you, please do not provide me with this at-
748
tribute.
749
In-context learning:
750
751
{
752
"type": "Sofa",
753
"material_frame": "Gold legs",
754
"style": "Modern, minimalist",
755
"size": "Three-Seater Sofa",
756
"color": "White",
757
"Padding": "High-Density Foam",
758
"Accessories": "Throw Pillows",
759
"Special Features": "Electric Reclining",
760
"Maintenance Requirements": "Dry Clean Only"
761
}
762
763
{
764
"brand": "Traditional Medicinals",
765
"type": "herbal tea",
766
"flavor": "eucalyptus and mint",
767
"caffeine_content": "caffeine-free",
768
"quantity": "16 tea bags",
769
"Packaging Type": "Tea bags",
770
"Storage conditions": "Dry and Well-
771
Ventilated Area",
772
"Processing Level": "Fermented",
773
"Aroma": "Rich",
774
"Tea Benefits": ["Refreshment", "Digestive
775
Aid"]
776
}
777
778
{
779
"brand": "Rugby",
780
"product_name": "Tab-A-Vite Multivitamin
781
Tablets",
782
"serving_size": "2 tablets",
783
"number_of_servings": 30,
784
"Dosage Form": "Tablet",
785
"Target Audience": "Adults",
786
"Indications": "Vitamin Supplementation",
787
"key_nutrients": [
788
"Thiamin (as Thiamine HCI)",
789
"Vitamin B6 (Pyridoxine HCI)",
790
"Calcium (as Dicalcium Phosphate)",
791
"Magnesium (as Magnesium Oxide)"
792
]
793
}
794
795
{
796
"Type": "Phone case",
797
"Material": "Silicone",
798
"Design": "Transparent",
799
"Function": "Shockproof",
800
"Compatibility": "Compatible Models",
801
"Color": "Black",
802
"Thickness": "Ultra-thin",
803
"Weight": "Lightweight",
804
"Texture": "Smooth"
805
}
806
807
Text information of the product:
808
Below is the text information of the product whose
809
attributes I want you to extract
810
Title: ...
811
Bullet point: ...
812
A.2
Prompt for merging the text attributes
813
and image properties of the product
814
In this step, we merge the image and text attributes
815
generated separately previously.
816
817
Task Description
818
The following is the information of the attribute
819
value pairs extracted from the image and text of
820
the same product respectively. Please help me
821
merge them into one. The same attributes will
822
be regarded as one after being merged.If I only
823
provide text or image information, then there is no
824
need to merge and directly output the text or image
825
information I provide.If you encounter a attribute
826
like Features, which is a bit general, try to give
827
more detailed attribute.Please let the output follow
828
the json format strictly and do not send me any
829
other text.
830
831
Image and text attributes
832
Image
attributes:{Attribute1:value1,
At-
833
tribute2:value2, ...}
834
11
Text
attributes:{Attribute1:value1,
At-
835
tribute2:value2, . . . }
836
837
A.3
Prompt for Generation of
838
Chain-of-Thought Tuning Data
839
840
Task Description
841
I will provide you with product images as well
842
as text information and attributes. Please judge
843
whether the product has this attribute.Please follow
844
the steps below to reason step by step and give
845
your reasoning process.
846
847
Five-step Chain of thought
848
Step 1: In this step, you need to determine
849
the type of product based on pictures and
850
text information, such as whether the product is
851
a mobile phone case, tea, or other types of products.
852
853
Step 2: This step requires analyzing the meaning
854
of the attributes.
If the attribute’s meaning is
855
unclear, we will make a preliminary determination
856
that it cannot be considered a product attribute. If
857
the intent of the attribute is clear, use common
858
sense to initially judge whether the attribute
859
matches the product type, and initially explain the
860
meaning of the attribute and why it may match the
861
product.
862
863
Step 3: If you preliminarily judge in the second
864
step that this type of product may have this
865
attribute, then please use the picture I provided to
866
guess its attribute value to confirm that the product
867
indeed has this attribute.
Since images do not
868
provide explicit attribute value information, there
869
is no need to derive exact attribute values. You
870
only need to determine a rough attribute value to
871
confirm.
872
873
Step 4: If you preliminarily judge that this
874
type of product may have this attribute in the
875
second step, then in this step, please use the text
876
I provided to guess its attribute value to confirm
877
that the product indeed has this attribute. If you
878
inferred an attribute value from the text, give the
879
exact attribute value
880
881
Step 5, please combine the reasoning from the
882
above steps to draw a conclusion whether the
883
product has this attribute.
884
885
Please mark the last paragraph with yes or no.
886
No other text is needed in this paragraph.
887
888
The product’s text and image information and
889
the attributes that need to be judged
890
Image:...
891
Title: ...
892
Bullet Point: ...
893
The attribute i want to judge is Attributei
894
895
B
Inference Instructions
896
B.1
Instructions for Batch Attribute
897
Generation
898
The list of instructions used for Batch Attribute
899
Generation is shown in Table 4. They present the
900
same meaning with natural language variance.
901
B.2
Deletion rules for similar meaning
902
attributes in rules
903
For two phrases, if the subwords of any phrase
904
correspond one-to-one with the subwords of the
905
other phrase, we determine that the two phrases are
906
synonyms.
907
To determine whether two subwords are simi-
908
lar, we use the word2vec-google-news-300 model,
909
calculate the cosine similarity of the two subword
910
vectors, and set the threshold.If one of the phrases
911
has only one subword, we will set a higher thresh-
912
old to determine similarity, because a subword is
913
more likely to correspond to a subword in another
914
phrase.
915
For two phrases that are judged to have similar
916
meanings, we must choose one to delete. Our judg-
917
ment rules can be summarized into the following
918
three rules.
919
1. Rule 1: If there is only one attribute with
920
one subword number of two attributes (assum-
921
ing that this attribute is attribute A and the
922
other phrase is attribute B), the attribute to be
923
deleted depends on the number of subwords
924
of attribute B. If it is equal to three, attribute
925
A is deleted, and if it is greater than three,
926
attribute B is deleted.
927
2. Rule 2: If two attributes have one attribute
928
with two subwords (assuming it is attribute A),
929
and the other attribute has three or more sub-
930
words (assuming it is attribute B), we choose
931
to delete attribute B.
932
12
Instructions
• “Extract the information from the title, bul-
let points, and product picture into JSON for-
mat.”
• “Convert the attribute values of the product
from the provided information into JSON for-
mat.”
• “Generate the product attribute values in JSON
format based on the provided title, bullet
points, and picture.”
• “Compile the product’s characteristic at-
tributes into JSON format according to the
provided information.”
• “Extract the product attribute information into
JSON format from the provided title, bullet
points, and picture.”
• “Parse the product’s features and attributes
into JSON format from the given informa-
tion.”
• “Extract the product’s characteristics into
JSON format using the provided title, bullet
points, and picture.”
• “Retrieve and organize the product’s attribute
values into JSON format from the provided
information.”
• “Compile the product’s attribute information
into JSON format based on the title, bullet
points, and picture content.”
• “Generate JSON-formatted product attribute
data based on the provided information.”
Table 4: List of instructions for Batch Attribute Genera-
tion.
3. Rule 3: If two attributes have the same num-
933
ber of subwords, we randomly pick one and
934
delete it.
935
B.3
Instructions for Filtering Wrong
936
Attributes
937
The list of instructions used for Filtering Wrong
938
Attributes is shown in Table 5. They present the
939
same meaning with natural language variance.
940
Instructions
• “I provide you with text and image informa-
tion of a product along with one attribute of
this product. Please determine if this product
possesses this attribute through the text and
image. Let’s think step by step, and please
provide your reasoning process.”
• “Here’s textual and visual data about a prod-
uct, along with a specific attribute. Your job is
to discern if this attribute applies to the prod-
uct, using both the text and the visuals. Let’s
methodically analyze the information, detail-
ing your reasoning process step by step.”
• “I present you with textual and visual data
about a product, along with a single attribute
associated with it. Your task is to determine
whether this product exhibits this attribute,
utilizing both the text and the image. Let’s
think step by step,and please provide your
reasoning process.”
• “I provide you with the text and picture infor-
mation of a product and an attribute of this
product, please help me judge whether this
product has this attribute through the text and
picture, we will think step by step, please give
your reasoning process.”
• “I will give you the text and image information
of a product, as well as one of its attributes.
Please use the text and image to help me de-
termine whether the product has this attribute.
Let’s think step by step, and please explain
your reasoning process.”
Table 5: List of instructions for Filtering Wrong At-
tributes.
13