`
fengshenbang
  • 浏览: 20283 次
  • 性别: Icon_minigender_1
  • 来自: 天津
文章分类
社区版块
存档分类
最新评论

weka的StringToWordVector类解析

阅读更多

weka的StringToWordVector类可以将给定的文档格式的内容转换为vms模型的内容,而后者是文本分类必须的模块。按照weka要求,生成arff格式的文本:

 

@relation D__java_weka_data
@attribute text string
@attribute class {test1,test2,test3}
@data
'here we go go go go to do ',test1
'Mostly, I expect we are interested in indexing XPath queries',test1
'so what do you think you can do anything?',test2
'Sparse ARFF files are very similar to ARFF files',test3
 

按照StringToWordVector类的命令格式,设定options:

 

String[] options = { "-C", "-T", "-i", "data//train.arff", "-o","data//res_train.arff", "-c", "last"};

 

生成结果如下:

 

@relation 'D_java_weka_data-weka.filters.unsupervised.attribute.StringToWordVector-R1-W1000-prune-rate-1.0-C-T-N0-stemmerweka.core.stemmers.NullStemmer-M1-tokenizerweka.core.tokenizers.WordTokenizer -delimiters \" \\r\\n\\t.,;:\\\'\\\"()?!\"'

@attribute class {test1,test2,test3}
@attribute I numeric
@attribute Mostly numeric
@attribute XPath numeric
@attribute are numeric
@attribute do numeric
@attribute expect numeric
@attribute go numeric
@attribute here numeric
@attribute in numeric
@attribute indexing numeric
@attribute interested numeric
@attribute queries numeric
@attribute to numeric
@attribute we numeric
@attribute anything numeric
@attribute can numeric
@attribute so numeric
@attribute think numeric
@attribute what numeric
@attribute you numeric
@attribute ARFF numeric
@attribute Sparse numeric
@attribute files numeric
@attribute similar numeric
@attribute very numeric

@data


{5 0.693147,7 1.609438,8 0.693147,13 0.693147,14 0.693147}

{1 0.693147,2 0.693147,3 0.693147,4 0.693147,6 0.693147,9 0.693147,10 0.693147,11 0.693147,12 0.693147,14 0.693147}

{0 test2,5 1.098612,15 0.693147,16 0.693147,17 0.693147,18 0.693147,19 0.693147,20 1.098612}

{0 test3,4 0.693147,13 0.693147,21 1.098612,22 0.693147,23 1.098612,24 0.693147,25 0.693147}

 

可以发现在dataset中attribute-class中少了test1的label。

 

StringToWordVector类在转化过程中经历了如下的步骤:首先将文档中attribute为string的属性,按给定的tokenizer 分词,并生成m_dictionary,可以按照数字 look up word,为了保证最后attribute value的形式统一,对于attribute非string的属性,其值在设定的时候为test1:attribute value = 0; test2: attribute value = 1; test3: attribute value = 2。在转化完成之后,新生成的instance传入SparseInstance,剔除掉value = 0的元素。所以test1的属性段就被剔除了。

 

究其根本,还是instance 类在设计的时候没考虑好attribute为class的时候,在以后sparse会将其value=0的剔除.

 

Bug修补方法:

 

重写SparseInstance方法,指定attribute.value(0)的字段不做sparse,当然这只是暂时的策略,最根本的还是要修改instance类中attribute value的赋值方法,但由于instance类是Weka中核心类,改起来牵扯的地方太多,还是这个方便点。

 

分享到:
评论
1 楼 pmh905001 2013-11-19  
看过,对于我这个刚入门的很有用,多谢!

相关推荐

Global site tag (gtag.js) - Google Analytics