How to Learn Klingon without a Dictionary: Detection and Measurement of Black Keywords Used by the Underground Economy

used vocabularies. Second, many black keywords are created in ungrammatical or even obfuscated forms (e.g., letter 'l' could be replaced by digit '1' in a word) while the NLP techniques are suitable for well-written text [4].

On the other hand, based on our prior experience in investigating underground economy, we found this challenge can be addressed through a pure data-driven approach. Many underground merchants rely on blackhat SEO (search engine optimization) to promote their business. Usually, plenty of black keywords are stuffed into one SEO page inside certain HTML tags (e.g., anchor tags) to fool the search engines, yet making themselves distinguishable under content analysis. We could extract them from the SEO pages but the irrelevant texts have to be pruned. It turns out that the search results associated with a candidate keyword can be leveraged to determine whether the keyword is "black": as revealed by our study, querying a black keyword usually returns multiple links alarmed by the existing scanners, so we can use the result as the main indicator. Additionally, we found our list of black keywords can be extended through related search, a feature presented by major search engines to correlate similar search terms based on users' searching behaviors. After these steps, a lot of black keywords can be discovered, but a large portion of them are long-tail keywords which contain words not of our interest (e.g., words except "heroin" in "where to buy heroin in Beijing"). To extract the core words (e.g., "heroin" in the above example), we devised a substring matching algorithm which can process the keywords very efficiently.

We developed KDES (Keywords Detection and Expansion System) and evaluated it on more than 2 million pages related to SEO, porn and gambling. We discovered 478,879 black keywords in total and extracted 1,522 core words (433,335 black keywords are covered). After sampling the detected keywords, we found that the accuracy can achieve 94.3%, suggesting KDES is effective. We applied our findings to Baidu and the feedback was very encouraging. Many of the detected keywords have been added into their internal blacklist.