Removing stop words in Java as part of data cleaning in Artificial IntelligenceData Science by Sunny Srinidhi - February 5, 2020February 5, 20200 More in The fastText Series. Working with text datasets is very common in data science problems. A good example of this is sentiment analysis, where you get social network posts as data sets. Based on the content of these posts, you need to estimate the sentiment around a topic of interest. When we're working with text as the data, there are a lot of words which we want to remove from the data to "clean" it, such as normalising, removing stop words, stemming, lemmatizing, etc. In this post, we'll see how we can remove stop words from our input text to clean our data so that our analysis is based only on the actual content of the data. But wait, what are stop
Optimising a fastText model for better accuracyData Science by Sunny Srinidhi - December 3, 2019December 19, 20190 More in The fastText Series. In our previous post, we saw what n-grams are and how they are useful. Before that post, we built a simple text classifier using Facebook’s fastText library. In this post, we’ll see how we can optimise that model for better accuracy. Precision and Recall Precision and recall are two things we need to know to better understand the accuracy of our models. And these two things are not very difficult to understand. Precision is the number of correct labels that were predicted by the fastText model, and recall is the number of labels, out of the correct labels, that were successfully predicted. That might be a bit confusing, so let’s look at an example to understand it better. Suppose for a sentence
Understanding Word N-grams and N-gram Probability in Natural Language ProcessingData Science by Sunny Srinidhi - November 26, 2019December 19, 20192 More in The fastText Series. N-gram is probably the easiest concept to understand in the whole machine learning space, I guess. An N-gram means a sequence of N words. So for example, “Medium blog” is a 2-gram (a bigram), “A Medium blog post” is a 4-gram, and “Write on Medium” is a 3-gram (trigram). Well, that wasn’t very interesting or exciting. True, but we still have to look at the probability used with n-grams, which is quite interesting. Why N-gram though? Before we move on to the probability stuff, let’s answer this question first. Why is it that we need to learn n-gram and the related probability? Well, in Natural Language Processing, or NLP for short, n-grams are used for a variety of things.
An intro to text classification with Facebook’s fastText (Natural Language Processing)Data Science by Sunny Srinidhi - November 25, 2019December 19, 20193 More in The fastText Series. Text classification is a pretty common application of machine learning. In such an application, machine learning is used to categorise a piece of text into two or more categories. There are both supervised and unsupervised learning models for text classification. In this post, we’ll see how we can use Facebook’s fastText library for some simple text classification. fastText, developed by Facebook, is a popular library for text classification. The library is an open source project on GitHub, and is pretty active. The library also provides pre-built models for text classification, both supervised and unsupervised. In this post, we’ll check out how we can train the supervised model in the library for some quick text classification. The library