Understanding Confusion Matrix | Model Evaluation Metrics

Example of confusion matrix used to evaluate the quality of the output of a classifier on our data set. The diagonal elements represent the number of points for which the predicted label is equal to the true label, while off-diagonal elements are those that are mislabeled by the classifier. The higher the diagonal values of the confusion matrix the better, indicating many correct predictions.

The figures show the confusion matrix with and without normalization by class support size (number of elements in each class). This kind of normalization can be interesting in the case of class imbalance to have a more visual interpretation of which class is being misclassified.

#Plotting confusion matrix
def plot_confusion_matrix(y_test, y_pred, classes,
normalize=False,
title=None,
cmap=plt.cm.Blues):
"""
This function prints and plots the confusion matrix.
Normalization can be applied by setting `normalize=True`.
"""
if not title:
if normalize:
title = 'Normalized confusion matrix'
else:
title = 'Confusion matrix, without normalization'

# Compute confusion matrix
cm = confusion_matrix(y_test, y_pred)
# Only use the labels that appear in the data
# For unique lebel read this:
# https://scikit-learn.org/stable/modules/generated/sklearn.utils.multiclass.unique_labels.html#sklearn.utils.multiclass.unique_labels

classes = unique_labels(y_test, y_pred)
if normalize:
cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
print("Normalized confusion matrix")
else:
print('Confusion matrix, without normalization')

print(cm)

fig, ax = plt.subplots()
im = ax.imshow(cm, interpolation='nearest', cmap=cmap)
ax.figure.colorbar(im, ax=ax)
# We want to show all ticks...
ax.set(xticks=np.arange(cm.shape[1]),
yticks=np.arange(cm.shape[0]),
# ... and label them with the respective list entries
xticklabels=classes, yticklabels=classes,
title=title,
ylabel='True label',
xlabel='Predicted label')

# Rotate the tick labels and set their alignment.
plt.setp(ax.get_xticklabels(), rotation=45, ha="right",
rotation_mode="anchor")

# Loop over data dimensions and create text annotations.
fmt = '.2f' if normalize else 'd'
thresh = cm.max() / 2.
for i in range(cm.shape[0]):
for j in range(cm.shape[1]):
ax.text(j, i, format(cm[i, j], fmt),
ha="center", va="center",
color="white" if cm[i, j] > thresh else "black")
fig.tight_layout()
return ax

1. Plot non-normalized confusion matrix

from sklearn.utils.multiclass import unique_labels

# Plot non-normalized confusion matrix
plot_confusion_matrix(y_test, y_pred, classes=[0,1],
title='Confusion matrix, without normalization')


2. Plot normalized confusion matrix

# Plot normalized confusion matrix
plot_confusion_matrix(y_test, y_pred, classes=[0,1], normalize=True,
title='Normalized confusion matrix')

Here the results are not as good as they could be as our choice for the regularization parameter C was not the best. In real life applications, this parameter is usually chosen using Tuning the hyper-parameters of an estimator.

Renaming Columns in pandas using pandas. DataFrame. rename() method

Pandas 0.21+ Answer

There have been some significant updates to column renaming in version 0.21.

  • The rename method has added the axis parameter which may be set to columns or 1. This update makes this method match the rest of the pandas API. It still has the index and columns parameters but you are no longer forced to use them.

Examples for Pandas 0.21+

Construct sample DataFrame:

df = pd.DataFrame({'$a':[1,2], '$b': [3,4],                    '$c':[5,6], '$d':[7,8],                    '$e':[9,10]})   $a  $b  $c  $d  $e0   1   3   5   7   91   2   4   6   8  10

Using rename with axis='columns' or axis=1

df.rename({'$a':'a', '$b':'b', '$c':'c', '$d':'d', '$e':'e'}, axis='columns')

or

df.rename({'$a':'a', '$b':'b', '$c':'c', '$d':'d', '$e':'e'}, axis=1)

Both result in the following:

   a  b  c  d   e0  1  3  5  7   91  2  4  6  8  10

It is still possible to use the old method signature:

df.rename(columns={'$a':'a', '$b':'b', '$c':'c', '$d':'d', '$e':'e'})

The rename function also accepts functions that will be applied to each column name.

df.rename(lambda x: x[1:], axis='columns')

or

df.rename(lambda x: x[1:], axis=1)

How to copy a file from a remote server to a local machine?

scp username@remote:/file/to/send /where/to/put

You can read man scp to get more ideas on this.

Point to be Noted

  • scp -r will copy recursively
  • Use scp -P 123 to specify a custom port
  • You should hit this command from local, not from the server

You can use rsync as an alternative. It is mainly for syncing files.. but you can use it for this purpose as well.

rsync -avzh --stats --progress remoteuser@/file/to/send /where/to/put

--progress and --stats are useful for real-time display of transfer.


 

Some Examples:

scp root@10.240.179.4:/root/Jmeter/reports.jtl Downloads/

scp sadananad@ipaddress:/home/demo/public_html/myproject.tar.gz .

scp -P2233 sadananad@ipaddress:/home/demo/public_html/myproject.tar.gz .

scp -r root@178.62.54.83:~/folderinremoteuserdir ~/folderinlocaluserdir

rsync -avzh --stats --progress remoteuser@remoteip localpath

rsync -e "ssh -P $port_value" remoteuser@remoteip localpath

Weighted Averages of Precision, Recall and F-Score | Model Evaluation Metrics

So if you have two classes, the weighted precision would be:

Where pc1 and pc2 are the precisions for class 1 and class 2, and |𝑐1| and |𝑐2| are the number of instances in class 1 and class 2. 

For example, we have two classes 0 and 1

from sklearn.metrics import classification_report
print(classification_report(y_test,y_pred))


Calculate Weighted Recall, Precision and F1 score:

1. Weighted Recall:

weighted_avg_recall = recall _0 * support_0 + recall _1 * support_1 / total support

2. Weighted Precision:

weighted_avg_precision = precision_0 * support_0 + precision_1 * support_1 / total support

3. Weighted F1 score:

weighted_avg_f1_score = f1_score_0 * support_0 + f1_score_1 * support_1 / total support

INPUT:

weighted_avg_precision = ((0.86 * 63) + (0.93 * 37) ) / 100
print("weighted_avg_precision: ",weighted_avg_precision)
weighted_avg_f1_score = ((0.91 * 63) + (0.82 * 37) ) / 100
print("weighted_avg_f1_score: ", weighted_avg_f1_score)

OUTPUT:

weighted_avg_precision:  0.8859weighted_avg_f1_score:  0.8767

Here support is the number of instances in a particular class

You can use Scikit learn methods to get those:

f1_score() method to get f1 score

INPUT:

print("weighted avg f1 score using f1_score method: ", f1_score(y_test,y_pred, average="weighted"))

OUTPUT:

weighted avg f1 score using f1_score method:  0.8763093622795115

Micro avg and Macro avg of Precision, Recall and F-Score | Model Evaluation Metrics

macro-average will compute the metric independently for each class and then take the average (hence treating all classes equally),

whereas a micro-average will aggregate the contributions of all classes to compute the average metric.

Note: In a multi-class classification setup, micro-average is preferable if you suspect there might be a class imbalance (i.e you may have many more examples of one class than of other classes).

In a multi-class setting micro-averaged precision and recall are always the same.


where c is the class label.

Since in a multi-class setting you count all false instances it turns out that

Hence P = R. In other words, every single False Prediction will be a False Positive for a class, and every Single Negative will be a False Negative for a class. If you treat a binary classification case as a bi-class classification and compute the micro-averaged precision and recall they will be the same.

Note: In the case of averaging binary precision and recall from multiple datasets. In which case the micro-averaged precision and recall are different.

USAGE on Multiple Datasets:

Macro-averaged metrics are used when we want to evaluate systems performance across different datasets.

Micro-averaged metrics should be used when the size of datasets are variable.

Accuracy, Recall , Precision, & F1 Score | Model Evaluation Metrics

Accuracy: Accuracy is the most intuitive performance metric, it is simply the ratio of correctly predicted observations to total observations.

Accuracy = correctly predicted observations / total observations.

Accuracy = TP + TN / TP + FP + FN + TN

Some people may think that if we have high Accuracy, then our model is the best.

Yes, accuracy is a great measure but only when you have symmetric datasets where values of false positive and false negatives are almost the same.

Therefore, you must review other parameters to evaluate the performance of the model. 

Precision: Precision is the ratio of correctly predicted positive observations of the total predicted positive observations. 

Precision = predicted positive observations / total predicted positive observations. 

Precision = TP/TP+FP

Recall (Sensitivity): The recall is the ratio of the positive observations correctly predicted to all observations in the actual class – yes. 

Recall = TP / TP + FN

F1 Score – The F1 score is the weighted average of Precision and Recall.

F1 score = 2 * (recall * accuracy) / (recall + accuracy)

Therefore, the score takes into account false positives and false negatives. Intuitively it’s not as easy to understand as accuracy, but F1 is usually more useful than accuracy, especially if you have uneven class distribution.

Point to be Noted: Accuracy is best if false positives and false negatives have similar costs. If the cost of false positives and false negatives is very different, then it’s best to look at Precision and Recall at the same time.

True Positives vs False Positives and True Negatives vs False Negatives | Model Evaluation Metrics

True Positives and True Negatives are correctly predicted observations and are therefore shown in GREEN.

We want to minimize False Positives and False Negatives so that they are displayed in RED.


True positive (TP): These are positive values for a correct prediction, which means that the value of the actual class is YES, and the value of the predicted class is also YES.

True Negative (TN): These are negative values that are correctly predicted, which means that the value of the actual class is NO and the value of the predictive class is also NO

False Positive (FP): When the actual class is NO and the predictive class is YES

False negative (FN): when the actual class is YES but the predicted class is NO


Note: False Positives and False Negatives, these values occur when your actual class conflicts with the predictive class.


ROC Curve

The true positive rate gives the proportion of actual positives which are correctly identified as such. (Higher is better)

The false positive rate gives the proportion of falsely identified positives amongst all actual negatives. (lower is better)


Best Instant Noodles in the World

Hey, what’s up, top ten spiciest instant noodles of all time 2019 edition all right so for the last nine months I’ve been pumped to do this list so let’s make it happen.

10: CarJEN Hot & Spicy Dry Curry With Chives Soup – Malaysia

spiciest instant noodles #10: CarJEN Hot & Spicy Dry Curry With Chives Soup – Malaysia

At number ten we’ve got car Jen with their hot and spicy dry curry with chives soup from Malaysia this is an interesting phenomenon because you’ll see this in Taiwan mostly I’d never seen it in Malaysia ah basically you’ve got a pack of noodles you’ve got a little extra sachet you make the noodles drain them make the broth out of it and so you have a sipping soup and then you’ve got some hot dry noodles these are curry they’re spicy don’t kick your butt most likely and they’re really good Sargent stuff has always impressed me and I’ve always liked it so yeah check out this curry in


9: Taihodien Restaurant Supreme Spicy Noodle – Taiwan

spiciest instant noodles #9: Taihodien Restaurant Supreme Spicy Noodle - Taiwan

at number nine we have Thai hojin restaurant with our supreme spicy noodle from Taiwan now this is something I tried pretty recently and I’ll tell you what it’s spicy now it’s not a spicy that you might be familiar with this is Szechuan pepper spices if you like that’s Wan pepper you’re probably gonna like this hi ho Tian is known as a spicy hot pot restaurant over in Taiwan I’ve never been there but I’d definitely like to try it but see amount of Szechuan pepper in this is jet in Danity Szechuan pepper everybody calls it a numbing buzzing kind of taste also it’s mentioned that it tastes like has a sensation rather of licking a 9-volt battery to all you people who haven’t had the sensation of licking a 9-volt battery go buy a lip 9-volt battery put it on your tongue there you go the noodles are pretty good yeah the suckers strong and speaking of Szechuan pepper


8: Samyang Foods Mala Buldak Bokkeummyun – South Korea

spiciest instant noodles #8: Samyang Foods Mala Buldak Bokkeummyun – South Korea

in number eight we have samyang foods with their mala type bull doc bo commune from South Korea this is what you would call point 5x spicy and it also has such one pepper in it it is half as spicy as the original bulldog bo community didn’t know that now you know and there have been people constantly going and saying hey I tried the four X spicy sorry kids no such thing maybe they’ll come out with them I hope they do I hope they come out with a 400 X spicy but because I hope so it doesn’t mean it’s gonna happen and not only that I think there’s a good that it just wouldn’t sell I mean how many people can handle that much heat this is like a lighter version of the original plus it has that added attribute of Szechuan pepper flavor in there too so pretty popular check it out in


 7. Paldo Teumsae Ramyun – South Korea

spiciest instant noodles #7: Paldo Teumsae Ramyun – South Korea

at number 7 we have Paulo with their Tuesday ramyun from South Korea so this is a big bowl of noodles I just did a spicy noodle look long with this one just to kind of test the water to see what I thought of it again you know it’s got a mean kick to it kick like a mule and the hardest part I found was just how much food there was here I generally don’t eat a lot of instant noodles you might think how’s that possible watch my new cooking show I’ll tell you about Petitte but yeah it’s it’s a lot of broth and it’s a lot of spicy broth so you kind of have to endure oh all through all that broth so it’s pretty fun noodles taste great and yeah this is a really good one and easy to find alright


6: Paldo Bulnak Bokkummyun Spicy Fried Octopus Ramyun – South Korea

spiciest instant noodles #6: Paldo Bulnak Bokkummyun Spicy Fried Octopus Ramyun – South Korea

at number 6 we have pal dough with their bull NOK Bo commune spicy fried octopus wrong in from South Korea this is the flavor of a sauce you would put on octopus that’s ultra spicy and it’s pretty good stuff it’s got a nice fit to it shouldn’t be too hard to find there was a bit of kerfuffle and where I had this on my list well a couple years back and honestly after retrying this and another one I pretty pretty darn comfortable with where it’s at right now you got a problem with it you know that’s cool but yeah this is worth that in


5: Paldo Volcano Curry Kkokkomyun – South Korea

spiciest instant noodles #5: Paldo Volcano Curry Kkokkomyun - South Korea

at number 5 we have pal dough with their volcano curry Coco Mian from South Korea so this is another dry stir noodle it’s really interesting because when they take curry I think of a strong curry flavor Japanese in the end no it’s definitely different and you’ll taste the difference in this one it’s good and it’s spicy so if you definitely want good spicy noodles this will do you’re right all right now’s where it starts getting interesting


4: Liangchengmei 30S” Hot Bird Noodle – China

spiciest instant noodles #4: Liangchengmei 30S

at number 4 we have Leung Chiang Mai with their 32nd hot bird noodle check this package out this is just awesome a chicken and a leather jacket yeah you can’t go wrong with that not only that you’ve got a fresh noodle in this box which kind of surprised me my wife spied this one when we were up in Canada and I’m stoked she did this is really interesting stuff honestly it tastes almost exactly like Bulldog Pokemon like a hinge let’s Pisces it’s it it might as well be exactly the same spicy it’s such a tiny amount different it gives me the same feeling but it has these noodles and it’s definitely a little bit of a cut-up in that side of things but tasting quality has nothing to do with this whatsoever this is about how hot it is so yeah alright we’re getting down there folks


3: Samyang Foods Buldak Bokkeummyeon – South Korea

spiciest instant noodles #3: Samyang Foods Buldak Bokkeummyeon – South Korea

at number three we have samyang foods with their fire noodle challenge bulldog pokemon this is spicy stuff it hits at four thousand four hundred and four Scoville heat units they use chromatography to find this out and it’s seriously good stuff people love it and globally and it means business this isn’t like oh it’s a little spicy ooh no this is this is clobber e stuff if it means business and it’s really good so if you can if you can get a hold of this one which honestly I would be floored if you have an Asian grocery near you and they do not carry this if they don’t wake them up they need the help they need to be a Vaness because this is like probably one of the most popular institute was in the world right all right hold on to your butts folks


2: Samyang Foods 2x Spicy Hek Buldak Bokkeummyun – South Korea

spiciest instant noodles #2: Samyang Foods 2x Spicy Hek Buldak Bokkeummyun – South Korea

at number 2 we have samyang foods 2x spicy heck nuclear boom box poke immune from Tammy and food South Korea this is hot this one 8,000 106 goal unit this has been known as the world spiciest instant noodle bar none for a while now and people do this challenge causally nuclear fire noodle check it’s hot don’t get me wrong this stuff is like sucking on the charcoal briquette yeah it’s so hot it’s ridiculously hot one thing I noticed about this one is if you’ve had it for two days in a row it definitely doesn’t seem as spicy on the second day yeah this is a very popular product very spicy noodles are great spicy sauce all over it sweet – yeah this is a good one you want a light and mouth on fire give this a try so last year I didn’t think there would ever be something to overtake the Semyon foods 2x spicy odd to be able to eat that easily I felt was quite an achievement


1: Isoyama Shoji 18 (Age 18 Prohibited/Restricted/Only La-Men Curry Taste) – Japan

spiciest instant noodles #1: Isoyama Shoji 18禁カレーラーメン (Age 18 Restricted La-Men Curry Taste) - Japan

now I have tried something that makes that taste like sugar water we have Isoyama Shoji with their 18 restricted curry flavored Lamya now this stuff will kick your butt or certainly did to me okay so first things first this variety requires 700 milliliters of water when I saw this thing somebody sent me one when I saw that like oh come on this has got to be a joke it’s gonna be that’s so much water I should probably add less just so I can paint it no that’s not the case that that water goes a long way the noodles are pretty good texture wise and everything and there’s a lot of them but the thing is you have one sachet and one sachet only of powder and you pour that in and stir it up and man you have Hellfire on your hands like nothing else I was told that this has boot Jolokia ghost pepper in it and I don’t know how much but I’m guessing probably equal to 20 of them I tried to do this twice I trained for it the second time and it just kicked my butt not only that it tastes horrible it’s absolutely horrible stuff I can’t handle it it’s so gross I love I absolutely love curry and when I’ve had this it makes me thank god this is just such a disservice to curry yeah but it also hurt my gut the first time I had a I was sick and I just thought now this will I know maybe it’s maybe it’s because I’m sick maybe that’s why I can’t handle it I’ve got a I’m sick I got I got something going on a bronchial something like that no I tried it the second time same thing happened and just trying to shovel it in and get it done and not breathe through my nose oh that if it did me in and my gut hurt for a good 45 minutes to an hour afterwards not just a normal kind of pain either it was just like a wrenching kind of pain it didn’t matter how much liquid I drank that sucker just burnt through burnt and Bert Bert Bert Bert booklet so I’ve never seen anybody eat a whole pack of this stuff not only that there are warnings all over it recommending that you make sure that you are healthy enough to do such a thing I may in the future try it again but right now no thanks I don’t want it no I’d take a bite and go God why’d I do that but yeah literally I don’t know the Scoville heat unit level on this stuff but I’ll tell you right now this could probably be 100 x-pipe I’m dead serious if you feel Yama Shoji it’ll show up as 18 prohibited 18 restricted 18 only.


 

Tags:

emart dare you,top 10 instant noodles 2019,top 10 instant noodles 2018,top 10 instant noodles 2017,top 10 instant noodles 2016,gookmul knight of ribs,best instant noodles singapore,samyang foods buldak bokkeummyeon,best korean cup noodles,paldo teumsae instant noodles with soup,mom’s dry noodle dan dan noodle,marutai kumamoto tonkotsu ramen,instant noodles fairprice,japanese nissin cup noodles singapore,samyang foods 2x spicy hek buldak bokkeummyun,top 10 instant noodles 2018,which cup noodles is best,top 10 instant noodles in india,best instant noodles korea,world’s spiciest ramen 2017,spicy instant noodles recipe

[NEW]: niki.ai Referral Code 2019

Niki.ai Referral Coderahul17654

  • First of all, Download App here
  • Install & open
  • Now Enter your Mobile number and verify it via OTP.
niki signup
  • In the next step, it will ask you to choose your Google account to automatically fill up Email option and Enter Niki Referral Coderahul17654
  • As soon as you signup using Niki Referral code, you will get 10% off (Maximum 500 Rs) Promo Code on your registered Email and Mobile number.

How does Referral work?

  • You’ll get a referral code that you can distribute to your network.
  • The new user will be asked to enter your referral code at the time of registration to ensure that you have the benefits.
  • For each unique / new user signing up for the Niki application (unique / new means a unique new phone number, a valid email ID and a mobile device), you and your friend get a special 5% instant CASHBACK code up to $ 100. Applicable to utilities.
  • The minimum order value for the offer is 200 rupees.
  • For new users, the promotion code applies only to the first transaction at all. The code applies only to utilities. Facilities include DTH, electricity, gas, forward pay, broadband, landlines, and Datacard. For the current user, the promotional code only applies after your friend has registered.
  • The Instant Cashback promotional code can be applied and refunded only once. You can choose to send your referral code directly from the app’s “Return & earn” page (available in the top corner).
  • If you allow us to send SMS messages to all contacts, we can do so from the end without having to spend a penny.

Referral Terms & Conditions

  • You can refer to as many people as you want. The promo code can not be reused once the value has been redeemed. If the transaction fails, the promo code can be reused. This offer can not be grouped with any other system. If the user has invited himself, he will not be considered a referral and will not create a referral promotion code. If a user invites someone using the same mobile device or device that is already registered, they will not be considered a referral and will not create a promotional code for the referral. Users are not allowed to distribute code or referral link on search engines and coupon sites.
  • If the user already has a subscription to another Niki account, a new account will not create a promotional code for the referral. The promotion code can not be activated for the referral earned through the cash referral program. It can only be used for transactions on Niki in cases where Niki accepts payment on the Niki application. Techbins Solutions Private Limited reserves the right to cancel / modify / change / add / delete any of the terms and conditions of the offer at any time in time without notice. Techbins Solutions Private Limited reserves the right to terminate the offer to all users or individuals at any time without notice. Techbins Solutions Private Limited reserves the right to refuse to honor the offer in case of fraudulent use or because of doubt or misuse of the offer by any customer without providing any notice or explanation. In no event shall Techbins be liable for any misuse of the Referral Code / Linkage by Customer’s negligence.

[Solved]: error: couldn't link model to 'en' spacy

USE this command:

pip install -U spacy --user && python -m spacy download en --user

Install in Root:

sudo pip install -U spacy && python -m spacy download en

Stack track:

Collecting spacy  Using cached https://files.pythonhosted.org/packages/55/2f/e4a9ab232152160f2f3e945d8e3f8e19558a63cbf0acc29a31c4121d95f3/spacy-2.1.3-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whlRequirement already satisfied, skipping upgrade: srsly<1.1.0,>=0.0.5 in /Users/admin/.local/lib/python3.6/site-packages (from spacy) (0.0.5)Collecting wasabi<1.1.0,>=0.2.0 (from spacy)  Using cached https://files.pythonhosted.org/packages/76/6c/0376977df1ba9f0ec27835d80456d9284c79737cb5205649451db1181f01/wasabi-0.2.1-py3-none-any.whlCollecting blis<0.3.0,>=0.2.2 (from spacy)  Using cached https://files.pythonhosted.org/packages/d1/92/84e017b19d988c55ef2e6fa50e51522f3a23a6461ef21f308d20b19dbe0c/blis-0.2.4-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whlRequirement already satisfied, skipping upgrade: murmurhash<1.1.0,>=0.28.0 in /Users/admin/anaconda3/lib/python3.6/site-packages (from spacy) (1.0.1)Requirement already satisfied, skipping upgrade: preshed<2.1.0,>=2.0.1 in /Users/admin/anaconda3/lib/python3.6/site-packages (from spacy) (2.0.1)Requirement already satisfied, skipping upgrade: numpy>=1.15.0 in /Users/admin/anaconda3/lib/python3.6/site-packages (from spacy) (1.15.4)Collecting thinc<7.1.0,>=7.0.2 (from spacy)  Using cached https://files.pythonhosted.org/packages/4d/76/45c09c2f511f6674bc2b25ac4b12be1c64b9f5d6a8eaaff77c672094dc7e/thinc-7.0.4-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whlRequirement already satisfied, skipping upgrade: plac<1.0.0,>=0.9.6 in /Users/admin/anaconda3/lib/python3.6/site-packages (from spacy) (0.9.6)Requirement already satisfied, skipping upgrade: cymem<2.1.0,>=2.0.2 in /Users/admin/anaconda3/lib/python3.6/site-packages (from spacy) (2.0.2)Requirement already satisfied, skipping upgrade: jsonschema<3.0.0,>=2.6.0 in /Users/admin/anaconda3/lib/python3.6/site-packages (from spacy) (2.6.0)Requirement already satisfied, skipping upgrade: requests<3.0.0,>=2.13.0 in /Users/admin/anaconda3/lib/python3.6/site-packages (from spacy) (2.21.0)Requirement already satisfied, skipping upgrade: tqdm<5.0.0,>=4.10.0 in /Users/admin/anaconda3/lib/python3.6/site-packages (from thinc<7.1.0,>=7.0.2->spacy) (4.26.0)Requirement already satisfied, skipping upgrade: urllib3<1.25,>=1.21.1 in /Users/admin/anaconda3/lib/python3.6/site-packages (from requests<3.0.0,>=2.13.0->spacy) (1.24.1)Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /Users/admin/anaconda3/lib/python3.6/site-packages (from requests<3.0.0,>=2.13.0->spacy) (2018.11.29)Requirement already satisfied, skipping upgrade: idna<2.9,>=2.5 in /Users/admin/anaconda3/lib/python3.6/site-packages (from requests<3.0.0,>=2.13.0->spacy) (2.8)Requirement already satisfied, skipping upgrade: chardet<3.1.0,>=3.0.2 in /Users/admin/anaconda3/lib/python3.6/site-packages (from requests<3.0.0,>=2.13.0->spacy) (3.0.4)Installing collected packages: wasabi, blis, thinc, spacySuccessfully installed blis-0.2.4 spacy-2.1.3 thinc-7.0.4 wasabi-0.2.1Collecting en_core_web_sm==2.1.0 from https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.1.0/en_core_web_sm-2.1.0.tar.gz#egg=en_core_web_sm==2.1.0  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.1.0/en_core_web_sm-2.1.0.tar.gz (11.1MB)    100% |████████████████████████████████| 11.1MB 332kB/s ta 0:00:01Installing collected packages: en-core-web-sm  Found existing installation: en-core-web-sm 2.0.0    Uninstalling en-core-web-sm-2.0.0:      Successfully uninstalled en-core-web-sm-2.0.0  Running setup.py install for en-core-web-sm ... doneSuccessfully installed en-core-web-sm-2.1.0✔ Download and installation successfulYou can now load the model via spacy.load('en_core_web_sm')✔ Linking successful/Users/admin/.local/lib/python3.6/site-packages/en_core_web_sm -->/Users/admin/.local/lib/python3.6/site-packages/spacy/data/enYou can now load the model via spacy.load('en')

Find list of substrings in List of strings python

listStrings = [ACDE, CDDE, BPLL, ... ]listSubstrings = [ACD, BPI, KLJ, ...]

The above entries are just examples. len(listStrings) is ~ 60,000, len(listSubstrings) is ~50,000-300,000, and len(listStrings[i]) is anywhere from 10 to 30,000.

Approach 1:

for i in listSubstrings:   for j in listStrings:       if i in j:          file.write(i+j)

While this works for my task, it’s horribly slow, using one core and taking on the order of 40 minutes to complete the task

Approach 2:

For a start, I’d suggest using the Aho-Corasick string matching algorithm. Basically, in exchange for some precompute work to build a matcher object from your set of fixed strings, you can scan another string for all of those fixed strings at once, in a single pass.

So instead of scanning 60K strings 50K+ times each (three BILLION scans?!?), you can scan them each once with an only slightly higher cost than a normal single scan, and get all the hits.

The best part is, you’re not writing it yourself. PyPI (the Python package index) already has the pyahocorasick package written for you. So try it out.

Example of use:

import ahocorasicklistStrings = [ACDE, CDDE, BPLL, ...]listSubstrings = [ACD, BPI, KLJ, ...]auto = ahocorasick.Automaton()for substr in listSubstrings:    auto.add_word(substr, substr)auto.make_automaton()...for astr in listStrings:    for end_ind, found in auto.iter(astr):        w.write(found+astr)

This will write multiple times if a substring (“needle”) is found in the string being searched (“haystack”) more than once. You could change the loop to make it only write on the first hit for a given needle in a given haystack by using a set to dedup:

for astr in listStrings:    seen = set()    for end_ind, found in auto.iter(astr):        if found not in seen:            seen.add(found)            w.write(found+astr)

You can further tweak this to output the needles for a given haystack in the same order they appeared in listSubstrings (and uniquifying while you’re at it) by storing the index of the words as or with their values so you can sort hits (presumably small numbers, so sort overhead is trivial):

from future_builtins import map  # Only on Py2, for more efficient generator based mapfrom itertools import groupbyfrom operator import itemgetterauto = ahocorasick.Automaton()for i, substr in enumerate(listSubstrings):    # Store index and substr so we can recover original ordering    auto.add_word(substr, (i, substr))auto.make_automaton()...for astr in listStrings:    # Gets all hits, sorting by the index in listSubstrings, so we output hits    # in the same order we theoretically searched for them    allfound = sorted(map(itemgetter(1), auto.iter(astr)))    # Using groupby dedups already sorted inputs cheaply; the map throws away    # the index since we don't need it    for found, _ in groupby(map(itemgetter(1), allfound)):        w.write(found+astr)

For performance comparisons, I used a variant that is more likely to contain matches, as well as enlarging the haystacks. First, setup code:

>>> from random import choice, randint>>> from string import ascii_uppercase as uppercase>>> # 5000 haystacks, each 1000-5000 characters long>>> listStrings = [''.join([choice(uppercase) for i in range(randint(1000, 5000))]) for j in range(5000)]>>> # ~1000 needles (might be slightly less for dups), each 3-12 characters long>>> listSubstrings = tuple({''.join([choice(uppercase) for i in range(randint(3, 12))]) for j in range(1000)})>>> auto = ahocorasick.Automaton()>>> for needle in listSubstrings:...     auto.add_word(needle, needle)...>>> auto.make_automaton()

And now to actually test it (using ipython %timeit magic for microbenchmarks):

>>> sum(needle in haystack for haystack in listStrings for needle in listSubstrings)80279  # Will differ depending on random seed>>> sum(len(set(map(itemgetter(1), auto.iter(haystack)))) for haystack in listStrings)80279  # Same behavior after uniquifying results>>> %timeit -r5 sum(needle in haystack for haystack in listStrings for needle in listSubstrings)1 loops, best of 5: 9.79 s per loop>>> %timeit -r5 sum(len(set(map(itemgetter(1), auto.iter(haystack)))) for haystack in listStrings)1 loops, best of 5: 460 ms per loop

So for checking for ~1000 smallish strings in each of 5000 moderate size strings, pyahocorasickbeats individual membership tests by a factor of ~21x on my machine. It scales well as the size of listSubstrings increases too; when I initialized it the same way, but with 10,000 smallish strings instead of 1000, the total time required increased from ~460 ms to ~852 ms, a 1.85x time multiplier to perform 10x as many logical searches.

For the record, the time to build the automatons is trivial in this sort of context. You pay it once up front not once per haystack, and testing shows the ~1000 string automaton took ~1.4 ms to build and occupied ~277 KB of memory (above and beyond the strings themselves); the ~10000 string automaton took ~21 ms to build and occupied ~2.45 MB of memory.

Decompress bz2 File in (Linux unix ubuntu Mac)

bzip2 is mono-threaded, which means it will take a long time to decompress a large file.

To decompress a .bz2 file multithreaded, you can use the free, open-source program lbzip2:

sudo apt-get install lbzip2lbzip2 -d file_name.bz2

-d indicates you wish to decompress the file. It would automatically determine how many threads it will use. To specify the exact number of threads you want to use, use the -n parameter, e.g.:

lbzip2 -d -n 32 file_name.bz2

A few more useful commands with lbzip2:

To compress a folder:

tar -c -I lbzip2 -f file_name.tar.bz2 folder_name

To uncompress a folder:

 tar -I lbzip2 -xvf file_name.tar.bz2

Parameters:

-I, --use-compress-program PROG      filter through PROG (must accept -d)-x, --extract, --get      extract files from an archive-v, --verbose      verbosely list files processed-f, --file ARCHIVE      use archive file or device ARCHIVE

Some alternatives to decompress a .bz2 file multithreaded:

pbzip2:

sudo apt-get install pbzip2pbzip2 -d my_file.bz2

mpibzip2: designed to be used on cluster machines.


If you need some large .bz2 files to experiment with: https://dumps.wikimedia.org/mirrors.html

For example (a 14 GB .bz2 file, 200 GB uncompressed):

wget http://dumps.wikimedia.your.org/wikidatawiki/20170120/wikidatawiki-20170120-pages-articles-multistream.xml.bz2 lbzip2 -d -n 32 wikidatawiki-20170120-pages-articles-multistream.xml.bz2 

How to append using tee command

The -a modifier is for ‘append’, or add to the end. Without -a the tee command overwrites the file

From man tee:

   Copy standard input to each FILE, and also to standard output.   -a, --append          append to the given FILEs, do not overwrite

Example:

1. The content of the run_script.sh file

curl -X GET "localhost:9200" | tee log.txt
echo "helelelelelelelele" | tee -a log.txt
curl -X GET "localhost:9200" | tee -a log.txt

2. execute is:

$ ./run_script.sh 

3. Content of log.txt

{
"name" : "SHUMrBf",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "beIL54B6SWSS1lb-r9mz9Q",
"version" : {
"number" : "6.5.4",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "d2ef93d",
"build_date" : "2018-12-17T21:17:40.758843Z",
"build_snapshot" : false,
"lucene_version" : "7.5.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
helelelelelelelele
{
"name" : "SHUMrBf",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "beIL54B6SWSS1lb-r9mz9Q",
"version" : {
"number" : "6.5.4",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "d2ef93d",
"build_date" : "2018-12-17T21:17:40.758843Z",
"build_snapshot" : false,
"lucene_version" : "7.5.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}

Detach a screen (Mac,Unix,Linux,Ubuntu,Centos,Fedora)

1. First List All screens:

screen -ls
$ screen -lsThere are screens on: 31643.screen_name1 (Attached) 30846.screen_name2 (Detached) 28743.screen_name3 (Detached) 31033.screen_name4 (Detached)4 Sockets in /var/run/screen/S-centos.

2. then detach screen whatever you want

screen -d screen_name1
OR
screen -d 31643

Note: if you want to reattach then use

screen -r screen_name1
OR
screen -r 31643

Python numpy Tutorial | Getting started with numpy From Beginner to Advanced Level

Tag:

numpy ndarray,numpy documentation,import numpy,scipy tutorial,numpy python 3,python numpy install,best numpy tutorial,numpy tutorial pdf tutorialspoint,numpy array append,numpy download,how to install numpy,scipy tutorial,numpy where example,shape python,numpy tutorial pdf,numpy python 3,travis oliphant,numpy array methods,python numpy array append,python tutorial for beginners with examples,numpy head,scipy in python,pandas in python,numpy for data science,pandas online course,how to learn python libraries,python numpy and pandas tutorial,create empty numpy array,pandas guru99,numpy tutorial datacamp,tensorflow tutorial guru99,numpy tutorial w3schools,numpy in python w3schools,numpy array creation example,numpy environment,numpy dtype tutorial,print(np.array([1,2,3]) +1),edureka matplotlib tutorial,matplotlib examples python,edureka python material,python pandas data analysis example,machine learning plus numpy,np bincount example,numpy sort matrix by column,numpy sequence,numpy beginner’s guide (second edition),numpy initialize array with value,python matrix analysis,numpy rss,learn pandas and numpy,how to learn python generators,python multivariable function

Matplotlib Tutorial Python | Getting started with Matplotlib from Beginner to Advanced Level


Tags:

pyplot scatter,matplotlib example,python plot histogram,python matplotlib install,python chart,python plot legend,matplotlib download,matplotlib documentation,matplotlib jupyter,plt.plot legend,python plot scatter,matplotlib tutorial pdf,python plot table,data processing in python,python matplotlib tutorial pdf,python numpy and pandas tutorial,matplotlib plot color,pyplot bar,pyplot subplot,graph data structure in python 3,install matplotlib python 3 windows,what is legend in matplotlib,matplotlib set window title,pyspark matplotlib example,python pandas data analysis example,matplotlib tutorial geeksforgeeks,python pandas case study,add_axes matplotlib example,box plot python tutorial,python data operations,python excel tutorialspoint,pyplot continuous plot,matplotlib practice,real python matplotlib,matplotlib cheat sheet dataquest,dataflair pandas,numpy dataflair,python libraries dataflair,what is scipy in python,learn pandas in python,data flair django,matplotlib plot marker,matplotlib legend,python matplotlib install,matplotlib python 3,matplotlib documentation,matplotlib tutorial pdf,matplotlib tutorial point,best matplotlib tutorial,how to import matplotlib in python

Case insensitive exact matches in Elasticsearch

Steps to Follow

Step 1: insert setting and mapping to elastic search

Note: if index is present then delete that.

curl -X PUT "localhost:9200/index_name" -H 'Content-Type: application/json' -d'
{ "settings": { "analysis": { "normalizer": { "my_normalizer": { "type": "custom", "char_filter": [], "filter": ["lowercase", "asciifolding"] } } } }, "mappings": { "_doc": { "properties": { "foo": { "type": "keyword", "normalizer": "my_normalizer" } } } }}

Step 2:  insert document

Note: you can store your data here

PUT index/_doc/1{  "foo": "BÀR"}PUT index/_doc/2{  "foo": "bar"}

Step 3: refresh index

POST index/_refresh

Step 4: search data from elastic search

GET index/_search{  "query": {    "term": {      "foo": "BAR"    }  }}GET index/_search{  "query": {    "term": {      "foo": "bAr"    }  }}

both will give same result.

 

How to send email from the Linux command line (CentOS, Redhat, Ubuntu and Debian)

Install mail Command:

Let’s use one of below command to install mail command as per your operating system. For RHEL based systems, this command installed from mailx rpm packages and on Debian based system this command found in mailutils package.

Install mail command on CentOS/Redhat 7/6/5:

yum install mailx

Install mail command on Fedora 22+:

dnf install mailx

Install mail command on Ubuntu/Debian/LinuxMint:

sudo apt-get install mailutils

Sending Test Email

After installing mail command packages in your system, send a test email using below command.

echo "Message Body" | mail -s "Message Subject" receiver@example.com

Embedding a Jupyter Notebook into Wordpress

1. Generate the Jupyter notebook

2. inline your graphics using %matplotlib -inline in the first cell as shown

3. export as an HTML file.

4. Upload the HTML to as a media file to the WordPress site.

5. click on Add media and insert the link of uploaded notebook

6. use an iframe to Add notebook to the post:

<iframe scr="link of notebook">

7. You are done

8. Adding this notebook to my website: Multiple linear regression

BCCI announces player salaries for 2019

BCCI announced its annual contract on Thursday evening, as the first class A-plus cricket body was reduced to just three members – skipper Virat Kohli, vice-captain Rohit Sharma and seamer Jaspet Boumerah. Last year, when the category was first announced by the Committee of Officials appointed by the Supreme Court, there were five players in the top category.

PeriodGrad  A+ Grade AGrade B Grade C
2019INR 7 CrINR 5 CrINR 3 CrINR 1 Cr 

Here is the list of contracted players:

Grade A+ (INR 7 Cr):

  1. Virat Kohli,
  2. Rohit Sharma,
  3. Jasprit Bumrah

Grade A (INR 5 Cr):

  1. R Ashwin,
  2. Ravindra Jadeja,
  3. Bhuvneshwar Kumar,
  4. Cheteshwar Pujara,
  5. Ajinkya Rahane,
  6. MS Dhoni,
  7. Shikhar Dhawan,
  8. Mohammed Shami,
  9. Ishant Sharma,
  10. Kuldeep Yadav,
  11. Rishabh Pant

Grade B (INR 3 Cr):

  1. KL Rahul,
  2. Umesh Yadav,
  3. Yuzvendra Chahal,
  4. Hardik Pandya

Grade C (INR 1 Cr):

  1. Kedar Jadhav,
  2. Dinesh Karthik,
  3. Ambati Rayudu,
  4. Manish Pandey,
  5. Hanuma Vihari,
  6. Khaleel Ahmed,
  7. Wriddhiman Saha.

Top 10 Ethnic Dresses for Women

1. Rain & Rainbow: Women Navy Blue Tropical Print Maxi Dress
1. Rain & Rainbow: Women Navy Blue Tropical Print Maxi Dress

PRODUCT DETAILS 

Navy and white printed woven maxi dress, has a round neck, three-quarter sleeves, hook and eye closures, flared hem

Material & Care

Cotton 
Machine-wash

Complete The Look

Look and feel your absolute best in this opulent Rain & Rainbow dress. This printed navy blue piece can be teamed with your favourite pair of heels and a chic jacket for a fun date night outfit.

Size & Fit

Semi-Fit
The model (height 5’8″) is wearing a size S

Buy From Myntra

2. AKS: Women Grey Printed Maxi Dress
2. AKS: Women Grey Printed Maxi Dress

PRODUCT DETAILS 

Grey printed woven maxi dress, has a V-neck, short sleeves, flared hem

Material & Care

Viscose Rayon
Machine-wash

Complete The Look

Give your closet a makeover with this stylish AKS dress. When you need an outfit for work and dinner after work, wear this printed grey piece with platform heels and some minimalistic silver jewellery.

Size & Fit

Semi-Fit
The model (height 5’8″) is wearing a size S

Buy From Myntra:

3. Nayo Women: Grey & Black Colourblocked Maxi Dress
3. Nayo Women: Grey & Black Colourblocked Maxi Dress

PRODUCT DETAILS 

Grey and black colourblocked woven maxi dress, has a round neck, three-quarter sleeves, flared hem

Material & Care

Cotton
Hand-wash

Complete The Look

Turn heads this season in this effortlessly stylish dress by Nayo. Wow, your date by pairing this colourblocked grey piece with platform heels and a chic blazer.

Size & Fit

Semi-Fit
The model (height 5’8″) is wearing a size S

Buy From Myntra:

4. AKS: Women Pink & White Printed Maxi Dress
4. AKS: Women Pink & White Printed Maxi Dress

PRODUCT DETAILS 

Pink and white printed woven maxi dress, has a round neck, short sleeves, concealed zip closure, flared hem

Material & Care

Cotton
Machine-wash

Complete The Look

Maintain a trendy style all year long with this stunning dress from AKS. Create an absolute style high by going with this printed pink piece in combination with a killer shoe and trendy bangles for a perfect brunch look.

Size & Fit

Semi-Fit
The model (height 5’8″) is wearing a size S

Buy From Myntra:

5. SASSAFRAS: Women Off-White Solid Fit and Flare Dress
5. SASSAFRAS: Women Off-White Solid Fit and Flare Dress

PRODUCT DETAILS 

Off-white and blue solid woven fit and flare dress with embroidered and tie-up detail, has a mandarin collar, three-quarter bell sleeves, button closure, flared hem

Material & Care

Cotton
Machine-wash

Complete The Look

Maintain a chic appearance all year long with this trendy SASSAFRAS dress. When you need an outfit for work and dinner after work, wear this solid off-white piece with platform heels and some minimalistic silver jewellery.

Size & Fit

Semi-Fit
The model (height 5’8″) is wearing a size S

Buy From Myntra:

6. Libas: Women Grey & White Solid Layered Maxi Dress
6. Libas: Women Grey & White Solid Layered Maxi Dress

PRODUCT DETAILS 

White solid woven maxi dress, has a round neck, sleeveless, flared hem, concealed zip closure 
Comes with a grey longline open front shrug, has a short sleeves

Material & Care

Cotton
Hand-wash

Complete The Look

Stand out from the crowd with this beautiful and comfortable Libas dress. Complement this self design grey piece with a pair of caged heels, your favourite bag and some minimalistic silver jewellery for a classy date ensemble.

Size & Fit

The model (height 5’8″) is wearing a size S

Buy From Myntra:

7. all about youfrom Deepika Padukone
7. all about youfrom Deepika Padukone

all about you from Deepika Padukone Women Cream-Coloured Printed Maxi Dress

PRODUCT DETAILS 

Cream-coloured printed woven maxi dress, has a shirt collar, three-quarter sleeves, zip closure, an attached lining, flared hem

Material & Care

100% polyester
Machine-wash

Complete The Look

This dress from all about you is a must-have item for any fashionista’s wardrobe. Wear your printed cream-coloured piece with strappy heels and your favourite clutch for a night out on the town with your friends.

Size & Fit

Semi-Fit
The model (height 5’8″) is wearing a size S

Buy From Myntra

8. Nayo: Women Black & White Checked Midi A-Line Dress
8. Nayo: Women Black & White Checked Midi A-Line Dress

PRODUCT DETAILS 

Black and white checked woven midi A-line dress, has a round neck, long sleeves, hook and eye closure, flared hem
Comes with a belt 

Material & Care

Cotton
Hand-wash

Complete The Look

Maintain a chic appearance all year long with this trendy Nayo dress. Create an absolute style high by going with this checked black piece in combination with a killer shoe and trendy bangles for a perfect brunch look.

Size & Fit

The model (height 5’8″) is wearing a size S

Buy From Myntra:

9. Tokyo Talkies Women Black Printed Maxi Dress
9. Tokyo Talkies Women Black Printed Maxi Dress

PRODUCT DETAILS 

Black printed woven maxi dress, has a round neck, long sleeves, concealed zip closure, attached lining, flared hem
Comes with a belt

Material & Care

100% polyester
Machine-wash

Complete The Look

You’re sure to love the style and comfort of this elegant Tokyo Talkies dress. Spruce up this printed black piece for work by layering it with a light sweater and some ballerinas.

Size & Fit

Semi-Fit
The model (height 5’8″) is wearing a size S

Buy From Myntra

10. Sangria: Women Navy Blue Solid Maxi Dress
10. Sangria: Women Navy Blue Solid Maxi Dress

PRODUCT DETAILS 

Navy Blue solid woven maxi dress, has a round neck, three-quarter sleeves, flared hem
Comes with an inner lining, has a round neck, sleeveless

Material & Care

Viscose
Machine-wash

Complete The Look

Replace some of your outdated essentials with this beautiful dress from Sangria. Create an absolute style high by going with this solid navy blue piece in combination with a killer shoe and trendy bangles for a perfect brunch look.

Size & Fit

Semi-Fit
The model (height 5’8″) is wearing a size S

Buy From Myntra: