Table Detection using Deep Learning

Step 1: Preprocessing Tables

We are using Luminoth(Tensorflow) as the backend. Luminoth only works on preprocessed(greyscale) images.  

Use the below script to convert images into grayscale images:

preprocess.py​

import osimport cv2import pandas as pdroot_dir = os.getcwd()file_list = ['train.csv', 'val.csv']image_source_dir = os.path.join(root_dir, 'data/images/')data_root = os.path.join(root_dir, 'data')for file in file_list:    image_target_dir = os.path.join(data_root, file.split(".")[0])    if not os.path.exists(image_target_dir):        os.mkdir(image_target_dir)    # read list of image files to process from file    image_list = pd.read_csv(os.path.join(data_root, file), header=None)[0]    print("Start preprocessing images")    for image in image_list:        # open image file        img = cv2.imread(os.path.join(image_source_dir, image))        img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)        # perform transformations on image        b = cv2.distanceTransform(img, distanceType=cv2.DIST_L2, maskSize=5)        g = cv2.distanceTransform(img, distanceType=cv2.DIST_L1, maskSize=5)        r = cv2.distanceTransform(img, distanceType=cv2.DIST_C, maskSize=5)        # merge the transformed channels back to an image        transformed_image = cv2.merge((b, g, r))        target_file = os.path.join(image_target_dir, image)        print("Writing target file {}".format(target_file))        cv2.imwrite(target_file, transformed_image)print("Finished preprocessing images")

Step 2: Create TFRecords

To train our model we need to convert our preprocessed images into TFrecords.

run this script. create TFRecords

lumi dataset transform --type csv --data-dir data/ --output-dir tfdata/ --split train --split val --only-classes=table

Step 3: Training our Model

Start training the luminoth network
Run this script: start_traning.sh
 

lumi train -c config.yml

config.yml

train:  # Name used to identify the run. Data inside `job_dir` will be stored under  # `run_name`.  run_name: table-area-detection-0.1  # Base directory in which model checkpoints & summaries (for Tensorboard) will  # be saved.  job_dir: jobs/  save_checkpoint_secs: 10  save_summaries_secs: 10  # Number of epochs (complete dataset batches) to run.  num_epochs: 10dataset:  type: object_detection  # From which directory to read the dataset.  dir: tfdata/classes-table/  image_preprocessing:    min_size: 600    max_size: 1024  data_augmentation:    - flip:        left_right: True        up_down: True        prob: 0.5model:  type: fasterrcnn  network:    # Total number of classes to predict.    num_classes: 1

Facts

It can take more than 1 day(24 hours) to train.

You will see last output like this:

INFO:tensorflow:Saving checkpoints for 3371 into jobs/table-area-detection-0.1/model.ckpt.
INFO:tensorflow:step: 3371, file: b'9527_018.png', train_loss: 2.469784736633301, in 23.64s
INFO:tensorflow:Saving checkpoints for 3372 into jobs/table-area-detection-0.1/model.ckpt.
INFO:tensorflow:step: 3372, file: b'9526_017.png', train_loss: 2.6023592948913574, in 21.86s
INFO:tensorflow:Saving checkpoints for 3373 into jobs/table-area-detection-0.1/model.ckpt.
INFO:tensorflow:step: 3373, file: b'1742_157.png', train_loss: 2.5478856563568115, in 22.36s
INFO:tensorflow:Saving checkpoints for 3374 into jobs/table-area-detection-0.1/model.ckpt.
INFO:tensorflow:step: 3374, file: b'9528_061.png', train_loss: 3.118919849395752, in 21.22s
INFO:tensorflow:Saving checkpoints for 3375 into jobs/table-area-detection-0.1/model.ckpt.
INFO:tensorflow:step: 3375, file: b'9516_001.png', train_loss: 3.0582146644592285, in 21.71s
INFO:tensorflow:Saving checkpoints for 3376 into jobs/table-area-detection-0.1/model.ckpt.
INFO:tensorflow:step: 3376, file: b'9509_040.png', train_loss: 2.7756423950195312, in 22.40s
INFO:tensorflow:Saving checkpoints for 3377 into jobs/table-area-detection-0.1/model.ckpt.
INFO:tensorflow:step: 3377, file: b'9508_065.png', train_loss: 3.152759552001953, in 22.57s
INFO:tensorflow:Saving checkpoints for 3378 into jobs/table-area-detection-0.1/model.ckpt.
INFO:tensorflow:step: 3378, file: b'5008_029.png', train_loss: 2.618196725845337, in 21.45s
INFO:tensorflow:Saving checkpoints for 3379 into jobs/table-area-detection-0.1/model.ckpt.
INFO:tensorflow:step: 3379, file: b'9518_018.png', train_loss: 2.6546759605407715, in 21.48s
INFO:tensorflow:Saving checkpoints for 3380 into jobs/table-area-detection-0.1/model.ckpt.
INFO:tensorflow:step: 3380, file: b'9515_024.png', train_loss: 3.292630434036255, in 21.28s
INFO:tensorflow:finished training after 10 epoch limit

To minimize training loss, you can train 20 epochs also.

Step 4: store last checkpoint

you have to store last checkpoint. it will be used in the prediction step.

create_checkpoint.sh

lumi checkpoint create config.yml

Step 5: Prediction

You can use luminoth in two form:

  1. command line
  2. luminoth web server
  3. Using python

1. Command line:

Run this script on cmd:

lumi predict --checkpoint c2155084dca6 data/val/9541_023.png

Run this script to start Luminoth web server:

lumi server web --checkpoint c2155084dca6

3. Using Python:

Follow this Link:

https://luminoth.readthedocs.io/en/latest/tutorial/07-using-luminoth-from-python.html

𝐖𝐇𝐘 𝐘𝐎𝐔 𝐒𝐇𝐎𝐔𝐋𝐃 𝐈𝐍𝐕𝐄𝐒𝐓 𝐈𝐍𝐓𝐎 𝐒𝐓𝐎𝐂𝐊 𝐌𝐀𝐑𝐊𝐄𝐓𝐒?

𝐖𝐇𝐘 𝐘𝐎𝐔 𝐒𝐇𝐎𝐔𝐋𝐃 𝐈𝐍𝐕𝐄𝐒𝐓 𝐈𝐍𝐓𝐎 𝐒𝐓𝐎𝐂𝐊 𝐌𝐀𝐑𝐊𝐄𝐓𝐒?

This thread will answer the most common question you ask yourselves several times, WHY should we be investing in stock markets.
If it brings clarity, share with people who are still thinking & retweet.

1. Inflation
Costs are rising and the benefit of buying influence of cash is diminishing. The inflation in India throughout the previous few years has been around 4-5%. In the event that you purchase good stocks,you can make a respectable steady return of 12-18%/annum

2. Put resources into Stocks Because You'll Make More Than Other Investments
Glancing back at history, by and large, financial backers have benefitted more from purchasing stocks than from purchasing bonds, purchasing a home, or most other venture alternatives.

3. Make money work for you
You simply need to sit idle after investing in good co.s. Your cash will develop as the organization flourishes. In the interim, you can utilize your opportunity to zero in on your essential work or in the manner in which you need.

4. They're Easy to Invest In
Stocks are quick, simple, and modest to exchange, though land and numerous different speculations are not. Stocks are frequently called "liquid resources," which simply implies they can be transformed into cash moderately rapidly.

5. Small cap
There is a typical misinterpretation among numerous individuals that they need an enormous sum to master the stock market. You can contribute a modest quantity of cash and begin getting returns. This alternative isn't accessible in other ventures like land.

6. Alternate revenue source aka diversification
Fundamentally, the more ways you need to bring in cash, the less you're in danger from falling into monetary difficulty if any one technique gets upset. Investing can be second kind of revenue.

7. Saving for retirement
Numerous individuals put resources into their retirement accounts. Instead putting resources into stocks or index funds can be an extraordinary method to put something aside for retirement.

8. You Can Own Part of a Company You Love
At the point when you purchase even a solitary portion of an organization, you're authoritatively a section proprietor.
You own a piece of business without even starting one.

Well,these were the key points of why one should be investing into stock markets!Simplifying your stock market journey with these threads.
Want more of them? Comment below the topic you need and don't forget to retweet.
Have a great evening.


You can follow @Abhishekkar_.

What are the best stocks to buy for 2021-2022 in India?

If you are looking to invest in Indian stock markets in 2021–22 then the focus should be on PHARMA, CHEMICALS, METALS, AND AGRICULTURE sector.

As currently these are the sectors that are in good momentum and the future looks bright.

Let us see which stocks can get some great returns in long term.

  1. PHARMA SECTOR:
    1. Sun pharma
    2. Cipla
    3. Auropharma
    4. Glenmark
  2. CHEMICALS SECTOR:
    1. Balaji Amines
    2. Deepak Nitrite
    3. Aarti ind
    4. Vinati organics
  3. METALS:
    1. Tata steel
    2. Hindalco
    3. SAIL
    4. JSW steel
  4. AGRICULTURE SECTOR:
    1. PI INDUSTRIES
    2. Bharat Rasayan
    3. UPL
    4. DCM Shriram

These are the 4 sectors we need to focus on! As the Indian economy is currently going through a locked-down and these sectors may perform well.


Advice for startup founders | best Tips for Startup Founders

Advice for startup founders:

When was the last time you bought a product or service, because the founder is from IIT, or because the founder hasn't drawn a salary for the past 6 months, or that the founder is going through depression?

You couldn't care less! No one cares a fuck about who you are. 

The market is the market!
It doesn't care about who you are, as a founder. Which school you went to. What you have given up to get to this point.

And that is hard hitting for a lot of people who are used to things working their way because of where they come from in life. 
As a founder, you are a celebrity, whether you like it or not.

Every word you speak will be analyzed, every action of yours will be reproduced, every standard you allow will set a new standard.
Think of how you speak, how you act and the standards you accept. 
Investors are in the business of entering and exiting businesses.
They are not in the business of making your business work.
If entering your business works for them, they will enter. But they will also eventually exit.
They NEED to exit.
That's the business they are in. 
Even a bad team can work in a good market.
But a good team can't make a bad market work.

The market is (almost) everything. 
As a founder, the only time you are called successful is when you take unpopular decisions and they work out.

Taking popular decisions is "easy".
And If your decisions don't work out, then you are as it is good for nothing.

That's massive odds against you. 
Early on, the only hiring technique that works is
Attitude >> Experience >> Education

At some point in your evolution, you will need
experience >> attitude.

But... 
you won't know when precisely.

So you will continue to hire for attitude, fuck up, put pressure on yourself to make them work and blame yourself a lot more than you ought to have. 
Everyone is fucked! EVERYONE!
But it is only in close quarters that they will admit to it, or you will be able to see it.
Until then, you will drown yourself in self doubt everyday, thinking you are the only fucktard who doesn't get it, when everyone else around you does. 
Your emotional and mental state as a founder has a direct impact on your startup.

People see that you are anxious.
They know when you are depressed.
Don't pretend to wear red underwear over your pants.
Act human. Be human.
Its ok - no one was born knowing how to be a founder 
Entrepreneurship is fucking hard.
The early excitement of building a team, planning a name, launching the first version will fade away.

And insane details that life has, will begin to emerge.
At that point, there is only one thing that will help.
The stories you tell yourself. 
Usain Bolt prepares for his race.
And then he runs the race. And wins.
That moment - when he crosses the finish line. That's etched in history.

Such moments are rare, if at all, when you are a founder.
It is the journey you have to fall in love with. There is no destination! 
You will have to layoff people. At some point.
And tell you what - it will be your fucking fault.
Not a pandemic, not your investors, not your customers.
YOUR FAULT.
And that will kill you within, the first time it happens. 
You will feel like a murderer. An asshole.
And nothing would have prepared you to deal with it.

It will be the first thing you will think of every morning.
Until one day, when it won't be. 
People will come to you with problems.
You'll think they have come for solutions.
No - they have come with problems.
But you will burden yourself with the search of the solution.
And if you don't find it, which most likely you won't, you will knowingly offer a lousy solution. 
You will take everything personally.

The stapler not working - your fault.
People leaving - your fault.
Office is far for people - your fault.
You are not growing - your fault.
You will share credit for all the good things.
And kill yourself within for all the things gone wrong. 
The world will continue to define success and failure for you
Raised money - success
Forbes list - success
Shut down - failure
Not growing - failure

You will live someone's life, unless you realize it.
And start living yours. 
In the end, keep reminding yourself of why you became a founder in the first place.

You wanted to be happy doing it, feel fulfilled being one, desired peace from it.
Or whatever else was it.

Because that is the only thing the matters.
Fuck the market. Do what is right for you.