Hello everyone! How are you doing?
In my last article we have about “Latest
PHP frameworks in 2014?”. Today I will impart to you one of my ongoing experience.
Recently I came across the task where I need
to import large csv records into mysql database. It’s quite tough because your
phpmyadmin will not allow you to import large file. That csv contains around 10
million records. You may think don't I use mysql LOADINFILE ? Due to some restriction I avoided and choosed my next option as csv spliting. First thing I tried to open that file, but it is displaying error as “File not
loaded completely”.
The reason behind the above error is excel can only handle 65,536 lines per sheet and it cannot open remaining records. I also tried to import this file directly into PhpMyadmin but the maximum time limit has exceeded. So the next thing what I did is I have increased the PHP "max
execution time" but it was taking too much time to process it.
Now you must be thinking that How do i solve this problem?
Here in this blog i will show you how to solve this issue with very simple steps. You can easily solve it simply by splitting the large CSV file into smaller chunks. By doing that script will not take much time process.
You can simply download FXFisherman’s CSV Spliter
program and run this program in your windows system. Installation is not required.
You can see below window when you run it.
Now you may browse your csv file which you
want to split, specify how many numbers of lines you want to have per csv and
also another way you can simply specify max number of pieces. Now simply click “Split
Now!”.
It will begin parting your csv into number of pieces. When this process is
completed you should see a screen below.
Once process is completed you will find a folder contains chunks of csv file into it, in the
same directory.
Thank God! It Saved My Day.
I
was able to import 3 millions of records at the end of the day.
Do
you have any other better or faster way to accomplish this job?
Hope
this article will be helpful. Please do not forget to drop your comments below and share
with your friends. It will encourage me to write many more such article for you.
Keep visiting for new posts. If you find my articles
are useful just follow my blog and subscribe to newsletter.
