FiveTech Support Forums

FiveWin / Harbour / xBase community
Board index FiveWin for Harbour/xHarbour BULK INSERT in MySQL
Posts: 1195
Joined: Mon Oct 17, 2005 05:41 AM
BULK INSERT in MySQL
Posted: Sun Sep 02, 2012 09:13 AM

Hi,

I'm converting data from an existing DBF to MySQL. I have tables with more than 2.000.000 records.
Now I'm going to the top of the DBF, and with an Do While !eof() I skip to the dbf and insert each record in the SQL.
It is very slow (more than 2 hours in some cases).
Is there a faster way to do this?
In SQLite I use 'Begin immediate' and 'Commit' before the loop, what resulted in a faster execution, but it doesn't work in MySQL.
Also using 'Start stransaction' and 'commit' doesn't help.

Thanks

Marc

Regards,

Marc



FWH32+xHarbour | FWH64+Harbour | BCC | DBF | ADO+MySQL | ADO+MariaDB | ADO+SQLite
Posts: 368
Joined: Sun May 31, 2009 06:25 PM
Re: BULK INSERT in MySQL
Posted: Sun Sep 02, 2012 01:44 PM
Read the values from the DBF to a temporary string and use INSERT statements with multiple VALUES lists to insert several rows at a time.
sql = "INSERT INTO your_table (field1, field2) VALUES ('string1', value1),('string2', value2),('string3', value3);"

You can also use a free tool like dbf2sql: http://www.tkinformidia.net/uploads/1/2/0/0/12005968/dbf2sql-3.0.zip
Regards,



André Dutheil

FWH 13.04 + HB 3.2 + MSVS 10
Posts: 1195
Joined: Mon Oct 17, 2005 05:41 AM
Re: BULK INSERT in MySQL
Posted: Mon Sep 03, 2012 06:32 AM

André,

Thanks for the information. I will try to insert more records at the same time.

Regards,

Marc

Regards,

Marc



FWH32+xHarbour | FWH64+Harbour | BCC | DBF | ADO+MySQL | ADO+MariaDB | ADO+SQLite
Posts: 1195
Joined: Mon Oct 17, 2005 05:41 AM
Re: BULK INSERT in MySQL
Posted: Fri Sep 07, 2012 07:38 AM
Hi,

I found a faster way to import/export into MySQL by using CSV-files.

The only problem I have is if I export to a CSV-file and do a append from into a DBF with the command
Code (fw): Select all Collapse
oSQL:query([SELECT * INTO OUTFILE 'C:/temp/TESTPLAN.CSV' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\n' FROM planning])


only 1 record is imported into the DBF. It seems that the LINES TERMINATED BY '\n' command is only a LF and not CRLF.
Does anyone know how to export to a CSV with CRLF?

Thanks
Marc
Regards,

Marc



FWH32+xHarbour | FWH64+Harbour | BCC | DBF | ADO+MySQL | ADO+MariaDB | ADO+SQLite
Posts: 1195
Joined: Mon Oct 17, 2005 05:41 AM
Re: BULK INSERT in MySQL
Posted: Fri Sep 07, 2012 08:34 AM
I found it!!
Code (fw): Select all Collapse
\r\n
:-)

Marc
Regards,

Marc



FWH32+xHarbour | FWH64+Harbour | BCC | DBF | ADO+MySQL | ADO+MariaDB | ADO+SQLite

Continue the discussion