FiveTech Support Forums

Hi Marco

Since it seems that the slowdown is linked to the opening of the index,

can you put the index formula and how many TAGs are present?

tia

The problem is the same even with only one tag.

Mauri, my answer is identical to enrico's one

Dear Enrico,

this is my logic to generate an index file in my PHP4DBF Lib.

I would like to compare this logic with the generation of in-memory indexes in Harbour.

Where can I find the source code? Can you tell me that?

Generating an index:

I first read all records in a loop and write all matches into

$matchRecords[] = [

'index' => $i,

'fieldValue' => $fieldValue, // Add fieldValue here

'recno' => $i + 1 // Add recno here

];

Then I sort $matchRecords[]

usort($matchRecords, function ($a, $b) {

return strcmp($a['fieldValue'], $b['fieldValue']);

});

and then I store all indexes in the sorted order, this is then my "index file"

$indexes = array_column($matchRecords, 'index');

To get the records sorted I use:

function php4dbf_getRecordByIndex($fileHandle, $index, $header, $fields, $loggingEnabled = true) {

$recordOffset = $header['headerLength'] + ($index * $header['recordLength']);

fseek($fileHandle, $recordOffset);

$record = fread($fileHandle, $header['recordLength']);



if ($record === false) {

    if ($loggingEnabled) {

        error_log("Error reading record at index $index");

    }

    return null;

}



$result = [];

$fieldOffset = 1; // Skipping the deleted flag byte

foreach ($fields as $field) {

    $fieldValue = trim(substr($record, $fieldOffset, $field['length']));

    $result[$field['name']] = my_utf8_encode($fieldValue); // Ensure UTF-8 encoding

    $fieldOffset += $field['length'];

}



return $result;

}

Best regards,

Otto

Hi Marco

not only how many tags but its formula for that single tag

e.g.: dtos( gio_drg) + str(gio_num,9) + str(gio_riga,3)

I often made too complex indexes which often slowed down the browse,

I would like to know if the slowness you see is on loop type : ' do while .. enddo '

or on a browse

Bye

I already tried with a simple formula like a single field: same problem. I only tried with a loop, not with a browse.

I like the idea of local indexes (temp) but in network system, once a other machine changes/adds data, the local indexes would be wrong not ?

In previous posts, I remember that this issue meanly is creating the right indexes... Using the RDD's optimised filters etc.. Mr. Rao onces started to explaine (thanks !!) but we didn't get to the final approuch (samples)

A quick Update,

the function in wich is very evident this slowness I insert an interval od Year + month permits to me to obtain an Excel containing all traffic from and to a particular number, Telephone traffic reading cdr.

For instance

202401 June 2024

202409 September 2024

And then I enter a telephone number to search for

I change every month the dbf table containing all Pbx traffic therefore only one file is opened by the program that stores data in it.

In this moment (today) only 202409.dbf is opened by the program who write all data coming from Pbx.

All other dbf tables ar not open from anybody, they are only used for statistics.

I have a For NEXT cicle that open a dbf file at a time and than a do while enddo cicle in quch analize every single

record that satisfye some conditions.

It is very quicly until it opens the table i_202409.dbf and i_202409.cdx that is opened by the other program that writes data.

For this last table it is very very slow.

I simply copy in local disk c: CDX files and open it instead the one on the lan.

The difference in performance are 29,77 seconds if I open cdx on lan 2,42 seconds If I copy cdx files in locel and open it

This is a great solution if I don't need to change anything in the table.

Hav a nice day

Marco

MarcoBoschi wrote:A quick Update,

I have a For NEXT cicle that open a dbf file at a time and than a do while enddo cicle in quch analize every single
record that satisfye some conditions.

Dear Marco,

The 500ms I mentioned above come from a database with 200,000 records. The database is 20 MB in size.

Could you maybe post the structure of the database after all? I would then fill it with dummy data for testing purposes out of curiosity.

It would also be interesting to know what you want to query.

Best regards,

Otto

Otto I've prepared a table in wich the contents of some fields are partially obscured

I send you via mail

Please your address?

Many thank

Dear Marco,

otto(@)atzwanger.com

Thank you in advance

Otto

Marco,

It seems that you pull all dfb's for all selected months and than process each record from every dbf's over the net. No filters, seeks etc. ? I think here is you problem, or maybe i'm missing something