FiveTech Support Forums

FiveWin / Harbour / xBase community
Board index FiveWin for Harbour/xHarbour Fulltextsearch (to Marco Boschi)
Posts: 6983
Joined: Fri Oct 07, 2005 07:07 PM
Fulltextsearch (to Marco Boschi)
Posted: Sun Nov 07, 2010 09:59 AM
Hello Marco,
did you ever found a solution to your question.
Thanks in advance
Otto
#include "FileIO.ch"

FUNCTION MAIN()

LOCAL cAllMemos

// open and read all dbf file into cStringa variable
nHandle := FOPEN( "memo2.fpt" )
nSize := FSeek( nHandle, 0, FS_END )
FSeek( nHandle, -nSize, FS_RELATIVE )
cAllMemos := SPACE( nSize )
nRead := FRead( nHandle, @cAllMemos, nSize )
FCLOSE( nHandle )
? cAllMemos
RETURN NIL

cAllMemos contains in a single fRead all fpt file.

For instance I search string "MARCO" in cAllMemos string.
Obviously I know structure of MEMO2.DBF

The question is: is it possible to get the number of record and the
name of memo field that corresponds to this search?

what are the rules for storing contents of memo fields in the file
fpt?
Does exist some any documentation?




@Harbour
FlexFile3

 *In this first memo field I write "MARCO"
¯  In this I write
"BOSCHI" ¯

Posts: 1091
Joined: Thu Nov 17, 2005 11:08 AM
Re: Fulltextsearch (to Marco Boschi)
Posted: Mon Nov 08, 2010 12:40 PM
Otto,
take a look at
http://groups.google.com/group/comp.lang.xharbour/browse_thread/thread/c48a5aff0123c410/fef17ea5e6d08c8d?lnk=gst&q=allmemo#fef17ea5e6d08c8d

Sorry but I'm very busy today and tomorrow

For the moment, I tell you only one thing:
for fast searches with good results I'm trying to load an enture table into an array and then search the array.
Bye
Marco Boschi
info@marcoboschi.it
Posts: 4840
Joined: Fri Nov 18, 2005 04:52 PM
Re: Fulltextsearch (to Marco Boschi)
Posted: Mon Nov 08, 2010 04:51 PM

Here is documentation for FTP memo fields:

http://www.clicketyclick.dk/databases/x ... FPT_STRUCT

Regards,
James

FWH 18.05/xHarbour 1.2.3/BCC7/Windows 10
Posts: 6983
Joined: Fri Oct 07, 2005 07:07 PM
Re: Fulltextsearch (to Marco Boschi)
Posted: Mon Nov 08, 2010 08:27 PM

Hello James,
thank you for your help.
If I use a text editor to look I have a value of 73610 for example inside the memo field.
Do you know what this value is. I think it is the offset of the memoblock. But how do I calculate?
I also saw that if a memoblock is edited and larger than 512 kB a new block is added. The old one still
remains and I didn’t find a delete mark.
As Marco said it seems impossible with the oneway design of memofields - the offset is only stored in the dbf file -
to find the record number corresponding to the memoblock.
What do you think?

Best regards,
Otto

Posts: 4840
Joined: Fri Nov 18, 2005 04:52 PM
Re: Fulltextsearch (to Marco Boschi)
Posted: Mon Nov 08, 2010 11:05 PM

Otto,

Sorry, I have never done any work requiring working with the structure of memo fields, I just happen to have the link to the reference in my notes.

Perhaps you can try Googling for "FTP memo field structure reading" or some such.

Regards,
James

FWH 18.05/xHarbour 1.2.3/BCC7/Windows 10
Posts: 6983
Joined: Fri Oct 07, 2005 07:07 PM
Re: Fulltextsearch (to Marco Boschi)
Posted: Mon Nov 08, 2010 11:10 PM

Hello James,
I spend my last weekend Googling for "FTP memo field structure" with no success.
Best regards,
Otto

Posts: 4840
Joined: Fri Nov 18, 2005 04:52 PM
Re: Fulltextsearch (to Marco Boschi)
Posted: Mon Nov 08, 2010 11:34 PM

Otto,

Well, so much for that idea!

Can you tell us more about what you are trying to do? I assume you need to search memo fields, but in how big of a file? Also, it is a one time search or multiple searches at the same time (by same time I don't mean all at once, but sequential searches all by the same user).

One of the difficulties, as you have mentioned, is unused blocks of text. Memo fields leave lots of unused blocks of text in the file which causes the file size to grow very quickly. To be efficient you have to either skip over those blocks or pack the file regularly.

I vaguely remember that there was a third-party commercial product that used a different file structure to store memo fields. I also remember working on one myself a number of years ago. Since you are using database objects it should be fairly simple to substitute a different database class (that doesn't use memo fields) into your system without any code changes to your system (only new code in the database class).

Anyway, if you tell me more about your problem, maybe I can come up with some ideas.

Regards,
James

FWH 18.05/xHarbour 1.2.3/BCC7/Windows 10
Posts: 663
Joined: Mon Dec 05, 2005 11:22 PM
Re: Fulltextsearch (to Marco Boschi)
Posted: Mon Nov 08, 2010 11:59 PM

I think you can any text ( including memo fields ) in a FTS (Fast Text Search) or Hyper-SEEK index. It will take all words in the text/memo field and index them.

Posts: 4840
Joined: Fri Nov 18, 2005 04:52 PM
Re: Fulltextsearch (to Marco Boschi)
Posted: Tue Nov 09, 2010 01:53 AM

Gale,

If I remember correctly, those are only available with the commercial version of xHarbour. Is that right?

Regards,
James

FWH 18.05/xHarbour 1.2.3/BCC7/Windows 10
Posts: 824
Joined: Thu Oct 13, 2005 07:39 AM
Re: Fulltextsearch (to Marco Boschi)
Posted: Tue Nov 09, 2010 08:57 AM
James,

HiPer-SEEK is available in free xharbour too. You can search a string over several fields.

Here is a sample (from xharbour)
#include "dbinfo.ch"

//REQUEST RMDBFCDX

PROCEDURE Main()
FIELD FIRST, LAST, STREET, CITY
LOCAL n, hs

if ascan( rddList(1), "RMDBFCDX" ) != 0
rddSetDefault( "RMDBFCDX" )
endif

use test
hs := HS_INDEX( "test", "FIRST+LAST+STREET+CITY", 2, 0, , .T., 3 )

/* Look for all records which have 'SHERMAN' string inside */
HS_SET( hs, "SHERMAN" )
while ( n := HS_NEXT( hs ) ) > 0
dbgoto( n )
if HS_VERIFY( hs ) > 0
? rtrim( FIRST+LAST+STREET+CITY )
endif
enddo
wait

/* Does RDD support Record Map Filters? */
if dbinfo( DBI_RM_SUPPORTED )
/* if yest then let set filter for all records with 'SHERMAN'
word and look at them in browser */
HS_FILTER( hs, "SHERMAN" )
dbgotop()
browse()
endif
HS_CLOSE( hs )
RETURN
kind regards

Stefan
Posts: 10733
Joined: Sun Nov 19, 2006 05:22 AM
Re: Fulltextsearch (to Marco Boschi)
Posted: Tue Nov 09, 2010 10:11 AM

Is RMDBFCDX available in free xHarbour? If so, what is the library to be linked to use RMDBFCDX ?

Regards



G. N. Rao.

Hyderabad, India
Posts: 824
Joined: Thu Oct 13, 2005 07:39 AM
Re: Fulltextsearch (to Marco Boschi)
Posted: Tue Nov 09, 2010 11:57 AM
RMDBFCDX is only available in commercial xharbour, but you don´t need it to run the sample. Only if it is linked in, it´s used. You can also comment out these lines:
if ascan( rddList(1), "RMDBFCDX" ) != 0
rddSetDefault( "RMDBFCDX" )
endif
kind regards

Stefan
Posts: 10733
Joined: Sun Nov 19, 2006 05:22 AM
Re: Fulltextsearch (to Marco Boschi)
Posted: Tue Nov 09, 2010 12:53 PM

I am getting HS_KEY, HS_FILTER as unresolved externals. What is/are the libaray(ies) I need to link to include these functions?

Regards



G. N. Rao.

Hyderabad, India
Posts: 663
Joined: Mon Dec 05, 2005 11:22 PM
Re: Fulltextsearch (to Marco Boschi)
Posted: Tue Nov 09, 2010 02:38 PM

I believe it is only in the commercial version.

Posts: 4840
Joined: Fri Nov 18, 2005 04:52 PM
Re: Fulltextsearch (to Marco Boschi)
Posted: Tue Nov 09, 2010 03:10 PM
Stefan,

HiPer-SEEK is available in free xharbour too. You can search a string over several fields.


Does this include memo fields?

Even so, I would think an index that indexes each word in a memo field would be much more efficient. This would be like a many-to-many relationship. Each word could be listed multiple times once for each record containing it.

Regards,
James
FWH 18.05/xHarbour 1.2.3/BCC7/Windows 10