FiveTech Support Forums

FiveWin / Harbour / xBase community
Board index FiveWin for Harbour/xHarbour Fulltextsearch (to Marco Boschi)
Posts: 1091
Joined: Thu Nov 17, 2005 11:08 AM
Re: Fulltextsearch (to Marco Boschi)
Posted: Wed Nov 10, 2010 09:54 AM

Otto,
I load an entire dbf into an Array using this function
...
LOCAL aClifor := {}
...
aClifor := load_dbf( "clifor" )

....
....
FUNCTION LOAD_DBF( cFile )
LOCAL aRecord := {}
LOCAL aFile := {}
LOCAL nFld
LOCAL nIni

SET DELETED OFF
SELECT 0
USE &cFile
GO TOP

nIni := SECONDS()

DO WHILE !EOF()
aRecord := {}
AADD( aRecord , IIF( DELETED() , "D", " " ) )
FOR nFld := 1 TO FCount()
// here if I remove condition I think I can include numeric
IF fieldtype( nFld ) = "C" .OR. fieldtype( nFld ) = "M"also
AADD( aRecord , UPPER(fieldget( nFld )) )
ENDIF
NEXT i

AADD( aFile , aRecord )
SKIP

ENDDO

SET DELETED ON

RETURN aFile

If I have to search in array call this function

sea_array( ALLTRIM("clifor"), aClifor, UPPER( field->words ) , field->progr )

at this point of the program I use cdx files created by function sea_array
// INDEX ON recno() TAG RICERCA CUSTOM TO &cCdxFile

SET INDEX TO &cIndSearch
SET ORDER TO TAG RICERCA

FUNCTION SEA_ARRAY( cFile , aFile, cCerca, cProgr )

LOCAL nPosLast
LOCAL nPos
LOCAL nFound
LOCAL iCerca
LOCAL nPos2
LOCAL nRecord
LOCAL n1 := SECONDS()
LOCAL aCerca := HB_ATokens( alltrim( cCerca ) , " " )
LOCAL aFound := {}
LOCAL nHandle

LOCAL cCdxFile := "search\" + cProgr

nPosLast := 0

DO WHILE .T.

  nPos := ASCAN( aFile , { | aItem | ASCAN( aItem, { | cItem | aCerca[1] $ cItem } ) > 0 } , nPosLast + 1 )

  IF nPos > 0
     nFound := 1
     FOR iCerca := 2 TO LEN( aCerca )
         nPos2 := ASCAN( aFile[ nPos ] , { | cItem | aCerca[ iCerca ] $ cItem } ) 
         IF nPos2 > 0
            nFound ++
         ENDIF
     NEXT iCerca
     IF nFound = LEN( aCerca )
        IF aFile[ nPos , U_DELETE  ] <> "D"
           AADD( aFound , nPos )
        ENDIF
     ENDIF
     nPosLast := nPos
  ELSE
     EXIT
  ENDIF

ENDDO

ferase( cCdxFile + ".cdx" )

IF LEN( aFound ) > 0

  SELECT &cFile

  INDEX ON recno() TAG RICERCA CUSTOM TO &cCdxFile
  FOR EACH nRecord In aFound
      GOTO nRecord
      OrdKeyAdd()
  NEXT
  SET INDEX TO

ELSE
nhandle := fcreate( cCdxFile + ".cdx" )
fclose( nHandle )
ENDIF

RETURN NIL

This is a client/server application
Server side I load into an array the content of dbf table at startup
At the end of this initial operation this app in polling mode wait for requests from clients.
A client makes a request to server that:
- perform search
- create index
the client when index is created use this index.

When a clients modifies a record of table (loaded into array) I update array in this way **

agg_rec( "clifor", aClifor, nRecno )

FUNCTION AGG_REC( cFile, aFile, nRecord )
LOCAL cDelete
LOCAL aRec := {}
LOCAL nFld, nNew
LOCAL nNuovi
LOCAL aNew := {}
LOCAL nStart := LEN( aFile )
LOCAL i
// se ci sono nuovi record prima crea i nuovi elementi dell'array vuoti
IF nRecord > LEN( aFile )

nNuovi := nRecord - LEN( aFile )

FOR nNew := 1 TO nNuovi
aRec := {}
FOR nFld := 1 TO FCount()
IF fieldtype( nFld ) = "C" .OR. fieldtype( nFld ) = "M"
AADD( aRec , "" )
ENDIF
NEXT i

   AADD( aFile , aRec )
   AADD( aNew , nStart + nNew )

NEXT nNew
ELSE
AADD( aNew , nRecord )
ENDIF

SELECT &cFile

FOR nNew := 1 TO LEN( aNew )

i := aNew[ nNew ]
GOTO aNew[ nNew ]

aFile[ i , U_DELETE ] := cDelete

aFile[ i , U_DELETE ] := " "
aRec := {}

AADD( aRec , IIF( DELETED() , "D", " " ) )
FOR nFld := 1 TO FCount()

    IF fieldtype( nFld ) = "C" .OR. fieldtype( nFld ) = "M"
       AADD( aRec , UPPER( fieldget( nFld ) ) )
    ENDIF

NEXT i

FOR nFld := 1 TO LEN( aFile[ i ] )
    IF aFile[ i , nFld ] <> aRec[ nFld ]
       aFile[ i , nFld ] := aRec[ nFld ]
    ENDIF
NEXT nFld

NEXT nNew

RETURN NIL

** I use another historical program that in real time update from main database (set of dbf tables). After each replace I write into a log file dbftable and record
Another server app read this registrations and replies anyu change in the secondary database

(You do not know how many times they saved my ass)

Marco Boschi
info@marcoboschi.it
Posts: 824
Joined: Thu Oct 13, 2005 07:39 AM
Re: Fulltextsearch (to Marco Boschi)
Posted: Wed Nov 10, 2010 12:55 PM
James,

James Bott wrote:Stefan,
Does this include memo fields?


I didn´t test it with memo fields so far, but I think, it should work. HSX uses its own index file, it has nothing to do with the other index files nor with any database relation.
kind regards

Stefan
Posts: 824
Joined: Thu Oct 13, 2005 07:39 AM
Re: Fulltextsearch (to Marco Boschi)
Posted: Wed Nov 10, 2010 01:06 PM
James,

James Bott wrote:Stefan,

In your example what is the var HS, an object, an order number?

Can these indexes be part of a single index file and be kept up automatically just like any other CDX index.

Regards,
James


hs is the handle of the index file. It is a single index file, but it is not automatically updated. I´m creating a new index ervery time my search function is called. Nevertheless it´s very fast.

I use hsx this way

Code (fw): Select all Collapse
SELECT Kunden

hHSX := HS_Index ("Kunden","Vorname+Name+Strasse+Ort",2,0,,.t.,3)

IF MsgGet ("Erweiterte Kundensuche","Suchtext eingeben: (Vorname,Name,Strasse oder Ort)",@cSearch)
  nRec := Kunden->(RecNo())
  Kunden->(OrdSetFocus ("KundenName"))
  Kunden->(dbGoTop())
  cSearch := Alltrim (Lower(cSearch))
 
  cSearch := AnsiToOem (cSearch)

  HS_Set (hHSX, cSearch)
  WHILE (n := HS_Next (hHSX)) > 0
    //? n
    Kunden->(dbGoTo (n))
    IF HS_Verify (hHSX) > 0

       AADD (aFound, {OemToAnsi(Kunden->Name),OemToAnsi(Kunden->Vorname),;
                      OemToAnsi(Kunden->Strasse),OemToAnsi (Kunden->Ort),;
                      Kunden->(RecNo()) } )
    ENDIF
  ENDDO
  HS_Close (hHSX)
ENDIF
kind regards

Stefan
Posts: 824
Joined: Thu Oct 13, 2005 07:39 AM
Re: Fulltextsearch (to Marco Boschi)
Posted: Wed Nov 10, 2010 01:16 PM
James,

James Bott wrote:
I do wonder about what is required to switch from CDX indexes to HiperSix indexes. Is this going to require a lot of code changes due to differences in syntax


there is already a userrdd called hsxrdd included in xharbour. But it seems you can´t use the normal index functions. Here is a sample from the docs
Code (fw): Select all Collapse
#include "dbinfo.ch"

REQUEST HSXRDD

PROC MAIN()
   FIELD FIRST, LAST, STREET, CITY
   LOCAL n, hs

   rddSetDefault("HSXRDD")
   dbCreate("_tst", {{"FIRST",       "C", 20,  0},;
                     {"LAST",        "C", 20,  0},;
                     {"STREET",      "C", 30,  0},;
                     {"CITY",        "C", 30,  0},;
                     {"STATE",       "C",  2,  0},;
                     {"ZIP",         "C", 10,  0},;
                     {"HIREDATE",    "D",  8,  0},;
                     {"MARRIED",     "L",  1,  0},;
                     {"AGE",         "N",  2,  0},;
                     {"SALARY",      "N",  6,  0},;
                     {"NOTES",       "C", 70,  0}})
   USE _tst
   HSX_CREATE( "_tst", "FIRST+LAST+STREET+CITY", 2, 0, .T., 3 )
   APPEND FROM test

   /* Look for all records which have 'SHERMAN' string inside */
   hs := HSX_HANDLE( "_tst" )
   HS_SET( hs, "SHERMAN" )
   WHILE ( n := HS_NEXT( hs ) ) > 0
      DBGOTO( n )
      IF HS_VERIFY( hs ) > 0
         ? RTRIM( FIRST+LAST+STREET+CITY )
      ENDIF
   ENDDO
   WAIT

   /* Does RDD support Record Map Filters? */
   IF DBINFO( DBI_RM_SUPPORTED )
      /* if yest then let set filter for all records with 'SHERMAN'
         word and look at them in browser */
      HS_FILTER( hs, "SHERMAN" )
      DBGOTOP()
      BROWSE()
   ENDIF
RETURN
kind regards

Stefan
Posts: 4840
Joined: Fri Nov 18, 2005 04:52 PM
Re: Fulltextsearch (to Marco Boschi)
Posted: Sat Nov 13, 2010 06:04 PM

Otto,

Have you been able to get a full-text search system going yet?

Regards,
James

FWH 18.05/xHarbour 1.2.3/BCC7/Windows 10
Posts: 6983
Joined: Fri Oct 07, 2005 07:07 PM
Re: Fulltextsearch (to Marco Boschi)
Posted: Sun Nov 21, 2010 09:12 AM
Hello James,

here is a screenshot from the application I would like to program.

Best regards,
Otto

Posts: 4840
Joined: Fri Nov 18, 2005 04:52 PM
Re: Fulltextsearch (to Marco Boschi)
Posted: Sun Nov 21, 2010 06:40 PM

Otto,

You just want to search those tiny memo fields? How many records are there (roughly)?

I would think just doing a DO WHILE ! EOF() would be sufficient as long as there are not too many records.

Regards,
James

FWH 18.05/xHarbour 1.2.3/BCC7/Windows 10

Continue the discussion