FiveTech Support Forums

FiveWin / Harbour / xBase community
Board index FiveWin for Harbour/xHarbour Why SQL Is Overrated – and Why Files, DBF, and Microservices Are the Future (Especially for Web and AI)
Posts: 6984
Joined: Fri Oct 07, 2005 07:07 PM
Re: Why SQL Is Overrated – and Why Files, DBF, and Microservices Are the Future (Especially for Web and AI)
Posted: Sun Jun 01, 2025 06:45 AM
Marcelo, you're right.
But it has some limits regarding access, security, navigation, etc.
That's exactly why we should work on the DBF format and on standardization.
The truth is, no one is really taking care of its development as a whole anymore.
But in our group, there are many database experts — and you're now pointing out what's missing.
I believe we could solve this quickly and extend DBF if we all pulled in the same direction.

I had the issue with mod\_harbour where almost 200ms pass just for startup and data access — that’s too slow for paging.
Another mod\_harbour development, version 2, is supposed to be faster in this regard.
But this group is again focusing on SQL.
So I started looking into the format itself and created a driver for PHP.
And I see now that this whole data access topic isn’t that complicated.
As a group with so many experts, we could quickly solve the few missing parts.
As far as I know, ADS is currently in maintenance mode.

Best regards,
Otto
Posts: 244
Joined: Mon Jun 05, 2006 09:39 PM
Re: Why SQL Is Overrated – and Why Files, DBF, and Microservices Are the Future (Especially for Web and AI)
Posted: Tue Jun 03, 2025 01:04 AM
Hello everyone. I have been using Ads since 2004.
I started with ads+dbf and around 2008 I started using ads with a data dictionary with adt tables.
I work with both ISAN and Static and I mix the two methods without any problems.
The speed is much higher.
Here is the tAds class that I created to be able to work with a data dictionary using DataSet style objects and with an example in the .\Samples folder.
This example will access a computer in my house. The computer is a dual core 1.2mhz. The internet is 200mb.
Since my internet belongs to private companies, access according to the source will not have a good speed. For this I would have to have a dedicated server in the cloud.
The example does not include tables.

https://mega.nz/file/7bZSEJYJ#1EQa6ckGxSyB1rKi-B6pliXRBcR_cvJUu2wEaN55ctc
Posts: 300
Joined: Wed Jul 11, 2007 11:06 AM
Re: Why SQL Is Overrated – and Why Files, DBF, and Microservices Are the Future (Especially for Web and AI)
Posted: Tue Jun 03, 2025 10:16 AM

Hello,

I develop software with MS Sql Server sinse 10 years .

I 'll no more use DBF in network development.

Select with linked table is very easy .

Have a good day .

Philippe

Posts: 990
Joined: Thu Nov 17, 2005 05:49 PM
Re: Why SQL Is Overrated – and Why Files, DBF, and Microservices Are the Future (Especially for Web and AI)
Posted: Wed Jun 18, 2025 05:43 PM
Ok, I will try to chime in this subject:
#1-- SQL hides your data behind layers of magic “You don’t own your data if you can’t see it.”
#2. Filesystems are more transparent, flexible, and AI-friendly “AI loves structure. Not secrecy.”
SQL is "magic" or "mysterious" or "incomprehensible" --yes, but only to anyone who does not understands it. Just like DBFs are incomprehensible to my cyclists friends who are great cyclists but have never seen the command prompt. SQL is not more complicated. Just different.
4. Query logic should live in your code – not buried in a database engine
This sounds nice, but in reality it is a lot easier to have your database reject certain transactions or behave in a specified manner than worrying about any possible way that any given code here or there that I may forget exists and writes or reads from a table may behave under certain unforeseen conditions. Why not just have the db centrally enforce the rule?
5. Want auditability? You won’t find it in SQL logs. * With files, auditing is easy:
* Use Git, rsync, `diff`, or timestamped snapshots
* With SQL:
Uffff, nothing further from the truth. I can't imagine asking a user to git, svn, diff, or anything else looking for how data changed, by whom and when. With SQL I simply have a trigger that writes to the audit table anytime a record is added or changed or deleted. Show audittrail table records that match a unique record id to see the audit trail on any given record. That's it. User gets to see full auditing. All you had to do was write the trigger that logs changes and you only write that once.

I'm sure a dbf user does not miss, money fields, or blob fields, or timestamp fields, or VarChar fields, or TinyInt fields. A DBF user does not miss long field names like: PropertyPhotoID and the such. **BUT** once you start using these fields, break through dBF limitations, start using triggers, have db enforce data integrity where no child is left orphaned... it will be hard to go back.

I can understand smaller systems with 5 or even 10 users and 10k records working with dbf fairly well. But if you ever aspire to have your system be used by 25 concurrent users, store millions of records, with with extended field types... I would think past dbfs. Even for smaller systems with less than 50 tables and 5k records, I find it is safer, easier, faster to write using some SQL.

I see Mr. Rao has done a great job at making it easier to transition to MySQL. He even has a method to import data from dbfs to MySql. I had to write my own since I was porting from ADS, including stored procedures and functions. My advise is to take advantage of Mrs. Rao's work and jump right in. Eventually SQL will be second nature and you will have a hard time staying with just ISAM on dbfs.

BTW-- any hosted server (Amazon, Oracle, DreamHost...) have MySQL db already configured so your system will run on any 3rd party server.

Just something to think about.
Posts: 76
Joined: Fri Aug 28, 2009 05:25 AM
Re: Why SQL Is Overrated – and Why Files, DBF, and Microservices Are the Future (Especially for Web and AI)
Posted: Thu Jun 19, 2025 04:16 AM

Dear All ,

SQL has strong features:

a) Powerful Indexing almost no rebuild is needed

b) SQL does aggregation in very few lines

c) Strong transaction management for OLTP Apps

d) Concurrent Management

many more...!

Thanks

Shridhar

Posts: 6984
Joined: Fri Oct 07, 2005 07:07 PM
Re: Why SQL Is Overrated – and Why Files, DBF, and Microservices Are the Future (Especially for Web and AI)
Posted: Thu Jun 19, 2025 06:19 AM

Hi Reinaldo,

Thanks for the input – from a technical point of view you're right: most hosted servers come preconfigured with MySQL and are ready to use.

However, things are quite different for us here in the EU, especially in Austria and Germany. We have to follow strict GDPR rules when working with personal data – and that makes using third-party hosting (like Amazon, Oracle, etc.) more complicated.

Key points we need to consider:

Data transfers to non-EU countries (like the US) are problematic due to legal access (e.g., the CLOUD Act).

A data processing agreement (DPA) is mandatory – and must be actively signed.

Technical and organizational measures (TOMs) must be clearly documented.

Full transparency is required about where and how data is stored and processed.

That’s why many of us prefer EU-based or even self-hosted solutions when dealing with sensitive user data.

Best regards from Austria!

Otto

Posts: 6984
Joined: Fri Oct 07, 2005 07:07 PM
Re: Why SQL Is Overrated – and Why Files, DBF, and Microservices Are the Future (Especially for Web and AI)
Posted: Thu Jun 19, 2025 06:27 AM

Hello,

I fully agree that SQL brings powerful features, especially for classic business systems:

a) Powerful indexing with minimal rebuilds

b) Fast aggregations in very few lines

c) Reliable transaction management for OLTP

d) Great concurrency handling

These are well-proven advantages, and they make SQL a great choice for established use cases like financial systems, administrative records, and structured reporting.

But I’m currently working on something different — a highly dynamic, context-driven workflow for processing human communication (e.g. anonymized emails, JSON mappings, context-aware responses). Everything is in constant flux:

The data structure is semi-structured at best (e.g. [Guest:ID], [Date:ID])

Placeholder formats evolve frequently

Contextual logic (e.g. temporal flow, RAG-augmented reasoning) is essential

Workflows need to adapt quickly — not go through migration scripts and schema locks

In such a case, I feel SQL might get in the way rather than help — and that file-based + script-based approaches (e.g. JSON + Python) offer more agility.

That said:

Are there scenarios where SQL could still add value in such dynamic, non-relational contexts?

Or would you agree this falls outside of where SQL shines?

I’d love to hear your thoughts — especially if you’ve built hybrid approaches.

Thanks!

Otto

Posts: 9022
Joined: Thu Oct 06, 2005 08:17 PM
Re: Why SQL Is Overrated – and Why Files, DBF, and Microservices Are the Future (Especially for Web and AI)
Posted: Thu Jun 19, 2025 07:17 AM

SQL is great for transaction based workflow, but it is a nightmare for strong interactive apps.

Posts: 76
Joined: Fri Aug 28, 2009 05:25 AM
Re: Why SQL Is Overrated – and Why Files, DBF, and Microservices Are the Future (Especially for Web and AI)
Posted: Thu Jun 19, 2025 09:03 AM

Dear All ,

UPI Indian Payment System is great example which is running on SQL database e.g. Oracle , PostgreSQL , MySQL approx. 1000 TPS in one bank.

Thanks

Shridhar

Posts: 6984
Joined: Fri Oct 07, 2005 07:07 PM
Re: Why SQL Is Overrated – and Why Files, DBF, and Microservices Are the Future (Especially for Web and AI)
Posted: Thu Jun 19, 2025 01:02 PM

Thanks Shridhar – technically you're right, but here in Europe a system like UPI would never be built with something like Harbour. It lacks scalability, security certifications, and enterprise support. Harbour is better suited for small, local business applications.

By the way – are you really considering Harbour for such systems, or was it just an example?

Best regards,

Otto

Posts: 76
Joined: Fri Aug 28, 2009 05:25 AM
Re: Why SQL Is Overrated – and Why Files, DBF, and Microservices Are the Future (Especially for Web and AI)
Posted: Mon Jun 23, 2025 12:21 PM

Dear Otto ,

As Harbour using C platform it should be used also to develop scalable apps. I am not expert in developing languages but I do not know what it is lagging HVM from JVM ( java virtual Machine ) ?

Thanks

Shridhar

Continue the discussion