Tensor Logic: from 0 to 100

Runnable companion: samples/ai/tltutor.prg — build it, run it, read tltutor.log next to this page

This tutorial takes you from "what is a tensor" to training a language model on FWH source code, using one single idea from Pedro Domingos' Tensor Logic (arXiv:2510.12269): a logical rule and an Einstein summation are the same operation. Every lesson below is a function in tltutor.prg — the outputs shown are its real output.

Full API: Tensor Logic reference.

How to use Tensor Logic — step by step

Step 0 — Prerequisites

• FiveWin for Harbour installed (this tree under C:\fwteam).
• A compiler variant already able to build samples (e.g. Harbour + BCC32 → hb32). See Build System.
• FWH libraries rebuilt so they include tensorlogic.prg and fwtensor.c:

  cd /d C:\fwteam
  C:\fwteam\mfwh_new.bat hb32

  (Use hm32, hm64, hg64, … for other variants.)
• Optional but recommended: OpenBLAS. Place libopenblas.dll on the PATH or next to your exe (a copy lives in samples\ai\libopenblas.dll). Without it the pure-C loops still work, just slower. Force portable mode with set FWT_BLAS_DLL=off.

Step 1 — Build and run the tutorial sample

cd /d C:\fwteam\samples\ai
C:\fwteam\samples\build_new.bat tltutor hb32
tltutor.exe
type tltutor.log

You should see nine lesson blocks ending with TUTORIAL COMPLETE — next: tltransformer.prg, tlminillm.prg, tlfwcode.prg. Keep tltutor.log open while you read the lessons below.

Step 2 — Mental model (30 seconds)

1. A tensor is an n-D array of float32 in C memory.
2. An einsum multiplies along shared indices (join) and sums away indices missing from the output (projection).
3. A tensor logic program is named tensors + equation strings evaluated by Run() (fixpoint) or Forward() (one pass for training).
4. The gradient of an equation is the same einsum with a factor and the output swapped — so any program is trainable with no hand-written backprop.

Step 3 — Your first program (copy-paste)

Create a small PRG, or type this into a scratch sample:

#include "FiveWin.ch"

FUNCTION Main()
   LOCAL oTL := TTensorLogic():New()

   oTL:Let( "W", { { 1, -1, 0 }, { 0, 1, -1 } } )
   oTL:Let( "X", { 1, 2, 0.5 } )
   oTL:Let( "B", { 2, -2 } )
   oTL:Eq( "Y[i] = step( W[i,j] * X[j] + B[i] )" )
   if ! oTL:Run()
      ? "Error:", oTL:cError
      RETURN NIL
   endif
   ? oTL:GetArray( "Y" )     // {1, 0}
   ? "BLAS active?", FWT_BlasActive()
RETURN NIL

C:\fwteam\samples\build_new.bat myfirst hb32
myfirst.exe

Step 4 — Train something (AND gate)

oTL := TTensorLogic():New()
oTL:Let(   "X", { { 0, 0 }, { 0, 1 }, { 1, 0 }, { 1, 1 } } )
oTL:Let(   "Ones", { 1, 1, 1, 1 } )
oTL:Param( "W", { { 0.1 }, { -0.1 } } )   // Param = learnable
oTL:Param( "B", { 0 } )
oTL:Eq( "Y[n,o] = sigmoid( X[n,f] * W[f,o] + Ones[n] * B[o] )" )
tT := FWT_FromArray( { { 0 }, { 0 }, { 0 }, { 1 } } )

for e := 1 to 300
   oTL:Forward()
   oTL:ZeroGrad()
   oTL:Seed( "Y", FWT_Scale( FWT_Sub( oTL:Get( "Y" ), tT ), 2 ) )
   oTL:Backward()
   oTL:ClipGrads( 1.0 )
   oTL:StepAdam( 0.05 )
next
// Y ≈ {0.00, 0.11, 0.11, 0.88} for targets {0,0,0,1}
oTL:SaveParams( "and.chk" )

That loop — Forward / ZeroGrad / Seed / Backward / Clip / Adam — is the same one used by the full language-model samples.

Step 5 — Climb the sample ladder

C:\fwteam\samples\build_new.bat tensorlogic hb32
tensorlogic.exe
type tensorlogic.log

Lesson 1 — Tensors

A tensor is an n-dimensional array of numbers. In FWH it is an FW_Tensor: a flat float32 buffer in C memory (fast, compact), created and inspected from PRG:

t := FWT_New( { 2, 3 } )                 // 2x3, zero-filled
FWT_Set( t, 1, 10 )                      // flat 1-based indexing, row-major
FWT_Set( t, 6, 60 )
? FWT_Shape( t )                         // {2, 3}
t := FWT_FromArray( { { 1, 2 }, { 3, 4 } } )   // from a nested array

Expected output (from tltutor.log):

=== Lesson 1: tensors ===
shape = {2, 3}, size = 6
t = {{10.00, 0.00, 0.00}, {0.00, 0.00, 60.00}}
from array = {{1.00, 2.00}, {3.00, 4.00}}

Lesson 2 — Einsum: the only operation you need

An einsum spec names each operand's indices and the output's. Two rules:

• JOIN: an index shared by two operands multiplies their matching elements.
• PROJECTION: an index missing from the output is summed away.

A := FWT_FromArray( { { 1, 2, 3 }, { 4, 5, 6 } } )
FWT_EinSum( "ij->i",     A )        // row sums      -> {6, 15}
FWT_EinSum( "ij->ji",    A )        // transpose
FWT_EinSum( "ij,jk->ik", A, B )     // matrix product (join j, project j)
FWT_EinSum( "i,j->ij",   v, w )     // outer product  (no projection)
FWT_EinSum( "i,i->",     v, v )     // dot product    (everything projected)

All of linear algebra is one primitive with different index patterns. When OpenBLAS is present these run on cblas_sgemm (see the reference for the 497 GFLOPS benchmark).

=== Lesson 2: einsum ===
row sums     'ij->i'    = {6.00, 15.00}
transpose    'ij->ji'   = {{1.00, 4.00}, {2.00, 5.00}, {3.00, 6.00}}
matmul       'ij,jk->ik' = {{14.00, 32.00}, {32.00, 77.00}}
dot 'i,i->' of {1,0,2} = 5.00

Lesson 3 — Your first program: equations

TTensorLogic holds named tensors and equations written as strings. This complete perceptron is ONE equation:

oTL := TTensorLogic():New()
oTL:Let( "W", { { 1, -1, 0 }, { 0, 1, -1 } } )
oTL:Let( "X", { 1, 2, 0.5 } )
oTL:Let( "B", { 2, -2 } )
oTL:Eq( "Y[i] = step( W[i,j] * X[j] + B[i] )" )
oTL:Run()
? oTL:GetArray( "Y" )        // {1, 0}

Read it as einsum: j is joined and projected (that is W·X), + B[i] adds a second term, step() is the nonlinearity. Available: step, sigmoid, relu, tanh, gelu, softmax, lnorm.

=== Lesson 3: first equation (a perceptron) ===
W.X + B = {1, -0.5} ; step -> Y = {1.00, 0.00}

Lesson 4 — Logical rules ARE equations

Store a relation as a boolean tensor (1 = fact holds). Then the Datalog rule Grandparent(x,z) <- Parent(x,y), Parent(y,z) is literally the same equation pattern: the shared variable y is the join, and step() turns "at least one path" into true:

oTL:Let( "Parent", { {0,1,0,0}, {0,0,1,0}, {0,0,0,1}, {0,0,0,0} } )   // 1->2->3->4
oTL:Eq( "Grandparent[x,z] = step( Parent[x,y] * Parent[y,z] )" )
oTL:Run()      // Grandparent: (1,3) and (2,4)

This is the paper's central claim, in working code: neural nets (lesson 3) and logic (lesson 4) run on the same engine.

=== Lesson 4: logic rules (Grandparent) ===
Grandparent = {{0,0,1,0}, {0,0,0,1}, {0,0,0,0}, {0,0,0,0}}

Lesson 5 — Recursion: forward chaining to a fixpoint

Run() re-evaluates all equations until nothing changes, so recursive rules just work. Transitive closure:

oTL:Let( "Ancestor", { {0,0,0,0}, {0,0,0,0}, {0,0,0,0}, {0,0,0,0} } )  // seed empty
oTL:Eq( "Ancestor[x,y] = step( Parent[x,y] + Ancestor[x,z] * Parent[z,y] )" )
oTL:Run( 20 )  // Ancestor of 1: {2,3,4}; of 2: {3,4}; of 3: {4}

=== Lesson 5: recursion to fixpoint (Ancestor) ===
Ancestor = {{0,1,1,1}, {0,0,1,1}, {0,0,0,1}, {0,0,0,0}}

Lesson 6 — Datalog over your DBF tables

An xBase table IS a relation. TTLDomain maps entity names to tensor indices, LetDbf() converts records, Query() decodes results back to names:

oDom := TTLDomain():New()
oDom:Harvest( "FAM", { "PARENT", "CHILD" } )       // register all names FIRST
oTL:LetDbf( "Parent", "FAM", { "PARENT", "CHILD" }, oDom )
oTL:Eq( "Grandparent[x,z] = step( Parent[x,y] * Parent[y,z] )" )
oTL:Run()
AEval( oTL:Query( "Grandparent", oDom ), ;
       {| a | QOut( a[1] + " -> " + a[2] ) } )     // abe -> bart, abe -> lisa

=== Lesson 6: Datalog over a DBF ===
grandparent: abe -> bart
grandparent: abe -> lisa

Tip: always Harvest() every relation before sizing tensors, so the domain is complete and dimensions match across rules.

Lesson 7 — Neural blocks: softmax and attention

A trailing . on a left-hand index marks the softmax axis. One attention head is six equations — compare with the hundreds of lines of a hand-written implementation:

oTL:Eq( "Q[p,k] = X[p,d] * WQ[d,k]" )
oTL:Eq( "K[p,k] = X[p,d] * WK[d,k]" )
oTL:Eq( "V[p,k] = X[p,d] * WV[d,k]" )
oTL:Eq( "S[p,q] = Q[p,k] * K[q,k] * 0.7071" )   // scaled scores (1/sqrt(dk))
oTL:Eq( "A[p,q.] = softmax( S[p,q] )" )          // rows sum to 1
oTL:Eq( "O[p,k] = A[p,q] * V[q,k]" )             // attended values

=== Lesson 7: attention in 6 equations ===
attention row 1 = {0.33, 0.33, 0.34}
row sum = 1.000000  (softmax => 1)

Lesson 8 — Autodiff: the gradient is another program

For Z = X1 * X2 * ... * Xn, the gradient w.r.t. any factor is the same einsum with that factor and the output swapped. The engine applies this rule mechanically to every equation, so no layer ever needs hand-written backprop:

oTL:Param( "W", { { 0.3, -0.2 }, { 0.1, 0.4 } } )   // Param = learnable
oTL:Eq( "Y[i] = sigmoid( W[i,j] * X[j] )" )
oTL:Forward()                                        // caches pre-activations
oTL:ZeroGrad()
oTL:Seed( "Y", FWT_Scale( FWT_Sub( oTL:Get("Y"), tTarget ), 2 ) )  // dL/dY
oTL:Backward()                                       // reverse sweep
? FWT_Get( oTL:GetGrad( "W" ), 1 )   // -0.19283922
// finite differences on the same weight: -0.19283742  — they match

=== Lesson 8: automatic differentiation ===
dL/dW[1,1] analytic = -0.19283922, finite differences = -0.19283742

Use Forward() (not Run()) when training: it caches the pre-activations that Backward() needs.

Lesson 9 — Training for real: Adam, clipping, batches

The professional loop adds an Adam optimizer (C kernel), global gradient clipping, mini-batch gradient accumulation and checkpointing. Learning the AND gate:

for e := 1 to 300
   oTL:Forward()
   oTL:ZeroGrad()
   oTL:Seed( "Y", FWT_Scale( FWT_Sub( oTL:Get("Y"), tT ), 2 ) )
   oTL:Backward()
   oTL:ClipGrads( 1.0 )
   oTL:StepAdam( 0.05 )
next        // Y -> {0.00, 0.11, 0.11, 0.88} for targets {0,0,0,1}

=== Lesson 9: Adam training (AND gate) ===
epoch  100  loss 0.179205
epoch  200  loss 0.073157
epoch  300  loss 0.039337
Y after training (targets 0,0,0,1) = {{0.00}, {0.11}, {0.11}, {0.88}}

For batches with TTransformerTL:

oT:oTL:ZeroGrad()
for i := 1 to nBatch
   oT:AccumStep( aWin[i], aTgt[i], 1 / nBatch )  // param grads accumulate
next
oT:oTL:ClipGrads( 1.0 )
oT:oTL:StepAdam( nLr )
oT:oTL:SaveParams( "model.chk", { "vocab" => aVocab } )

Lesson 10 — Transformer facade and beyond

You do not have to write every equation by hand. TTransformerTL builds a GPT-style stack for you:

oT := TTransformerTL():New( 2, 8, 2, nVocab )   // layers, d_model, heads, vocab
aProbs := oT:ForwardSeq( { 2, 3, 4, 5, 6 } )
nLoss  := oT:TrainStep( aIds, aTargets, 0.01 )
aGen   := oT:Generate( { 1 }, 8 )

Under the hood that is still a TTensorLogic program — inspect oT:oTL:aEqs or open samples/ai/tltransformer.prg.

• A real transformer: samples/ai/tltransformer.prg writes ~15 equations per layer and reproduces the classic Transformer class output closely with the same weights (test: tl_parity_forwardseq).
• A modern LLM: samples/ai/tlminillm.prg adds DeepSeek/MiMo techniques — MLA latent attention, RoPE (a signed-permutation join!), mixture-of-experts, multi-token prediction — all still just equations.
• Train on real data: samples/ai/tlfwcode.prg trains a char-level model on FWH source code with warmup+cosine LR, train/val split, best-val checkpoints and resume (set TLFW_RESUME=1). Tune with TLFW_STEPS, TLFW_BATCH, TLFW_LAYERS, TLFW_DMODEL, TLFW_HEADS, TLFW_WINDOW, TLFW_LR, …
• Synthetic code training: samples/ai/tlc.prg — generator + oracle, execution learning, grammar-constrained decoding (100% valid programs even untrained).
• Speed: samples/ai/tlbench.prg — with OpenBLAS the einsum reaches 497 GFLOPS on a 1024 matmul (PyTorch/MKL: 982 on the same machine).

The mental model to keep: everything — a rule, a layer, a gradient — is a set of tensor equations, and an equation is one einsum plus a nonlinearity.

Cheat sheet

See Also

• Tensor Logic (reference)
• Transformer Neural Network
• Build System