Traceoid

Product

traceoid.ai ¹ is developing a programming language, traceoid, tailored towards machine learning and optimization problems in general. The stated goal of this language is to facilitate training of large-scale energy-based models.

What are energy-based models?

Energy-based models (EBMs) provide a superior approach to machine learning, an approach that learns an accurate representation of the training data. This is in stark contrast with autoregressive models which learn a representation distorted through the lens of averages. Essentially all mainsteam approaches incl. LLMs, diffusion models, etc fall under the umbrella of autoregressive model.

The difference stems from the different notions of probability each approach uses. While autoregressive models rely on standard, normalized probability, EBMs use unnormalized probability (in physics called the energy function).

Compared with autoregressive models, EBMs have the following advantages:

• Interpretability: EBMs learn an accurate, interpretable models of the world. This is in contrast with autoregressive models which are not interpretable because they reduce the their model of the world to averages hindering interpretability. For example, imagine three parking spaces arranged as follows: free, occupied, free (_🚗_). The average free parking space is the occupied one. How can this information be used to figure out where to park? How can we distinguish it from the scenario (🚗_🚗) which has the same average free parking space?

• Unsupervised learning: EBMs do not need labeled data to learn, they are an instance of unsupervised learning.

• Composability: Smaller EBMs can be combined to form larger models without a need for retraining.

• Universal architecture: There are no architectural differences between single energy-based models.

• Autonomy: Current models are ill-suited for autonomous operation over longer periods of time. Autoregressive models capture a distorted model of the world and this distortion only grows over time to the point that eventually, there is no resemblance between the model and the true state of the world.

• No hallucinations: EBMs are much less prone to halluciations compared with autoregressive models. Averages also cause hallucinations. Closely related to lack of interpretability. Averages compress compress with too little of a cramped space, it is no surprise that some information. (??)

• Noise resistant: EBMs are resistant to noisy inputs. Autoregressive models are prone to Since there are averages, noisy data cannot will not skew these averages.

• Biologically plausible: Karl Friston, a preeminent neurobiologist and one most cited scientists, argues that operations of the brain should be interpreted through the lens of energy functions.

EBMs are the favorite approach of Yann LeCun. In addition, Geoff Hinton and John Hopfield received the Nobel prize in physics for their work on EBMs.

• each of these would be a huge step forward

Unfortunately, energy-based models cannot be trained using standard approaches as they do not scale.

Insight

Our product, the traceoid programming language, revolves around a novel algorithm which faciliates, among others, training of large-scale energy-based models.

This core algorithm is a subsumes and generalizes automatic differentiation, the current method for model training.

Mainstream adoption of automatic differentiation in the mid-2010s was one of the fuses of the ongoing machine learning revolution. Despite this, automatic differentiation itself has changed very little.

AGI is not achievable with automatic differentiation and a fundamental change is in order. Our core algorithm is .

This algorithm is a vast generalization of automatic .

Applications of this algorithm

If we success, ..

Current wisdon argues that agi will be achieved by scaling current approaches.

Why a new language?

Modern machine learning frameworks are compilers for Python, the world's most OK langugage. Python, in it's natural state, does not lends itself to numerical computation, and in fact, a staggering amount of man hours went into making...

traceoid relies on language constructs that would be difficult to showhorn to existing

In addition to the advantages inherent in EBMs, Traceoid provides the following advantages:

• Inference speedup: During training, our algorithm extracts more structure from the data. This additional structure translates into a significantly faster inference.

• Controlled autoscaling: Traceoid programs are structured in a manner where the traceoid runtime automatically partitions program operations across threads (both CPU and GPU), as well as across a computing cluster. No more Kubernetes headaches. No more handwritten CUDA kernels. All this without sacrificing control.

• Minimal programs: traceoid redefines the boundary between the programmer and the program, requiring the programmer to provide only a minimial specification while retaining full control over the program if needed. We argue that AI coding assistants exist due to shortcomings of our languages, and

Get in touch

• X: @adamnemecek1
• Discord
• email: adam at company-name dot ai

Investments

We invite potential investors to reach out over email. Even though we are currently not raising funding, we will be at some point in the near future and it is much better to start the conversation sooner rather than later.

Join us

While we are currently not actively hiring, there is always space for the right candidate. We hire from a wide pool

• You will be a joining a mostly PhD (and )
• Full remote.