Website

https://helmholtz-blablador.fz-juelich.de

Play around! 🐶

OUTLINE

Past
Present
Future

Blablador

/ˈblæblæˌdɔɹ/
Bla-bla-bla 🗣️ + Labrador 🐕‍🦺
A stage for deploying and testing large language models
Models change constantly (constantly improving rank, some good, some awful)
A mix of small, fast models and large, slower ones - changes constantly, keeps up with the state of the art
It is a web server, an api server, model runner, and training code.

Blablador

Past: Why?

AI is becoming basic infrastructure
Which historically is Open Source
We (as we in scientists) train a lot, deploy little: Here is your code/weights, tschüssi!
Little experience with dealing with LLMs
From the tools point of view, this is a FAST moving target 🎯💨
Acquire local experience in issues like
- data loading,
- quantization,
- distribution,
- fine-tune LLMs for specific tasks,
- inference speed,
- deployment

Why, part 2

Projects like OpenGPT-X, TrustLLM need a place to run
The usual: we want to be ready when the time comes
- The time is now!
TL;DR: BECAUSE WE CAN! 🚀

Privacy is our selling point

No data collection at all. I don’t keep ANY data whatsoever!
- You can use it AND keep your data private
- No records? Privacy (and GDPR is happy)

Deployment as a service

Scientists from FZJ can deploy their models on their own hardware and point to blablador
This solves a bunch of headaches for researchers:
- Authentication
- Web server
- Firewall
- Availability
- Etc
If you have a model and want to deploy it, contact me!

OpenAI-compatible API

Users import openai-python from OpenAI itself
All services which can use OpenAI’s API can use Blablador’s API (VSCode’s Continue.dev, LangChain, etc)
The API is not yet rate-limited or monitored (other than unique visits/day)

PRESENT

Usage

Web UI only
API usage wasn’t recorded until we moved to a new host, devs still migrating
Some healthy usage by tools e.g. B2DROP assistant: Around 400 requests/day (count as a single ip from B2DROP’s server)

The LLM ecosystem

For open models: If it isn’t on huggingface, it doesn’t exist
The “open” ecosystem is dominated by a few big players: Meta, Mistral.AI, Google
Meta has the best open frontier-level models; but no training data
Mistral is a close second - also no training data
Google is catching up FAST on closed models (and leaving open behind)
AllenAI is catching up and it’s open

The LLM ecosystem (cont)

Microsoft has tiny, bad ones (but I wouldn’t bet against them)
Apple is going their own way + using ChatGPT
Twitter/X has Grok-3 and lies about their benchmarks
Anthropic is receiving billions from Amazon but Claude is completely closed
Amazon released its “Nova” models recently. 100% closed, interesting tiering (similar to blablador)

The LLM ecosystem

Evaluation is HARD
- Benchmarks prove little
- Absolute majority of new models are fine-tuned versions of existing ones…
- On the benchmarks themselves!
  - Is this cheating?

Evaluating LLM models

LMSys’ arena: https://lmarena.ai
Doesn’t differentiate between open and closed
Open source gets buried under all private ones

Open Source?

Very few models are really open source
Most are “open source” in the sense that you can download the weights
Either the code isn’t available, or the data

Open Source

German academia: OpenGPT-X
- Trained in Jülich and Dresden
- For German businesses and academia
- Yet unclear if training data will be open
EU: TrustLLM
- Trained in Jülich
- “Trustworthy” LLM
- Says it will be fully open
Laion from FZJ (and others) is also open source
- Provides datasets, audio, image, video, text encoder models

Open source

Outside of Academia, there’s OLMo from AllenAI
- Has training code, weights and data, all open
Intellect-1 was trained collaboratively
- Up to 112 H100 GPUs simultaneously
- They claim overall compute utilization of 83% across continents and 96% when training only in the USA
- Fully open

Nous Research

Announced recently
Training live, on the interent
Previous models were Llama finetuned on synthetic data

DeepSeek

Deepseek V3 came out last year, competitive to the biggest models
Paper in may/24 described exactly what they were doing
“Discovered” reasoning by accident, released R1 in January
Western market freaked out for a month
Bubble is over
Released their former proprietary inference engine as open source

Non-transformer architectures

First one was Mamba in February
Jamba from A21 in March
Last I checked, LFM 40B MoE from Liquid/MIT was the best one (from 30.09)
Performs well on benchmarks
What about real examples?
Some mathematical discussions about it being turing-complete (probably not)
Other example is Hymba from NVIDIA (tiny model, 1.5B)

EU AI Act

“GPAI models present systemic risks when the cumulative amount of compute used for its training is greater than 10^25 floating point operations (FLOPs)”
Moore’s law has something to say about this
TrustLLM and OpenGPT-X will have to comply
- To be used commercially
Research-only models are exempt
Bureaucratic shot in the foot: the EU AI Act will make it harder for EU models to compete internationally

Juelich Supercomputing Centre

Humbler beginnings…

Small cluster inherited from other projects
Started small, with three nodes
Runs Slurm, has a parallel FS/Storage, uses EasyBuild

User demand is growing, we need more hardware

Currently around 300+ unique users/day on the website
API usage is higher, growing and heavier

Big models need big hardware

Llama 3.1 405b runs on WestAI nodes
- Launched with UNICORE
DeepSeek R1 takes 16 H100 Gpus

It’s being used in the wild!

https://indico.desy.de/event/38849/contributions/162118/

Controlling a radiotelescope with AI agents

It’s being used in the wild!

Someone reverse-engineered the API and created a python package

It’s being used in the wild!

https://git.geomar.de/everardo-gonzalez/blablador-python-bindings

It’s being used in the wild!

GEOMAR created a chatbot for their website
Scanned their material and created embeddings (RAG)
Calls Blablador’s API with embeddings and gets answers
Product of the NFDI hackathon DataXplorers // and beyond
It’s called TL;DR (Too Long; Didn’t Read) - Source code

It’s being used in the wild!

It’s being used in the wild!

EUDAT is a collection of data management services
Has an instance of NextCloud (File Share, Office, Calendar etc)
It’s integrating AI deeply into its services, backed by Blablador!

It’s being used in the wild!

Blassistant, a Blablador Nextcloud Assistant: https://github.com/EUDAT-B2DROP/assistant

It’s being used in the wild!

FZJ’s IEK7 (Stratosphere) is also using Blablador on their IEK7Cloud

It’s being used in the wild!

https://github.com/haesleinhuepf/bia-bob

FUTURE

Vision for the (near) future

Blablador, the brand, as an umbrella for ALL inference at JSC

JSC Inference umbrella

Blablador as LLM is the first step
Grow to include other types of models for Science and Industry

Use cases so far

LLMs for science (e.g. OpenGPT-X, TrustLLM, CosmoSage)
Prithvi-EO-2.0: Geospatial FM for EO (NASA, IBM, JSC). The 300M and 600M models will be released today
Terramind: Multi-Modal Geospatial FM for EO (ESA Phi-Lab, JSC, DLR, IBM, KP Labs). Model will be released in spring 2025
Helio NASA’s FM model for Heliophysics, for understating complex solar phenomena (first study design of the model is available)
JSC/ESA/DLR’s upcomping model FAST-EO
Health: Radiology with Aachen Uniklinik
JSC/Cern/ECMWF’s Atmorep
Open models:
- Pango weather
- Graphcast
With privacy!

Todo

Multi-modality: video, audio, text, images
Auto-RAG with privacy:
- Easy to do badly. Hard to do securely.
Everything from the previous slide

Potato

Take the slides with you

Questions?

No dogs have been harmed for this presentation

Extra slides

LLMOps resource

A No-BS Database of How Companies Actually Deploy LLMs in Production: 300+ Technical Case Studies, Including Self-Hosted LLMs in https://www.zenml.io/llmops-database

“I think the complexity of Python package management holds down AI application development more than is widely appreciated. AI faces multiple bottlenecks — we need more GPUs, better algorithms, cleaner data in large quantities. But when I look at the day-to-day work of application builders, there’s one additional bottleneck that I think is underappreciated: The time spent wrestling with version management is an inefficiency I hope we can reduce.”

Andrew Ng, 28.02.2024

“Building on top of open source can mean hours wrestling with package dependencies, or sometimes even juggling multiple virtual environments or using multiple versions of Python in one application. This is annoying but manageable for experienced developers, but creates a lot of friction for new AI developers entering our field without a background in computer science or software engineering.”

Andrew Ng, 28.02.2024

API Access

Create a token in the helmholtz codebase settings - (https://codebase.helmholtz.cloud) with the api scope
Copy the token
Go to https://api.helmholtz-blablador.fz-juelich.de/docs
Click on Authorize, and paste the token

Demo: API

Go to /v1/models
Click on Try it out
Click on Execute

Demo: API

Demo: cURL

curl --header "Authorization: Bearer MY_TOKEN_GOES_HERE"   https://api.helmholtz-blablador.fz-juelich.de/v1/models

Demo: VScode + Continue.dev

Yes. It DOES run with Emacs too. Ask your favorite Emacs expert.
Yes, vim too!
Add continue.dev extension to VSCode
On Continue, choose to add model, choose Other OpenAI-compatible API
Click in Open Config.json at the end

Demo: VScode + Continue.dev

Inside config.json, add at the "models" section:

 {
      "model": "AUTODETECT",
      "title": "Blablador",
      "apiKey": "glpat-YOURKEYHERE",
      "apiBase": "https://api.helmholtz-blablador.fz-juelich.de/v1",
      "provider": "openai"
    }

Try with the other models you got from the API!

Demo: VScode + Continue.dev

Select some code in a python file
Type Control-I (cmd-I on Mac) to edit the code, or Control-L to “talk” to blablador about this code
Ask Blablador to explain this code!
Can also fix, add tests, etc