CaloryTalk benchmark
How accurately does the AI behind CaloryTalk estimate calories and macronutrients? Three runs of 50 queries, checked against the USDA FoodData Central SR Legacy database. Data and method are open.
At a glance. Energy: about 13–16 % weighted error, with a bias within ±4 %. Macronutrients are less accurate (roughly 15–37 % weighted error), and fat was the weakest nutrient in all three runs. This is an exploratory test under favourable conditions (exact grams, English); see limitations.
Methodology
- Reference: USDA FoodData Central, SR Legacy (April 2018), per-100 g values for energy, protein, carbohydrates and fat. Baby foods and foods with names longer than 70 characters are excluded (so several fit in one query).
- Queries: each query lists M randomly chosen USDA foods, each with a random amount from 30 to 300 g, e.g.
150 g of Cheese, cheddar. With several foods they are joined with;and “and”. Every run draws different foods. - Runs: N = 50 queries each, with M = 1, 3 and 5 foods per query, executed on 10 Nov 2026.
- AI: the same model (
@cf/google/gemma-4-26b-a4b-iton Cloudflare Workers AI), prompt and response validation that the CaloryTalk app uses. One call per query. - Comparison: the AI’s numbers are compared with USDA’s. With several foods per query, the sum of the AI’s estimates is compared with the sum of the USDA values, like a real meal.
Results
Values per nutrient over 50 queries. Errors are AI minus USDA, in kcal or grams. Field definitions are below.
1 food per query (50 queries)
| MAE | MAPE % | Weighted error % | Bias | Bias % | Pearson r | |
|---|---|---|---|---|---|---|
| Energy (kcal) | 63.35 | 20.4 | 16 | -15.39 | -3.9 | 0.963 |
| Protein (g) | 3.62 | 31.1 | 25.1 | 0.03 | 0.2 | 0.9 |
| Carbohydrates (g) | 5.83 | 25.7 | 16.2 | 0.77 | 2.1 | 0.97 |
| Fat (g) | 4.08 | 32 | 18.3 | -2.53 | -11.3 | 0.983 |
Cases used for MAPE (of 50): kcal 49, protein 37, carbs 37, fat 35.
3 foods per query (50 queries)
| MAE | MAPE % | Weighted error % | Bias | Bias % | Pearson r | |
|---|---|---|---|---|---|---|
| Energy (kcal) | 142.32 | 16.5 | 15.9 | -10.15 | -1.1 | 0.889 |
| Protein (g) | 10.22 | 31.4 | 19.4 | -3.21 | -6.1 | 0.884 |
| Carbohydrates (g) | 34.37 | 31.3 | 34.4 | 0.21 | 0.2 | 0.68 |
| Fat (g) | 12.02 | 123.8 | 36.5 | 5.11 | 15.5 | 0.808 |
Cases used for MAPE (of 50): kcal 50, protein 50, carbs 48, fat 48.
5 foods per query (50 queries)
| MAE | MAPE % | Weighted error % | Bias | Bias % | Pearson r | |
|---|---|---|---|---|---|---|
| Energy (kcal) | 232.38 | 13.3 | 13.4 | -26.66 | -1.5 | 0.866 |
| Protein (g) | 14.28 | 17.2 | 17.3 | -7.77 | -9.4 | 0.821 |
| Carbohydrates (g) | 27.06 | 16.6 | 14.5 | -10.02 | -5.4 | 0.935 |
| Fat (g) | 20.2 | 36.9 | 26.6 | 1.6 | 2.1 | 0.83 |
Cases used for MAPE (of 50): kcal 50, protein 50, carbs 50, fat 50.
Calculated fields
- MAE – mean absolute error
- Average of |AI − USDA|, in kcal or grams: how far off the AI is on average.
- MAPE % – mean absolute percentage error
- Average of |AI − USDA| / USDA × 100, computed per query and then averaged. References below 10 kcal or 1 g are left out, because a tiny error on a tiny amount is a huge percentage (the number of cases used is shown under each table). It can still jump on small foods.
- Weighted error %
- Sum of |AI − USDA| divided by the sum of USDA, × 100. Large foods weigh more and small ones do not inflate it, so it is the most stable summary of the error.
- Bias and bias %
- Average of (AI − USDA), keeping the sign, in kcal or grams, and the same as a percentage (sum of the errors / sum of USDA × 100). Positive means the AI tends to overestimate, negative to underestimate. Near zero means errors cancel out instead of pointing the same way.
- Pearson r
- Correlation between USDA and AI values, from −1 to 1. It shows whether the AI goes up and down with the real value, even if it is always too high or too low (that is what bias measures). With few or very different foods it is easy to get a high r, so it is only a sanity check.
Data
CSV files, comma-separated, decimal point, UTF-8. In a Spanish-locale Excel use Data → From Text/CSV.
- 1 food per query: summary · raw data
- 3 foods per query: summary · raw data
- 5 foods per query: summary · raw data
Raw data has one row per query: query, status (ok when the AI answered), foods_requested, foods_returned, text (what was sent), and for each nutrient (kcal, protein, carbs, fat) the columns _usda, _ai, _error (AI − USDA) and _error_pct.
Summary has one row per nutrient: queries, foods_per_query, nutrient, cases, mae, mape_pct, mape_cases, mape_excluded, weighted_error_pct, bias, bias_pct and pearson_r.
Limitations
- Exploratory test, not a formal validation: 50 queries per run, so figures can shift by a few points between runs.
- USDA values are generic American averages from 2018. Many descriptions are ambiguous (raw or cooked, which cut of meat…), so part of the error is not the AI’s.
- The sample favours foods with short names, and only English and exact grams were tested. Household portions (“a slice”, “a bowl”), other languages and composite dishes are not measured, and real-world error will be larger.
- Results apply to this model and this prompt; they must be repeated if either changes.
- CaloryTalk provides rough estimates, not medical or dietary advice.