Build Local AI Skills to Load Grocy From Photos

I don’t run one giant chatbot prompt for this. I run three narrow skills bolted onto one boring, predictable inventory client:

1. A grocery intake skill reads groceries, packaging, or a receipt and adds the purchases to Grocy.
2. A meal consumption skill translates “I made scrambled eggs” or a meal photo into conservative stock deductions.
3. An alcohol intake skill identifies bottles, estimates the remaining liquid, and records bar stock in milliliters.

The model does the fuzzy part - reading labels, understanding a meal, guessing a fill line, knowing when to ask. A plain deterministic script does the risky part - pulling live inventory, validating units, rejecting ambiguous matches, showing a dry run, and actually calling the Grocy API.

That’s why I keep the two apart. Let the model talk straight to your inventory API with no validation in between, and the first slick demo eventually turns into an impressively wrong stock count.

This is the setup I run at home, with the household-specific addresses, IDs, and credentials stripped out. It’s the skill layer - the part that turns photos and plain language into inventory operations. It doesn’t try to build a whole recipe engine or a Home Assistant integration from scratch; those are their own guides, for meals and for the bar.

This guide uses Codex agent skills, Grocy, and optional Home Assistant REST actions. The same architecture also works with another local inventory system if it has a usable API.

Food-safety note: An inventory best-before reminder is not a guarantee that food is safe. Prefer printed package dates and use authoritative storage guidance for fallback reminders. When food looks questionable or has been stored improperly, inventory software should not overrule common sense.

What a skill is, and what it isn’t

A Codex skill is a directory containing a required SKILL.md file and optional scripts, references, and assets. Codex initially sees the skill name and description. It loads the full instructions only when the request matches or the user explicitly invokes the skill.

So the description matters more than it looks. It’s the router that decides when the skill even fires.

A skill is also not the place to hide a 400-line API client, an API key, or a giant product catalog. Keep the workflow and decision rules in SKILL.md; put deterministic code in scripts/ and conditional detail in references/.

Use three skills instead of one because their triggers and risk profiles differ:

The local-first architecture

The complete flow should look like this:

Photo or written description
        ↓
Codex selects one focused skill
        ↓
Skill reads the live Grocy catalog
        ↓
Skill writes a temporary JSON manifest
        ↓
Deterministic client validates and prints a dry run
        ↓
The same unchanged manifest is committed
        ↓
Grocy is queried again to verify the result
        ↓
Optional Home Assistant refresh action

That temporary JSON manifest is doing real work. It lets you look at the model’s interpretation before it turns into an actual inventory change.

What you need

Before writing the skills, set up:

• A local Codex installation that can use skills and image inputs.
• A working Grocy instance reachable from the machine running Codex.
• Products, locations, and quantity units in Grocy.
• A Grocy API key stored outside the skill.
• Python 3 or another language for the deterministic client.
• Optional Home Assistant access if another integration caches Grocy data and needs an explicit refresh.

Grocy exposes an integrated Swagger interface at your instance’s /api route, and its web interface uses the same REST API. Authentication uses the GROCY-API-KEY header. Check the Swagger page on your installed version before hard-coding endpoints or request fields; that is more reliable than copying a random old example from a forum.

Keep Grocy local. If you need remote access, use a VPN or another authenticated private path. Do not expose an inventory API to the public internet just so a local skill can reach it.

Prepare Grocy before writing prompts

No prompt is going to save you from a badly set-up inventory. Decide how Grocy will count things first.

For ordinary groceries:

• Track eggs as individual pieces if you want “used three eggs” to work naturally.
• Track packaged products as packages when that is how you think about stock.
• Enable partial stock units for products such as milk, flour, and rice when you want fractional deductions.
• Set useful default locations such as Fridge, Freezer, Pantry, and Cupboard.
• Add purchase-to-stock conversions where the ratio never varies.

For bar stock:

• Create a Bar location.
• Create a milliliter quantity unit.
• Track spirits, liqueurs, syrups, and mixers in milliliters, not fractional bottles.
• Decide whether recipes distinguish only broad families such as bourbon, or specific products such as green-chartreuse and yellow-chartreuse.

Grocy supports separate purchase and stock units, product-specific conversions, and partial stock. Its own food tutorial gives the useful example of buying a six-pack of eggs while tracking six individual eggs in stock.

Create the three skill folders

For skills that belong to one repository, use .agents/skills in that repository. For personal skills that should be available in any project, use $HOME/.agents/skills.

.agents/skills/
├── add-groceries-from-photo/
│   ├── SKILL.md
│   ├── agents/
│   │   └── openai.yaml
│   ├── references/
│   │   ├── expiration-rules.md
│   │   └── manifest-schema.md
│   └── scripts/
│       └── grocy_inventory.py
├── consume-meal-from-grocy/
│   ├── SKILL.md
│   ├── references/
│   │   ├── inference-rules.md
│   │   └── manifest-schema.md
│   └── scripts/
│       └── grocy_inventory.py
└── add-bar-stock-from-photo/
    ├── SKILL.md
    ├── references/
    │   ├── bottle-classifications.json
    │   ├── ingredient-ids.json
    │   └── manifest-schema.md
    └── scripts/
        └── grocy_bar_inventory.py

You do not need every optional folder on day one. Start with SKILL.md. Add a script when the workflow needs deterministic validation or external tooling; add a reference when detailed rules would make the main file harder to scan.

Codex includes a Skill Creator. Invoke $skill-creator, describe the job and trigger phrases, and ask it to create an instruction-only skill or include scripts. You can also create the folders manually. Codex detects skill changes automatically; restart it if a new skill does not appear.

Build the shared inventory client first

The grocery and meal skills can share the same client. The alcohol skill can reuse its HTTP layer while applying different units and classification rules.

Give the client a small command-line contract:

grocy_inventory.py catalog
grocy_inventory.py add --input /tmp/grocery-add.json
grocy_inventory.py add --input /tmp/grocery-add.json --commit
grocy_inventory.py remove --input /tmp/meal-remove.json
grocy_inventory.py remove --input /tmp/meal-remove.json --commit

The default behavior must be a dry run. --commit should be the only path that changes stock.

The client should:

1. Retrieve the API key from an environment variable or operating-system secret store.
2. Read the live product catalog, stock, locations, and quantity units.
3. Load a non-empty JSON manifest.
4. Validate positive quantities and real ISO dates.
5. Prefer exact product IDs from the live catalog.
6. Reject ambiguous fuzzy matches instead of choosing the first plausible result.
7. Reject removals larger than current stock.
8. Print a structured plan containing valid, errors, and items.
9. Rebuild and revalidate the plan immediately before a commit.
10. Return a structured list of the records actually changed.

The core pattern is intentionally simple:

def run(manifest, commit=False):
    snapshot = fetch_live_grocy_catalog()
    plan = validate_and_resolve(manifest, snapshot)

    if not commit:
        return plan

    if not plan["valid"]:
        raise RuntimeError("Invalid plan; no changes made")

    changed = []
    for item in plan["items"]:
        changed.append(apply_one_grocy_change(item))

    return {"committed": True, "changed": changed}

Do not treat a multi-item loop as automatically transactional. A network failure can happen after item one succeeds and before item two runs. After any uncertain failure, fetch the live catalog again. Never blindly retry the entire manifest.

Keep credentials out of the skill

The script can accept configuration through two environment variables:

GROCY_BASE_URL=http://grocy.local/api
GROCY_API_KEY=your-secret

Those names are an interface, not a recommendation to save secrets in a repository, shell profile, manifest, or SKILL.md. Use macOS Keychain, a Linux secret service, a password manager CLI, or the secure environment facility used by your local runner. The script should retrieve the secret at runtime and never print it.

Skill 1: add groceries from a photo or receipt

The grocery skill needs to separate recognition from inventory policy. A receipt line such as HYV GRD A LG 12CT can become a readable product name, but tax, coupons, deposits, loyalty lines, and subtotals should not become inventory.

Use a manifest like this:

{
  "items": [
    {
      "name": "Grade A Large Eggs",
      "product_id": 5,
      "quantity": 12,
      "location": "Fridge",
      "expiration_date": "2026-09-28",
      "expiration_source": "visible",
      "price": 4.99
    }
  ]
}

Always record whether the date was visible or assumed. That prevents the final response from accidentally presenting a model estimate as printed package information.

Here is a compact SKILL.md starter:

---
name: add-groceries-from-photo
description: Analyze grocery, food-package, shopping-bag, refrigerator, and receipt photos and add identified food purchases to Grocy. Use when the user asks to add, stock, inventory, record, check in, or put away pictured groceries. Do not use for removing or consuming stock.
---

# Add Groceries From Photo

Add photographed food through `scripts/grocy_inventory.py`. Never place credentials in a manifest or response.

## Workflow

1. Inspect every attached image at high detail when labels, dates, quantities, or receipt text are small.
2. Run `scripts/grocy_inventory.py catalog` and use live product IDs, names, locations, stock, and default shelf lives.
3. Identify food products only. Ignore taxes, fees, deposits, coupons, subtotals, and non-food items unless requested.
4. Consolidate duplicate lines and translate register abbreviations into readable names.
5. Prefer a visible package date. Otherwise use the existing Grocy default, then a conservative category reminder from `references/expiration-rules.md`.
6. Prefer a confident existing product ID. Do not create a near-duplicate to avoid an ambiguous match.
7. Write one temporary JSON manifest using `references/manifest-schema.md`.
8. Dry-run the addition and resolve every error.
9. Commit the unchanged manifest only when the user has asked to add the items.
10. Report product, quantity, date, whether the date was visible or assumed, and every skipped line.

## Guardrails

- Treat separate packages as separate quantities unless a multipack count is printed.
- Do not interpret a price or weight as a unit count without supporting evidence.
- Do not decode an unlabeled lot number as an expiration date.
- Ask one concise question when an ambiguous product or date would materially change the result.
- Never commit an invalid dry run or blindly retry an uncertain commit.

Expiration rules that do not pretend to be certainty

Use this priority:

1. Printed EXP, Expires, or Use by date.
2. Printed Best by or Best before date.
3. Printed Sell by date if no better date is visible.
4. The existing Grocy product default.
5. A conservative category reminder.

Base assumed dates on the receipt purchase date when it is visible; otherwise use the current local date. Keep category defaults in references/expiration-rules.md so they can be reviewed separately from the core workflow.

For refrigerated meat, poultry, fish, eggs, deli products, and leftovers, base fallback reminders on the FoodSafety.gov cold storage chart, not a model’s memory. The FDA also notes that many package dates relate to quality rather than safety, so label these values as inventory reminders and preserve visible wording where useful.

Skill 2: consume a described or photographed meal

Meal consumption is harder because the input is incomplete by nature. A photo can show bread and turkey; it cannot reliably prove mayonnaise, cooking oil, or the exact cheese hidden inside.

The safe rule is: deduct evidence, not imagination.

Use a manifest like this:

{
  "items": [
    {
      "name": "Grade A Large Eggs",
      "product_id": 5,
      "quantity": 3,
      "spoiled": false
    },
    {
      "name": "Whole Milk 59 fl oz",
      "product_id": 7,
      "quantity": 0.0678,
      "spoiled": false
    }
  ]
}

In the second row, four fluid ounces of milk is 4 / 59 = 0.0678 of the stocked carton. Grocy deductions must use the product’s stock unit, not whatever unit appeared in the sentence.

Here is a SKILL.md starter:

---
name: consume-meal-from-grocy
description: Deduct food and ingredients from Grocy based on a written meal, recipe, snack, drink, or prepared-food photo. Use when the user says what they made, cooked, ate, drank, served, or packed and asks to consume, deduct, subtract, record, or remove it from inventory. Do not use for grocery intake.
---

# Consume Meal From Grocy

Translate a described or photographed meal into conservative stock-unit deductions. Treat every commit as destructive.

## Workflow

1. Determine whether the user describes ingredients used to prepare a batch, food eaten, or both.
2. For photos, use accompanying text as stronger evidence than visual inference.
3. Run `scripts/grocy_inventory.py catalog` and match only products currently in stock.
4. Prefer explicit ingredients and quantities, then only core ingredients necessarily implied by an unambiguous dish.
5. Convert physical quantities into each product's Grocy stock unit.
6. Consolidate duplicate ingredients and write a temporary manifest using `references/manifest-schema.md`.
7. Dry-run all proposed deductions.
8. Commit only when matches, servings, and unit conversions are high confidence.
9. Report deductions, assumptions, and intentionally omitted ingredients.

## Guardrails

- Deduct a prepared batch's ingredients once. Do not deduct them again when leftovers are eaten.
- Prefer a stocked finished product when the user ate packaged food or a previously recorded leftover.
- Never deduct both a finished product and its component ingredients.
- Do not infer hidden ingredients, oils, seasonings, sauces, or condiments from appearance.
- Do not guess between similar stocked variants.
- Never remove more than current stock, silently cap a quantity, or mark normal consumption as spoiled.
- Ask one concise question when a material ambiguity remains.
- Never commit an invalid dry run or blindly retry an uncertain commit.

Rules for preparation versus eating

Put the detailed inference rules in references/inference-rules.md. Useful examples include:

• “I made scrambled eggs with three eggs and half a cup of milk” means the ingredients used for that batch.
• “I ate a frozen burrito” means one matching stocked burrito, not reconstructed tortilla, beans, meat, and cheese.
• “I ate leftover chili” means a stocked finished chili item if one exists. Otherwise ask whether its ingredients were already recorded when it was cooked.
• “We made a pot of pasta” needs batch quantities or serving context before a destructive deduction.

When both the package size and consumed amount are known, calculate the exact fraction. When they are not known, ask before making a material deduction. A tiny disclosed estimate may be acceptable for a low-impact ingredient, but never invent a loaf’s slice count, a scoop size, or a package yield.

Skill 3: add photographed alcohol and bar stock

Bar inventory looks like grocery intake but needs a different unit model and a more careful taxonomy.

Use milliliters:

amount_ml = container_ml × remaining_fraction × bottle_count

A 750 mL bottle that looks 60% full contributes 450 mL. If you can’t see the liquid line, pick a documented household default (75% works) and label the quantity assumed.

Use a manifest like this:

{
  "items": [
    {
      "name": "Example Kentucky Bourbon",
      "ingredient": "bourbon",
      "container_ml": 750,
      "remaining_fraction": 0.6,
      "quantity_source": "visible",
      "location": "Bar"
    }
  ]
}

Here is a SKILL.md starter:

---
name: add-bar-stock-from-photo
description: Analyze photos of liquor, wine, liqueur, bitters, cocktail syrup, and mixer bottles and add estimated remaining liquid to Grocy in milliliters. Use when the user asks to add, stock, inventory, record, or check in pictured bar bottles. Do not use for ordinary groceries or prepared drinks.
---

# Add Bar Stock From Photo

Add photographed bottles through `scripts/grocy_bar_inventory.py`. Never place credentials in a manifest or response.

## Workflow

1. Inspect every image and identify each distinct bottle, printed capacity, visible fill level, and printed date.
2. Read the live catalog and prefer an existing tagged product when the label matches.
3. Choose the most specific stable ingredient ID from `references/ingredient-ids.json`.
4. Use printed capacity. Use 750 mL only for a visibly standard spirits or wine bottle with an unreadable capacity.
5. Use a visible remaining amount or fill fraction. If neither is visible, use the documented household default and mark it `assumed`.
6. Use printed dates when present. Treat shelf-stable spirits according to the household's non-expiring policy; give opened syrups and perishable mixers a reviewed shelf-life rule.
7. Write one consolidated manifest using `references/manifest-schema.md`.
8. Dry-run and resolve every error.
9. Commit the unchanged validated manifest only when the user asked to add the bottles.
10. Report product, ingredient class, capacity, fill estimate, total milliliters, date, and whether the quantity was visible or assumed.

## Guardrails

- Store bar inventory in milliliters, not fractional bottles.
- Count every separately visible bottle and consolidate identical bottles safely.
- Do not create a duplicate when a matching untagged product already exists.
- Do not classify cooking wine, pancake syrup, or unrelated grocery liquids as bar stock unless requested.
- Never substitute a nearby flavor or broad family when the distinction matters to recipes.
- Never commit an invalid dry run or blindly retry an uncertain commit.

Build a taxonomy before the bottle collection gets large

Ingredient IDs should be stable, lowercase, and machine-friendly:

[
  "bourbon",
  "london-dry-gin",
  "green-chartreuse",
  "yellow-chartreuse",
  "maraschino-liqueur",
  "orange-bitters",
  "ginger-syrup"
]

Do not classify by marketing color alone. Production style, base spirit, aging, origin, and recipe behavior matter more. If your cocktail planner treats two bottles differently, the inventory taxonomy must preserve that distinction.

For distinctive labels, add deterministic name rules in references/bottle-classifications.json and test them. This prevents a future model run from quietly changing a specialty bottle into a broad parent category.

[
  {
    "patterns": ["(?i)example\\s+green\\s+herbal\\s+liqueur"],
    "ingredient": "green-chartreuse"
  }
]

You don’t need every bottle tagged down to the brand. You just need to keep the distinctions that change how a recipe matches or substitutes.

Optional skill metadata

Add agents/openai.yaml when you want a friendly name and starter prompt in the UI:

interface:
  display_name: "Add Groceries from Photo"
  short_description: "Add photographed groceries to Grocy."
  default_prompt: "Analyze the attached groceries or receipt and add the recognized food to Grocy."

Implicit invocation is enabled by default. That is appropriate when the skill description is narrow and the user still must use an action phrase such as “add,” “record,” or “deduct.” If a skill should run only when explicitly selected, set policy.allow_implicit_invocation to false.

Add Home Assistant only where it helps

Grocy should remain the inventory source of truth. Home Assistant can display or use that data, but the skill should not maintain two competing stock ledgers.

If a Home Assistant integration caches Grocy or recipe-planner data, call its refresh action only after a successful Grocy commit. Home Assistant’s REST API uses bearer-token authentication and calls actions at:

POST /api/services/<domain>/<service>

Store the Home Assistant token with the same care as the Grocy key. If no refresh action exists, report that the dashboard may remain stale until its next scheduled scan.

Do not “refresh” a real integration by writing directly to /api/states/<entity_id>. Home Assistant’s documentation is explicit that this only changes the representation of an entity; it does not communicate with the underlying integration or device.

Test the trigger before testing the API

A technically correct skill that activates at the wrong time is still broken.

Test positive prompts:

• “Add everything in this grocery receipt to Grocy.”
• “I made three turkey sandwiches; deduct the ingredients.”
• “Inventory these bottles and estimate how much is left.”

Test near misses:

• “What groceries are in this photo?” should not mutate inventory without an add request.
• “What can I cook tonight?” should not invoke meal consumption.
• “Suggest a bourbon cocktail” should not invoke bar intake.
• “Remove this spoiled milk” belongs in a separate disposal or grocery-removal skill.

Then test behavior:

• A duplicate receipt line is consolidated.
• An ambiguous product match blocks the affected item.
• A visible date wins over a default.
• An assumed date is reported as assumed.
• A meal never removes more than current stock.
• A prepared batch is not deducted twice when leftovers are eaten.
• A bottle with an invisible fill line uses and discloses the fallback.
• A specialty bottle cannot be assigned an incompatible broad category.
• --commit cannot bypass a failed dry run.
• A simulated mid-commit failure is recovered by rereading live stock, not replaying the whole request.

For deterministic rules, write ordinary unit tests. Bottle-name classification, unit conversion, date parsing, fuzzy-match thresholds, stock-limit checks, and marker preservation are all good candidates. Avoid tests that merely compare the model’s prose.

Common mistakes

One giant kitchen skill

Broad descriptions activate unpredictably and mix add, remove, and planning permissions. Split skills by user intent and inventory action.

Credentials in SKILL.md

Skills are instructions, not secret stores. A manifest may be logged, inspected, or shared. Keep credentials out of both.

Fuzzy matching that is too eager

“Whole milk” and “2% milk” are similar strings but materially different products. Require a high score and a clear margin over the second-best candidate, or ask.

Units that describe packaging instead of stock

“One bottle” is useless if recipes consume 45 mL. “One carton” is hard to deduct when a recipe uses half a cup. Choose stock units around how the household consumes the product.

Treating model confidence as validation

A confident label guess can still map to a nonexistent product, an unsupported location, or insufficient stock. The live catalog and client validation remain authoritative.

Retrying after an uncertain failure

A partially completed request can double-add or double-remove stock. Refresh the catalog, compare intended and actual state, and continue only with the missing rows.

Hiding assumptions

An estimated fill line, inferred serving, or fallback date should be visible in the final report. Quiet assumptions make inventory drift impossible to debug.

The finished behavior

Once the three skills are installed, a normal interaction becomes pleasantly short:

User: Add these groceries to Grocy.
Assistant: Added 12 eggs, one carton of milk, and two yogurts. The egg and yogurt dates were visible; the milk date was estimated from its existing Grocy shelf-life setting. I skipped one unreadable receipt line.

User: I made scrambled eggs with three eggs and half a cup of milk. Deduct it.
Assistant: Deducted three eggs and 0.0678 of the 59 fl oz milk carton. I did not infer butter, oil, salt, or pepper.

User: Inventory these bottles.
Assistant: Added 450 mL of bourbon from a visibly 60%-full 750 mL bottle. I added 562.5 mL for the second bottle using the stated 75% fallback because its fill line was hidden.

The whole thing ends up as natural language at the front, conservative interpretation in the middle, and boring, reviewable API calls at the back.

Sources and further reading

• OpenAI: Build skills
• Grocy REST API and integrated Swagger UI
• Grocy food tutorial: products, units, purchases, and consumption
• Grocy OpenAPI specification
• Home Assistant REST API
• FoodSafety.gov cold food storage chart
• FDA: Are You Storing Food Safely?