Skip to content

Local out of tree usermods fail to find dependencies #5802

Description

@lost-hope

Issue description

Problem

An out-of-tree usermod wired in via a plain local path, e.g.

custom_usermods =
  ${env:esp32dev.custom_usermods}
  symlink://../my-usermod

fails to compile with:

fatal error: wled.h: No such file or directory

even though the symlink resolves correctly and the usermod's own sources are
found and compiled (they just don't see WLED's own include path).

This only happens for local symlink:///file:// entries. The same line
written as github.com/user/my-usermod or https://github.com/user/repo.git
works fine, and the explicit named form

custom_usermods =
  my-usermod = symlink://../my-usermod

is also a working workaround.

Root cause

pio-scripts/load_usermods.py decides whether a resolved library dependency
is a WLED module (and therefore gets WLED's include path added to it) in
is_wled_module():

def is_wled_module(dep: LibBuilderBase) -> bool:
  return (
    usermod_dir in Path(dep.src_dir).parents
    or str(dep.name).startswith("wled-")
    or dep.name in _custom_usermod_names
  )

For an out-of-tree usermod that is neither inside usermods/ nor named
wled-*, this relies entirely on dep.name being present in
_custom_usermod_names, which is populated ahead of time by
_predict_dep_name() while parsing the raw custom_usermods option.

_predict_dep_name() only extracts a name from the URL path when
parsed.netloc is github.com, gitlab.com or bitbucket.com:

if _URL_SCHEME_RE.match(entry):
  parsed = urlparse(entry)
  if parsed.netloc in ('github.com', 'gitlab.com', 'bitbucket.com'):
    parts = [p for p in parsed.path.split('/') if p]
    if len(parts) >= 2:
      name = parts[1]
    else:
      name = Path(parsed.path.rstrip('/')).name.strip()
    if name.endswith('.git'):
      name = name[:-4]
    return name or None
# falls through to SSH / '@' / plain "owner/Name" checks, none of which
# match a "symlink://..." string, so the function returns None

For symlink://../my-usermod (or any other non-git-host URL, including
symlink:///absolute/path and file://...), parsed.netloc is not one of
the three known hosts, so the if body - and therefore the return - is
never reached. The function falls through the remaining SSH/@/plain-name
checks (none of which match a URL string) and returns None at the very
end. A WARNING: Cannot determine library name for custom_usermods entry
is printed, but the build continues without the name being registered - and
because the module was silently not recognised as a WLED module, WLED's
include path is silently missing from that compilation unit, which only
surfaces later as a wled.h: No such file or directory in each of that
usermod's source files.

Worth noting: urlparse() puts the first path segment after // into
.netloc, not .path, whenever there is one (e.g.
urlparse("symlink://../foo")netloc='..', path='/foo'); only a URL
with an empty authority section (symlink:///abs/path, three slashes)
gets the whole thing in .path. Any fix needs to account for both forms.

Possible Fix

Give the else branch (any URL whose host isn't a known git forge -
i.e. every local symlink:///file:// path, plus self-hosted git servers)
the same "last path segment is the library name" fallback the git-host
branch already uses, instead of falling through to None. netloc and
path are stitched back together first so both symlink://../foo (netloc
non-empty) and symlink:///abs/path/foo (netloc empty) resolve to the same
foo:

  # URL scheme: extract name from path
  if _URL_SCHEME_RE.match(entry):
    parsed = urlparse(entry)
    if parsed.netloc in ('github.com', 'gitlab.com', 'bitbucket.com'):
      parts = [p for p in parsed.path.split('/') if p]
      name = parts[1] if len(parts) >= 2 else Path(parsed.path.rstrip('/')).name.strip()
    else:
      # Local paths (symlink://, file://) and any other host: fall back to
      # the last path segment, same as PlatformIO does for a local/symlinked
      # library without a manifest override. urlparse puts the first path
      # component into .netloc when the URL's authority section isn't
      # empty (e.g. "symlink://../foo" -> netloc="..", path="/foo"), so
      # stitch them back together before taking the last segment - this
      # also covers absolute paths ("symlink:///abs/path/foo") where
      # netloc is already empty.
      full_path = parsed.netloc + parsed.path if parsed.netloc not in ('', 'localhost') else parsed.path
      name = Path(full_path.rstrip('/')).name.strip()
    if name.endswith('.git'):
      name = name[:-4]
    return name or None

Disclaimer: Issue found and proposed fix made by AI.

Metadata

Metadata

Assignees

Labels

AIPartly generated by an AI. Make sure that the contributor fully understands the code!bug

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions