mirror of
https://github.com/brunopostle/ifcurl.git
synced 2026-07-12 10:18:14 +00:00
git.py: LRU disk cache eviction; service.py: configurable in-memory limits
Disk cache (bare repo clones under ~/.cache/ifcurl/): - IFCURL_CACHE_MAX_GB env var sets the max total size; when exceeded after a new clone the oldest-accessed repos are removed until under the limit - remote_url stored per clone entry so ifcurl cache list can show the URL - _touch_cache() updates mtime on every repo open for LRU tracking In-memory caches (service.py): - _T2_MAX and _T3_MAX now read from IFCURL_T2_MAX / IFCURL_T3_MAX env vars (defaults: 8 IFC blobs, 64 selector results) — configurable without rebuild New CLI: ifcurl cache list | prune [--max-gb N] | clear Tests: 6 new unit tests for _dir_size, _touch_cache, _repo_cache_entries, _evict_if_needed. 118 → 124 passing. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
5d5b8c6516
commit
335349db1a
5 changed files with 222 additions and 3 deletions
|
|
@ -22,7 +22,7 @@
|
|||
{"id":"ifcurl-ach","title":"Deploy Phase 3b footer template to Forgejo","description":"The footer.tmpl file that injects a 'View in 3D' button on .ifc file pages has been written but not yet deployed. Deploy it to /etc/forgejo/templates/custom/footer.tmpl and verify the button appears when viewing an .ifc file on localhost:3000.","status":"closed","priority":2,"issue_type":"task","owner":"bruno@postle.net","created_at":"2026-04-23T05:47:53Z","created_by":"Bruno Postle","updated_at":"2026-04-23T06:14:35Z","closed_at":"2026-04-23T06:14:35Z","close_reason":"Split into atomic tasks: ifcurl-ohq, ifcurl-49o","dependencies":[{"issue_id":"ifcurl-ach","depends_on_id":"ifcurl-ahd","type":"blocks","created_at":"2026-04-23T07:02:40Z","created_by":"Bruno Postle","metadata":"{}"}],"dependency_count":1,"dependent_count":1,"comment_count":0}
|
||||
{"id":"ifcurl-69l","title":"License headers for Forgejo integration files","description":"The Go patch files (ifc_url.go, ifc_url_test.go) already carry MIT headers matching Forgejo's per-file convention. The JS/HTML assets have no license header: viewer.html, viewer-url.js, footer.tmpl. Review: (1) confirm MIT is correct for files intended for Forgejo contribution (Forgejo project is GPL-3.0 but individual contributed files use MIT); (2) check compatibility with ThatOpen components licence (viewer.html loads @thatopen/components from CDN); (3) add SPDX licence headers to the three unlicensed asset files. The Python/CLI core remains LGPL-3.0-or-later as it targets IfcOpenShell contribution.","status":"closed","priority":3,"issue_type":"task","owner":"bruno@postle.net","created_at":"2026-04-24T07:09:46Z","created_by":"Bruno Postle","updated_at":"2026-04-24T07:29:22Z","closed_at":"2026-04-24T07:29:22Z","close_reason":"Added MIT SPDX headers to viewer.html, viewer-url.js, footer.tmpl — ThatOpen components also MIT-licensed so no compatibility issue","dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||
{"id":"ifcurl-8s8","title":"viewer.html: interactive tool to set clipping planes","description":"Currently clip= planes can only be set by hand-editing the URL. Add an interactive clipping tool to the viewer: a UI control (e.g. toolbar button) that lets the user drag/place a clipping plane against the loaded model. The plane position and normal should be written back into the ifc:// URL as clip= parameters so the view is shareable. Consider using ThatOpen's clipper component (OBC.Clipper) if available, or a Three.js-based approach.","status":"closed","priority":3,"issue_type":"feature","assignee":"Bruno Postle","owner":"bruno@postle.net","created_at":"2026-04-24T05:49:22Z","created_by":"Bruno Postle","updated_at":"2026-04-24T06:23:14Z","started_at":"2026-04-24T06:11:49Z","closed_at":"2026-04-24T06:23:14Z","close_reason":"setupClipper() uses OBC.Clipper: seeds URL planes as draggable SimplePlanes, toolbar '✂ clip' toggle for double-click placement, '⊗ clips' clear button. syncCameraUrl extended to serialise current planes back to URL.","dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||
{"id":"ifcurl-07a","title":"Cache size control for bare repo clones","description":"git.py clones remote repos as bare repos under ~/.cache/ifcurl/\u003chash\u003e/repo.git with no size limit or eviction policy. Large repos accumulate indefinitely. Add a configurable max cache size (e.g. via env var IFCURL_CACHE_MAX_GB or a config file) with an LRU eviction policy based on last-access time of each repo directory. Should also expose a CLI command (ifcurl cache --list, --prune, --clear) to inspect and manage the cache manually.","status":"open","priority":3,"issue_type":"feature","owner":"bruno@postle.net","created_at":"2026-04-24T05:45:59Z","created_by":"Bruno Postle","updated_at":"2026-04-24T05:45:59Z","dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||
{"id":"ifcurl-07a","title":"Cache size control for bare repo clones","description":"git.py clones remote repos as bare repos under ~/.cache/ifcurl/\u003chash\u003e/repo.git with no size limit or eviction policy. Large repos accumulate indefinitely. Add a configurable max cache size (e.g. via env var IFCURL_CACHE_MAX_GB or a config file) with an LRU eviction policy based on last-access time of each repo directory. Should also expose a CLI command (ifcurl cache --list, --prune, --clear) to inspect and manage the cache manually.","status":"closed","priority":3,"issue_type":"feature","assignee":"Bruno Postle","owner":"bruno@postle.net","created_at":"2026-04-24T05:45:59Z","created_by":"Bruno Postle","updated_at":"2026-04-24T08:06:23Z","started_at":"2026-04-24T07:48:28Z","closed_at":"2026-04-24T08:06:23Z","close_reason":"IFCURL_CACHE_MAX_GB env var for LRU disk eviction; IFCURL_T2_MAX/T3_MAX for in-memory limits; ifcurl cache list/prune/clear CLI; remote_url stored per clone","dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||
{"id":"ifcurl-ts5","title":"Assess Go test strategy for Forgejo plugin patch","description":"The Forgejo integration is a low-maintenance source patch (footer.tmpl, viewer.html, /viewer and /proxy HTTP handlers). Evaluate whether adding Go tests is feasible without coupling to Forgejo's full test suite: can the handler logic be extracted into a small testable package, or do tests have to live inside the Forgejo tree? If the cost of keeping tests green across Forgejo upstream upgrades is too high, document the decision and close. Outcome: either a small test file or a documented rationale for no Go tests.","status":"closed","priority":3,"issue_type":"task","assignee":"Bruno Postle","owner":"bruno@postle.net","created_at":"2026-04-24T05:45:57Z","created_by":"Bruno Postle","updated_at":"2026-04-24T06:51:21Z","started_at":"2026-04-24T06:36:35Z","closed_at":"2026-04-24T06:51:21Z","close_reason":"Go tests are feasible and low-maintenance. 7 tests in ifc_url_test.go live in the same package (markdown_test), use goldmark.New() directly (no Forgejo test infrastructure needed beyond TestMain which is already there). Only deps: setting.IfcURL.PreviewServiceURL (ours), goldmark (stable), testify. Test file copied to forgejo/modules/markup/markdown/ in ifcurl repo as patch artifact.","dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||
{"id":"ifcurl-hvw","title":"Add JavaScript tests for viewer.html URL logic","description":"viewer.html contains pure-JS functions (parseIfcUrl, buildIfcUrl, toRawUrl, syncCameraUrl) with no test coverage. Add a lightweight JS test harness (e.g. a standalone test.html or Node-runnable test.js using the native test runner) that unit-tests these functions: round-trip URL parse/build, toRawUrl for each host type (GitHub, GitLab, Forgejo), and camera parameter serialisation. No browser automation needed — pure function tests.","status":"closed","priority":3,"issue_type":"task","assignee":"Bruno Postle","owner":"bruno@postle.net","created_at":"2026-04-24T05:45:56Z","created_by":"Bruno Postle","updated_at":"2026-04-24T06:30:47Z","started_at":"2026-04-24T06:26:01Z","closed_at":"2026-04-24T06:30:47Z","close_reason":"32 Node tests in tests/test_viewer_url.mjs covering parseIfcUrl, buildIfcUrl, toRawUrl. Pure functions extracted to viewer-url.js; viewer.html imports from it. Also caught + not being decoded in URL building.","dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||
{"id":"ifcurl-b9e","title":"Audit and extend Python test coverage","description":"103 tests exist across test_url.py, test_git.py, test_auth.py, test_render.py, and test_service.py. Audit what is missing: check branch/edge coverage in git.py (offline fallback, mutable ref fetch, auth injection), service.py (tier 2 cache eviction, mutable ref bypass), and the CLI entry points in __main__.py. Add tests for any significant gaps found.","status":"closed","priority":3,"issue_type":"task","assignee":"Bruno Postle","owner":"bruno@postle.net","created_at":"2026-04-24T05:45:40Z","created_by":"Bruno Postle","updated_at":"2026-04-24T07:37:10Z","started_at":"2026-04-24T07:33:43Z","closed_at":"2026-04-24T07:37:10Z","close_reason":"Added 15 tests: SSRF protection (8), GET /preview endpoint (3), tier-2 LRU eviction (3), heads/\u003cbranch\u003e ref form (1)","dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||
|
|
|
|||
|
|
@ -115,6 +115,34 @@ def main() -> None:
|
|||
),
|
||||
)
|
||||
|
||||
cache_parser = subparsers.add_parser(
|
||||
"cache",
|
||||
help="Inspect and manage the local git repository cache",
|
||||
description=(
|
||||
"Manage bare git repositories cached under the OS cache directory\n"
|
||||
"(~/.cache/ifcurl/ on Linux). Use 'list' to see what is cached,\n"
|
||||
"'prune' to enforce a size limit, or 'clear' to remove everything."
|
||||
),
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
)
|
||||
cache_sub = cache_parser.add_subparsers(dest="cache_cmd")
|
||||
|
||||
cache_sub.add_parser("list", help="Show cached repos with size and last-access time")
|
||||
|
||||
prune_parser = cache_sub.add_parser(
|
||||
"prune",
|
||||
help="Remove oldest repos until cache is under the size limit",
|
||||
)
|
||||
prune_parser.add_argument(
|
||||
"--max-gb", type=float, default=None, metavar="GB",
|
||||
help=(
|
||||
"Maximum total cache size in GB. Defaults to IFCURL_CACHE_MAX_GB "
|
||||
"environment variable; required if that is not set."
|
||||
),
|
||||
)
|
||||
|
||||
cache_sub.add_parser("clear", help="Remove all cached repos")
|
||||
|
||||
# Print help when called with no arguments
|
||||
if len(sys.argv) == 1:
|
||||
parser.print_help()
|
||||
|
|
@ -126,6 +154,8 @@ def main() -> None:
|
|||
_cmd_render(args)
|
||||
elif args.command == "serve":
|
||||
_cmd_serve(args)
|
||||
elif args.command == "cache":
|
||||
_cmd_cache(args)
|
||||
|
||||
|
||||
def _cmd_render(args: argparse.Namespace) -> None:
|
||||
|
|
@ -203,5 +233,57 @@ def _cmd_serve(args: argparse.Namespace) -> None:
|
|||
uvicorn.run(app, host=args.host, port=args.port)
|
||||
|
||||
|
||||
def _cmd_cache(args: argparse.Namespace) -> None:
|
||||
import datetime
|
||||
import shutil as _shutil
|
||||
|
||||
from ifcurl.git import _evict_if_needed, _get_max_cache_bytes, _repo_cache_entries
|
||||
|
||||
if args.cache_cmd is None or args.cache_cmd == "list":
|
||||
entries = _repo_cache_entries()
|
||||
if not entries:
|
||||
print("No cached repositories.")
|
||||
return
|
||||
total = 0
|
||||
for atime, size, cache_dir, url in entries:
|
||||
dt = datetime.datetime.fromtimestamp(atime).strftime("%Y-%m-%d %H:%M")
|
||||
mb = size / (1024 ** 2)
|
||||
total += size
|
||||
print(f" {url:<55} {mb:>7.1f} MB {dt}")
|
||||
total_mb = total / (1024 ** 2)
|
||||
max_bytes = _get_max_cache_bytes()
|
||||
limit_str = f" limit: {max_bytes / (1024**3):.1f} GB" if max_bytes else ""
|
||||
print(f"Total: {total_mb:.1f} MB ({len(entries)} repos){limit_str}")
|
||||
|
||||
elif args.cache_cmd == "prune":
|
||||
max_gb = args.max_gb
|
||||
if max_gb is None:
|
||||
if os.environ.get("IFCURL_CACHE_MAX_GB"):
|
||||
max_gb = float(os.environ["IFCURL_CACHE_MAX_GB"])
|
||||
else:
|
||||
print("Error: --max-gb is required when IFCURL_CACHE_MAX_GB is not set", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
os.environ["IFCURL_CACHE_MAX_GB"] = str(max_gb)
|
||||
before = sum(s for _, s, _, _ in _repo_cache_entries())
|
||||
_evict_if_needed()
|
||||
after = sum(s for _, s, _, _ in _repo_cache_entries())
|
||||
freed = (before - after) / (1024 ** 2)
|
||||
remaining = after / (1024 ** 2)
|
||||
print(f"Freed {freed:.1f} MB — {remaining:.1f} MB remaining.")
|
||||
|
||||
elif args.cache_cmd == "clear":
|
||||
entries = _repo_cache_entries()
|
||||
if not entries:
|
||||
print("Nothing to clear.")
|
||||
return
|
||||
total = sum(s for _, s, _, _ in entries)
|
||||
for _, _, cache_dir, _ in entries:
|
||||
try:
|
||||
_shutil.rmtree(str(cache_dir))
|
||||
except OSError as exc:
|
||||
print(f"Warning: could not remove {cache_dir}: {exc}", file=sys.stderr)
|
||||
print(f"Cleared {total / (1024**2):.1f} MB ({len(entries)} repos).")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
|
|||
|
|
@ -36,6 +36,7 @@ import hashlib
|
|||
|
||||
# allows git import even if git executable isn't found during import
|
||||
import os
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
from platformdirs import user_cache_dir
|
||||
|
|
@ -99,6 +100,72 @@ def _get_repo(ifc_url: IfcUrl, token: str | None = None) -> git.Repo:
|
|||
return _open_remote(ifc_url.git_remote_url(), ifc_url.is_mutable_ref(), token=token)
|
||||
|
||||
|
||||
def _touch_cache(cache_dir: Path) -> None:
|
||||
"""Update mtime of *cache_dir* to now for LRU access tracking."""
|
||||
try:
|
||||
os.utime(str(cache_dir), None)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def _dir_size(path: Path) -> int:
|
||||
"""Return total byte size of all files under *path*."""
|
||||
return sum(f.stat().st_size for f in path.rglob("*") if f.is_file())
|
||||
|
||||
|
||||
def _get_max_cache_bytes() -> int | None:
|
||||
"""Return configured max disk cache size from IFCURL_CACHE_MAX_GB, or None."""
|
||||
val = os.environ.get("IFCURL_CACHE_MAX_GB")
|
||||
if val is None:
|
||||
return None
|
||||
try:
|
||||
return int(float(val) * 1024 ** 3)
|
||||
except (ValueError, OverflowError):
|
||||
return None
|
||||
|
||||
|
||||
def _repo_cache_entries() -> list[tuple[float, int, Path, str]]:
|
||||
"""Return all cached repo dirs as ``(last_access, size_bytes, path, remote_url)``.
|
||||
|
||||
Sorted oldest-first by last access. Dirs without a ``repo.git``
|
||||
subdirectory are ignored.
|
||||
"""
|
||||
base = Path(user_cache_dir("ifcurl"))
|
||||
if not base.exists():
|
||||
return []
|
||||
entries = []
|
||||
for entry in base.iterdir():
|
||||
if not entry.is_dir() or not (entry / "repo.git").exists():
|
||||
continue
|
||||
try:
|
||||
atime = entry.stat().st_mtime
|
||||
size = _dir_size(entry)
|
||||
url_file = entry / "remote_url"
|
||||
url = url_file.read_text().strip() if url_file.exists() else "?"
|
||||
entries.append((atime, size, entry, url))
|
||||
except OSError:
|
||||
pass
|
||||
entries.sort(key=lambda e: e[0])
|
||||
return entries
|
||||
|
||||
|
||||
def _evict_if_needed() -> None:
|
||||
"""Remove oldest cached repos until total disk usage is under the configured limit."""
|
||||
max_bytes = _get_max_cache_bytes()
|
||||
if max_bytes is None:
|
||||
return
|
||||
entries = _repo_cache_entries()
|
||||
total = sum(s for _, s, _, _ in entries)
|
||||
for _, size, cache_dir, _ in entries:
|
||||
if total <= max_bytes:
|
||||
break
|
||||
try:
|
||||
shutil.rmtree(str(cache_dir))
|
||||
total -= size
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def _open_local(repo_path: str) -> git.Repo:
|
||||
"""Open a local git repository."""
|
||||
try:
|
||||
|
|
@ -176,6 +243,8 @@ def _open_remote(remote_url: str, is_mutable: bool, token: str | None = None) ->
|
|||
|
||||
if not git_dir.exists():
|
||||
repo = _clone_bare(auth, git_dir, remote_url)
|
||||
(cache_dir / "remote_url").write_text(remote_url)
|
||||
_evict_if_needed()
|
||||
else:
|
||||
repo = git.Repo(str(git_dir))
|
||||
if is_mutable:
|
||||
|
|
@ -198,6 +267,7 @@ def _open_remote(remote_url: str, is_mutable: bool, token: str | None = None) ->
|
|||
except git.exc.GitCommandError:
|
||||
pass # offline — use cached data
|
||||
|
||||
_touch_cache(cache_dir)
|
||||
return repo
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -95,7 +95,7 @@ def _is_private_ip(host: str) -> bool:
|
|||
# Tier 2: (commit_hexsha, path) → IFC bytes
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_T2_MAX = 8
|
||||
_T2_MAX: int = int(os.environ.get("IFCURL_T2_MAX", "8"))
|
||||
_t2_cache: OrderedDict[tuple[str, str], bytes] = OrderedDict()
|
||||
_t2_lock = threading.Lock()
|
||||
|
||||
|
|
@ -122,7 +122,7 @@ def _t2_put(hexsha: str, path: str, data: bytes) -> None:
|
|||
# Tier 3: (commit_hexsha, path, selector) → frozenset[GlobalId]
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_T3_MAX = 64
|
||||
_T3_MAX: int = int(os.environ.get("IFCURL_T3_MAX", "64"))
|
||||
_t3_cache: OrderedDict[tuple[str, str, str], frozenset[str]] = OrderedDict()
|
||||
_t3_lock = threading.Lock()
|
||||
|
||||
|
|
|
|||
|
|
@ -97,3 +97,70 @@ class TestInvalidRepo:
|
|||
url = IfcUrl.parse(f"ifc://{tmp_path}@HEAD?path=model.ifc")
|
||||
with pytest.raises(ValueError):
|
||||
fetch_ifc(url)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Cache management helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
from ifcurl.git import _dir_size, _evict_if_needed, _repo_cache_entries, _touch_cache
|
||||
|
||||
|
||||
def _make_fake_repo_cache(base: "Path", url: str, size_bytes: int = 1024) -> "Path":
|
||||
"""Create a fake cached-repo directory structure under *base*."""
|
||||
import hashlib
|
||||
key = hashlib.sha256(url.encode()).hexdigest()[:24]
|
||||
entry = base / key
|
||||
git_dir = entry / "repo.git"
|
||||
git_dir.mkdir(parents=True)
|
||||
(git_dir / "dummy").write_bytes(b"\x00" * size_bytes)
|
||||
(entry / "remote_url").write_text(url)
|
||||
return entry
|
||||
|
||||
|
||||
class TestCacheHelpers:
|
||||
def test_dir_size_counts_nested_files(self, tmp_path):
|
||||
(tmp_path / "a").write_bytes(b"x" * 100)
|
||||
(tmp_path / "sub").mkdir()
|
||||
(tmp_path / "sub" / "b").write_bytes(b"y" * 200)
|
||||
assert _dir_size(tmp_path) == 300
|
||||
|
||||
def test_touch_cache_updates_mtime(self, tmp_path):
|
||||
import time
|
||||
old_mtime = tmp_path.stat().st_mtime
|
||||
time.sleep(0.05)
|
||||
_touch_cache(tmp_path)
|
||||
assert tmp_path.stat().st_mtime > old_mtime
|
||||
|
||||
def test_repo_cache_entries_lists_repos(self, tmp_path, monkeypatch):
|
||||
monkeypatch.setattr("ifcurl.git.user_cache_dir", lambda *a, **kw: str(tmp_path))
|
||||
_make_fake_repo_cache(tmp_path, "https://example.com/org/repo")
|
||||
_make_fake_repo_cache(tmp_path, "https://github.com/org/other")
|
||||
entries = _repo_cache_entries()
|
||||
urls = {e[3] for e in entries}
|
||||
assert "https://example.com/org/repo" in urls
|
||||
assert "https://github.com/org/other" in urls
|
||||
|
||||
def test_repo_cache_entries_ignores_non_repo_dirs(self, tmp_path, monkeypatch):
|
||||
monkeypatch.setattr("ifcurl.git.user_cache_dir", lambda *a, **kw: str(tmp_path))
|
||||
(tmp_path / "not_a_repo").mkdir() # no repo.git subdir
|
||||
entries = _repo_cache_entries()
|
||||
assert len(entries) == 0
|
||||
|
||||
def test_evict_removes_oldest_over_limit(self, tmp_path, monkeypatch):
|
||||
monkeypatch.setattr("ifcurl.git.user_cache_dir", lambda *a, **kw: str(tmp_path))
|
||||
monkeypatch.setenv("IFCURL_CACHE_MAX_GB", "0.000001") # ~1 KB limit
|
||||
old = _make_fake_repo_cache(tmp_path, "https://old.example.com/repo", size_bytes=512)
|
||||
import time; time.sleep(0.05)
|
||||
new = _make_fake_repo_cache(tmp_path, "https://new.example.com/repo", size_bytes=512)
|
||||
_evict_if_needed()
|
||||
assert not old.exists()
|
||||
assert new.exists()
|
||||
|
||||
def test_evict_no_limit_does_nothing(self, tmp_path, monkeypatch):
|
||||
monkeypatch.setattr("ifcurl.git.user_cache_dir", lambda *a, **kw: str(tmp_path))
|
||||
monkeypatch.delenv("IFCURL_CACHE_MAX_GB", raising=False)
|
||||
entry = _make_fake_repo_cache(tmp_path, "https://example.com/repo")
|
||||
_evict_if_needed()
|
||||
assert entry.exists()
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue